Summative Feedback Research Articles

Architecture students’ uptake and use of formative and summative feedback

Increasingly, attention is being turned toward the actions of students in relation to feedback. This paper presents the outcomes of a questionnaire study researching what undergraduate and postgraduate architecture students perceive as the purpose of feedback, how they use feedback on their coursework, and what makes them more likely to act on it. An overarching theme in their conception of feedback’s purpose was that of a guide for improving coursework and learning. Two key dimensions that facilitated use over the longer term were feedback being generative and informing their working methods. The study revealed students engaging in a range of strategies with both formative and summative feedback to prioritise their next work; these could form the basis of facilitating wider adoption of these methods across the student body. There were similarities, but also substantial differences, between how undergraduate and postgraduate students reported perceiving and utilising feedback, which may have impact on feedback provision to enhance uptake and use. Whereas undergraduate students saw feedback’s purpose in terms of directing improvement, postgraduates were more likely to see their own role in the process. Postgraduates were more likely to question feedback’s validity over the longer term.

A Systematic Literature Review of Gameful Feedback in Computer Science Education

For the past decade, video game- and gamification-elements get used in different fields of research. However, a contextualized usage of these elements is still underrepresented in the current research. For that reason, this research tries to identify contextualized game-elements in e-learning environments for computer science education. A systematic literature review examines the current overlap of feedback in computer science education by the use of game-elements. The relevant papers were identified by a combination of search-terms and analyzed according to a defined scope, that focuses on formative and summative feedback. In a nutshell, the majority of provided feedback in computer science education, that is not just given by an instructor, is often implemented by automated code tests. These are supported through techniques to monitor the performance of the student and their progress towards the set goal. Game- or gamification-elements do play a subordinate role, when providing feedback and are often just to enhance the monitoring process.

Analyzing Self-Efficacy and Summary Feedback in Automated Social Skills Training.

Goal: Although automated social skills training has been proposed to enhance human social skills, the following two aspects have not been adequately explored: what types of feedback are effective from virtual agents and the extent to which such systems enhance users' social self-efficacy. Methods: We developed an automated social skills trainer+ that follows human-based social skills training processes and implemented two types of feedback: 1) a summary of the displayed feedback and 2) feedback based on the results of their previous training. Using our developed system, we measured social self-efficacy, feedback evaluations, and the third-party ratings of participants between pre- and post-training as well as their social responsiveness scales. Results: Self-efficacy is significantly correlated to the social responsiveness scale (r = −0.72) and can be improved with our system (mean improvement of 0.68, p < 0.05). The participants highly rated the feedback that was compared to their past training (14 out of 16, p < 0.05) more than the cases without it and the displayed summary feedback (11 out of 16, p = 0.21) more than the verbal comments. Conclusions: Our system effectively summarized user feedback in terms of user self-efficacy and third-party ratings.

The Effect of Frequency of Summary Feedback on the Proprioception in Knee Extension of Healthy Subjects

Purpose: This study investigated the effect of the frequency of summary feedback on proprioception during knee extension in healthy subjects. Methods: Forty-eight young adults volunteered to participate and were randomly assigned to one of four groups (1*summary, 5*summary, 10*summary, and control). The power and velocity errors for proprioception in knee extension were measured using a PRIMUS RS system (BTE Tech., Hanover, USA). Multivariate analysis of variance (MANOVA) was used to determine the effect of the frequency of summary feedback on proprioception in knee extension of healthy participants. Scheffe post hoc tests were applied, and the level of significance was set at α=0.05. Results: In the acquisition phase to practice the effect, power, and velocity error scores in all summary groups were significantly lower than those in the control group (p 0.05). In the retention phase to assess the learning effect, the power and velocity error scores in all summary groups were also significantly lower than those in the control group (p <0.05), and the score in the 5*summary group was significantly lower than those in the other summary groups (p<0.05). Conclusion: 5*summary feedback was effective for motor learning while reducing the error of proprioception.

Feedback during summative clinical assessments: Experiences of diagnostic radiography students at a higher education institution in South Africa

IntroductionSummative clinical assessments and feedback, conducted at clinical training sites, are vital for successfully preparing undergraduate diagnostic radiography students to become competent, skilled diagnostic radiographers. However, providing appropriate feedback in a clinical context is a complex matter, as studies show that students often feel intimidated by feedback and may only accept feedback selectively, so as to pass their assessment or course. This article reports on the experiences of radiography students regarding summative clinical assessment feedback. MethodsA qualitative, exploratory, descriptive, and contextual research design was used. Data were gathered from radiography students at one higher education institution in South Africa who were registered in year three and year four of a four-year professional Diagnostic Radiography degree. Five semi-structured focus group interviews were conducted, with 26 participants in total. Focus group interviews were audio recorded, transcribed verbatim and coded using Tesch's data analysis method. ResultsTwo principle themes emerged from the data analysis. Theme 1 unpacked whether radiography students viewed the assessor as an ally or foe. It was found that various assessor-related characteristics influenced radiography students receptivity towards feedback and this was related to how they viewed the assessor. Theme 2 concerned the radiography students' perceptions of the key elements influencing the nature of a feedback process. These elements were influenced primarily by the assessor, but the attitudes of radiography students towards the feedback process were also important. ConclusionRadiography students reported various positive experiences regarding assessor feedback during the summative clinical assessments. However, they were also dissatisfied with numerous aspects of how the assessors provided feedback often viewing the assessor as one who is a foe or who is opposed to their success. Subsequently the negative aspects of the feedback process hampered the learning experience of the participants. Implications for practiceAwareness and the development of feedback skills for assessors and radiography students would be beneficial to the learning process. Radiography students, the radiography profession and ultimately the patients would benefit from the effects of effective feedback.

Digital manikins to self-report pain on a smartphone: A systematic review of mobile apps.

BackgroundChronic pain is the leading cause of disability. Improving our understanding of pain occurrence and treatment effectiveness requires robust methods to measure pain at scale. Smartphone‐based pain manikins are human‐shaped figures to self‐report location‐specific aspects of pain on people's personal mobile devices.MethodsWe searched the main app stores to explore the current state of smartphone‐based pain manikins and to formulate recommendations to guide their development in the future.ResultsThe search yielded 3,938 apps. Twenty‐eight incorporated a pain manikin and were included in the analysis. For all apps, it was unclear whether they had been tested and had end‐user involvement in the development. Pain intensity and quality could be recorded in 28 and 13 apps, respectively, but this was location specific in only 11 and 4. Most manikins had two or more views (n = 21) and enabled users to shade or select body areas to record pain location (n = 17). Seven apps allowed personalising the manikin appearance. Twelve apps calculated at least one metric to summarise manikin reports quantitatively. Twenty‐two apps had an archive of historical manikin reports; only eight offered feedback summarising manikin reports over time.ConclusionsSeveral publically available apps incorporated a manikin for pain reporting, but only few enabled recording of location‐specific pain aspects, calculating manikin‐derived quantitative scores, or generating summary feedback. For smartphone‐based manikins to become adopted more widely, future developments should harness manikins’ digital nature and include robust validation studies. Involving end users in the development may increase manikins’ acceptability as a tool to self‐report pain.SignificanceThis review identified and characterised 28 smartphone apps that included a pain manikin (i.e. pain drawings) as a novel approach to measure pain in large populations. Only few enabled recording of location‐specific pain aspects, calculating quantitative scores based on manikin reports, or generating manikin feedback. For smartphone‐based manikins to become adopted more widely, future studies should harness the digital nature of manikins, and establish the measurement properties of manikins. Furthermore, we believe that involving end users in the development process will increase acceptability of manikins as a tool for self‐reporting pain.

Virtual Standardized Patients vs Academic Training for Learning Motivational Interviewing Skills in the US Department of Veterans Affairs and the US Military

Despite the need for effective and scalable training in motivational interviewing (MI) that includes posttraining coaching and feedback, limited evidence exists regarding the effectiveness of using virtual (computerized) standardized patients (VSPs) in such training. To evaluate the efficacy of training with a VSP on the acquisition and maintenance of MI skills compared with traditional academic study. This study was a 2-group, parallel-training randomized trial of 120 volunteer health care professionals recruited from a Department of Veterans Affairs and Department of Defense medical facility. Motivational interviewing skill was coded by external experts blinded to training group and skill assessment time points. Data were collected from October 17, 2016, to August 12, 2019. After a computer course on MI, participants trained during two 45-minute sessions separated by 3 months. The 2 randomized training conditions included a branching storyline VSP, which provided MI skill rehearsal with immediate and summative feedback, and a control condition, which included academic study of content from the computerized MI course. Measurement of MI skill was based on recorded conversations with human standardized patients, assessed using the Motivational Interviewing Treatment Integrity 4.2.1 coding system, measured at baseline, after training, and after additional training in the randomized condition 3 months later. A total of 120 volunteers (83 [69%] women), with a mean (SD) of 13.6 (10.3) years of health care experience, participated in the study; 61 were randomized to receive the intervention, and 59 were randomized to the control group. Those assigned to VSP training had significantly greater posttraining improvement in technical global scores (0.23; 95% CI, 0.03-0.44; P = .02), relational global scores (0.57; 95% CI, 0.33-0.81; P = .001), and the reflection-to-question ratio (0.23; 95% CI, 0.15-0.31; P = .001). Differences were maintained after the 3-month additional training session, with more improvements achieved after the 3-month training for the VSP trainees on the reflection-to- question ratio (0.15; 95% CI, 0.07-0.24; P = .001). This randomized trial demonstrated a successful transfer of training from a VSP to human standardized patients. The VSP MI skill outcomes were better than those achieved with academic study and were maintained over time. Virtual standardized patients have the potential to facilitate dissemination of MI and may be useful for training in other evidence-based skills and treatments. ClinicalTrials.gov Identifier: NCT04558060.

Evaluating the nature of summative feedback on electronically submitted work at a UK University: implications for post-compulsory education

ABSTRACT Summative feedback provides individualised support for students in formal learning situations; however, numerous studies show that academic practice in summative feedback is highly uneven, that the purposes are poorly understood and that the provision of developmental feedback is often subsidiary to the justification of the assessment decisions. This study considers the nature of marker feedback on 105 pieces of summative work that were electronically submitted across 16 first degrees in sports at a UK University. The study explores the patterns of feedback practice and relates these to ideas of dialogic feedback, to the concept of Feedback Mark 2, and to feedback frameworks in the literature. The analysis shows that feedback practice is often monologistic, predominantly addressing the structure of student work and is characterised by the identification of deficiencies rather than the support of reflective learning. However, the study finds that the feedback on non-anonymised texts was more extensive and dialogic than that on anonymised texts. Overall, the study confirms that feedback practice is more aligned to transmissive approaches than to the empowerment of students through and identifies the implications for practice in post-compulsory education.

Loma Linda University School of Medicine.

Loma Linda University School of Medicine.

University of Florida College of Medicine.

University of Florida College of Medicine.

McGill University Faculty of Medicine.

McGill University Faculty of Medicine.

Geisinger Commonwealth School of Medicine.

Geisinger Commonwealth School of Medicine.

The SAFE-T upper endoscopy tool: a web-based application for the point-of-care evaluation of gastroenterology fellow performance in upper endoscopy.

BackgroundAttending assessment is a critical part of endoscopic education for gastroenterology fellows. The aim of this study was to develop and validate a concise, web-based assessment tool to evaluate real-time fellow performance in upper endoscopy.MethodsWe developed the Skill Assessment in Fellow Endoscopy Training (SAFE-T) upper endoscopy tool to capture both summative and formative feedback in a concise, five-part questionnaire. The tool mirrors the previously validated SAFE-T colonoscopy tool and is administered electronically via a web-based application. We evaluated the tool in a prospective study of 15 gastroenterology fellows (5 fellows each from Years 1–3 of training) over the 2018–2019 academic year. An independent reviewer evaluated a subset of these procedures and completed both the SAFE-T and Assessment of Competency in Endoscopy (ACE) upper endoscopy forms for reliability testing.ResultsTwenty faculty completed 413 SAFE-T evaluations of the 15 fellows in the study. The mean SAFE-T overall score differentiated each sequential fellow year of training, with first-year cases having lower performance than second-year cases (3.31 vs 4.25, P < 0.001) and second-year cases having lower performance than third-year cases (4.25 vs 4.56, P < 0.001). The mean SAFE-T overall score decreased with increasing case-complexity score, with straightforward compared with average cases (3.98 vs 3.39, P < 0.001) and average compared with challenging cases (3.39 vs 2.84, P = 0.042). In dual-observed procedures, the SAFE-T tool showed excellent inter-rater reliability with a Kappa agreement statistic of 0.815 (P = 0.001). The SAFE-T overall score also highly correlated with the ACE upper endoscopy overall hands-on score (r = 0.76, P = 0.011).ConclusionsWe developed and validated the SAFE-T upper endoscopy tool—a concise and web-based means of assessing real-time gastroenterology fellow performance in upper endoscopy.

Faculty Feedback Program Evaluation in CIMS Multan, Pakistan.

Faculty feedback program (FFP) at CMH Multan Institute of Medical Sciences (CIMS) was conducted for obtaining feedback for basic medical sciences faculty and evaluated to highlight its weaknesses for future improvement. The evaluation design was utilization-focused evaluation (UFE) keeping in mind its two essential elements. First element is the primary intended users (PIU) of the evaluation, namely the college faculty and students which were clearly identified and personally engaged to investigate intended use of the evaluation. Second element required the evaluator to ensure that the intended use of evaluation by PIU guide all other decisions made about the evaluation process. It was a mixed method study (qualitative and quantitative methods both) conducted from August 2018 to August 2019 in CIMS Multan with IRB approval following the steps of UFE. The whole program evaluation was conducted in two parts - first part constituted the 2018 manual FFP evaluation that provided suggestions for a FFP conducted in 2019 online. In step 2 the 2019 online FFP was evaluated again forming basis for future recommendations. Hence the PIUs response was recorded twice in the evaluation cycle - initially after the manual 2018 basic science FFP (response rate: 53%) - after which based on our findings a report was generated and recommendations suggested which were implemented in the 2019 online FFP and response observed again (response rate: 85.7%) to complete the evaluation cycle. Open-end questions were asked from faculty (qualitative analysis) with three themes emerging regarding FFP procedure, questionnaire and timing. An acknowledgement of shift of FFP procedure from manual (2018) to online system (2019) was observed in which faculty praised the ease (72.2%), confidentiality (66.6%), anonymity (50%) and transparency (33.3%) of the online system compared to manual FFP, which was reported to be a rather tense experience (83%). Regarding questionnaire, 38% faculty members reported feedback questions asked from students to be vague and 66.6% claimed that the timing was inappropriate and should have been end of academic year. When asked for suggestions for improvement in 2018 FFP, 72% faculty suggested training students on providing feedback and making the procedure user friendly (83%). Student response regarding both feedback was obtained online by a survey with closed ended questions (quantitative study). Fifty-three percent college students were satisfied with the online FFP giving an average rating of 3.2 to the software user interface and 85% affirmed that using the online software aided in providing anonymous responses helping them provide candid feedback. Seventy-five percent students agreed that online feedback system in 2019 had streamlined the feedback process and made it more efficient compared to the paper-based manual survey of 2018. After evaluating 2019 online FFP, few suggestions were recommended for future FFP including obtaining formative as well as summative faculty feedback, supplementing feedback with teacher’s self-assessment/pen picture and incorporating 360 multi-source feedback.

Randomised control language intervention for infants of adolescent mothers.

Create a Language ENvironment Analysis (LENA)-based intervention to increase adolescent and infant speech and improve 12-month language outcomes. Randomised control trial of adolescent (15-19years) mother-infant pairs comparing language-motor (intervention) and motor (control) groups. Intervention included reviewing language-motor curriculums, formative feedback on 4 LENA recordings (baseline, post-curriculum, 4 and 12-months) and 16-weekly language-motor texts. Controls reviewed a motor curriculum, summative feedback of four recordings after study completion and 4-monthly motor texts. Primary outcome was 12-month MacArthur scores. Secondary outcomes were LENA counts and social impacts to language outcomes. A total of 108 infants were randomised. Groups had similar baseline characteristics and LENA counts. Both groups had low maternal Peabody Picture Vocabulary age-equivalents (14.2years). On post-curriculum recording, intervention infants had higher vocalisations (188 vs 109, P=.02) and conversations (49 vs 30, P=.005) than controls. Group 4-month and 12-month LENA counts and 12-month MacArthur scores were similar. In regression analyses, more people in the home and cohabiting with the infant's father were associated with higher MacArthur scores. Linguistic feedback and a simple curriculum resulted in short-term increased vocalisations and conversational turns for infants of adolescent mothers that were not sustained over time. Household characteristics provided protective effects on outcomes.

Achieving expert performance through simulation-based education and application of mastery learning principles

Medical education has undergone significant change as we have moved from a purely apprenticeship model to one of competence. Simulation-based education can and does play a significant role in the development of an expert surgeon. Proven concepts of simulation-based education include distributed deliberate practice, content that challenges the learner, formative and summative feedback. The application of mastery learning principles is critical to achieve the best retention and outcomes related to an educational program. In this manuscript, we explore the methodology and justification for Mastery Learning as the gold standard of surgical education.

Identifying Key Physical Activity Categories and Topics to Include in Health Professional Training Programs

Phenomenon: Healthcare is an important sector in promoting physical activity (PA). However, few health professional training programs include PA, nor does standardized guidance exist on implementing it in the curriculum. This study aimed to consolidate health professional expert opinion on key PA categories and topics that should be included in the curriculum of health professional training programs. Approach: A three-round, modified e-Delphi process examined the opinions of 73 experts from seven health professions (clinical nutrition, exercise physiology, medicine, nursing, occupational therapy, physical therapy, physician assistants). In Round 1, panelists reported importance, ranked, and scored five broad PA categories, and responded to open-ended prompts for additional categories. In Round 2, panelists received summary feedback, re-ranked and re-scored PA categories, and suggested key PA topics within the five categories. In the final round, panelists viewed, ranked, and scored the PA topics. Findings: Expert panelists felt that all PA categories were important, with Health Behavior Change ranking the highest (98.7%) followed by Cellular and Systemic Implications of Exercise, Clinical Exercise Physiology, and PA and Public Health. The Administrative Aspects of Integrating PA into Health Systems ranked least important (48.0%). A consensus on the key PA categories was considered reached after two rounds. Five to eight specific PA topics were generated within each PA category and ranked in order of importance. Insights: These findings highlight several key PA categories and topics that can serve as a foundation for a diverse number of health professional training programs.

Development, Implementation, and Evaluation of a Faculty Development Workshop to Enhance Debriefing Skills Among Novice Facilitators.

IntroductionEffective debriefing during simulation-based training (SBT) is critical to promote learning outcomes. Despite debriefing’s central role in learning and various published debriefing methods and techniques, little is known about faculty development structure for debriefing training among novice facilitators. Continuing medical education courses often use simulation-based methods but provide minimal training in debriefing techniques to novice facilitators. We describe the development, implementation, and evaluation of a structured debriefing training workshop for novice facilitators.MethodsDesigned and conducted by simulation debriefing experts, a debriefing workshop was provided to novice facilitators serving as faculty during the simulation-based Sedation Provider Course (PC) at the 2018 Society of Pediatric Sedation conference. Emphasizing evidence-based key elements of effective debriefing, the workshop was divided into three components: 1) an introductory 30 minute didactic, 2) 75 minutes role modeling of simulated effective and ineffective debriefing 3) 120 minutes repetitive deliberate practice sessions with summative and formative feedback. Effective transfer of learned debriefing skills was assessed during facilitators’ PC debriefing using the Objective Structured Assessment of Debriefing (OSAD) tool, facilitators’ self-efficacy, and PC student learners’ evaluation of facilitator debriefings during the PC.ResultsSixteen facilitators participated in the 4-h workshop and the next day served as PC faculty. The median OSAD score was 31 (13-40) for all facilitators. OSAD components with lowest and highest performance were “Establishing Learning Environment” with a median score of 1 (1-5) and “Engagement of Learners,” with a median score of 4.75 (2.5-5). Facilitators' self- assessment in debriefing significantly improved on the 5-point Likert scale pre- and post-workshop, respectively. PC student learners’ evaluations revealed high degrees of satisfaction with debriefing quality.ConclusionsA proposed model integrating full-length debriefing and repetitive practice paired with summative and formative feedback provides a feasible and effective approach for debriefing training of novice facilitators for simulation-based educational courses.

Emodash: A dashboard supporting retrospective awareness of emotions in online learning

Abstract We present Emodash , an interactive dashboard supporting tutors’ retrospective awareness of learners’ emotions in a video-conferencing learning environment. Socio-affective relationships play an important role in learning processes and learning outcomes, but they are harder to develop in online-learning. This can be explained by a lack of emotion awareness due to the asynchronous interactions, technical challenges, and tutors’ focus on properly conducting the learning activity and gearing towards pedagogical outcomes. We conducted an eight-week long field study with five professional tutors on how they used Emodash while writing feedback to learners after language learning sessions. We found that Emodash led tutors who were already sensitive to learners’ emotions to incorporate more affective elements in their reports, suggesting a stronger awareness of learners’ emotions. Tutors also wrote more formative and less summative feedback. Furthermore, our results suggest that glanceable visualizations of learners’ emotions may be preferred and sufficient to foster tutors’ awareness of learners’ emotions. Finally, the dashboard led tutors to reflect on the way they conduct their lessons, using learners’ positive emotions as a proxy evaluation of their teaching.

Behavioural couple therapy: measuring therapist competence

Abstract This paper outlines a pilot validation study of the newly developed Behavioural Couple Therapy Scale for Depression (BCTS-D). The BCTS-D aims to assess therapist competence in delivering behavioural couple therapy (BCT) and provide therapists with summative and formative feedback on their performance. Completed by both therapist and supervisor, this will aid therapists’ reflection on practice and improve performance. This paper will report on two stages in the development of the BCTS-D: (a) a study evaluating content validity, face validity and usability and (b) a focus group examining usability and utility. Both parts of the study were conducted in the context of a BCT training course and included 20 participants who were either BCT supervisors or BCT trainees. Results suggest that the BCTS-D has good face validity, content validity and usability, and provides a useful tool for promoting self-reflection and providing formative feedback. The studies also provided insight into the strengths of the scale and into areas of refinement, and a number of modifications were undertaken to improve the BCTS-D in response to feedback collected. Future research will need to focus on evaluating the psychometric properties of the BCTS-D and continue to adapt the scale to its users’ needs. Key learning aims (1) Readers will understand the importance of measuring therapists’ competence to improve practice. (2) Readers will understand the development of the BCTS-D scale and its initial psychometric properties. (3) Readers will know how to use the BCTS-D in everyday clinical practice. (4) Readers will know about the challenges of developing a therapist competence measure within a real-world clinical context.