Interim Feedback Research Articles

The effects of interim feedback on perceptions and outcomes of assessment in a first-year history course

ABSTRACT Research on the first year of higher education (FYHE) has shown that students benefit from scaffolding to ease the transition to university. Constructively aligned assessment and feedback are integral to a scaffolded approach. Low-stakes interim assessments and feedback can reduce uncertainties around higher-stakes assessments and help students meet key learning outcomes. This small-scale study investigated the effects of one-on-one interim feedback conversations (FCs) with students in a first-year modern history course. Two datasets were used and compared. First, the students were surveyed about their perspectives on FCs after they took place. Second, analytics data provided insights into engagement and assessment performance across the cohort. Considering these datasets alongside each other suggests that FCs, though a popular and beneficial feedback mode, are unlikely to override other known success factors. However, the positive reception and apparent benefits of FCs make them a viable option for implementing interim feedback in the FYHE.

Abstract P119: Patient Evaluation of Technology and Coach Interaction in the Maintain Prime Weight Loss Maintenance Trial in Primary Care

Background: Participant experiences with lifestyle interventions can influence long-term engagement and outcomes. MAINTAIN PRIME is a pragmatic clinical trial adapted from an effective EHR-based weight maintenance program and designed for implementation by primary care staff. The trial assesses if health coaching with lifestyle tracking tools through the EHR patient portal (MyChart) helps patients better maintain weight loss compared to tracking tools alone. In this report we evaluated interim feedback on technology and coaching. Methods: Participants were adults with BMI ≥25 and intentional weight loss of ≥5% through healthy lifestyle changes. Participants were randomized to health coaching with tracking tools (coaching) or tracking tools alone (tracking) over 24 months. We conducted surveys after 12 months of participation. The System Usability Scale (SUS) and a team-developed survey on MyChart tracking tools and overall satisfaction were given to both groups, while questions adapted from the Measure of Coach and Teacher Alliance-Teacher Report (Alliance) were given to the coaching group only. The SUS and Alliance were analyzed using published methods, and the MyChart survey was analyzed descriptively. Results: Out of 163 participants completing 12 months in MAINTAIN PRIME, 65 (40 in coaching and 25 in tracking groups, 39.9% response rate) responded to at least 1 survey by time of analysis. At baseline, the mean age was 48.1 years, and females represented 64.6% (42/65) of responders. Demographics of responders were similar to non-responders across a range of characteristics (p>0.05). Participants rated the program usability as “good” with mean SUS score 70.2 (SD 18.4). There was no significant difference in SUS score by intervention arm (p=0.52). Most participants (56.9%, 29/51) were satisfied with their experience in the program and 51.0% (26/51) were likely to recommend it to others. Participants rated tracking tools as helpful for daily weight (51.0%, 26/51), steps (37.3%, 19/51), physical activity minutes (33.3%, 17/51), calories (23.5%, 12/51), and fat grams (11.8%, 6/51); 41.2% (21/51) of participants reported no use of tracking methods. Overall, 39.5% (17/43) of participants found tracking tools helpful. On the Alliance questionnaire participants rated aspects of health coaching as 3.3, 3.6, and 2.8 (on a 5-point Likert scale) for the coaching relationship, coaching process, and investment, respectively. Conclusions: MAINTAIN PRIME participants rated the usability of MyChart positively for health coaching and tracking tools, though less than half found tracking tools helpful. Health coaching was rated neutrally, suggesting further evaluation of transferability into a primary care and patient portal-based format is needed. Participant interviews are planned to further assess the acceptability of MyChart for health tracking and coaching.

Moonshots

In the last half-century, technological progress has stagnated. Rapid advances in information technology disguise the slow pace of productivity growth in other fields. Reigniting technological progress may require firms to invest in moonshots—long-term projects to commercialize innovations. Yet all but a few giant tech firms shy away from moonshots, even when the expected returns would justify the investment. The root of the problem is corporate structure. The process of developing a novel technology does not generate the kind of interim feedback that shareholders need to monitor managers and managers need to motivate employees. Managers who anticipate these agency problems invest in incremental innovations instead.
 In the last few years, a new structure designed to commercialize long-term innovations has emerged—the venture carveout. A venture carveout is a private company with one or two public company parents, outside private investors, and employee ownership. The parents provide intellectual property and a long-term strategic commitment. The private investors supply patient capital that insulates the project from short-term shareholder pressure. The employees’ equity motivates them to bring a product to market. The first venture carveouts are attempting to commercialize autonomous vehicles. If they succeed, they will validate a new model for innovation. This Article argues that venture carveouts could enable more companies to invest in moonshots, compete with the tech giants that dominate our economy, and accelerate technological progress.

Psychological pressure and the right to determine the moves in dynamic tournaments – evidence from a natural field experiment

In this paper, we show that the right to determine the sequence of moves in a dynamic team tournament improves the chances of winning the contest. Because studying dynamic team tournaments – like R&D races – with interim feedback is difficult with company data, we examine decisions of highly paid professionals in soccer penalty shootouts and show that teams whose captains can decide about the shooting sequence are more likely to win the shootout. So, managerial decisions matter for outcomes of dynamic tournaments and we discuss potential reasons for this finding.

A Law School Course in Applied Legal Analytics and AI

Technological advances in artificial intelligence (AI) are affecting the legal profession. Machine learning (ML) and natural language processing (NLP) enable new legal apps that, to some extent, can analyze contracts, answer legal questions, or predict the outcome of a case or issue. While it is hard to predict the extent to which these techniques will change law practice, two things are certain: legal professionals will need to understand the new text analysis techniques and how to use and evaluate them, and law faculties face the question of how to teach law students the required skills and knowledge to do so. At the University of Pittsburgh School of Law, the authors have co-designed a semester-long course entitled, Applied Legal Data Analytics and AI, and twice taught it to combined groups of law students and students from technical departments. The course provides a hands-on practical introduction to applying ML and NLP to extract information from legal text data, the ways text analytics have been applied to support the work of legal professionals, researchers, and administrators, and the techniques for evaluating how well they work.
 The article introduces the new text analytic techniques and briefly surveys law schools’ current efforts to incorporate instruction on computer programming and machine learning in legal education. Then it describes the 2020 version of the course, including the students, instructors, and course sessions in overview. We explain how we taught law students skills of programming and experimental design and engaged them in assignments that involve using Python programming environments to analyze legal data.
 The course culminated in joint projects engaging small teams of law and technical students in applying machine learning and data analytics to legal problems. The article explains how the instructors prepare the students for the final course projects, beginning early in the term with project ideas and databases of text, forming teams, working on the projects as a team and obtaining interim feedback, and finally completing the projects and reporting results. We draw some salient comparisons between the 2019 and 2020 versions of the course and report what worked well and what did not, the students’ reactions, and lessons learned for future offerings of the course.

Uncertainty-Driven Cooperation

We consider dynamic team production in the presence of uncertainty. Team members receive interim feedback that depends on both their current effort level and the project’s uncertain prospects. In this environment, each member can encourage the others by making them more optimistic about the project’s prospects. We study the extent to which this incentive counters the usual free-riding incentive. Restricting the agents’ access to feedback can increase their equilibrium effort levels by mitigating the ratchet effect. In this case, using joint performance measures can be beneficial even when individual measures are available.

Psychological Pressure and the Right to Determine the Moves in Dynamic Tournaments – Evidence from a Natural Field Experiment

In this paper, we show that the right to determine the sequence of moves in a dynamic team tournament improves the chances of winning the contest. Because studying dynamic team tournaments – like R&D races – with interim feedback is difficult with company data, we examine decisions of highly paid professionals in soccer penalty shootouts and show that teams whose captains can decide about the shooting sequence are more likely to win the shootout. So, managerial decisions matter for outcomes of dynamic tournaments and we discuss potential reasons for this finding.

Psychological Pressure and the Right to Determine the Moves in Dynamic Tournaments: Evidence from a Natural Field Experiment

In this paper, we show that the right to determine the sequence of moves in a dynamic team tournament improves the chances of winning the contest. Because studying dynamic team tournaments – like R&D races – with interim feedback is difficult with company data, we examine decisions of highly paid professionals in soccer penalty shootouts and show that teams whose captains can decide about the shooting sequence are more likely to win the shootout. So, managerial decisions matter for outcomes of dynamic tournaments and we discuss potential reasons for this finding.

Uncertainty‐driven cooperation

We consider dynamic team production in the presence of uncertainty. Team members receive interim feedback that depends on both their current effort level and the project's uncertain prospects. In this environment, each member can encourage the others by making them more optimistic about the project's prospects. We study the extent to which this incentive counters the usual free‐riding incentive. Restricting the agents' access to feedback can increase their equilibrium effort levels by mitigating the ratchet effect. In this case, using joint performance measures can be beneficial even when individual measures are available.

Grading and interim feedback processes for a case method course

The authors’ purpose is to describe a grading system that reasonably assesses student performance relative to course goals while at the same time provides interim feedback that is accurate, facilitates learning and motivates students. This method has been effectively used in several iterations of a finance course. The authors explain how to design a course around the four categories of knowledge and six cognitive processes proposed by Bloom; design rubrics for a quiz, a final exam, and classroom contribution; and effectively utilize a spreadsheet to calculate grades and provide timely and transparent interim feedback to students.

Sourcing Innovation: On Feedback in Contests

It is notoriously difficult to provide outside parties with adequate incentives for innovation. Contests—in which solvers compete for a prize offered by the contest holder—have been shown to be an effective incentive mechanism. Despite considerable interest in this concept, we lack a thorough understanding of important aspects of contests; in particular, feedback from the contest holder to the solvers has received only limited attention. This paper discusses how contest holders can improve contest outcomes by devising an optimal information structure for their feedback policies. We first identify when, and when not, to give feedback as well as which type of feedback to give: public (which all solvers can observe) or private (which only the focal solver can observe). We uncover a nontrivial relationship between contest characteristics and optimal feedback choices. Second, we examine whether the contest holder should mandate interim feedback or instead allow solvers to seek feedback at their own discretion. Third, we discuss how changing the granularity of feedback information affects its value to solvers.This paper was accepted by Ashish Arora, entrepreneurship and innovation.

Effects of Training and Feedback on Accuracy of Predicting Rectosigmoid Neoplastic Lesions and Selection of Surveillance Intervals by Endoscopists Performing Optical Diagnosis of Diminutive Polyps

Real-time differentiation of diminutive polyps (1-5 mm) during endoscopy could replace histopathology analysis. According to guidelines, implementation of optical diagnosis into routine practice would require it to identify rectosigmoid neoplastic lesions with a negative predictive value (NPV) of more than 90%, using histologic findings as a reference, and agreement with histology-based surveillance intervals for more than 90% of cases. We performed a prospective study with 39 endoscopists accredited to perform colonoscopies on participants with positive results from fecal immunochemical tests in the Bowel Cancer Screening Program at 13 centers in the Netherlands. Endoscopists were trained in optical diagnosis using a validated module (Workgroup serrAted polypS and Polyposis). After meeting predefined performance thresholds in the training program, the endoscopists started a 1-year program (continuation phase) in which they performed narrow bandimaging analyses during colonoscopies of participants in the screening program and predicted histological findings with confidence levels. The endoscopists were randomly assigned to groups that received feedback or no feedback on the accuracy of their predictions. Primary outcome measures were endoscopists' abilities to identify rectosigmoid neoplastic lesions (using histology as a reference) with NPVs of 90% or more, and selecting surveillance intervals that agreed with those determined by histology for at least 90% of cases. Of 39 endoscopists initially trained, 27 (69%) completed the training program. During the continuation phase, these 27 endoscopists performed 3144 colonoscopies in which 4504 diminutive polyps were removed. The endoscopists identified neoplastic lesions with a pooled NPV of 90.8% (95% confidence interval 88.6-92.6); their proposed surveillance intervals agreed with those determined by histologic analysis for 95.4% of cases (95% confidence interval 94.0-96.6). Findings did not differ between the group that did vs did not receive feedback. Sixteen endoscopists (59%) identified rectosigmoid neoplastic lesions with NPVs greater than 90% and selected surveillance intervals in agreement with those determined from histology for more than 90% of patients. In a prospective study following a validated training module, we found that a selected group of endoscopists identified rectosigmoid neoplastic lesions with pooled NPVs greater than 90% and accurately selected surveillance intervals for more than 90% of patients over the course of 1 year. Providing regular interim feedback on the accuracy of neoplastic lesion prediction and surveillance interval selection did not lead to differences in those endpoints. Monitoring is suggested, as individual performance varied. ClinicalTrials.gov no: NCT02516748; Netherland Trial Register: NTR4635.

Helping Hospitals Improve Patient Centeredness: Assessing the Impact of Feedback Following a Best Practices Workshop.

Regularly reported patient surveys are an important dimension of hospital quality management. This study investigates whether providing hospital staff with interim feedback on patient survey results following a best practices workshop can help hospitals improve patient centeredness. Standardized surveys with consecutive patient samples were administered in accredited breast cancer center (BCC) hospitals in one German state (18 million inhabitants), over a 6-month period, in 2012. Two studies were conducted by applying a combination of regression point displacement (RPD) and interrupted time series (ITS) designs. In Study 1, 2 of the 27 hospitals that had previously participated in a best practices workshop to discuss patient-centeredness issues were randomly chosen and were provided interim feedback of patient survey results and workshop minutes. In Study 2, 4 randomly chosen hospitals of 32 that had not participated in the workshop also received interim feedback but no workshop minutes. Control hospitals in both studies neither received feedback nor workshop minutes. The impact of interim feedback was evaluated by applying graphical assessments and multiple regression analyses. Both graphical assessments (locally weighted scatterplot smoothing (LOESS) lines, RPD plots) suggested an effect of interim feedback. Multiple regression results did not unambiguously support these findings. The suggested design approach may prove particularly useful to assess effects in pilot studies, when resources are not available to conduct a randomized study or when its conduct is contingent on initial, positive evidence.

Interim Feedback in Tournaments for Ideas: Trading Off Participation for Quality

Performance feedback is ubiquitous in competitive settings where new products are developed. This paper introduces a fundamental tension between incentives and improvement in performance evaluation. Using a sample of four thousand commercial logo design tournaments, I show that feedback reduces participation but improves the quality of subsequent submissions, with an ambiguous effect on high-quality output. To evaluate this tradeoff, I develop a procedure to estimate agents’ effort costs and simulate counterfactuals under alternative feedback policies. The results suggest that feedback on net increases the number of high-quality ideas produced and may thus be desirable for a principal seeking innovation.

THE IMPACT OF PRIVATE AND PUBLIC FEEDBACK ON WORKER PERFORMANCE—EVIDENCE FROM THE LAB

Previous empirical studies have demonstrated the positive incentive effects of private performance feedback as well as public recognition on work performance. In our laboratory experiments, we directly compare these two feedback schemes. Moreover, we compare their incentive effects across two real‐effort tasks that differ in regard to how interesting the subjects perceive working on them. We find that both feedback schemes lead to performance increases compared to the tasks' respective control treatments. In particular subjects' ability and a positive interim feedback enhance performance. Competitive preferences significantly affect performance only in the less enjoyable and challenging task. (JEL C9, J3, M1)

Improved Asset Management and Inventory Development through Sample Analysis and Vendor–Client Communication

AbstractThis study compared output from mobile inventory data collection vehicles to manually collected data techniques with a focus on two-way communications primarily through the submission of a sample data set to be analyzed prior to the submission of a full data set. The interim submittal and feedback to the vendors based on that resulted in a marked improvement in data quality for 5 of the 28 assets studied. After feedback, it is apparent that highway data collection vendors can accurately locate the vast majority of assets, with the primary exception being those that are occluded by vehicles or surrounding landscaping, such as those assets in the median. Along with the locations of assets, vendors showed promise at collecting many of the feature descriptions such as asset type and condition. Many of the elements (location, type, etc.) for a particular asset type that created collection difficulty were only problematic for a particular vendor, which suggests that further improvements may be achieved ...

Frequency of examinations and student achievement in a randomized experiment

We carry out a randomized experiment involving undergraduate students enrolled at an Italian University attending two introductory economics classes to evaluate the impact on achievement of examination frequency and interim feedback provision. Students in the treated group were allowed to undertake an intermediate exam and were informed about the results obtained, while students in the control group could only take the final exam. The results show that students undertaking the intermediate exam perform better both in terms of the probability of passing the exams and of grades obtained. High ability students appear to benefit more from the treatment. The experiment design allows us to disentangle “workload division or commitment” effects from “feedback provision” effects. We find that the estimated treatment impact is due exclusively to the first effect, while the feedback provision has no positive effect on performance. Finally, the better performance of treated students in targeted examinations seems not to be obtained at the expenses of results earned in other examinations.

Legacy TRIPSEs (Tri‐Partite Problem‐Solving Exercises): Fostering Student Engagement and Learning in Large Classes

The TRIPSE is a process‐oriented problem solving exercise that mimics the scientific process. Using limited data, students: 1) frame hypotheses; 2) design experimental tests; and 3) later with new information, reassess answers. Earlier we validated it as an evaluative exercise in a large class (The FASEB Journal. 2008; 22:767.1). We report here the use of TRIPSEs to promote student engagement and learning in a freshmen introductory biology class. In part 1, students learned about gene expression and cell signaling. In part 2, students (groups of 5) incorporated elements of what they had learned into designing novel problems that could be left as a “legacy” for future classes. Groups were told to: 1) write challenging problems based on published material; 2) state 4 plausible hypotheses and for each hypothesis indicate appropriate experimental tests; and 3) provide annotated references. Interim feedback was given. Groups were assessed on clarity, creativity, plausibility of hypotheses, relevance of the experiments proposed and corroboration for statements made. On an exit survey, a total of 322 students from 2 separate years rated this exercise the most valuable amongst several used.

Knowledge to Action: The Development of Training Strategies

This paper presents an overview of curriculum and program development activities at the four Canadian Regional Training Centres directed towards the goal of achieving increased knowledge to action. The RTCs have initiated learning opportunities to increase the skills of graduate students in conducting knowledge translation and exchange (KTE). The authors describe similar as well as unique approaches used at each centre to hone understanding and skills. RTC activities include the development of a new four-year residency program for doctoral students, new Web-based and real-time interactive theory courses and new linkages with departments of journalism. While formal evaluation is yet to be completed, interim feedback from participating graduate students has been encouraging.

Working in teams: An examination of the variables that improve team performance in accounting students

The accountancy profession requires professional accountants to acquire skills such as intellectual skills, technical and functional skills, personal skills, interpersonal and communication skills, and organizational and business management skills. In the academic environment, accounting students are often required to work in teams and may therefore acquire some of these skills, for example, the ability to work in a collaborative environment and better interpersonal skills.The paper examines the internal and external factors that may influence the performance of accounting student teams. The data was collected in a questionnaire survey and then analysed to determine what factors influence team performance. To operationalize performance, team members’ willingness to continue working together and the team’s mark were used. Results indicate that factors such as meeting management skills and interpersonal skills have a positive effect on the students’ satisfaction with the team and that interim feedback has a positive effect on students’ performance.