Data-driven iterative refinements to educational development services: directly measuring the impacts of consultations on course and syllabus design

Abstract

Evidence-based practice in educational development includes leveraging data to iteratively refine center for teaching and learning (CTL) services. However, CTL data collection is often limited to counts and satisfaction surveys rather than direct measures of outcomes. To directly assess impacts of consultations on course and syllabus design, we analyzed 94 clients’ syllabi (32 faculty, 62 graduate students and postdocs) before and after consultations. Faculty and non-faculty clients demonstrated significant change following consultations (6% and 10% gains in syllabus rubric scores, representing 50% and 31% of possible gains and effect sizes of 0.73 and 1.04 standard deviations, respectively). We compared faculty clients to quasi-experimental control groups that did not receive consultations. Syllabi from non-clients scored lower and did not demonstrate similar changes across semesters. Attendance at a CTL seminar on course and syllabus design did not explain variation in clients’ syllabi. We discuss implications for assessment of CTL services and how we leveraged formative assessments to inform and iteratively refine our educational development practices.