BUZZING—A Theory-Based Impact Evaluation Design

Abstract

BUZZING is a pilot project which introduces a new way of working with disengaged and long-term unemployed young people (aged 15 to 24 years) to support their transition into employment using gamification and online platforms to drive engagement. The project is funded by the Australian Government's Department of Employment under their Empowering YOUth Initiatives, and delivered by not-for-profit organisation United Synergies. This article gives focus to the design considerations that underpin the work of an internal evaluator working on the BUZZING project, and enrolled in an evaluation unit within a postgraduate course. The step-by-step process used to create an evaluation plan inclusive of purpose, context, scope, method and methodology are illustrated, supported by evidence-based justifications. Furthermore, counter-bias design considerations such as a mix of different methodology inquiry, data sources and researchers (triangulation) are discussed and recommendations given for an ethical evaluation framework. The practice of using evidence-based rigour in evaluation is promoted for organisations which seek to demonstrate positive social change within complex environments. The capacity building and professional development of internal evaluators in the not-for-profit sector is therefore recommended to drive accountability, ethical practices and continual improvement for public good from within the organisation. Finally, White's (2009) theory-based impact evaluation is adopted as the evaluation method as it maps out the causal chain from inputs to outcomes and impact, testing the underlying assumptions to understand why a program has, or has not, had an impact.