Measure what matters:Evaluating the impact of public administration on citizensA mixed methods approach to developing and testing the Experience Effectiveness (XE) framework in Australia and Canada

Abstract

Public administration has an essential role in enabling citizen compliance behaviour. When public administration is effective, it persuades citizens of its value and fairness. It also supports those attempting to respond appropriately and ensures they are rewarded by easy and seamless services that are transparent and unobtrusive. An effective public sector is also economically proficient and guarantees that those who attempt to evade their responsibilities are unsuccessful. In contrast, when the administration is ineffective it obfuscates the rules and becomes an obstacle that dutiful citizens struggle to comprehend and overcome. The administration becomes an enemy to be feared rather than a friend to be respected.No holistic evaluation of the public sector has been conducted that measures both the impact of its actions on citizens’ lives as well as its ongoing performance. This thesis outlines the design and application of a novel framework for measuring and evaluating the effectiveness of experiences or services created by the public sector. The framework provides two multi-dimensional evaluation tools: the Full Experience Effectiveness (XE) Measurement Scale and the XE Expert. These tools enable the evaluation of both citizen experiences and the impacts of public sector administration. The framework details a typology for describing and documenting citizen designs for evaluation. The core dimensions of XE relate to the design of the experience in terms of products, processes, service and emotions. It also includes viewpoints of the experience from multiple perspectives such as the citizen, the service delivery staff (public sector employees), administrative specialists and related service intermediaries.The approach used in this research is novel in that it draws from multiple domains to construct and measure XE including: (i) the field of new public administration which provides insight into the role of citizens in co-producing the service experience; (ii) services marketing which provides extensive understanding of customer satisfaction for service development; (iii) human factors which suggest practical methods to design systems that anticipate and prevent human error; and (iv) human-centred design which creates ownership and advocacy by engaging the people in the design of services that they will have to use.The research utilised a pragmatic bricolage of methods. The mixed methods approach follows a pluralist view and utilises the iterative, collaborative approach well-established in action research.The research program involved three phases. Phase 1 included the development of the XE Measurement Scale prototype through a comprehensive review of existing measures and evaluation methods including an expert review of the proposed scale and its subsequent refinement. Phase 2 applied the XE Measurement Scale prototype in an Australian context (starting a small business with a tax compliance focus). During this phase, I refined the prototype, added more components and enhanced the model for operationalisation. In Phase 3, I applied the refined XE Framework, which included a full evaluation scale (Full XE Measurement) and the XE Expert evaluation tool. The application of the full suite of tools evaluated six projects impacting citizens in the Canadian public sector. The environment for design and evaluation was also assessed to determine the influence of the organisation culture and context on the outcome of the delivered citizen experiences.Throughout this research, the objective was to provide a practical method for identifying and measuring the factors that impact citizen interactions with public services. The creation of a method to evaluate the effectiveness of the service provided by a public administrator is a responsibility that governments must take seriously to improve compliance and increase productivity. An objective measurement approach of public administration service, unfettered by political influence, is currently unavailable but would be of great benefit to the community. The adoption of the evaluation framework presented in this thesis has the potential for significant impact in all areas of public administration requiring goodwill, adherence to obligations and mutual respect. An evaluation framework encourages self-assessment but also enables the community to hold the bureaucracy accountable, redistribute control and initiate public participation in new forms of collaborative government.