An evaluation of evaluation: problems with performance measurement in small business loan and grant schemes

Abstract

This project researched the problems of evaluating public sector small business loan and grant schemes. The methodology was to use case studies of four loan and grant schemes run by enterprise agencies on contract from task forces or city challenges. These case studies were selected following a mapping exercise of 27 finance schemes in north east London. Research showed that evaluation relied on performance indicators. The main measures used were deployment of funds, job creation, leverage of additional funds, lender of last resort, ethnicity of recipients, default rate, and enquiry levels. Four technical problems with these performance indicators were found. First, the interpretation of indicators varied between organisations. Second, performance indicators judged agencies on areas which were outside their control. Third, indicators failed to take account of the full range of work involved in managing a loan and grant scheme. Fourth, lender of last resort and leverage were incompatible in their preferred risk position and no attempt was made to reconcile the two. That these technical weaknesses were not taken into account in the weight given to performance indicators led to four organisational effects. First, agencies seemed to have made their own attempt to render the indicators meaningful, reflecting the conditions under which the schemes had been first established. The existence of strong political, auditing, and time pressures led to three management styles designated as client oriented, bureaucratic, and entrepreneurial. Second, the system encouraged managing agencies to distort figures to produce more favourable results. For instance, rescheduling of debts allowed the agency to avoid recording high levels of default. Third, the focus of information gathering on external needs (those of the funders) seemed to have discouraged agencies from contemplating and developing their own data systems. This might reflect either a lack of resources or a fear that any information could be used against them. Several possible explanations for these problems are explored in this report. Objectives were ambiguous because of the under-development of programme theory. Data was not always available because its collection was not of value to the enterprise agency. Communication between funders and agencies was poor because funders had come to see their main power as residing in the re-tendering of contracts. These points present part of the explanation, but a further level of analysis is possible. That theory is under-developed, data unavailable, and communication weak are symptoms of a deeper problem. This is the problem that performance management is operating under a positivist theory of knowledge. Several possible improvements to the evaluation of small business loan and grant schemes are explored in this paper. Key among these is an acknowledgement of the limitations of evaluation through adopting a developmental approach within a culture of organisational learning. This paper illustrates the value of adopting a developmental approach, but also the fundamental transformation which it implies. “With each new evaluation, the evaluator sets out, like an ancient explorer, on a quest for useful knowledge, not sure whether seas will be gentle, tempestuous, or becalmed. Along the way the evaluator will often encounter any number of challenges: political intrigues wrapped in mantles of virtue; devious and flattering antagonists trying to co-opt the evaluation in service of their own narrow interests and agendas; unrealistic deadlines and absurdly limited resources; gross misconceptions about what can actually be measured with precision and definitiveness; deep-seated fears about the evils-incarnate of evaluation, and therefore, evaluators; incredible exaggerations of evaluators' power; and insinuations about defects in the evaluator's genetic heritage ( Patton, 1997: 38; Utilisation Focused Evaluation).”