Evaluation Advisory Groups

Abstract

Evaluation (like applied social science research and social work research) exists to be practical. This means that its very being and certainly its job in the world is to be used—used to help individuals learn, assess, reflect, and be useful in imagining, proposing, deciding, and recommending whether and how to refine, improve, close, or otherwise respond to the realities of organized efforts to ‘‘make a difference’’ in the material and other worlds of individuals, groups, organizations, programs, or the like. It exists to repair (Spelman, 2002). To do this right and well requires facts, values, beliefs, and ideas which can be placed over against criteria of rightness, goodness, appropriateness, efficiency, and effectiveness so as to conclude that a process, practice intervention, activity was/was not consequential and effective, that is, whether it ‘‘worked,’’ and if so, in the ways expected and desired, without major, troubling, or other anticipated and unanticipated consequences on/for the same or other persons, things, or processes. Conventionally, these ideas have informed the creation of the now widely accepted standards of program evaluation (Yarbrough, Shulha, Hopson, & Caruthers, 2011). To do this work well requires technical competence, facts and other information, thought, and wisdom. Needed is both Aristotle’s epistome and phronesis—practical wisdom. Doing evaluation well—responsibly, with credibility means to do it accurately, fully, openly, transparently, with awareness of one’s stance, biases in all directions, strengths and limitations, along with those of one’s approach, methods, tools, and hermeneutic practices, at least. It can, indeed should, mean to include as a corrective and for utility, that is, for use, evaluation should reflect multiple perspectives be multivocal, and include the ways of seeing, knowing, and doing, as well as the content of what is seen, known, and done, of multiple others, those with interest in what is being evaluated, how the evaluation is undertaken and completed, and how it can and well be used—for policy, program improvement, decision making, and conceptual clarification and refinement. The involvement of others can be as respondents, informants, observers, evaluation workers, or as advisers. Our focus here is on the latter. Advice is a primordial, socio-intellectual process—a system of soliciting and providing, of asking others for ‘‘input’’—perspective, technical ‘‘know how,’’ thoughts, and meanings. And of deciding whether or not to use what is offered. Advice is a way to help correct one’s limitations, or a way to solicit support, a way to increase the likelihood that a study will meet its life purpose by being used in the everyday world in ways practical, useful, meaningful, and consequential. Most researchers, including evaluators, bring advice to their work. This can be in the form of solicited and unsolicited others or interior self-inquiry—reflective practice, where one interrogates oneself to ensure, self-consciously, that one’s biases are explicit and clear; this is one way to minimize the errors brought by arrogance, incompetence, or normal human failing. Like all advice, it can prove preventive and corrective. Less common and the focus of this essay is the inclusion of advice using formal structures—groups, such as an advisory committee or council, or individuals, who can offer consultations and technical assistance on the evaluation enterprise, from contract through research to report and use. A review of the literature on the use of advisory structures in social work research texts shows that, just as with the literature on advisory structures in evaluation research texts, more is hortatory than practical, far more promotes this practice than describes, suggests, or evaluates advisory structures. Here are the two literature reviews. Both literature reviews used a similar method. Texts were found using general searches in Google Scholar, Google, Amazon.com, and the University of Minnesota library system. Publisher-specific searchers were also run on major evaluation and social work publishers. Text book indexes and table of contents were initially searched for the phrases: advice, advisory, advisory board, advisory committee, advisory group, steering committee, committee, and working group. Full text searches were made of texts available on Google Books (http://books. google.com). When such titles were available in table of contents, texts were also searched for phrases like ‘‘stakeholder engagement,’’ ‘‘advisory board,’’ ‘‘participatory evaluation,’’ ‘‘evaluation management,’’ and so forth. Few textbooks and handbooks available at the time of this survey (2011–2012) included the mention of evaluation or research advisory boards/groups/committees. No specific sections of any books examined contain specific or substantive