Evaluation Of Social Programs Research Articles

Toward an evaluation of evaluation methods: a commentary on the experimental approach in the fields of employment, work, and professional training

This paper compares two families of evaluation techniques for social programmes in the field of employment and income. The first refers to the experimental methods and in particular to the randomised control trials, which are of increasing use in the evaluation of social programmes. The second refers to the structural economic modelling of individual behaviours supposed to be important determinants of the policy being evaluated. Although the emphasis of the paper is more on experimental methods, because of their relative novelty in the economic discipline, it essentially argues that no method should be preferred to the other on a priori grounds. They both have advantages and limitations in their applicability. Ideally they should be used concomitantly as they have some power to partially compensate the deficiencies of each other.

Prioridades regionales y nacionales de investigación en salud, Perú 2010-2014: un proceso con enfoque participativo y descentralista

Peru has performed many efforts to identify national health research priorities since 1974 through processes historically planned based on expert opinions, with little impact. It was decided to generate a change in the management of research in order to overcome the weaknesses of the previous processes, applying a methodology with a participative and decentralized approach. In order to establish the regional and national research priorities of the key stakeholders, the Instituto Nacional de Salud (Peru) developed a process of citizenship consult through three phases i) advocacy and workshops in 20 regions; ii) a workshop for the analysis of the Concerted National Health Plan with 200 experts in Lima; iii) the national forum, with 500 representatives in 50 working tables. The research priorities of Peru for the period 2010- 2014 are: research to recognize the problems of health human resources, to recognize the mental health problems, impact evaluations of the social programs for reduction of children malnutrition, impact evaluation of social programs of the actual interventions in maternal mortality and operative research and impact evaluation of interventions in communicable diseases.

The Role for an Evaluator: A Fundamental Issue for Evaluation of Education and Social Programs

This paper discusses one of the fundamental issues in education and social program evaluation: the proper role for an evaluator. Based on respective and comparative analysis of five theorists’ positions on this fundamental issue, this paper reveals how different perspectives on other fundamental issues in evaluation such as value, methods, use and purposes can result in different roles for evaluators, and how such difference can affect evaluators’ responsibilities in different stages of an evaluation. Then the paper proposes its own resolution of the issue of evaluator’s role and discusses its implication and limitations.

The cost of crime to society: New crime-specific estimates for policy and program evaluation

Estimating the cost to society of individual crimes is essential to the economic evaluation of many social programs, such as substance abuse treatment and community policing. A review of the crime-costing literature reveals multiple sources, including published articles and government reports, which collectively represent the alternative approaches for estimating the economic losses associated with criminal activity. Many of these sources are based upon data that are more than 10 years old, indicating a need for updated figures. This study presents a comprehensive methodology for calculating the cost to society of various criminal acts. Tangible and intangible losses are estimated using the most current data available. The selected approach, which incorporates both the cost-of-illness and the jury compensation methods, yields cost estimates for more than a dozen major crime categories, including several categories not found in previous studies. Updated crime cost estimates can help government agencies and other organizations execute more prudent policy evaluations, particularly benefit–cost analyses of substance abuse treatment or other interventions that reduce crime.

Programme evaluation: Maintaining quality in higher education

The evaluation of educational or social programmes is paramount for establishing success or impact in higher education. Evaluation questions about programme goals (e.g. better performance of first-year students) or about the quality of programme strategies (design and implementation) and effectiveness of delivery (coordinator inputs and management) reflect not only sound research methodology, but also the desire for real knowledge claims regarding programme effect. The model for programme evaluation presented in this article is generic and could be applied to any academic or support programme. The research is presented here as it pertains to a structured mentoring programme for undergraduate students and focuses on programme implementation and programme monitoring. While programme monitoring focuses more on programme delivery, outcome evaluation or impact assessment looks at the actual benefits experienced by the clientele or beneficiaries (Babbie and Mouton 2003), in this case students in higher education. The evaluation of programmes is important to establish success or impact of interventions. Therefore, a logic model to enable programme coordinators to evaluate their work is presented and discussed.

Timing and Duration of Exposure in Evaluations of Social Programs

Impact evaluations aim to measure the outcomes that can be attributed to a specific policy or intervention. Although there have been excellent reviews of the different methods that an evaluator can choose in order to estimate impact, there has not been sufficient attention given to questions related to timing: How long after a program has begun should one wait before evaluating it? How long should treatment groups be exposed to a program before they can be expected to benefit from it? Are there important time patterns in a program's impact? Many impact evaluations assume that interventions occur at specified launch dates and produce equal and constant changes in conditions among eligible beneficiary groups; but there are many reasons why this generally is not the case. This paper examines the evaluation issues related to timing and discusses the sources of variation in the duration of exposure within programs and their implications for impact estimates. It reviews the evidence from careful evaluations of programs (with a focus on developing countries) on the ways that duration affects impacts.

The Generalist Model: Where do the Micro and Macro Converge?

Although macro issues are integral to social work, students continue to struggle with the acquisition of knowledge and skills pertaining to larger systems. Educators have developed innovative methods to integrate learning across systems of various sizes however it appears an imbalance persists. This challenge is supported by baccalaureate student responses to a social work program evaluation. Four years of data from 295 undergraduate students revealed that they felt less prepared to practice with larger, macro systems. Changes in curriculum to reflect collaboration and holism, and more research are needed to adequately provide macro learning and macro practice opportunities within the generalist model and in the context of the current socio-economic-political environment.

Avaliação de programas sociais: comparando estratégias de análise de dados

Este artigo apresenta uma discussão sobre questões metodológicas em relação à avaliação de programas sociais, tomando como exemplo o caso do Plano Nacional de Qualificação do Trabalhador - PLANFOR. O delineamento da pesquisa foi de um estudo quase-experimental, com três grupos: dois experimentais e um controle. A pesquisa foi realizada durante os anos de 2001 a 2004, com coleta de dados antes e depois do programa, englobando 360 instituições, sendo uma por município, cobrindo todas as regiões brasileiras. Para a coleta de dados, utilizou-se um instrumento com indicadores quantitativos (sociais e econômicos). Foram comparadas quatro estratégias de análise de dados: análise de variância (ANOVA) de mensurações repetidas (pré e pós-teste), ANOVA com escores de ganho, ANOVA com categorização dos sujeitos no pré-teste (blocking) e análise de covariância (ANCOVA). Foram ainda consideradas interpretações alternativas para os efeitos encontrados. Por fim, discutiu-se a adequação de tais estratégias para a avaliação de programas sociais.

Combining survey and ethnographic methods to improve evaluation of conditional cash transfer programs

Drawing on evaluations of conditional cash transfer programs for the governments of Nicaragua and Turkey, this article explores how survey and ethnographic methods have been combined to improve program evaluation. Quantitative and qualitative research designs for the two countries are described, benefits of each method discussed, and examples provided of how findings of the different approaches complemented, explained, illuminated, or contradicted each other. The surveys provide reliable measures of program impacts on human capital. The qualitative research provides explanations of why we do or do not find these impacts, and explores how social processes and social relations are affected by, and in turn shape responses to, the programs. While many official evaluations now require mixed methods, and these have impacts on policy, there is still a considerable way to go with respect to how methods are integrated and how mixed approaches are appreciated in social program evaluation.

Between Individual and Collective Action: Citizen Participation and Public Oversight in Mexico's Oportunidades Programme

Programmes against poverty implemented in Mexico over the last 30 years have become experimental laboratories for citizen participation in social policy: food programmes in the 1970s and 1980s, implemented through the establishment of committees for rural supply; Solidarity National Programme (Pronasol) in the 1990s, also implemented by establishing thousands of solidarity committees; and the PROGRESA/Oportunidades programme, implemented between the end of last century and early years of the current one. All those programmes have different institutional designs in order to boost (but often in fact limiting and guiding) the forms and capacities of people’s real participation in the design, operation, societal control and evaluation of social programmes (Isunza and Hevia 2006).

Incorporating nonmarket time into benefit–cost analyses of social programs: An application to the self-sufficiency project

Benefit–cost analysis is used extensively in the evaluation of social programs. Often, the success or failure of these programs is judged on the basis of whether the calculated net benefits to society are positive or negative. Almost all existing benefit–cost studies of social programs count entire increases in income accruing to participants in a social program as net benefits to society. However, economic theory implies that the conceptually appropriate measure of the impact of a government program on any group of individuals is the net change in their surplus (or economic rent), rather than the net change in their income. For example, if a social program causes increases in income by increasing work hours, then the lost nonmarket time that accompanies these increases has value that needs to be counted as a cost when assessing the merits of that program. In this paper, we develop a methodology for incorporating lost nonmarket time into benefit–cost analyses of social programs. We apply our methodology to the Self-Sufficiency Project (SSP), an experimental welfare-to-work program tested on a pilot basis in two provinces in Canada during the 1990s. We find that if losses in nonmarket time are ignored, SSP yields a substantial positive net benefit to society. However, if losses in nonmarket time are taken into account, the net societal benefits are greatly reduced, even becoming negative in certain instances. We conclude that future benefit–cost analyses of social programs must take effects on nonmarket time into account in order to give a more accurate picture of the net benefits of the program.

Do We Know What Works? A Systematic Review of Impact Evaluations of Social Programs in Latin America and the Caribbean

Poverty is a multidimensional phenomenon closely related to the low accumulation of human capital and scant economic opportunities for the poor. Government efforts to help households overcome poverty require actions on multiple fronts. The governments of Latin-America and the Caribbean are making important efforts to scale up innovative programs to increase the capabilities and economic opportunities of the poor. Increasingly, the implementation of these programs has been accompanied by significant efforts and resources to evaluate their impact and effectiveness. These evaluations have produced many useful lessons. This study reviews a set of rigorous impact evaluations, placing emphasis on extracting lessons to assess the development effectiveness and cost-effectiveness of these interventions. The results presented in this review show a mostly positive picture of the impact of the programs evaluated. Results of the evaluations have also proven to be useful for identifying program weaknesses, such as problems with targeting mechanisms or groups that are not reaping the full benefits of the program, and inducing the adjustments necessary to increase program effectiveness. The evaluation results and the experience in implementing these programs also raise various issues for the reform of social and fiscal policies in order to make public social expenditure more effective, pro-poor, and fiscally sustainable. Scaling up well-targeted programs is key to improving the distributive impact of spending.

New Challenges Facing Interinstitutional Social Science and Educational Program Evaluation Research at Academic Health Centers: A Case Study from the ELAM Program

Since the mid-1990s, the protection of human subjects through institutional review boards (IRBs) has progressively broadened in scope. In this case study, the authors describe their challenges in effectively handling IRB processes to conduct educational and social sciences research within academic health centers, particularly (1) complications in conducting longitudinal interinstitutional research that involves multiple IRBs, each with different procedures that changed over ten years; and (2) factors affecting consent form and survey response rates when applying the biomedical IRB process to obtain the consent of human subjects for participation in social and educational research. The authors had a unique opportunity to follow the effect of changes in consent forms (from no form to a one-page form to a three-page form requiring signature of a witness), ways of administration (in person or by mail), and time of administration (at the time of the program or years later) on consent form and survey response rates among medical and dental school faculty members. The authors explore the extended timelines required for data collection and increased costs in dealing with these issues, as well as the effects on response rates of consent form language and administration procedures. The authors recommend strategies to lessen adverse effects of dealing with multiple IRBs at different institutions for social science and educational research, and discuss policy implications for funders, institutions and investigators.

My life as an applied social psychologist

I want to describe a personal odyssey that interweaves my career with three main areas of my work: social psychology, program evaluation, and mental health services research. This essay is a personal reflection on the many social, personal, and random factors that influenced the research paths that I followed. The organization is primarily chronological and I pick up the threads that run though my professional life in no particular order. Writing a professional autobiography is by its very nature an egocentric activity. Some who know me will think that this should pose no challenge to me.

Developing a Stakeholder-Driven Anticipated Timeline of Impact for Evaluation of Social Programs

The authors present a stakeholder-driven method, the earliest anticipated timeline of impact, which is designed to assess stakeholder expectations for the earliest time frame in which social programs are likely to affect outcomes. The utility of the anticipated timeline of impact is illustrated using an example from an evaluation of a comprehensive community initiative in which such a timeline was developed using the concept-mapping methodology. The benefits of such a timeline, including for planning programs and evaluations, are explored. Some potential problems that might arise when developing a stakeholder-driven timeline of impact are also discussed.

Evaluation Research: An Overview

Evaluation research can be defined as a type of study that uses standard social research methods for evaluative purposes, as a specific research methodology, and as an assessment process that employs special techniques unique to the evaluation of social programs. After the reasons for conducting evaluation research are discussed, the general principles and types are reviewed. Several evaluation methods are then presented, including input measurement, output/performance measurement, impact/outcomes assessment, service quality assessment, process evaluation, benchmarking, standards, quantitative methods, qualitative methods, cost analysis, organizational effectiveness, program evaluation methods, and LIS-centered methods. Other aspects of evaluation research considered are the steps of planning and conducting an evaluation study and the measurement process, including the gathering of statistics and the use of data collection techniques. The process of data analysis and the evaluation report are also given attention. It is concluded that evaluation research should be a rigorous, systematic process that involves collecting data about organizations, processes, programs, services, and/or resources. Evaluation research should enhance knowledge and decision making and lead to practical applications.

Evaluation and Communication

This article identifies a surprising dearth of studies that explicitly link communication and evaluation at substantive, theoretical, and methodological levels. A three-fold typology of evaluation studies referring to communication is proposed and examples given. The importance of organizational communication in program delivery is stressed and illustrative studies reviewed. It is proposed that organizational communication should be considered in all program evaluations and that this should be approached through communication audit. Communication audits are described with particular reference to established survey questionnaire instruments. Two case studies exemplify the use of such instruments in the evaluation of educational and social programs.

Psychology and the Art of Programme Evaluation

This article suggests that psychologists may find value in the literature on programme evaluation, both theoretically and methodologically. Programme evaluation is an eclectic and diverse field and its literature reflects the contributions of persons trained within a variety of disciplines. It draws on a number of fields, which include management and organisational theory, policy analysis, education, sociology, social anthropology and the literature on social change. As such, the literature on programme evaluation may have value for psychologists planning evaluations of social programmes, in providing access to evaluation approaches and models developed within these different traditions. In terms of the breadth of perspectives and research traditions on which the evaluation literature draws, different forms of evaluation research can contribute to a psychology in South Africa which deals with multiple values and issues. On a theoretical level, this article suggests that the issues and debates reflected in the evaluation literature (e.g., those on empowerment) mirror debates that have occurred within the mainstream of psychology over the past 20 years. For this reason, the issues raised in the evaluation literature are relevant to the development of psychology as a discipline. The approaches and models proposed for the evaluation of social programmes are also potentially useful on a methodological level, particularly to those psychologists who work in community settings.

Forum: intervención social, juventud y salud sexual

In recent decades one of the most visible facets in the youth issue has been the formulation and development by public agencies and organized civil society of various social programs and services targeting low-income youth in the fields of health work culture and leisure. However the educational practices and effects of these actions are largely unknown. With the objective of providing support for the implementation and evaluation of social programs for low-income youth the Ford Foundation supported research in Rio de Janeiro Brazil and Bogota and Cali Colombia concerning the repercussions that such social interventions have on young peoples lives and the theoretical and methodological basis for these educational actions. Despite their specificities the common thread in the studies is the use of qualitative methodologies in the analysis of the social groups cultural manifestations and material living conditions in the two countries. (excerpt)

The ethics of federal social program evaluation: A response to Jan Blustein

The ethics of federal social program evaluation: A response to Jan Blustein