Quasi-experimental Evaluations Research Articles

Systematic review of the effects of schools and school environment interventions on health: evidence mapping and synthesis

BackgroundIn contrast to curriculum-based health education interventions in schools, the school environment approach promotes health by modifying schools' physical/social environment. This systematic review reports on the health effects of the school environment and processes by which these might occur. It includes theories, intervention outcome and process evaluations, quantitative studies and qualitative studies.Research questionsResearch question (RQ)1: What theories are used to inform school environment interventions or explain school-level health influences? What testable hypotheses are suggested? RQ2: What are the effects on student health/inequalities of school environment interventions addressing organisation/management; teaching/pastoral care/discipline; and the physical environment? What are the costs? RQ3: How feasible/acceptable and context dependent are such interventions? RQ4: What are the effects on student health/inequalities of school-level measures of organisation/management; teaching/pastoral care/discipline; and the physical environment? RQ5: Through what processes might such influences occur?Data sourcesA total of 16 databases were searched between 30 July 2010 and 23 September 2010 to identify relevant studies, including the British Educational Index, the Cumulative Index to Nursing and Allied Health Literature, the Health Management Information Consortium, EMBASE, MEDLINE and PsycINFO. In addition, references of included studies were checked and authors contacted.Review methodsIn stage 1, we mapped references concerning how the school environment affects health and consulted stakeholders to identify stage 2 priorities. In stage 2, we undertook five reviews corresponding to our RQs.ResultsStage 1: A total of 82,775 references were retrieved and 1144 were descriptively mapped. Stage 2: A total of 24 theories were identified (RQ1). The human functioning and school organisation, social capital and social development theories were judged most useful. Ten outcome evaluations were included (RQ2). Four US randomised controlled trials (RCTs) and one UK quasi-experimental study examined interventions building school community/relationships. Studies reported benefits for some, but not all outcomes (e.g. aggression, conflict resolution, emotional health). Two US RCTs assessed interventions empowering students to contribute to modifying food/physical activity environments, reporting benefits for physical activity but not for diet. Three UK quasi-experimental evaluations examined playground improvements, reporting mixed findings, with benefits being greater for younger children and longer break times. Six process evaluations (RQ3) reported positively. One study suggested that implementation was facilitated when this built on existing ethos and when senior staff were supportive. We reviewed 42 multilevel studies, confining narrative synthesis to 10 that appropriately adjusted for confounders. Four UK/US reports suggested that schools with higher value-added attainment/attendance had lower rates of substance use and fighting. Three reports from different countries examined school policies on smoking/alcohol, with mixed results. One US study found that schools with more unobservable/unsupervised places reported increased substance use. Another US study reported that school size, age structure and staffing ratio did not correlate with student drinking. Twenty-one qualitative reports from different countries (RQ5) suggested that disengagement, lack of safety and lack of participation in decisions may predispose students to engage in health risks.LimitationsWe found no evidence regarding health inequalities or cost, and could not undertake meta-analysis.ConclusionsThere is non-definitive evidence for the feasibility and effectiveness of school environment interventions involving community/relationship building, empowering student participation in modifying schools' food/physical activity environments, and playground improvements. Multilevel studies suggest that schools that add value educationally may promote student health. Qualitative studies suggest pathways underlying these effects. This evidence lends broad support to theories of social development, social capital and human functioning and school organisation. Further trials to examine the effects of school environment modifications on student health are recommended.FundingThe National Institute for Health Research Public Health Research programme.

Crime as Pollution

T he point of departure of this provocative article by John and Emily Eck (2012, this issue) is a striking empirical regularity uncovered by Sherman, Gartin, and Buerger (1989). Sherman et al. found that only 3% of addresses and intersections (“places,” as they were called) in Minneapolis produced 50% of all calls to the police. Weisburd and Green (1995) found that 20% of all disorder crime and 14% of crimes against persons in Jersey City, NJ, originated from 56 drug crime hot spots. Twenty five years later in a study of Seattle, WA, Weisburd, Bushway, Lum, and Yang (2004) reported that between 4% and 5% of street segments in the city accounted for 50% of crime incidents for each year over a 14-year period. Other more recent studies finding comparable crime concentrations include Brantingham and Brantingham (1999), Eck, Gersh, and Taylor (2000), and Roncek (2000). The observation that crime is concentrated at a discrete location gave rise to what has come to be called “place-based” policing in which police are deployed and managed to prevent crime at specific locations. The quintessential example of placed-based policing is greatly increasing police presence at crime hot spots. Braga’s (2008) informative review of hot spots policing summarizes the findings from nine experimental or quasi-experimental evaluations. The studies were conducted in five large U.S. cities and one suburb of Australia. Crime-incident reports and citizen calls for service were used to evaluate impacts in and around the geographic area of the crime hot spot. The targets of the police actions varied. Some hot spots were generally high-crime locations, whereas others were characterized by specific crime problems like drug trafficking. All but two studies found evidence of significant reductions in crime. Furthermore, no evidence was found of material crime

‘Policing Schools’ Strategies: A Review of the Evaluation Evidence

Background: Schools experience a wide range of crime and disorder, victimizing students and staff, and undermining attempts to create a safe and orderly environment for student learning. Police have long established programs with schools, but there has been no systematic review of evaluations of these programs, outside of police-led prevention classroom curriculum programs such as D.A.R.E. Purpose: This paper documents a systematic search to identify experimental and quasi-experimental evaluations that assess the effectiveness of non-educational policing strategies and programs in schools. Setting: Included studies took place in or around K-12 schools in the United States, Canada, and the United Kingdom. Intervention: Studies were included if they reported on a specific school-based strategy that heavily involved police and did not exclusively involve the police teaching a curriculum or program such as Drug Abuse Resistance Education (D.A.R.E.). Research Design: Systematic review of experimental or quasi-experimental evaluations Data Collection and Analysis: Only those impact studies that used experimental or quasi-experimental design, had at least one outcome measure of school crime or disorder, and were available through December 2009 were eligible. Electronic searches and other methods were used to identify published and unpublished evaluation reports. Findings: The searches identified a total of eleven quasi-experimental studies. Ten of the eleven studies would likely have received a “3” on the Maryland Scientific Methods Rating Scale, a common approach to classifying studies on the basis of internal validity. If evidence rating criteria from the U.S. Department of Education’s What Works Clearinghouse (WWC) were applied, only one study would likely receive a grade of “Level 2” evidence (acceptable with reservations) and the other ten studies would likely not meet WWC evidence screening criteria. Keywords: systematic review; police and schools; crime prevention

Assessing the effectiveness of drug courts on recidivism: A meta-analytic review of traditional and non-traditional drug courts

PurposeThe objective of this research was to systematically review quasi-experimental and experimental evaluations of the effectiveness of drug courts in reducing offending. MethodsOur search identified 154 independent evaluations: 92 evaluations of adult drug courts, 34 of juvenile drug courts, and 28 of DWI drug courts. The findings of these studies were synthesized using meta-analysis. ResultsThe vast majority of adult drug court evaluations, even the most rigorous evaluations, find that participants have lower recidivism than non-participants. The average effect of participation is analogous to a drop in recidivism from 50% to 38%; and, these effects last up to three years. Evaluations of DWI drug courts find effects similar in magnitude to those of adult drug courts, but the most rigorous evaluations do not uniformly find reductions in recidivism. Juvenile drug courts have substantially smaller effects on recidivism. Larger reductions in recidivism were found in adult drug courts that had high graduation rates, and those that accepted only non-violent offenders. ConclusionsThese findings support the effectiveness of adult drug courts in reducing recidivism. The evidence assessing DWI courts' effectiveness is very promising but more experimental evaluations are needed. Juvenile drug courts typically produce small reductions in recidivism.

The Effects of Focused Deterrence Strategies on Crime

Objective. Focused deterrence strategies are increasingly being applied to prevent and control gang and group-involved violence, overt drug markets, and individual repeat offenders. Given the growing popularity of this approach, a systematic review and meta-analysis of the extant evaluation evidence is needed to determine the crime reduction benefits of the approach. Methods. Our examination of the effects of focused deterrence strategies on crime followed the systematic review protocols and conventions of the Campbell Collaboration. As a preliminary examination of the effects of focused deterrence strategies on crime, the authors used a vote counting procedure. In our closer examination of program effects, meta-analyses were used to determine the size, direction, and statistical significance of the overall impact of focused deterrence strategies on crime. Results. We identified 10 quasi-experimental evaluations and 1 randomized controlled trial. Our meta-analysis suggests that focused deterrence strategies are associated with an overall statistically significant, medium-sized crime reduction effect. However, the strongest program effect sizes were generated by evaluations that used the weakest research designs. Conclusion. The authors conclude that this approach seems very promising in reducing crime but a more rigorous body of evaluation research needs to be developed. While the results of this review are very supportive of deterrence principles, the authors believe that other complementary crime control mechanisms are at work in the focused deterrence strategies described here that need to be highlighted and better understood.

Guardianship for crime prevention: a critical review of the literature

Cohen and Felson’s (Cohen and Felson American Sociological Review 44(4):588–608, 1979) routine activity theory posits that for a crime to occur three necessary elements must converge in time and space: motivated offenders, suitable targets, and the absence of capable guardianship. Capable guardians can serve as a key actor in the crime event model; one who can disrupt, either directly or indirectly, the interaction between a motivated offender and a suitable target. This article critically reviews the literature on guardianship for crime prevention. Our specific focus is two-fold: (1) to review the way guardianship has been operationalized and measured, and (2) to review experimental and quasi-experimental evaluations and field tests of guardianship. Research on routine activities has had an uneven focus resulting in the neglect of the guardianship component (Reynald Crime Prevention and Community Safety 11(1):1–20, 2009; Sampson et al. Security Journal 23(1):37–51, 2010; Tewksbury and Mustaine Criminal Justice and Behavior 30(3):302–327, 2003; Wilcox et al. Criminology 45(4):771–803 2007). Evaluations of guardianship-related interventions demonstrate support for the theoretical construct; however, high-quality field tests of guardianship are wholly lacking. Implications for theory and research are discussed.

Interventions for disorder and severe intoxication in and around licensed premises, 1989–2009

To systematically review rigorous evaluation studies into the effectiveness of interventions in and around licensed premises that aimed to reduce severe intoxication and disorder. A systematic search was conducted. Papers that rigorously evaluated interventions based in and around licensed premises to reduce disorder or intoxication were included. Fifteen studies were identified, three randomized controlled trials and 12 non-randomized quasi-experimental evaluations. Outcome measures were intoxication (n = 6), disorder (n = 6) and intoxication and disorder (n = 3). Interventions included responsible beverage service training (n = 5), server violence prevention training (n = 1), enhanced enforcement of licensing regulations (n = 1), multi-level interventions (n = 5), licensee accords (n = 2) and a risk-focused consultation (n = 1). Intervention effects varied, even across studies using similar interventions. Server training courses that are designed to reduce disorder have some potential, although there is a lack of evidence to support their use to reduce intoxication and the evidence base is weak.

Do driver training programs reduce crashes and traffic violations? — A critical examination of the literature

This paper reviews the evaluation literature on the effectiveness of classroom and behind-the-wheel driver training. The primary focus is on North America programs as originally taught in high schools but now also by private instructors. Studies from the United Kingdom, Australia, New Zealand and Scandinavia are also included.By far the most rigorous study to date was the experimental study in DeKalb, Georgia, U.S.A. This study used a randomized design including a control group and a very large sample size to provide reasonable statistical precision. I reexamine the DeKalb data in detail and conclude that the study did show evidence of small short-term crash and violation reductions per licensed driver. However, when the accelerated licensure caused by the training is allowed to influence the crash and violation counts, there is evidence of a net increase in crashes.The other studies reviewed present a mixed picture but the better designed quasi-experimental evaluations usually showed no effects on crash rates but almost all suffer from inadequate sample size. I show that as many as 35,000 drivers would be required in a two group design to reliably detect a 10% reduction in crash rates.The advent of GDL laws in North America and other countries has largely remedied the concern over accelerated licensure of high risk teenage drivers by delaying the progress to full licensure. Conventional driver training programs in the U.S. (30h classroom and 6h on-the-road) probably reduce per licensed driver crash rates by as little as 5% over the first 6–12months of driving. The possibility of an effect closer to 0 cannot be dismissed.Some GDLs contain an incentive for applicants to complete an advanced driver training program in return for shortening the provisional period of the GDL. The results of Canadian studies indicate that any effects of the driver training component are not sufficient to offset the increase in accidents due to increased exposure.There is no evidence or reason to believe that merely lengthening the number of hours on the road will increase effectiveness. Programs directed toward attitude change and risk taking better address the underlying cause of the elevated crash risk of young drivers but these behaviors are notoriously resistant to modification in young people.

PROTOCOL: The impact of juvenile system processing on delinquency

PROTOCOL: The impact of juvenile system processing on delinquency

Of taps and toilets: quasi-experimental protocol for evaluating community-demand-driven projects

Sustainable and equitable access to safe water and adequate sanitation are widely acknowledged as vital, yet neglected, development goals. Water supply and sanitation (WSS) policies are justified because of the usual efficiency criteria, but also major equity concerns. Yet, to date there are few scientific impact evaluations showing that WSS policies are effective in delivering social welfare outcomes. This lack of an evaluation culture is partly because WSS policies are characterized by diverse mechanisms, broad goals and the increasing importance of decentralized delivery, and partly because programme administrators are unaware of appropriate methods. We describe a protocol for a quasi-experimental evaluation of a community-demand-driven programme for water and sanitation in rural India, which addresses several evaluation challenges. After briefly reviewing policy and implementation issues in the sector, we describe key features of our protocol, including control group identification, pre-post measurement, programme theory, sample sufficiency and robust indicators. At its core, our protocol proposes to combine propensity score matching and difference-in-difference estimation. We conclude by briefly summarizing how quasi-experimental impact evaluations can address key issues in WSS policy design and when such evaluations are needed.

Physical activity and food environments: solutions to the obesity epidemic.

Environmental, policy, and societal changes are important contributors to the rapid rise in obesity over the past few decades, and there has been substantial progress toward identifying environmental and policy factors related to eating and physical activity that can point toward solutions. This article is a status report on research on physical activity and food environments, and it suggests how these findings can be used to improve diet and physical activity and to control or reduce obesity. This article summarizes and synthesizes recent reviews and provides examples of representative studies. It also describes ongoing innovative interventions and policy change efforts that were identified through conference presentations, media coverage, and websites. Numerous cross-sectional studies have consistently demonstrated that some attributes of built and food environments are associated with physical activity, healthful eating, and obesity. Residents of walkable neighborhoods who have good access to recreation facilities are more likely to be physically active and less likely to be overweight or obese. Residents of communities with ready access to healthy foods also tend to have more healthful diets. Disparities in environments and policies that disadvantage low-income communities and racial minorities have been documented as well. Evidence from multilevel studies, prospective research, and quasi-experimental evaluations of environmental changes are just beginning to emerge. Environment, policy, and multilevel strategies for improving diet, physical activity, and obesity control are recommended based on a rapidly growing body of research and the collective wisdom of leading expert organizations. A public health imperative to identify and implement solutions to the obesity epidemic warrants the use of the most promising strategies while continuing to build the evidence base.

Evaluating Crime Prevention: The Potential of Using Individual-Level Matched Control Designs

The relative merits of quasi-experimental area-level place-based evaluations and randomized control trials are the subject of contemporary debate in the criminological literature. Pertinent issues in the application of these approaches to crime prevention include the experimental validity of the former and the real-world practicality of the latter. In this article, we propose a 'middle-ground' and discuss the use of matched control designs and survival analysis conducted at the individual level. This is illustrated using information relating to a target-hardening scheme implemented in the UK. Some advantages of this approach are that it enables the tracking of samples of action and control households over time, does not demand random allocation to groups from the outset and provides insights into the longevity of the impact of measures. Disadvantages lie in the extensive data preparation required and the dependency of the process on detailed records from agencies. Language: en

The Handbook of Leadership Development Evaluation

Foreword by Laura C. Leviton. Preface. Acknowledgments. The Authors. Introduction (Jennifer W. Martineau, Kelly M. Hannum, and Claire Reinelt). PART ONE: DESIGNING LEADERSHIP DEVELOPMENT EVALUATION. 1 Experimental and Quasi-Experimental Evaluations (S. Bartholomew Craig and Kelly M. Hannum). 2 Leading with Theory: Using a Theory of Change Approach for Leadership Development Evaluations (Manuel Gutierrez and Tania Tasse). 3 EvaluLEAD: An Open-Systems Perspective on Evaluating Leadership Development (John T. Grove, Barry M. Kibel, and Taylor Haas). 4 Making Evaluation Work for the Greater Good: Supporting Provocative Possibility and Responsive Praxis in Leadership Development (Hazel Symonette). 5 Measuring Return on Investment in Leadership Development (Jack J. Phillips and Patti Phillips). PART TWO: LEADERSHIP DEVELOPMENT EVALUATION IN CONTEXT. 6 Building Leadership Development, Social Justice, and Social Change in Evaluation Through a Pipeline Program (Prisca M. Collins and Rodney K. Hopson). 7 From the Inside Out: Evaluating Personal Transformation Leadership Efforts (Sally Leiderman). 8 Evaluating Leadership Development and Organizational Performance (Nancy Vollmer LeMay and Alison Ellis). 9 The Importance of Local Context in Leadership Development and Evaluation (Larry Peters and John Baum). 10 Evaluating Community Leadership Programs (Teresa R. Behrens and Maenette K. P. Benham). 11 Evaluating Leadership as a Strategy to Transform Complex Systems (Kimberly Jinnett and Todd Kern). 12 Evaluating Leadership Development for Social Change (Kim Ammann Howard and Claire Reinelt). 13 Evaluating Youth Leadership Development Through Civic Activism (Hanh Cao Yu, Heather K. Lewis-Charp, and Michelle Alberti Gambone). 14 Evaluating Leadership Efforts for Neighborhood Transformation (Nilofer Ahsan). PART THREE: INCREASING IMPACT THROUGH EVALUATION USE. 15 Strategic Uses of Evaluation (E. Jane Davidson and Jennifer W. Martineau). 16 Evaluation for Planning and Improving Leadership Development Programs: A Framework Based on the Baldrige Education Criteria for Performance Excellence (Karl E. Umble). 17 Communication in Evaluation: A Systems Approach (Darlene F. Russ-Eft). 18 Accelerating Learning About Leadership Development: A Learning Community Approach (Deborah Meehan and Claire Reinelt). 19 Continuous Learning (Rosalie T. Torres). Afterword: Future Directions for Leadership Development Evaluation (Kelly M. Hannum, Jennifer W. Martineau, and Claire Reinelt). Name Index. Subject Index. About the Center for Creative Leadership.

Twenty-Five Years of Self-Efficacy in Career Assessment and Practice

In this article, the ways in which self-efficacy has influenced career assessment and practice during the past 25 years are explored. Focus is on research describing or evaluating specific theory-based interventions or programs with particular attention to experimental, quasi-experimental, and program evaluations studies published in refereed professional journals. Suggestions for further research and practice are given.

Randomized and Quasi-Experimental Evaluations of Program Impact in Child Welfare in Canada: A Review

Abstract: Respondents to a recent survey identified the evaluation of service effectiveness as the most pressing priority for child welfare research in Canada. After a comprehensive search, we located and reviewed 10 peer-reviewed impact evaluations, published during 1995-2005, of interventions in Canadian child welfare. Four evaluations were based on randomized controlled trials, and six on non-randomized, quasi-experimental designs. After a critical review of each study, we formulated the implications of the review, for the design and evaluation of child welfare interventions in Canada, in terms of three main needs: for more high-quality impact evaluations; for the evaluation of the effectiveness of a wider range of interventions; and for the implementation and evaluation, in the Canadian context, of interventions of mainly U.S. origin that incorporate the principles of the increasingly influential perspective of evidence-based practice.

Addressing self-selection bias in quasi-experimental evaluations of whole-school reform. A comparison of methods.

This article discusses potential sources of self-selection bias in quasi-experimental evaluations of whole-school reform models and considers how individual student-level data might be used to provide valid impact estimates. Although repeated pretreatment and posttreatment measures of student performance can provide unbiased estimates under relatively weak assumptions, such data are difficult to obtain. The article develops an instrumental variable strategy that can be used to improve on common value-added estimators when only posttreatment measures of performance are available. Using data from New York City, the author shows that the instrumental variable strategy can provide estimates of model impacts similar to those provided by a difference-in-differences estimator provided that appropriate instruments are used.

A Meta-Analysis of Corrections-Based Education, Vocation, and Work Programs for Adult Offenders

Corrections-based education, vocation, and work programs are a major rehabilitative component of our correctional system. Recent reviews of the research literature raise doubt about the effectiveness of these programs in reducing recidivism. This study meta-analyzed the recidivism outcomes of 33 independent experimental and quasi-experimental evaluations of education, vocation, and work programs and found that program participants recidivate at a lower rate than nonparticipants. The generally weak methodological character of these studies, however, prevents attributing this observed effect on criminal behavior to the activities of the programs. Future evaluative research in this area could be strengthened through the incorporation of theoretical links between the program activities and future criminal involvement and through designs that control for self-selection bias beyond basic demographic differences.

Environmental and policy interventions to promote physical activity

Background: Because most adults in industrialized countries do not meet physical activity guidelines, population-wide interventions are needed. Environmental and policy interventions are based on ecological models of behavior and have the potential to influence entire populations. Ecological models are particularly applicable to physical activity because the behavior must be done in specific physical settings. Cross-sectional data indicate that environmental and policy variables are associated with physical activity behaviors of young people and adults. Method: Seven published evaluations of environmental and policy interventions to increase physical activity were reviewed. Results: Two studies showed that placing signs encouraging stair use can be effective. Quasi-experimental evaluations provided limited evidence that broad environmental changes can be effective. Large-scale policy interventions are currently being conducted in several countries. Proposed Model: A model describing the development of policy and environmental interventions is proposed, in the hope of stimulating more research in this area. Advocacy or planning groups identify and work with agencies that control policies and environments that can be altered to increase physical activity. Educational and policy/environmental interventions are seen as complementary. Conclusion: Lack of conceptual models and the inherent difficulties of evaluation have hampered research on environmental and policy interventions. Further research is needed, and practitioners and researchers should work together to evaluate programs.

Illuminating Negative Results in Evaluation of Smoking Prevention Programs

Evaluation of program implementation can help illuminate negative results of school-based smoking prevention programs. In three conventional quasiexperimental evaluations, no statistically significant impacts of smoking prevention programs on children's knowledge, attitudes, intentions, or behavior were detected. Complementary evaluations of program implementation along several dimensions using naturalistic methods suggested reasons for null effects were different at each site. These data were used to form hypotheses and recommendations for future interventions.

The Use of Qualitative Evaluation Methods to Test Internal Validity

Since the late 1970s, qualitative research methods have gained recognition for their potential contribution to the evaluation of health promotion and educationprograms. Today, the issue is no longer whether to use quantitative or qualitative methods, but rather, how they can be combined to provide more complete assessments of the impact of a program. Each method is based on different yet complementary assumptions and each method has certain strengths that can be used to compensate for the limitations of the other. This article illustrates how qualitative and quantitative methods were combined in the evaluation of a health promotion program conducted in 23 rubber-producing plants in the United States. The qualitative case study and monitoring data were used to complement the quantitative study through the testing of threats to internal validity, and through the detection of a Type III error, that is, the inadequate implementation of the primary program intervention. In experimental and quasi-experimental evaluations it is recommended that selected in-depth case studies of both experimental and control sites be conducted Such case studies are essential for answering the question of why program activities were or were not carried out.