Quasi-experimental Evaluation Research Articles

The Effectiveness of Recipe4Health: A Quasi-Experimental Evaluation

IntroductionFood as Medicine is increasingly recognized as an important strategy for addressing the related challenges of food insecurity and nutrition-related chronic conditions. Food as Medicine refers to integration of food-based nutrition interventions into healthcare to prevent and treat disease. However, there is limited evidence to understand the effectiveness of Food as Medicine. MethodsRecipe4Health, a comprehensive Food as Medicine program, was implemented in 4 Federally Qualified Health Centers in California for patients with food insecurity and/or nutrition-related chronic conditions. Patients were referred by a healthcare provider to a ‘Food Farmacy’ (16 weekly produce home deliveries) alone or in combination with a ‘Behavioral Pharmacy’ (16 weekly group visits). A quasi-experimental study with pre/post surveys (4 months) and propensity score matched controls for Electronic Health Record (EHR) outcomes over 12 months was conducted. Participants were 2,643 Recipe4Health patients and 2,643 controls identified from 1/2020 to 12/2022; data were analyzed from 2023-2024. ResultsThere was a significant increase in produce consumption from baseline to four months (0.41 servings/day [0.11, 0.72], p=0.007) in the Food Farmacy in combination with Behavioral Pharmacy. Compared to controls, there were improvements in non-HDL cholesterol for the Food Farmacy alone (-17.1 mg/dl[-26.9, -7.2], p<0.001) and in combination with Behavioral Pharmacy (-17 mg/dl [-28.3, -5.8], p=0.003) at 12 months. Compared to controls, HbA1c significantly decreased in the Food Farmacy alone at 12 months (-0.37%, 95% CI [-0.65, -0.08]; p=0.01), but not the Food Farmacy with Behavioral Pharmacy. ConclusionsRecipe4Health resulted in improvements in diet and multiple clinical health outcomes, such as non-HDL cholesterol and HbA1c.

Implementing and evaluating the School-Wide Positive Behaviour Support (SWPBS) model in Norwegian primary schools

ABSTRACT The Norwegian version of the School-Wide Positive Behaviour Support (SWPBS) model was tested in two quasi-experimental evaluation studies with 65 primary schools (grade 4–7). The impact of the N-PALS model on externalising behaviour and learning climate in class was tested at 28 elementary schools that implemented the full-scale model, while 17 schools implemented a short version with emphasis on the universal level, and 20 schools served as controls with regular practice. After 3 years, the staff at the schools implementing the full model reported significantly fewer behavioural incidents and more use of praise and encouragement compared to the staff at the control schools. And four months after the collective training of staff at schools implementing the universal short version, significantly fewer negative behavioural incidents were reported. School size and implementation quality moderated the outcomes in the two studies. In conclusion, the school-wide intervention model effectively reduced the negative behavioural incidents and improved the learning climate and social relations in the classrooms.

The basic income for care leavers in Wales pilot evaluation: Protocol of a quasi-experimental evaluation.

This study will evaluate the Basic Income for Care Leavers in Wales pilot (BIP), which is the most generous basic income scheme in the world. A cohort of care-experienced young people who become aged 18 during a 12-month enrolment period (July 2022-June 2023) are receiving £1,600 (before tax) per month for two years, and the Welsh Government intends this to have a range of benefits. This evaluation will examine the impact of BIP, the implementation of the pilot and how it is experienced, and its value for money. The study is a theory-based quasi-experimental evaluation, and the design and methods are informed by ongoing co-production with care-experienced young people. We will estimate the impact of BIP on participants using self-reported survey data and routinely collected administrative data. This will include outcomes across a range of domains, including psychological wellbeing, physical and mental health, financial impact, education, training and volunteering. Comparisons between temporal (Welsh) and geographical (English, using administrative data) controls will be done using coarsened exact matching and difference in differences analysis. The process evaluation will examine how BIP is implemented and experienced, primarily through monitoring data (quantitative) and interview, observational, and focus group data (qualitative). The economic evaluation will take a public sector and a societal perspective to identify, measure and value the costs and outcomes of BIP, and to synthesise the evidence to inform a social cost-benefit analysis at 24 months post-intervention. BIP is unusual in that it targets a wide range of outcomes and is available to an entire national cohort of participants. The evaluation also has several practical constraints. Therefore, the study will use a range of methods and triangulate between different analyses to assess how successful it is. Findings will inform policy in relation to care leavers, social security and basic income studies worldwide.

Understanding the mechanisms of change in social norms around tobacco use: A systematic review and meta-analysis of interventions.

Tobacco use spreads through social networks influencing social norms around tobacco use. However, the social norms scholarship is extremely diverse and occasionally conflicting, complicating efforts to understand how best to leverage social norms to reduce tobacco use. This study systematically reviewed and meta-analyzed this vast terrain by focusing on social norms measurement and mechanism, and intervention effectiveness and modality aimed at changing social norms around tobacco use and actual tobacco use. We searched Scopus, PubMed, PsycInfo, Clinicaltrials.gov, ProQuest Dissertations, the Cochrane Trial Registry, as well as the websites of the Society for Research on Nicotine and Tobacco, the Open Science Framework, medrXiv and the Truth Initiative for experimental and quasi-experimental evaluation studies of interventions designed to shift tobacco use. We included studies written in English from inception to 30 May 2024. We only included studies which noted social norms or social influence as part of the intervention design or set of measured variables. We excluded studies with only one time point, without an intervention being evaluatedand those not published in English. Study screening followed the PRISMA 2020 guidelines, and was conducted by at least two independent reviewers who resolved discrepancies through discussion and consensus. All included studies were analyzed in a narrative synthesis, and those providing sufficient statistics for tobacco and social norms outcomes were included in meta-analyses, which were performed separately for tobacco outcomes and social norms outcomes. Study outcomes were transformed into a standardized mean difference (Hedges' g) and several meta-regressions were fit to explore sources of heterogeneity using a robust variance estimation specification to handle effect size dependency. A total of 95 studies met inclusion criteria for the narrative synthesis, 200 effect sizes from 86 studies were included in the tobacco outcomes meta-analysis, and 66 effect sizes from 29 studies were included in the social norms outcomes meta-analysis. Nearly 90% of included studies were conducted in high-income settings, with the remainder conducted in middle-income settings. No studies were conducted in Latin America or on the African continent. Social norms change interventions had a statistically significant effect on reducing tobacco use and pro-tobacco social normative perceptions [g = 0.233, 95% confidence interval (CI) = 0.166, 0.301, P < 0.001 and g = 0.292, 95% CI = 0.090, 0.494, P = 0.007, respectively]. Interventions were commonly conducted among schoolchildren in classrooms through multicomponent education sessions, often coupled with regular 'booster' sessions over time. Among adult populations, motivational interviewing and other counselling approaches were used in some cases, and few interventions leveraged digital technologies to change social norms. Social norms change interventions appear to be effective for reducing tobacco use and pro-tobacco social normative perceptions. In particular, leveraging role models appears to be the most effective social norms change mechanisms for tobacco control.

A quasi-experimental evaluation of RADAR: response awareness, de-escalation, and referral for people with behavioral health issues

ABSTRACT This paper describes the implementation and evaluation of RADAR (Response Awareness, De-Escalation, and Referral), a hybrid CIT/co-response program for people with behavioral health issues (PBHI). RADAR emphasized information sharing between police, PBHI, and behavioral health professionals and targeted outreach to PBHI by a police officer and clinician working in partnership. We evaluated RADAR using a quasi-experimental analysis of calls for service and incident data, an officer survey, and interviews with key personnel. While RADAR did not significantly reduce calls, incidents, or physical contact and resistance relative to a comparison jurisdiction, the survey and interviews suggest that it did help to shift the department’s culture and organizational perceptions around collaborative responses to PBHI. The practical implications of our findings for police agencies seeking to implement co-responder programs include the need to focus on sustainability; relationship building between the police, PBHI, and clinicians; and technological solutions to improve data collection and evaluation efforts.

Impact of a clinical pharmacist-led, artificial intelligence–supported medication adherence program on medication adherence performance, chronic disease control measures, and cost savings

BackgroundChronic diseases are the leading cause of disability and death in the United States. Clinical pharmacists have been shown to optimize health outcomes and reduce health care expenditures in patients with chronic diseases through improving medication adherence. ObjectiveThe primary objective of this study was to evaluate a pharmacist-led, artificial intelligence (AI)–supported medication adherence program on medication adherence, select disease control measures, and health care expenditures. MethodsThis was a multicenter, retrospective, quasi-experimental evaluation from January 2019 to December 2019 (preimplementation) and January 2021 to December 2021 (postimplementation). This pharmacy-driven service focuses on improving medication adherence and patient outcomes through AI-supported analytics, individual patient case review, and pharmacist-led individual patient outreach. The primary end point was to determine whether implementation improved medication adherence in 3 medication-related measures: medication adherence for hypertension (MAH), medication adherence for cholesterol (MAC), and medication adherence for diabetes (MAD). Secondary outcomes were to evaluate reductions in select chronic diseases control measures and cost savings of this service after implementation of this service. ResultsThis medication adherence service was deployed across 10,477 patients: 60.6% of patients were in at least 1 medication-related measure, generating 2762 actionable medication adherence gaps. After the implementation of this pharmacist-led program, medication adherence improved in all 3 disease state measures (MAH 5.9% improvement, MAC 7.9% improvement, MAD, 6.4% improvement), and Medicare Star ratings also improved. The percentage of patients with diabetes who reached their A1c goal also increased (75.5%-81.7%). Furthermore, reductions in overall health care expenditures were seen per member per month in patients who were adherent in comparison with those who were nonadherent (hypertension 31% cost savings, hyperlipidemia 25% cost savings, diabetes 32% cost savings). ConclusionThis clinical pharmacist–driven service leveraged technology and patient connection to increase medication adherence in patients with chronic disease states and led to improvement in select disease control measures and substantial health care cost savings.

Effects of nutritional education on metabolic syndrome parameters in a sample of adults in Subotica, Northern Serbia

The aim of this study is to assess the changes in metabolic syndrome parameters among adults after the implementation of nutrition education. The research was conducted as a prospective, longitudinal, quasi-experimental evaluation study from May to December 2022. The research sample included 31 employees at the College of Vocational Studies in Subotica. The study was conducted in three phases: Initial assessment (anthropometric, clinical, biochemical measurements and eating habits), Nutrition educations (onloine and face-to-dace) and Evaluative assessment (six months after the initial assessment). Data analysis included methods of descriptive and comparative statistical analysis. Nutrition education was conducted for 31 participants, but only 18 responded to the evaluative assessment. Two thirds (62%) of the participants were overweight ad had elevated cholesterol levels. Elevated triglyceride values were registered in more than a half of the participants and high blood glucose levels in 13% of participants. More than a third of the participants had three or four risk factors for the development of metabolic syndrome. After nutrition education, distribution of participants according io BMI significantly changed (I—normal: 37%; overveigt: 31%; obese:29%; E—normal:10%; overveigt:29%; obese:19%; p = 0,001). Significant changes in the distribution of total cholesterol levels (I: 5.7 mmol/l; E: 5.4 mmol/l) was determined at the evaluative assessment (p = 0.005). Statistically changes in the distribution of breakfast (p = 0.001) and vegetables (p = 0.013) intake were observed. There were not determined significant changes in other antropometric, clinical and biochemical parameters after nutrition education. Although the nutrition education intervention showed some changes in the prevalence of risk factors, significant changes require more intensive, long-term interventions tailored to individual needs.

Parenting pain away: Quasi-experimental study of an eHealth learning platform to evaluate acceptability, feasibility, and utilisation of parent-led pain management.

To evaluate the impact of an eHealth educational resource about infant procedural pain management, given during the prenatal period, on feasibility, acceptability, knowledge, self-efficacy, and involvement. Routine health care requires newborns to have painful procedures (e.g., intramuscular injection). The impacts of untreated pain in neonates are widely recognised but adoption of effective procedural pain management strategies in clinical practice varies. There is clear evidence supporting the effectiveness of parent-led pain management during procedures (e.g., skin-to-skin care) and reputable resources to raise awareness among parents are warranted. Our team co-created Parenting Pain Away, a website to equip parents with evidence to assist with managing the pain of procedures and empower them to be involved. A quasi-experimental evaluation using a pre/post intervention design with low-risk expectant parents. Of the 41 participants, before intervention exposure participants were familiar or had used skin-to-skin care (n = 33), breastfeeding (n = 30) and sucrose (n = 13) as pain management. Most participants (n = 38) desired more information on how to be involved. Providing access to Parenting Pain Away during pregnancy was supported and participants ranked the website above average using the System Usability Scale. Parenting Pain Away did not have a statistically significant influence on outcomes. Participants reported variation in clinical support with parent-led pain management. A multifaceted approach is recommended to maintain infant procedural pain management. Equipping parents with knowledge related to infant pain management using an eHealth approach satisfied their information desires. The study findings are important considerations for perinatal care providers, policy makers, and families to finally achieve adequate procedural pain management. This study used the STROBE checklist, adhering to EQUATOR guidelines. A stakeholder group (expectant parents, parents, perinatal researchers, clinicians, and administrators) was created to inform the study design and intervention.

Quasi-experimental quality evaluation of educational-purposed user-generated contents under a stochastic multi-criteria environment

ABSTRACT As social platforms experience an influx of diverse content from users, the need to determine high-quality contributions becomes crucial, especially for educational purposes. This paper highlights the pivotal role of quality in assessing how educational-purposed user-generated content (UGC) shapes user experiences, fosters engagement, and establishes credibility. This study proposes a computational framework using a quasi-experimental evaluation through the sorting-based ELimination Et Choice TRanslating Reality, termed ELECTRE-SORT, with a dataset randomly generated from normally distributed user evaluations. Considering the diverse nature of contents, the method evaluates 16 educational-purposed UGC videos from different online media platforms (i.e. Facebook, YouTube, TikTok). These videos were categorized based on their concordance and discordance to three (3) main criteria: content quality, design quality, and technology quality. Employing the ELECTRE-SORT reveals that most UGC videos (i.e. 14 out of 16) fall into the “medium quality” category, possessing a considerable standard for the quality of educational purpose content. Their characteristics generally satisfy the quality attributes and can be used to guide the development of future relevant UGC videos. Finally, to demonstrate the robustness of the proposed approach, we presented a sensitivity analysis by designing different weight assignments to the quality attributes. Practical insights are outlined in this work.

Supporting Prospective Teachers in Learning Psychology: A Digital Case Exploration Tool

This article describes the implementation of a digital case-exploration tool and its use in teacher education. The cases were constructed according to empirically based psychological theories and represent school-related scenarios in which the students could choose among options for action. After selecting an action, its psychological consequences are presented in text or video including theoretical explanations how and why the selected action would lead to the presented consequences. We present first quasi-experimental evaluation data demonstrating the tool's usefulness regarding learning outcomes, students’ active participation, and motivation.

Medical training program size and clinical staff productivity and turnover.

The objective of this analysis was to evaluate the effect of resident program training size on clinician productivity and turnover in the Veterans Health Administration (VHA), the largest education and training platform for medical professionals in the United States. We retrieved administrative data on training programs and training facilities from the VA Office of Academic Affiliations and the VHA Corporate Data Warehouse. Data on primary care physician shortage areas were retrieved from the Health Resources and Services Administration. We used a quasi-experimental instrumental variables 2SLS design and constructed an exogenous predicted training allocation treatment variable as a function of the total national training program allocation. The outcome was clinical staff productivity and turnover. Secondary analyses stratified results using Health Professional Shortage Areas data (HPSA). Data were obtained for a national dataset of 141 VHA medical facilities and 26 specialties that hosted training programs across 11 years from 2011 to 2021 (N = 132,177). Instrumental variables results showed that on average, an increase of one training slot in a specialty leads to a decrease of 0.039 visits per standardized clinic day (p < 0.001) and a 0.02 percentage point increase in turnover (p < 0.001). The direction of this association varied by specialty: while psychiatry and psychology specialties saw a decline in productivity, fields such as primary care and cardiology experienced an increase in productivity. HPSA stratified results indicate that negative effects on productivity and turnover are driven by areas with little to no primary care physician shortage, whereas shortage areas experienced a small increase in productivity and no effect on turnover. This quasi-experimental evaluation indicates that resident training program size is associated with reduced productivity and increased turnover in specialties such as psychiatry and in facilities with high baseline productivity. However, in specialties like primary care and cardiology, as well as areas with shortages of primary care, larger training programs are associated with increased productivity.

Regional differences in police officer misperceptions: a quasi-experimental evaluation of sexual assault investigations training in Kentucky

ABSTRACT Our study fills an important research gap by investigating the differences in myths and misperceptions about sexual assault survivors among police officers (N = 388) and evaluating the effects of sexual assault investigations training across geographic regions. First, we assessed police officers’ pretraining rape myth acceptance and misperceptions of crime victim reporting behaviours. Second, we used a Solomon four-group quasi-experimental design to assess pretesting effects and evaluate the effect of training and jurisdiction type on officers’ adherence to rape myths and misperceptions of trauma. We used Ordinary Least Squares (OLS) models to evaluate regional differences in officers’ adherence to myths and misperceptions about survivors, the main effects of training, effects of training when considering jurisdiction type, and the moderating effects of officer jurisdiction type on training outcomes. Results showed lower pretraining scores for urban/suburban officers and significant improvements in post-training scores across geographic regions. In addition, officer jurisdiction type failed to moderate –change—the relationship between training and outcomes. This research improves our understanding of officer misperceptions regarding sexual assault survivors and the impact of specialised sexual assault training in different geographic contexts.

Can we Break the School to Prison Pipeline through Partnerships? A Quasi-experimental Evaluation of the Effects of School Justice Partnerships

Can we Break the School to Prison Pipeline through Partnerships? A Quasi-experimental Evaluation of the Effects of School Justice Partnerships

Results of the Sukuma Ndoda ("Stand up, Man") HIV Self-Screening and Assisted Linkage to Care Project in Johannesburg: A Quasi-Experimental Pre-Post Evaluation.

HIV testing rates among South African men lag behind rates for women and national targets. Community-based HIV self-screening (HIVSS) distribution and follow-up by community health workers (CHWs) is a scalable option to increase testing coverage, diagnosis, and treatment initiation. We provided HIVSS and assisted linkage to care to men not recently tested (within the past 12 months) residing in high-HIV-burden areas of Johannesburg. CHWs distributed HIVSS in 6 clinic catchment areas. Follow-up to encourage confirmatory testing and antiretroviral therapy initiation was conducted through personal support (PS) or an automated short message service (SMS) follow-up and linkage system in 3 clinic areas each. Using a quasi-experimental pre-post design, we compared differences in the proportion of men testing in the clinic catchment areas during the HIVSS campaign (June-August 2019) to the 3 months prior (March-May 2019) and compared treatment initiations by assisted linkage strategy. Among 4793 participants accepting HIVSS, 62% had never tested. Among 3993 participants with follow-up data, 90.6% reported using their HIVSS kit. Testing coverage among men increased by 156%, from under 4% when only clinic-based HIV testing services were available to 9.5% when HIVSS and HIV testing services were available (z = -11.6; P < 0.01). Reported test use was higher for men followed through PS (99% vs. 68% in SMS); however, significantly more men reported reactive self-test results in the SMS group compared with PS (6.4% vs. 2.0%), resulting in more antiretroviral therapy initiations in the SMS group compared with PS (23 vs. 9; P < 0.01). CHW HIVSS distribution significantly increases testing among men. While PS enabled personalized follow-up, reporting differences indicate SMS is more acceptable and better aligned with expectations of privacy associated with HIVSS.

A Quasi-Experimental Evaluation of the Lifelong Links Intervention for Reducing Homelessness amongst Care Leavers

Abstract Despite the fact that care leavers are at significantly higher risk of homelessness than their peers who have not experienced care, there is a paucity of information on evidence-based interventions to address this risk. Lifelong Links is a program developed and delivered by the Family Rights Group in England. The intervention aims to ensure that a child in care has a positive support network around them to help during their time in care and into adulthood. In this article, we analyse the outcomes of a quasi-experimental evaluation of the Lifelong Links program. Specifically, we make use of a combination of coarsened exact matching, and a triple-differences approach (or difference in difference in differences analysis). Results of our analyses show that Lifelong Links is associated with a reduction in the risk of becoming homeless for care leavers aged eighteen to twenty in the years following its implementation. Our most robust model showed a reduction of around 10 per cent in the risk of a young care leaver being deemed at risk of, or experiencing homelessness, suggesting that improving ties between young people in care and their birth families (and/or building other sources of support) could have beneficial impacts on housing outcomes into adulthood.

Evaluating the Effectiveness of Expert Modeling Videos on Nursing Student Competency Behaviors.

Expert modeling videos (EMVs) have shown promise in improving students' performance in simulation. However, research evaluating the impact of EMVs is limited to major performance areas, lacking exploration into specific student competency behaviors. This study evaluated the effect of an EMV intervention on undergraduate nursing students' behavioral competencies as measured by the Creighton Competency Evaluation Instrument (CCEI). Using a quasi-experimental pragmatic evaluation design, students in a medical surgical nursing course (n=160) viewed either an expert model demonstration video (experimental) or expert model discussion video (control). Students' behavioral competencies were measured and compared between groups using the CCEI. Students who viewed an expert model demonstration video performed at a higher level of competency in 11 of the 18 CCEI behaviors. Using EMVs in nursing simulation may improve students' ability to achieve clinical competency in nursing specific behaviors.

An exploratory analysis of moderating effects in a non-randomized, quasi-experimental evaluation of a college-based mental health literacy curriculum

Objective: To assess possible moderating variables that influence the effectiveness of a college-based mental health literacy curriculum, Mental Health Awareness and Advocacy (MHAA). Participants: Students at a Western University in the United States (MHAA treatment group: N = 474; Comparison group: N = 575). Methods: Using a non-randomized, quasi-experimental design over six semesters (2017–2019), students in the treatment and control groups reported on key demographic factors, experiences with mental health, and their knowledge, self-efficacy, and behaviors related to mental health literacy in a pretest/post-test format. Linear mixed effects models were used to assess moderation. Results: MHAA curriculum’s efficacy varied based on participants’ current depression and anxiety levels, whether they had prior personal experience with a mental health concern, whether they had been diagnosed with a mental health concern, and whether they had experienced suicidal thoughts. Conclusions: Reducing anxiety and depression among students may increase the efficacy of the MHAA program.

Effects of a structured educational program on nursing students' level of knowledge and awareness toward child neglect and abuse: A quasi-experimental evaluation.

Nurses have a key role in the recognition, reporting, and channelization of child abuse and neglect. Nurses should have sufficient knowledge about the presentation of child abuse and neglect. They also need to understand their role in reporting any instance where they detect cases of abuse and neglect. The aim of this study is to determine the effect of a structured Child Neglect and Abuse Educational Program (CNaA-EP) on the level of knowledge and awareness of nursing students about child neglect and abuse. This study conducted with nursing students used a pretest-post-test quasi-experimental design. The third-year nursing students were divided into two groups: experimental (n = 25) and control groups (n = 30). The intervention group received CNaA-EP. In both groups, pretests were performed using the "Child Abuse and Neglect Awareness Scale," and "Child Abuse and Neglect Knowledge Test" The posttest was conducted 1 week after the 14-week CNaA-EP course was completed. Numbers, percentages, chi-square test, t-test, Kolmogorov-Smirnov test, Fisher's exact test were conducted using SPSS 22.00 software. The education group's scores of child abuse/neglect awareness (t = -2.768 p < 0.05) and child abuse/neglect knowledge (t = -12.416 p < 0.01) were significantly higher than those of the control group. After the education program, a significant difference was found between the education and control groups on the core measures. A child neglect and abuse education program was an effective method of educating nursing students on knowledge and awareness of child abuse and neglect. For this reason, education programs should be planned and implemented to increase awareness about abuse and neglect problems that threaten children's health and development.

Proactive policing and traffic disturbances: A quasi-experiment in three Israeli cities

Abstract This paper describes a quasi-experimental evaluation of a reform in Israel (‘EMUN’), which attempted to institutionalize problem-oriented policing on a national scale. The current study examines the effect of this reform on tackling traffic disturbance and road bullying offences. We compared three police stations that chose to deal with traffic offences using the tools and techniques provided through the reform (treatment stations) with five police stations that were matched - using a specially designed algorithm - on several criteria, including similar trends of traffic offences (comparison stations). Each treatment station was compared to two comparison stations using a difference-in-differences approach. In five out of six comparisons there were large and significant reductions in documented traffic disturbances in the targeted areas of the treatment stations compared to the control stations. We also found evidence of significant diffusions in crime control benefits in two of the treatment stations. However, there was evidence of significant geographical displacement to the buffer zone in the largest treatment stations. We attribute this to differences in the nature of the areas targeted and discuss the relative harms and benefits. The findings of the study show that institutionalizing a variety of evidence-based policing strategies has a promise not only for classic crimes (such as property and violence), but also for incivilities and quality-of-life offenses.

Towards more robust evaluation of policies and programmes in education: identifying challenges in evaluating DEIS and Reading Recovery

ABSTRACT In recent years, countries including the UK and USA have seen advancements in the use of Randomised Controlled Trials in education, progress that has not been mirrored in Ireland. Ireland does not have a strong tradition of using experimental or quasi-experimental evaluation designs for monitoring and evaluation of education policy despite well-articulated commitments to the latter activities. An example is the evaluation of the Delivering Equality of Opportunity in Schools (DEIS) programme where little is known about the causal impact of DEIS on student outcomes, despite considerable State investment over almost 20 years. There appears to have been comparatively little critical review of the literacy and numeracy programmes offered to DEIS schools, with those included at the outset largely still on offer. Here, we draw on the wider literature on the evaluation of complex interventions and consider what lessons may be learned for Ireland, illustrating challenges and opportunities by focusing on DEIS and Reading Recovery (a literacy support programme widely used in DEIS schools). We advocate for greater use of experimental designs where feasible and appropriate; further gathering of comparable measures of achievement at pupil level to facilitate monitoring; and detailed consideration of quasi-experimental methods that can support causal conclusions.