PROTOCOL: The Effects of School‐Based Decision Making on Educational Outcomes in Low and Middle Income Contexts: A Systematic Review

Abstract

Education is internationally understood to be a fundamental human right that offers individuals the opportunity to live healthy and meaningful lives. Evidence from around the world also indicates that education is vital for economic and social development, as it contributes to economic growth and poverty reduction, sustains health and well-being, and lays the foundations for open and cohesive societies (UNESCO, 2o14). In recognition of the vital importance of education, governments across the globe have made a substantial effort to expand and improve their education systems, as they strive to meet the Education for All goals, adopted by the international community in 1990. These efforts have borne remarkable results; it is estimated that the number of out-of-school children has halved over the last decade (ibid, p. 53). However, there are still serious barriers to overcome, particularly in terms of access, completion and learning (Krishnaratne, White, & Carpenter, 2013). Access to education - particularly for girls, poor children and children in conflict-affected areas - remains a crucial issue. The 2013 Global Monitoring Reports claims that an estimated 57 million children are still out of school, over half of whom are in sub-Saharan Africa (UNESCO, 2014, p.53).1 Furthermore, despite increases in enrolment numbers, there has been almost no change since 1999 in the percentage of students dropping out before the end of the primary cycle. The evidence also indicates that many children enrolled in school are not learning. Recent estimates suggest that around 130 million children who have completed at least four years of school still cannot read, write or perform basic calculations (UNESCO, 2014, p. 191). Many governments have attempted to address this worrying situation, while also improving efficiency and reducing costs within the education sector, by decentralising decision-making processes. Decisions about curricula, finance, management, and teachers can all be taken at one or more of several administrative levels: centrally at the national or federal state level, by provinces/regions within a country, by districts or by schools. The devolution of decision-making authority to schools has been widely adopted as the preferred model by many international agencies, including the World Bank, the U.S. Agency for International Development (USAID) and the UK Department for International Development (DFID), as it is assumed that locating decision-making authority within schools will increase accountability, efficiency and responsiveness to local needs (Gertler, Patrinos, & Rubio-Codina, 2008). Often described as ‘school-based’ or ‘community based’ management, the devolution of decision-making authority to schools includes a wide variety of models and mechanisms. These differ in terms of which decisions are devolved (and how many), to whom decision-making authority is given, and how the decentralisation process is implemented (i.e., through ‘top-down’ or ‘bottom-up’ processes). School-based decision-making can be used to describe models in which decisions are taken by an individual principal or head teacher, by a professional management committee within a school, or by a management committee involving local community members. This last model may simply imply an increased role for parents in the management and activities of the school, or it may result in more active provision of training and materials to empower broader community involvement (Krishnaratne et al., 2013). The devolved decisions can be financial (e.g. decisions about how resources should be allocated within a school; decisions about raising funds for particular activities within a school; etc.), managerial (e.g. human resource decisions, such as the monitoring of teacher performance and the power to hire and fire teachers; decisions relating to the management of school buildings and other infrastructure; etc) or related to the curriculum and/or pedagogy (e.g. decisions related to the articulation of a school's curriculum; decisions about how elements of a national curriculum will be taught and assessed within a given school; etc.). In order to support the process of decision-making, many models involve some means of providing information to community members on the performance of an individual school (or school district) relative to other schools (Barrera-Osorio & Linden, 2009). All of these models and mechanisms are considered to potentially increase accountability and responsiveness to local needs by bringing local community members into more direct contact with schools, and to increase efficiency by making financial decisions more transparent to communities, thereby reducing corruption and incentivising investment in high quality teachers and materials. For the purposes of this review, ‘school-based decision-making’ includes any model in which at least some of the responsibility for making decisions about planning, management and/or the raising or allocation of resources is located within schools and their proximal institutions (e.g. community organisations), as opposed to government authorities at the central, regional or district level. The ‘intervention’ considered within this review, therefore, is any reform in which decision-making authority is devolved to the level of the school. Within this broad definition, we anticipate that the available evidence will relate to the three main mechanisms outlined above: (1) devolving decision-making around management to the school level; (2) devolving decision-making around funding to the school level; and (3) devolving decision-making around curriculum, pedagogy and other aspects of the classroom environment to the school level. School-based decision-making is widely promoted by donors in lower-income countries as a means for improving educational quality and is often taken up enthusiastically by national governments. Both generally articulate the ultimate outcome of school-based decision-making models as being a positive change in student outcomes (including but not restricted to learning outcomes). In addition to learning outcomes (most often measured through standardised tests of cognitive skills), there are many other possible student learning outcomes which may be valued by schools, donors and governments, such as improved student ability to demonstrate psychosocial and ‘non-cognitive’ skills. Changes in student aspirations, attitudes (such as increased appreciation of diverse perspectives) and behaviours (such as the adoption of safe sex practices) could also be considered important educational outcomes. However, it is clear that devolving decision-making to the level of the school does not lead directly to such outcomes. Rather, school-based decision-making is likely to impact on outcomes via a number of causal pathways. Reforms that increase accountability and responsiveness to local needs are assumed to lead to positive stakeholder perceptions of (and engagement in) educational provision, which, in turn, is expected to increase enrolment, attendance and retention and to reduce corruption within schools. It is also presumed that increased accountability will encourage schools to make recruitment decisions on the basis of teacher performance, rather than mechanically relying on qualifications or allowing for nepotism to interfere. Such personnel practices, in turn, are seen to lead to reduced teacher absenteeism, increased teacher motivation and, ultimately, improvements in the quality of teaching within schools. It is also assumed that local communities will encourage schools to adopt more locally relevant curricula, which can then have a positive impact on the quality of teaching and student opportunities to learn. At the same time, decentralised funding mechanisms and other reforms aimed at increasing efficiency within schools, particularly when combined with efforts to increase community participation, are presumed to result in more resources being available to schools, another important factor in improving educational quality (Krishnaratne et al., 2013). Increased efficiency is, in turn, assumed to affect the cost of educational provision, a proximal outcome highly valued by governments in less well-resourced settings. School-based decision-making mechanisms, therefore, result in a number of proximal (or intermediate) outcomes, in addition to the final outcomes mentioned above. These proximal outcomes include increased enrolment, improved equality of access, improved attendance, improved retention, improved progression, and higher quality educational provision. Furthermore, there is growing evidence that decentralisation reforms may actually have unintended and sometimes negative effects in certain political and economic circumstances (Banerjee et al., 2008; Bardhan & Mookherjee, 2000, 2005; Carr-Hill, Hopkins, Lintott, & Riddell, 1999; Condy, 1998; Glassman, Naidoo & Wood, 2007; Pherali, Smith & Vaux, 2011; Rocha Menocal & Sharma, 2008; Rose, 2003; Unterhalter, 2012). Decentralising decision-making may lead to elite capture at the local level and/or further corruption within school systems, for example, or may limit educational opportunity for marginalised ethnic groups. There is some consensus in this literature that decentralisation is only likely to have a positive impact on outcomes when (a) there is clear government policy and/or regulations about the powers and role played by different agencies and stakeholders; (b) there are sufficient financial resources available within the system; and (c) there is some form of democratic culture (see De Grauwe et al., 2005; Lugaz et al., 2010; Pherali et al., 2011). This body of evidence highlights the contingency of the effects of decentralisation, linked to important interactions between formal structures of decision-making and informal structures of power and authority within bureaucracies, communities and schools. In addition to the ways in which enabling or constraining conditions and circumstances can alter the outcomes of school-based decision-making reforms, it is clear that differences in implementation can also affect outcomes. Those vested with the authority to make decisions on behalf of the school must have the capacity and knowledge to make such decisions, or their decisions are unlikely to have a positive impact on outcomes (World Bank, 2004). Furthermore, each link in the causal chain rests on certain assumptions which must be met in order for a change in the location of decision-making to have the desired effect(s). For instance, the assertion that involving parents and community members in the hiring and firing of teachers (an ‘accountability’ mechanism employed in many contexts) will improve quality of teaching rests on the assumption that (a) parents and community members will be able to identify high quality teachers who should be retained and/or rewarded and (b) the incentives provided will positively impact student learning. This is not always achieved. In some contexts, teacher incentive schemes have been found to have a negative impact on overall student learning, if, for instance, they create perverse incentives for teachers to block the enrolment of low-performing students in order to maintain high average test scores within their classrooms (Glewwe, Ilias, & Kremer, 2003). The impact of school-based decision-making models is, therefore, likely to differ depending on a wide variety of implementation factors, relating to the objective of the reform, the particular decisions that are devolved, the individuals given decision-making authority and the nature of the decision-making process. Figure 1 (below) is a visual depiction of our understanding of the causal pathways, contributing factors and underlying processes that appear to affect the impact of school-based decision-making on educational outcomes. Our conceptual framework is not presented here as a definitive map of the existing evidence. Rather, it is proposed as a ‘working hypothesis’ to help guide the implementation of this review (Oliver, Dickson & Newman, 2012, p. 68). As such, we have used the framework to generate specific review questions and define our review methodology (as recommended by Anderson et al., 2011). We plan to significantly revise, modify and potentially simplify (or disaggregate) the framework during the review process, in order to more accurately reflect the current body of evidence related to school-based decision-making in lower-income contexts. This may include articulating separate theories of change for some of the individual mechanisms, depending on the evidence available. Conceptual Framework Source: Original Although the rhetoric around decentralisation suggests that school-based management has a positive effect on educational outcomes, there is limited evidence from low-income countries of this general relationship. In reality, much of the decentralisation literature focuses exclusively on the proximal outcomes of school-based decision-making (described above). This is likely due to the relative ease of measuring such outcomes, as well as the shorter time period generally required to identify impact on intermediate outcomes. Evidence from the U.S. suggests that there can be a time lag of up to 8 years between the implementation of a school-based management model and any observable impact on student test scores, although intermediate effects may be more rapidly identifiable (World Bank, 2007, p. 13). This may explain why studies with different time scales have found mixed evidence around the impact of school-based management models on student learning outcomes (Barrera-Osorio & Linden, 2009; Jimenez & Sawada, 2003; Sawada & Ragatz, 2005). As a result of these trends within the empirical literature, existing reviews on school-based decision-making have also tended to focus on proximal outcomes (e.g. Guerrero, Leon, Zapata, Sugimaru, & Cueto, 2012, on teacher absenteeism; Petrosino, Morgan, Fronius, Tanner-Smith, & Boruch, 2012, on student enrolment). There are very few that consider the full range of relevant outcomes, including student learning. Those that do have tended to focus exclusively on one particular mechanism (e.g. Bruns, Filmer & Patrinos, 2012, on accountability reforms), rather than considering the full range of school-based decision-making models. The comprehensive reviews that do exist (e.g. Santibanez, 2007; World Bank, 2007) need updating, as they (a) rely on literature that is now nearly ten years out of date, (b) focus almost exclusively on Central America, referencing almost no evidence from other low- or middle-income countries, and (c) do not report the use of systematic searches, critical appraisal and statistical synthesis of study effect sizes. There is, therefore, a need for a current globally-comprehensive systematic review of the impact of school-based decision making on a wide range of educational outcomes. Furthermore, existing reviews on this topic tell us almost nothing about why school-based decision-making has positive or negative effects in different circumstances. The exclusive focus on evidence collected through impact evaluations and quasi-experimental designs has significantly limited the policy relevance of these reviews as this approach has (a) resulted in a very small (<60) number of studies and (b) prevented any analysis of the conditions and circumstances under which school-based decision-making models can have a positive impact. We anticipate that the outcomes of this review will be useful for a wide range of stakeholders. In particular, policy-makers, at both the national and supranational levels, will benefit from the evidence linking decentralised decision-making processes to a wide range of potential outcomes and the analysis of underlying conditions that affect impact. School-based management is a key component of education reform across the world, and it is a particular focus of education activities sponsored by many of the core development agencies, including the World Bank, USAID and DFID. It is, therefore, crucial that we gain deeper understanding of how school-based decision-making affects a broad range of educational outcomes in both positive and negative ways and how such models can be strengthened and improved. The timing of this review will help to increase the potential impact of the results, as it coincides with ongoing conversations within the development community around the most appropriate focus (and strategies) for the next round of international development goals post-2015 (see http://post2015.org/; http://www.beyond2015.org). This review aims to answer the following overarching review question: What is the evidence around how decentralising decision-making to the school level affects educational outcomes in low and middle income contexts (LMICs)? The primary objective of the study, therefore, is to gather, assess and synthesise the existing evidence around how the decentralisation of decision-making to schools affects a broad range of educational outcomes in LMICs (question 1 above). This objective will be accomplished by examining the results of causal studies (e.g. those with an appropriate counterfactual) that consider the impact of at least one model of school-based decision-making on any of the proximal or final outcomes depicted in the conceptual framework above. Such analysis will allow us to report on all relevant quantitative measures of educational outcomes. Although we recognise that focusing on quantitative studies may preclude our ability to discuss outcomes usually considered harder-to-measure, we anticipate that the results will be useful, both for illuminating the ways in which school-based decision-making models do impact outcomes and for highlighting the current gaps in the evidence base. We also aim to draw conclusions about why particular models of school-based management work in some lower-income country contexts (and not in others), in order to make determinations about the particular contextual and implementation factors which act as barriers to – or enablers of – effective outcomes (question 2 above). This objective will be accomplished by examining evidence collected through a broader range of studies, including but not limited to that obtained from the included studies referenced in response to question 1. Given the broader scope of this second review question, studies do not need to be causal in nature in order to be included. In addition to examining the overall (positive and negative) effects of decentralisation processes on outcomes, we aim within this review to examine how changes in decision-making processes might impact differentially on diverse groups within societies. We are particularly concerned with groups which have historically experienced poor service delivery and/or demonstrated poor educational outcomes (e.g. marginalised or low-performing students). This will be accomplished by examining: (1) whether the interventions outlined in the included studies specifically target particular populations and (2) whether the included studies report any sub-group analysis for such populations. These objectives will be accomplished through the implementation of a high quality systematic review, relying on existing methodological guidance from the Campbell Collaboration and the EPPI-Centre at the Institute of Education (e.g. Becker et al., undated; Gough, Oliver & Thomas, 2012; Hammerstrom, 2009; Shadish & Myers, 2004). As this review aims to both aggregate the demonstrated effects of school-based decision-making on educational outcomes and draw conclusions around the conditions and circumstances that can affect outcomes, we have elected to conduct a mixed methods review, following the guidelines developed by Snilstveit (2012) for ‘effectiveness plus’ systematic reviews in international development. As such, we will use our conceptual framework throughout the review to guide the search strategy, decisions regarding the inclusion and exclusion of studies, coding, and synthesis. In keeping with ‘effectiveness plus’ review methodology, we will also consider different kinds of evidence in relation to our two review sub-questions. As the first review question is an ‘effectiveness’ question, studies included for synthesis will need to have an appropriate comparator or control group (or to have employed an appropriate method of constructing a counterfactual or control for confounding during analysis). However, a broader range of evidence, including studies based on qualitative data, will be reviewed in response to the second sub-question, as other methods are likely to be particularly useful for clarifying which external conditions and/or implementation factors may substantially affect outcomes. Studies will be included in the review if they meet the following selection criteria. We will be looking exclusively at evidence related to primary and secondary schools in LMICS. Studies of both public and private sector provision will be included. In order to be included, studies must be based in at least one context classified (at the start of a given intervention) as either ‘low’ or ‘middle’ income, according to the World Bank classification. We will exclude evidence collected in LMICs located within Central and Eastern Europe or the former USSR. We have defined ‘school-based decision-making’ quite broadly for the purposes of this review. There were two reasons for this decision: (1) As impact evaluation has been used only sparingly in the literature, we felt it important to use a broad definition in order to capture adequate breadth of literature to respond to the review questions; and (2) By constraining our search to only particular models of school-based decision-making, we thought it likely that we would miss potentially common features across models which may be found to have a significant impact on particular outcomes. Given the need for breadth, we have elected to include any study that empirically investigates an intervention utilising at least one of the three school-based decision-making mechanisms outlined in the conceptual framework (i.e. school management reforms, funding reforms, or curricular/pedagogical reforms). This is likely to include a long list of particular interventions, such as school management committees, school ‘report cards’, and capitation grants (both school and individual). An exhaustive list of intervention models has not been developed a priori, so as to allow for the broader possible range of potentially includable studies. In reference to the first review question, we are likely to find comparisons between groups in which no school-based decision-making reform has been attempted and groups in which some school-based decision-making reform has been attempted. We may also find comparisons between groups in which different school-based decision-making reforms have been attempted (e.g. funding reforms versus school management reforms). Both will be included, although they will be distinguished from one another during synthesis. Comparisons must be contemporaneous (i.e. in comparisons between interventions, the interventions must have been implemented during the same time period, and, in comparisons between a reform group and a non-reform group, data must reflect the same time period). Comparison groups are not a prerequisite for inclusion in relation to the second review question. As school-based models of decision-making can yield a wide range of outcomes (both positive and negative), we will not be excluding studies on the basis of a pre-determined list of outcomes. However, for inclusion in reference to both review questions, studies must empirically investigate the connection between school-based decision-making and at least one educational outcome (either proximal, e.g. attrition, equality of access, increased enrolment; or final, e.g. student learning as captured by test scores, psychosocial and non-cognitive skills). Studies are eligible which analyse data at the level of the child or at community or sub-national (e.g. district) level, as well as the level of the school. Studies reporting analyses based on these different methods and levels of data will be separated in the synthesis (see below). Studies will be excluded in relation to this question which do not present quantitative information on proximal or final outcomes, or which present comparison groups at country level or higher. Given the wide diversity of studies likely to be included in the review, we will assess the validity of all included studies prior to synthesis using risk of bias categories (see below). Included studies in reference to the second review question will need to meet the standards of transparency, appropriateness, rigour, validity, reliability and cogency set out in the DFID ‘How to note’ on ‘Assessing the Strength of Evidence’ (2014) in order to be included for synthesis. Studies of any follow-up duration and studies with multiple follow-ups will be included. However, during coding, the specific time-lag will be captured for each included study, so that we can consider temporal differences that are likely to affect synthesis. Language: As members of our team are fluent in English, French, Spanish and Portuguese, we intend to include studies written in any of these languages. Studies written in other languages will be excluded, unless English translations are available. Publication Status: We will include journal articles, books, conference papers and institutional grey literature, including reports and process evaluations, in the review. Unpublished papers showing empirical evidence (such as dissertations and theses, empirical studies showing null and/or negative results and the like) will also be included. The first four search strategies will be conducted at the start of the review process. Existing systematic reviews will first be identified through the 3ie Database of Systematic Reviews, the EPPI-Centre Database of Education Research, and the Campbell Collaboration Library. The reference lists for any potentially relevant reviews will be screened for potentially includable studies. We will then conduct detailed searches, with the support of our colleagues at the EPPI Centre, in the following databases and websites: These resources have been selected because they are likely to yield evidence that is relevant to the review questions while also representing a wide range of disciplinary perspectives. We have also made an explicit effort to include resources that are likely to help us identify grey literature and literature published within LMIC contexts. In addition, we will search for potentially relevant articles in the following academic journals: Compare, Comparative Education Review, International Journal of Educational Development, Journal of Development Economics, Economics of Education Review, Education Economics, World Development, World Bank Economic Review, and World Bank Research Observer. We will also reach out to a small list of experts who are known to have published widely on school-based management, in order to determine if there might be potentially relevant studies that have been completed but are not yet published. Existing systematic reviews (e.g. Petrosino et al., 2012) have indicated a lack of relevant studies on education decentralisation in developing countries published prior to 2000. We will therefore limit electronic searches and journal hand searches to 2000 onwards. However, we will search reference lists of existing literature reviews (e.g. Santibanez, 2007 and World Bank, 2007) and systematic reviews (e.g. Petrosino et al., 2012) to identify additional relevant literature, including studies published before 2000. Once the initial search has been completed, all potential titles and abstracts will be imported into EPPI-Reviewer, and a duplicate check will be completed. We will then begin the process of screening and coding studies (described in more detail below). Once we have decided on our list of studies for quality appraisal, we will complete our final search strategy by checking the reference lists of all included studies – and consulting the Web of Science, Google Scholar and Scopus to track citations of our included studies – in order to identify any key sources that we might have missed during the initial search. If any such sources are identified, they will be included prior to quality appraisal. Relying on the expertise of the EPPI Centre, we have assembled a list of controlled terms which tend to be used in the main electronic databases in reference to Concept2. The list of search terms involved in Concept 1 has been developed through an iterative process. First, members of the review team proposed a list of models, mechanisms and common phrases which have dominated the literature on school-based management in recent years. A test search was then conducted in ERIC and the IIEP decentralisation database, using this initial list of terms, plus some controlled terms for ‘primary education’ and LMICs and the date restriction ‘published since 2000’. The test search yielded 170 records in the IIEP database and 152 records in ERIC. A repeated search in ERIC, without the primary school terms, yielded 483 records. A sample of 350 of these records, plus all of the records generated by the first two searches, were then hand-screened by the review team to generate further search terms for inclusion in the final search strategy. Our final search will be conducted using the following search concepts/terms. In order to be captured by the search, studies must reference at least one term from each concept in either the title or the abstract. In databases allowing for complex Boolean searches, controlled terms will also be searched (designated as SU terms in the search concept below). These terms vary by database, so the term