Impact Evaluation Tools Research Articles

The Social Justice Educational Assessment Scale: Filling a Gap in Social Justice Education Assessment and Evaluation

Nationally recognized social justice standards guide educators in developing social justice education. Absent from the guidance are tools to conduct initial formative assessment or to measure the impact of related instruction. To fill that gap, an academic researcher and 10th-grade teacher used a 3-phased, 9-step process to develop, pilot test, and evaluate a self-assessment tool called the Social Justice Educational Assessment Scale (SJEAS). In Phase 1, the team created the SJEAS items, aligning each with the Learning for Justice 9th–12th-grade social justice learning outcomes to ensure content validity. In Phase 2, they pretested the questions, revised the scale, and administered it to 322 student participants. Next, they conducted inter-item and total-item correlation tests and principal component analysis to ensure internal consistency. In Phase 3, they evaluated the SJEAS. Preliminary results indicate the SJEAS could provide educators and curriculum coordinators with a practical formative assessment and impact evaluation tool for social justice education aligned with nationally recognized learning outcomes. Complementary data, including classroom assignments and student discussions, would further enhance its value.

Co-learning through participatory evaluation: an example using Theory of Change in a large-scale EU-funded tourism intervention

Tourism interventions, as tools for social change and preservation of natural and cultural assets are inherently complex. This study presents an improved method for the evaluation of complex tourism interventions. We argue that participatory methods can promote a culture of evaluation that supports partners throughout evidencing project impacts, eliminating negative attitudes to evaluation resulting from fear of being judged on performance. We demonstrate that Theory of Change (ToC) is an effective tool that allows organisations to actively co-create and own an evaluation strategy to ensure the delivery of project outcomes. We show how ToC can be applied as a useful process and impact evaluation tool. This paper represents a novel methodological application of ToC based on participatory approaches to evaluation to disseminate knowledge and to improve decision-making in the field of tourism interventions and tourism policy making.

Economic impact of nature-based tourism.

Protected areas (PAs) can help address biodiversity loss by promoting conservation while fostering economic development through sustainable tourism. Nature-based tourism can generate economic benefits for communities in and around PAs; however, its impacts do not lend themselves to conventional impact evaluation tools. We utilize a Monte Carlo simulation approach with econometric estimations using microdata to estimate the full economic impact of nature-based tourism on the economies surrounding three terrestrial and two marine PAs. Simulations suggest that nature-based tourism creates significant economic benefits for communities around PAs, including the poorest households, and many of these benefits are indirect, via income and production spillovers. An additional tourist increases annual real income in communities near the PAs by US$169-$2,400, significantly more than the average tourist's expenditure. Conversely, lost tourism due to the COVID-19 pandemic and economic costs of human-wildlife conflict have disproportionately large negative impacts on local incomes.

Lack of adaptability in Brazilian social housing: impacts on residents

Brazilian social housing is used to address the country’s housing deficit through government programs with the massive production of standardised affordable housing. This non-systemic approach is shown to neglect residents’ changing needs over time and results in inflexible housing in relation to environmental, constructive, and social issues. The use of inappropriate spatial patterns fails to account for residents’ current socioeconomic and spatial realities. Quantitative/qualitative methods involving residents (<em>n</em> = 162) and two case studies are used to assess resilience in social housing. The costs and implications arising from a lack of housing adaptability are identified. An ‘impact assessment’ tool is employed to document the shocks and tensions experienced by residents along with the negative effects on the house and its occupants. These impacts increase the vulnerability of families and hinder their adaptive capacity. The identification of major impacts and residents’ perceptions can be used to establish priorities for action. <em><strong>Policy relevance</strong></em> Post-occupancy evaluation (POE) processes provide a more systemic approach to identifying and resolving the problems of creating housing fit for purpose. An impact evaluation tool was developed to help with a positive transformation. This indicates how architects, urban planners, and communities can act to mitigate and solve the problems experienced by users of social housing. The use of systemic POE in social housing can inform policies and planning practices that address the needs of residents. It is also intended to provide design information to agents involved in the production of social housing in Brazil, notably designers and users, in order to foster effective strategies to promote more resilient housing, especially in scenarios of intervention and renovations during use. These tools and processes contribute to fulfilling Habitat III’s New Urban Agenda and the United Nations’ Sustainable Development Goals (SDGs 1, 10, and 11).

GRADE IT-A Literacy-Based Assessment Tool for Generating Research-Based Assessment Data to Evidence the ImpacT of Anti-doping Education via Athletes' Capability to Make the Right Decision.

The International Standard for Education (ISE) mandates Code Signatories to plan, deliver, and evaluate anti-doping education. Performance-based evaluation of anti-doping education requires alignment between educational goals, content, and defined outcomes. Based on an existentialist teaching and learning philosophy, we aimed to develop and test an anti-doping impact evaluation tool, to assess the impact of anti-doping education on doping awareness, literacy (DAL), perceived trust, and legitimacy. We propose that the impact of anti-doping education is best captured through assessment of situation-specific (social) cognitive mediators of actions that influence athletes' choices in the context of sport-related goals. In phase one, we aimed to develop and test the Generating Research-based Assessment Data to Evidence the ImpacT of anti-doping education (GRADE IT) evaluation tool that comprised a set of social cognitive components: anti-doping knowledge, DAL, perceived trust, and legitimacy of anti-doping (organizations). In phase two we assessed whether anti-doping education impacts knowledge, the three DAL stages (functional, interactive, and critical literacy), perceived trust and legitimacy. Phase one enrolled 986 junior elite athletes, and we showed that all GRADE IT components performed well. After revision of the tool for phase two, we validated the assumption that anti-doping education impacts the likelihood that athletes will make the “right” choice (based on a new set of data from 1,255 junior elite athletes). Comprehensive education was associated with higher scores for all stages of DAL, as well as perceived trust and legitimacy. Even athletes reporting no education had positive scores for all included outcomes, supporting the assumption that most athletes wish to engage in clean sport behaviors and might need anti-doping education not to prevent them from doping, but rather to reinforce their commitment to clean sport. In conclusion, GRADE IT, which is available in 23 languages, is a suitable tool for application to young, emerging athletes to satisfy the ISE requirement for evaluating anti-doping education programs. Researchers and practitioners alike are advised to collect additional data to further validate the tool for adult athletes, and to apply it longitudinally to identify if changes in doping prevention policies have a delayed effect on DAL, perceived trust, and legitimacy.

Self-evaluating participatory research projects: A content validation of the InSPIRES online impact evaluation tool

Abstract Research projects involving science shops and citizen science in their promotion of participatory approaches are flourishing globally. However, an instrument evaluating the impacts of such approaches at different stages of a participatory research processes has yet to be validated. The InSPIRES H2020 project developed an impact evaluation tool for just this purpose, consisting of 64 items that reflect upon the dimensions of knowledge democracy, citizen-led research, participatory dynamics, transformative change, and integrity. In this article, we seek to test the content validity of this tool and to provide recommendations that can ensure its validity. A panel of nine experts was created to evaluate each item as regards the following three criteria: representativeness, relevance, and clarity. The Aiken’s V and Wilson Score methods were used to assess the tool’s content validity based on the experts’ ratings. Experts’ written comments were also reviewed. At the panel level, 75% of the items were considered satisfactory in relation to each of the three validity criteria. However, at the population level, 72% of the items suggested that parts of the tool were not valid and required revision. The main suggestions from the experts pointed to the need to reformulate items in which the separation between science and society appeared reinforced and to develop more items about the gender perspective of a research project. The revised version of the tool should serve as a well-founded, comprehensive evaluation instrument for on-going and future projects whose goal is to self-reflect and compare participatory research processes.

When the means modify the ends. (E)Valuating and transforming the purpose of a nonprofit organization through impact assessment.

In a context marked by a trend towards marketization and a rise in performance measures, nonprofit organizations (NPOs) suffer pressures to show that their activities have visible impacts. Based on the French pragmatist sociology’s Economies of Worth framework, we explore how NPOs may come to change what they value and what is their fundamental purpose. More specifically, we delve into the process of valuation triggered by the intervention of consultants to help a NPO develop an impact evaluation tool aimed at legitimizing its actions. Our findings show how this process led to a complete redefinition of the NPO’s “theory of change”, ultimately distantiating the NPO from its initial, explicit purposes.

When the means modify the ends. (E)Valuating and transforming the purpose of a nonprofit organization through impact assessment.

In a context marked by a trend towards marketization and a rise in performance measures, nonprofit organizations (NPOs) suffer pressures to show that their activities have visible impacts. Based on the French pragmatist sociology’s Economies of Worth framework, we explore how NPOs may come to change what they value and what is their fundamental purpose. More specifically, we delve into the process of valuation triggered by the intervention of consultants to help a NPO develop an impact evaluation tool aimed at legitimizing its actions. Our findings show how this process led to a complete redefinition of the NPO’s “theory of change”, ultimately distantiating the NPO from its initial, explicit purposes.

Sustainable development in the mining industry: towards the development of tools for evaluating socioeconomic impact in the Canadian context

In spite of its economic outputs, the Canadian mining sector has suffered from several setbacks that have compromised its reputation. This has led many of its stakeholders to adopt a shift towards a new paradigm focused on sustainable development. In this review of the literature, we identify the impact that mining activities have had on various social, economic and environmental issues in Canadian society. We also survey the criteria and indicators included in various models that may be used to evaluate the adequacy of the transition of mines towards sustainable development. Finally, the strengths and weaknesses of eight tools that have been applied to the socioeconomic impact of mines are analysed. A clear vision of prospects for promoting sustainable development in Canadian mining emerges from this survey. Our analysis reveals important factors not given adequate consideration in the models used currently to evaluate and suggests opportunities to improve these evaluation tools through identification of more relevant criteria and indicators. The research undertaken should incite stakeholders to develop social and economic impact evaluation tools that are better adapted to the Canadian context.

Teachers’ educational beliefs change through lesson study: implications for school culture

PurposeThe study aims at exploring the perspective of three English as a Foreign Language teachers after their year-long involvement in the Lesson Study project in the context of Kazakhstan in order to capture and list any perceived changes in teachers’ educational beliefs over the period of the Lesson Study intervention. The main argument of the study suggests that the school-based Lesson Study initiative is conducive to triggering changes in teachers’ educational beliefs, and thus, might lead to positive changes in school culture in Kazakhstani schools. Shaped following Hill et al., (1982) in Swales, 1990 hour-glass model of a research project (Swales, 1990), the article reflects the third concluding part of the Ph.D. thesis focusing on the implementation of the Lesson Study methodology in Kazakhstan.Design/methodology/approachThe study adopts the qualitative research design and follows the narrative inquiry methodology. The three narrative interviews (Bauer, 1996) are utilized as the main method of data collection. The data were analyzed as text following a general inductive approach (Thomas, 2003), where emerging themes were identified employing data reduction and further sub-categorized through the conceptual and theoretical lenses of the study. The emerged categories reflecting the perceived shifts in teachers’ educational beliefs were dialectically linked to implications for school culture in Kazakhstani schools.FindingsAs data suggest, the respondents’ active engagement in the Lesson Study professional learning community and exercising leadership through implementing changes in their classroom practice has made a positive impact on teachers’ rethinking their teaching practice, attitudes to students and their learning, collegiality, and professional self-identification. We conclude that, if organized properly, Lesson Study has enormous potential to facilitate changes of teachers’ educational beliefs: from direct transmission beliefs toward constructivist beliefs, from restricted professionals’ beliefs toward reflective practitioner beliefs and attitudes, toward beliefs in the power of student’s voice, and collaboration. Those shifts are linked to establishing a more positive, child-friendly and rights-based school culture with teachers’ shared visions and capacity for innovation.Research limitations/implicationsWe acknowledge that the abundance of the reported positive changes or perceived shifts in teachers’ thinking might not be the indicators of actual changes in their beliefs. We emphasize that the study was carried in a controlled context, i.e. the three ELF teachers were constantly supported, and the teacher talk was systematically guided by a trained facilitator. Warned by Giroux et al. (1999), we are aware of the major challenge of the fundamental assumption of critical pedagogy that teachers are willing and able to undertake “the practice of analyzing their practice” (p. 27) voluntarily. Thus, the question remains open: if the facilitator’s support is eliminated, will the results point to the occurrence of the disruption and disorientation as a necessary condition for the beliefs change?Originality/valueCarried out in the largely overlooked by the academic literature context of the Reform at Scale (Wilson et al., 2013) in Kazakhstan and building on the original combination of conceptual and theoretical lenses, the research contributes to the academic literature by connecting teachers’ educational beliefs, Lesson Study and school culture. The findings might be of value for the school leaders, educators, teacher trainers, and policymakers to advocate Lesson Study as a systematic approach to the whole-school improvement, as a tool to facilitate positive changes in school culture, as well as give impetus to studies employing the school culture perspective in developing Lesson Study impact evaluation tools.

Two reading assessments for youth in alternative basic skills and livelihood skills training programs

This article describes two assessment tools that have been used to assess the reading skills of youth participating in alternative basic skills and livelihood skills training programs. The Rapid Assessment of Reading Skills (RARS) was developed to identify potential participants who needed to improve their reading skills before beginning training and to assign them to the proper level of basic reading instruction. The Out-of-School Literacy Assessment incorporates RARS into a more complex impact evaluation tool. Both assessment tools can also be used to inform program design, define benchmarks, and assess impact.

CORRIM: The 20th Anniversary That Was More Than 40 Years in the Making*

Abstract In commemorating almost anything, it can be easy to forget what came before the event celebrated. In the case of the Committee on Renewable Resources for Industrial Materials (CORRIM) or, more accurately CORRIM II, the roots are deep, extending at least back to 1970. This report chronicles the events leading up to the creation of the original CORRIM effort, outlines the early history of CORRIM II, and includes observations regarding future directions. The story begins around the time of the first Earth Day, involves a congressionally authorized initiative focused on industrial raw materials that largely ignored renewables, leads to action by the wood science and technology community to include consideration in federal policymaking of wood and agriculturally derived material, and results in the formation of CORRIM I. After a period of several decades during which raw material issues loom progressively larger, the story resumes with the reengagement of the wood science community concurrent with development of a new environmental impact evaluation tool—life-cycle assessment—and concludes with incorporation of CORRIM II in 1996. The end of the story, however, is also the beginning, with much already accomplished but much yet to be done.

An evidential reasoning-based AHP approach for the selection of environmentally-friendly designs

Due to the stringent environmental regulatory requirements being imposed by cross-national bodies in recent years, manufacturers have to minimize the environmental impact of their products. Among those environmental impact evaluation tools available, Life Cycle Assessment (LCA) is often employed to quantify the product's environmental impact throughout its entire life cycle. However, owing to the requirements of expert knowledge in environmental science and vast effort for data collection in carrying out LCA, as well as the common absence of complete product information during product development processes, there is a need to develop a more suitable tool for product designers. An evidential reasoning-based approach, which aims at providing a fast-track method to perform design alternative evaluations for non-LCA experts, is therefore introduced as a new initiative to deal with the incomplete or uncertain information. The proposed approach also enables decision makers to quantitatively assess the life cycle phases and design alternatives by comparing their potential environmental impacts, thus effectively and efficiently facilitates the identification of greener designs. A case application is carried out to demonstrate the applicability of the proposed approach.

Green Template for Life Cycle Assessment of Buildings Based on Building Information Modeling: Focus on Embodied Environmental Impact

The increased popularity of building information modeling (BIM) for application in the construction of eco-friendly green buildings has given rise to techniques for evaluating green buildings constructed using BIM features. Existing BIM-based green building evaluation techniques mostly rely on externally provided evaluation tools, which pose problems associated with interoperability, including a lack of data compatibility and the amount of time required for format conversion. To overcome these problems, this study sets out to develop a template (the “green template”) for evaluating the embodied environmental impact of using a BIM design tool as part of BIM-based building life-cycle assessment (LCA) technology development. Firstly, the BIM level of detail (LOD) was determined to evaluate the embodied environmental impact, and constructed a database of the impact factors of the embodied environmental impact of the major building materials, thereby adopting an LCA-based approach. The libraries of major building elements were developed by using the established databases and compiled evaluation table of the embodied environmental impact of the building materials. Finally, the green template was developed as an embodied environmental impact evaluation tool and a case study was performed to test its applicability. The results of the green template-based embodied environmental impact evaluation of a test building were validated against those of its actual quantity takeoff (2D takeoff), and its reliability was confirmed by an effective error rate of ≤5%. This study aims to develop a system for assessing the impact of the substances discharged from concrete production process on six environmental impact categories, i.e., global warming (GWP), acidification (AP), eutrophication (EP), abiotic depletion (ADP), ozone depletion (ODP), and photochemical oxidant creation (POCP), using the life a cycle assessment (LCA) method. To achieve this, we proposed an LCA method specifically applicable to concrete and tailored to the Korean concrete industry by adapting the ISO standards to suit the Korean situations.

Using Impact Evaluation Tools to Unpack the Black Box and Learn What Works

Researchers and policy makers are increasingly dissatisfied with the “average treatment effect.” Not only are they interested in learning about the overall causal effects of policy interventions, but they want to know what specifically it is about the intervention that is responsible for any observed effects. This discusses Peck's (2003) approach to creating symmetrically-predicted subgroups for analyzing endogenous features of experimentally evaluated interventions and then it identifies several possible extensions that might help evaluators better understand complex interventions. It aims to enrich evaluation methodologists’ toolbox, to improve our ability to analyze “what works” in addressing important questions for policy and program practice.

Development of a family physician impact assessment tool in the district health system of the Western Cape Province, South Africa.

BackgroundPolicy makers in Africa are ambivalent about the need for family physicians to strengthen district health services. Evidence on the impact of family physicians is therefore needed. The aim was to develop a tool to evaluate the impact of family physicians on district health services according to the six expected roles that have been defined nationally.MethodsMixed methods were used to develop, validate, pilot and test the reliability of the tool in the Western Cape Province, South Africa. An expert panel validated the content and construction of the tool. The tool was piloted by 94 respondents who evaluated eight family physicians. Cronbach alpha scores were calculated to test the reliability of the tool. The impact of these family physicians in the pilot study was also analysed.ResultsA draft tool was successfully developed, validated, and proved reliable (Cronbach alpha >0.8). The overall scores (scale of 1–4) were: Care provider = 3.5, Consultant = 3.4, Leader and champion of clinical governance = 3.4, Capacity builder = 3.3, Clinical trainer and supervisor = 3.2 and Champion of community-orientated primary care (COPC) = 3.1. The impact on COPC was significantly less than the impact of other roles (p < 0.05).ConclusionThe Family Physician Impact Evaluation Tool can be used to measure the impact of family physicians in South Africa. The pilot study shows that the family physicians are having most impact in terms of clinical care and clinical governance, and a lesser impact in terms of clinical training, capacity-building and especially COPC.Electronic supplementary materialThe online version of this article (doi:10.1186/s12875-014-0204-7) contains supplementary material, which is available to authorized users.

The full road to impact: the experience of the Global Environment Facility Fourth Overall Performance Study

The Evaluation Office of the Global Environment Facility (GEF) has developed an approach to impact evaluation that aims to go beyond causalities and counterfactuals into replication, up-scaling and longer-term lasting systemic changes. Starting from the OECD Development Assistance Committee glossary definition of impact and building on a theory-based approach to evaluation, several methods and tools have been developed that cover various aspects of the long road from outcomes to impact. This article presents both the development over time of these methods and tools and methodological considerations. Two full-scale impact evaluations were undertaken: the first on protected areas in Eastern Africa and the second on the GEF portfolio of projects supporting the removal from production and consumption of ozone-depleting substances in countries with economies in transition. In addition several other impact studies were undertaken. The evaluations of various aspects of the progress from outcome to impact were gathered and as far as possible aggregated in the Fourth Overall Performance Study of the GEF, which was presented in 2009 to the replenishment process of the GEF. An overview is presented of progress toward impact of the GEF portfolio of finished projects on climate change mitigation and on stemming the loss of biodiversity. The Office will continue to develop and use the variety of impact evaluation tools at its disposal.

Evaluation methods for a combined research and extension program used to address starter phosphorus fertilizer use for corn in New York

Ketterings, Q. M., Czymmek, K. J. and Swink, S. N. 2011. Evaluation methods for a combined research and extension program used to address starter phosphorus fertilizer use for corn in New York. Can. J. Soil Sci. 91: 467–477. Since there is no substitute for phosphorus (P), judicious use of P fertilizer is needed to protect water quality, world P reserves and farm economics. In 2001–2003, 78 on-farm corn (Zea mays L.) P trials were conducted on New York State (NY) dairy farms as part of a statewide, integrated (research and extension), outcome-focused project. The data showed fertilizer P could be eliminated or reduced to less than 28 kg P2O5 ha−1 for fields very high or high in soil test P, respectively. We conducted: (1) case study P fertilizer management evaluations using 30 NY, and (2) a producer survey using a postcard evaluation tool, to determine project outcome potential and farmer intent to change P management. The two impact evaluations showed (1) P use could be reduced to 17 kg of P2O5 ha−1 or less with soil-test based decision making, and (2) 81% of producers were likely to change P use in future years. Statewide P fertilizer sales decreased from 12 603 Mg (8.6 kg ha−1) in 2000 to 10 092 tons (6.8 kg ha−1) in 2007, a 20% reduction in P use. These results reflect the effectiveness of the impact evaluation tools and of outcome-based projects where extension and research are integrated and farmers and farm advisors are key participants in the design and implementation of the project.

Tools for evaluation of impact associated with MSW incineration: LCA and integrated environmental monitoring system

The identification of significant pollutants emitted from the contamination source is the first step in evaluating the impact associated with anthropic activity. Municipal solid waste (MSW) incinerators are still generally perceived as great pollutant sources, in particular due to their gaseous emissions from the stack, which constitute the major effluent from the plant. In this work a life cycle assessment and an integrated environmental monitoring system were applied together, in order to obtain complete information about the incineration process and its environmental impact. The former is a proven methodology, but its application to waste management systems constitutes a relatively new field of application with a great developmental potential. The contribution of the incineration process to the different environmental impact categories was investigated, finding many avoided impacts due to energy recovery. The latter is an innovative approach that allows a remarkable understanding of impact due to a contamination source; interesting correlations were found between heavy metals both in gas emissions and in natural matrices in the surroundings.

Evaluating the purpose and benefits of continuing education in nursing and the implications for the provision of continuing education for cancer nurses.

In the light of increasing emphasis upon mandatory continuing education in nursing, the need for cancer nurses to update and improve their knowledge and skills is heightened. The challenges lie in how to keep nurses up-to-date, broaden their outlook and ensure that patients benefit. In this paper the literature on the purpose and benefits of continuing education is reviewed, and the implications of the Post-Registration Education and Practice Project (PREPP) for cancer nurses are addressed. Arguments about a definition of 'continuing learning' rather than 'continuing education' are discussed, since outcomes vary depending upon the definition adopted. Inconsistencies between the benefits claimed and those measured are highlighted. Attention is drawn to the inconclusive nature of the studies that have measured impact; some of the problems encountered in the provision of adequate education and its accessibility to nurses; and the continuing discrepancy of perceived need between nurse and manager. Specific impact evaluation studies relating to continuing education in cancer nursing are reviewed. Recommendations for providing education with measurable outcomes are drawn from the literature, while a case is made for the alternative perspective of lifelong learning where outcomes are neither specific nor measurable. In conclusion, the questions of who should be responsible for education and the need for further development of impact evaluation tools are addressed.