Evaluation Of Development Programs Research Articles

Estimating Treatment Effects with Big Data When Take-up is Low: An Application to Financial Education

Abstract Low take-up of interventions is a common problem faced by evaluations of development programs. A leading case is financial education programs, which are increasingly offered by governments, nonprofits, and financial institutions, but which often have very low voluntary participation rates. This poses a severe challenge for randomized experiments attempting to measure their impact. This study uses a large experiment on more than 100,000 credit card clients in Mexico. The study shows how the richness of financial data allows combining matching and difference-in-difference methods with the experiment to yield credible measures of impact, even with take-up rates below 1 percent. The findings show that a financial education workshop and personalized coaching result in a higher likelihood of paying credit cards on time, and of making more than the minimum payment, but do not reduce spending, resulting in higher profitability for the bank.

Evaluation Program for the Career Development of Indonesian Navy Civilian Personnel using the Cipp Evaluation Model and Analytical Hierarchy Process (Ahp)

Nowadays, the implementers of Civilian Personnel (Pegawai Negeri Sipil/PNS) Career Development Programs within the Indonesian Navy are still one with military personnel where their existence is only complementary so that they become less optimal/less focused. In its implementation, it is only carried out by officials at the level of Head of Affair (Superintendent/PNS class III/c), the impacts are low of performance, competence and confidence that can be related with weakening work performance. This study aims to provide an evaluation of Civilian Personnel career development programs within the Indonesian Navy by using the CIPP evaluation approach (Content, Input, Process, Product) and Analytical Hierarchy Process (AHP) methods. Based on the results of the program evaluation study, it can be seen that the results of the Context aspect evaluation amounted to 85.84% with the Excellent category; Input aspects amounted to 76.38% with good categories; Process aspects were 79.77% with good categories; Product aspect is 82.48 in the excellent category. So, the overall evaluation of the Civilian Personnel career development program is 81.12% with an Excellent category. The results of the recommendations state that career development programs need to be revised on several aspects of the criteria.

A web-based clinical pedagogy program to enhance registered nurse preceptors' teaching competencies – An innovative process of development and pilot program evaluation

BackgroundRegistered Nurse Preceptors guide students to integrate theory into practice, assess clinical competencies, and enhance problem-solving skills. Researches have indicated that the teaching competencies of nurse preceptors can be transferred to students' clinical learning. ObjectivesThe aims of the study are to develop a web-based clinical pedagogy (WCP) program for Registered Nurse Preceptors and conduct pilot program evaluation. DesignA three-step process was applied to integrate the theoretical framework, evidence from the systematic review, and content validity by the experts and pilot test with the Registered Nurse Preceptors in the content and technical development of the program. The WCP program has unique features including use of dashboard, interactive videos, consultation with experts, discussion forum and backend data analysis. ResultsA committee of six content experts evaluated the comprehensiveness, appropriateness, and relevancy of the program. The item-Content Validity Index (CVI) score ranged from 0.83 to 1.00 and the scale-CVI score was 0.87, which indicated that the WCP program had a strong content validity. Ten nurse preceptors were invited to use the WCP program. Preceptors shared that the website was easy to use and navigate. They commented that the videos in each module are beneficial for nurses to understand the real situation in the clinical setting. This feature also makes the website more interactive. Feedback from preceptors was subsequently used to further refine the program. Discussion and conclusionThe WCP program is an evidence-based program that provides a comprehensive coverage on clinical teaching pedagogy and assessment strategies. The unique web-based technology and interactive features provide a platform for nurse preceptors to discuss clinical encounters with peers and consult experts. The flexible and resource-rich nature of web-based learning encourages nurses to use it for continuing education.

Theories of Change in Rural Policy Evaluation

AbstractRural policy evaluation helps to understand the extent to which policies have met pre‐defined objectives, achieve value for money and learn from implementation failures. However, there is increasing debate over the quality of policy evaluation and the extent to which its methods can fully contribute to an understanding of rural policy. Responding to these calls, this article employs a theory driven approach to policy evaluation to assess the social impacts of attempts to reduce animal disease on farms in England. Popular in other policy arenas, theory driven evaluation relies on developing a theory of change to examine the interactions between policy contexts and mechanisms and policy outcomes and determine what works for whom. Drawing on longitudinal qualitative and quantitative research, the article identifies two mechanisms of change to evaluate the Badger Vaccine Deployment Project (BVDP) in England to reduce incidence of bovine Tuberculosis (bTB) in cattle. The article shows how these mechanisms – ‘seeing is believing’ and ‘practice similarity’ – are triggered by different contextual factors leading to the failure to deliver expected policy outcomes. We also consider the advantages and limitations to theory based evaluation, and the contribution it can make to the evaluation of other rural development programmes.

The state of higher education leadership development program evaluation: A meta-analysis, critical review, and recommendations

There is a widespread use of leadership development (LD) for students in higher education; however, less is known about the effectiveness of such practices. We provide a summative and meta-analytic review to identify the state of LD programs for students in higher education (i.e., undergraduate and graduate students). The overall objective is to demonstrate whether LD programs are implementing the most effective strategies with any discrepancy revealing a gap between management science and higher education practice. Our results suggest that LD programs within higher education work, but evaluation studies need to more effectively address endogeneity concerns. As a way moving forward, we provide recommendations for conducting a LD program evaluation study and for conducting a meta-analysis on evaluation studies. This meta-analysis can be used as a starting point for the discussion on these issues. We hope that our findings can guide the future development of LD programs.

Community Participation and Government Role in Using Village Funds in Dlingo Village

This study discusses community participation and the role of the government in the utilization of village funds in Dlingo Village. The aim of this study was to find out about the level of community participation in the use of village funds in Dlingo Village and to find out about the role of the government in determining the use of village funds in Dlingo Village. This study uses Arnstein's theory of participation ladder, which is used to determine the level of community participation at each stage of development, namely planning, implementation, and monitoring and evaluation. The research method used is qualitative descriptive research method by collecting qualitative data in the form of interviews, document studies, and observations. The results of this study indicate that: (1) Community Participation in Development Program Planning is at the level of community participation at the partnership level. Community participation in the process is quite high, reaching 90% attendance at the village development planning meeting forum. (2) Community Participation in the Implementation of the Development Program is at the stage of placation, this is evidenced by the implementation of the program involving almost the entire community. The role of the village government as the manager of village funds in terms of implementing the program is as a coordinator so that each program can be implemented properly. (3) Community participation in the Development Program Monitoring and Evaluation is in the consultation stage where the community becomes a consultant to the village government. The role of the government is in the monitoring and evaluation process in terms of being a driver of supervision and evaluation among the community.

KONSEP DASAR EVALUASI DAN IMPLIKASINYA DALAM EVALUASI PEMBELAJARAN PENDIDIKAN AGAMA ISLAM DI SEKOLAH

This paper aims to discover the basic concepts of evaluation and its implications on evaluating Islamic Religious Education learning in Schools. This paper employs a qualitative approach and literature study method. Based on the results of the discussion, the evaluation consisted of measurement and assessment. In the context of learning, evaluation has a very important and strategic position since it is included in the steps of learning. The aim is to determine the effectiveness and efficiency of the learning system. The area of discussion includes evaluation in learning programs, learning processes, and learning outcomes. In general, the principle of evaluation consists of continuity, comprehensiveness, fairness, objectivity, cooperativeness, and practicality. In specific, the principle consists of integration, coherence, pedagogical, and accountability. In terms of type, learning evaluation consists of evaluation of planning, development, monitoring, impact, efficiency, and comprehensive programs. In terms of object, evaluation of learning includes input, transformation, and output. The subject includes the educator, the officer who has been trained, and the students that can evaluate themselves. In terms of the technique, it consists of tests and non-tests. The implication is that the evaluation of the learning of Islamic Religious Education in schools needs to be carried out in a continuous, comprehensive and integrated manner. Thus, Islamic Religious Education educators must be able to evaluate the development of students, including in the aspects of aqliyah, qolbiyah, and amāliyah.

Evaluation of Student Development Programs in the State Universities and Colleges of Samar Island

Evaluation of Student Development Programs in the State Universities and Colleges of Samar Island

Exploring academic development programme evaluation at a university: A systematic scholarly approach

The systematic review, critique and evaluation of Academic Development (henceforth AD) programmes are increasingly becoming important for informing ongoing and future programme design. Ensuring the relevance requires a thorough understanding of the programme to be evaluated (Chalmers and Gardiner, 2015). This paper reports on the findings from a study exploring the nature of AD programmes at the University – with the aim of constructing a conceptual framework for thinking about AD programmes. This is necessary, in order to determine and evaluate the characteristics of nine programmes; and to determine whether these include the promotion of conceptual change – to thinking differently about teaching and learning in Higher Education. The process of re-contextualising teaching, learning and assessment at the University from a common-sense approach – to a more scholarly approach – includes evaluative-research methodologies, as well as critical reflection within a paradigm of system theories. The research outcomes include a conceptual framework and a blueprint for a generic AD programme design.

Faculty development program evaluation: a need to embrace complexity.

Faculty development is essential for renewing and assisting faculty to maintain teaching effectiveness and adapt to innovations in Health Professions educational institutions. The evaluation of faculty development programs appears to be a significant step in maintaining its relevance and efficiency. Yet, little has been published on the specific case of faculty development program evaluation in spite of the availability of general program evaluation models. These models do not measure or capture the information educators want to know about outcomes and impacts of faculty development. We posit that two reasons account for this. The first is the evolving nature of faculty development programs as they adapt to current reforms and innovations. The second involves the limitations imposed by program evaluation models that fail to take into account the multiple and unpredictable outcomes and impacts of faculty development. It is generally accepted that the outcomes and impacts are situated at various levels, ranging from the individual to the institutional and cultural levels. This calls for evaluation models that better capture the complexity of the impacts of faculty development, in particular the reciprocal relationships between program components and outcomes. We suggest conceptual avenues, based on Structuration Theory, that could lead to identifying the multilevel impacts of faculty development.

Developmental health in the context of an early childhood program in Brazil: the "Primeira Infância Melhor" experience.

Design and evaluation of early child development (ECD) programs are poorly documented in low- or middle-income countries. The study aimed to identify family and child characteristics associated with developmental health outcomes among children aged from 4 to 6 years who participated in the "Primeira Infância Melhor" - PIM (Better Early Childhood), a home visiting program in Rio Grande do Sul State, Brazil. We also evaluated the impact of PIM on developmental vulnerability at school entry using a comparison group. Multistage sampling was first used to select cities, then families, in different regions of the state, resulting in a sample of eight cities and 571 children (364 PIM; 207 comparison). We used a sociodemographic questionnaire, completed by parents, and the Early Development Instrument (EDI), completed by teachers. Among PIM children, lower family income, time of exit from the program, city, and younger age were associated with higher risk of developmental vulnerability and/or with lower mean scores in EDI domains. Multivariate analysis controlling for covariates found no differences between the study groups in EDI outcomes even though the gaps in equity of the outcomes were smaller in the PIM group. These results are discussed in the context of challenges faced by home visiting programs in addressing complex social conditions of high-risk families and difficulties in finding an adequate comparison group in communities where an ECD program is universally accessible. We also note the importance of setting structured and longitudinal monitoring systems together with the implementation of ECD policies.

Manajemen Pengembangan Kultur Literasi di MTs Negeri Kota Batu

Literacy is identified with reading and writing activities. With the passage of time literacy activities not only read and write. Literacy is defined as the ability to support, challenge, create, communicate and count using printed material and writing related to various contexts. Literacy that supports learning is literacy that allows individuals to achieve what they want to know, develop their knowledge and talents and support their participation in the community. This study aims to describe the management of the development of literacy culture, the implementation of the development of a culture of literacy and the evaluation of literacy culture development programs in MTs Negeri Kota Batu. This research uses qualitative with descriptive research types. The data sources used are primary and secondary. Data analysis used the Miles and Huberman analysis models. The results show the management carried out by the MTs Negeri Kota Batu under the coordination of the kepala madrasah and supported by the deputy of kepala madrasah in the fields of curriculum, infrastructure, public relations and financing. Not only that, for the success of the school management program, it also involves the madrasah committee. The implementation of the program in the form of habituating the dhuha prayer and reading Al-Qur’an, the habit of silent reading before learning, holding reading corners, programming the language of the month, and scientific writing training for teacher in collaboration with universities in Malang. Program evaluation carried out by the madrasah management is still anticipatory action. Its consideration is that the program is just implemented.

Low carbon supplier development

PurposeThe purpose of this paper is to provide an analytical model for low carbon supplier development. This study is focused on the level of investment and collaboration decisions pertaining to emission reduction.Design/methodology/approachThe authors’ model includes a fuzzy c-means (FCM) clustering algorithm and a fuzzy formal concept analysis. First, a set of suppliers were classified according to their carbon performances through the FCM clustering algorithm. Then, the fuzzy formal concepts were derived from a set of fuzzy formal contexts through an intersection-based method. These fuzzy formal concepts provide the relative level of investments and collaboration decisions for each identified supplier cluster. A case from the Indian renewable energy sector was used for illustration of the proposed analytical model.FindingsThe proposed model and case illustration may help manufacturing firms to collaborate with their suppliers for improving their carbon performances.Research limitations/implicationsThe study contributes to the low carbon supply chain management literature by identifying the decision criteria of investments toward low carbon supplier development. It also provides an analytical model of collaboration for low carbon supplier development. Though the purpose of the study is to illustrate the proposed analytical model, it would have been better if the model was empirically validated.Originality/valueThough the earlier studies on green supplier development program evaluation have considered a set of criteria to decide whether or not to invest on suppliers, these are silent on the relative level of investment required for a given set of suppliers. This study aims to fulfill this gap by providing an analytical model that will help a manufacturing firm to invest and collaborate with its suppliers for improving their carbon performance.

A Practical Solution to Developing County Extension Director’s Leadership Skills: Exploring the Design, Delivery and Evaluation of an Online Leadership Development Program

County Extension Directors (CED) are tasked with a myriad of responsibilities, many of which are directly related to leadership skills. Despite the identification of competencies and skills needed by CEDs in order to maintain successful programs, little research has been conducted on actual Extension leadership programming. Even more uncommon in the literature is the evaluation of CED leadership development programs that utilize online delivery. Thus, the purpose of this study was to explore the outcomes of a primarily online leadership development program with the goal of developing 21st century soft skills in CEDs. The Leadership Short Course was built upon the leadership development foundations of Moore and Rudd (2005), Owen (2004) and Sanders (2014), while the program evaluation framework utilized the Kirkpatrick (1976) model to evaluate program participant reactions, learning, and behavior changes. Key findings indicate that the design and delivery of the program resulted in high participant satisfaction and significant increases in leadership knowledge and skill level. Positive outcomes in this type of online programming has implications for the design and implementation of future CED leadership programs.

Report from the International Workshop on the Role of Public Administration in Public Policies’ Design & 15th Anniversary of the Central European Public Administration Review

In April 2018, the Faculty of Administration of the University of Ljubljana organised a two-day international workshop on the role of public administration in public policies' design. The workshop consisted of four parts: three sessions and one round table. In the first session, discussion was about evaluating public administration and public governance. The second session focused on the identification of the key success factors for effective public policies in Slovenia. These sessions were initiated based on the research project “Development of the model for monitoring and evaluation of development programmes and projects in public sector”, known as the ATENA project. The project is co-funded by the Slovenian Research Agency for the period 2016–2019 (no. J5-7557) and led by prof. dr. Mirko Vintar (cf. Mencinger et al., 2017). The third session was motivated by the European research project EUPACK (European Public Administration Country Knowledge), focused on the analysis of public administration characteristics and performance in EU Member States (see Thijs, Hammerschmid & Palaric 2018). A special part of the workshop was devoted to the 15th anniversary of the Central European Public Administration Review. Here, a round table was conducted with the editors-in-chief of established public administration journals from the region, followed by an editors and reviewers recognition awards ceremony. The discussions were all very fruitful, also thanks to the participation of several internationally recognised scholars from the Netherlands, Croatia, Germany, Slovakia, the Czech Republic, Romania and Slovenia, as well as around twenty representatives of Slovenian ministries, other administrative authorities and non-governmental organisations. In a dynamic debate that comprehensively covered the evaluation in public policy cycle and the role of public administration and university therein, numerous issues were discussed. Below is a report on the main topics discussed in the workshop.

Critical evaluation of professional development and training programs for civil engineers in Baghdad

Construction industry comprises a high degree of competition; the performance of construction projects is highly affected by the performance of civil engineers who are the core of construction productivity. Professional development is one of the essential milestones toward improving the performance of civil engineers. This paper, critically, evaluates the professional development and training of civil engineers in Baghdad. This encompassed the study of the existing programs offered for the professional development, the expected impact, and suggestions. Data were collected from 113 professional civil engineers about the existing professional development programs and training courses and analyzed. Large variations were found in terms of the time spent for professional development, despite the fact that they are comparable to some developed countries. Pitfalls in the field of professional training and development were clear. Remedial suggestions including professional development program and appraisal system were presented. It was emphasized that collaborative efforts by all stakeholders are required to ensure sustainable professional development.

What policymakers need from implementation evaluations of early childhood development programs.

As early childhood interventions move from small- to large-scale programs, the partnerships among researchers, policymakers, communication experts, practitioners, and local communities become increasingly critical for successful implementation. The results from evaluations of programs need to guide policymakers so that they can make informed administrative and fiscal decisions. This commentary makes five recommendations for researchers to consider when designing early childhood programs for scale.

Does Context Matter? Convergent and Divergent Findings in the Cross-Institutional Evaluation of Graduate Teaching Assistant Professional Development Programs.

Graduate teaching assistants (GTAs) play important instructional roles in introductory science courses, yet they often have little training in pedagogy. The most common form of teaching professional development (PD) for GTAs is a presemester workshop held at the course, department, or college level. In this study, we compare the effectiveness of presemester workshops at three northeastern research universities, each of which incorporated scientific teaching as the pedagogical content framework. The comparison of GTA PD program outcomes at three different institutions is intended to test theoretical assertions about the key role of contextual factors in GTA PD efficacy. Pretest and posttest surveys were used to assess changes in GTA teaching self-efficacy and anxiety following the workshops, and an objective test was used to assess pedagogical knowledge. Analysis of pretest/posttest data revealed statistically significant gains in GTA teaching self-efficacy and pedagogical knowledge and reductions in teaching anxiety across sites. Changes in teaching anxiety and self-efficacy, but notpedagogical knowledge, differed by training program. Student ratings of GTAs at two sites showed that students had positive perceptions of GTAs in all teaching dimensions, and relatively small differences in student ratings of GTAs were observed between institutions. Divergent findings for some outcome variables suggest that program efficacy was influenced as hypothesized by contextual factors such as GTA teaching experience.

Ex-post evaluation of regional bureaucracy development programs

We study the influence of the program-targeted planning mechanisms on the civil service efficiency in the subjects of the Russian Federation. Our hypothesis is that the criteria for assessing effectiveness of civil service development programs are to increase the efficiency of its functioning and focus on solving problems of socio-economic development. The research methodology is based on the generalization of statistical data on the regions of the Central Federal District (CFD) of Russia in 2003-2016, and on the construction of some indices for civil service and socio-economic development. We have analyzed 67 programs of civil service development. Due to lack of financing Russian regions are forced to abandon many measures of the civil service reform. The average values of efficiency coefficients do not reach 30%, which indicates the existence of significant reserves for increasing the civil service development programs effectiveness. We have revealed that a crisis condition of socio-economic development resists the effective civil service development in the regions. At the same time, the region’s advanced positions in the economy do not automatically lead to a comparable high level of civil service development. This conclusion is important for extrapolating to regions of other federal districts and to the federal level in Russia.

Case study as a means of evaluating the impact of early years leaders: Steps, paths and routes

The paper argues that case study will need to play an increasingly important role in the evaluation of leadership development programmes as both formal and substantive theories of leadership place greater emphasis upon the role played by organizational context on leaders ability to bring about change.Prolonged engagement within a case study provides researchers with opportunities to capture the dynamics between leaders and their organisational contexts. However, adopting a case study approach is no substitute for inadequate theorization of the link between leadership approaches and leadership effects. The paper argues for the use of inclusive and expansive theoretical notions of leadership and its relationship to organisational context.The evaluation used to illustrate these arguments was based on a longitudinal multi-site case study methodology. The case studies ran over a three-year period and tracked the effect of 42 leaders on the quality of provision in some 30 early years settings. Both individual and collective theoires of leadership were used to trace leaders’ steps, paths and routes to improvement. Three overlapping theoretical lenses were used to study the dynamics of these leaders interactions with a key aspect of their organisational contexts - the existing formal and informal leadership structures – and how these affected their attempts to improve the quality of provision of their settings.The analysis, and related findings, were tiered in order to provide progressively more detailed descriptions of the relationships between leaders’ approaches and changes in their settings’ quality of provision. Each layer of analysis operated with a causal logic that became gradually less general and linear and increasingly more ‘local’ and complex.