Program Evaluation Process Research Articles

Review and Implications of the AutoCarto Six Retrospective Project

A previous IJAGR paper, using the Retrospective Approach to Commemorate AutoCarto Six (Wellar, 2014), presented the reasons for using a retrospective approach to re-visit papers that were published 30 years ago (1983) in the proceedings of the Sixth International Symposium on Automated Cartography. This paper addresses four important topics that arise from producing AutoCarto Six Retrospective. First, in response to requests for more information about the “retro experience”, the research design of the retrospective project is reviewed in terms of lessons learned. Second, the contribution that the retrospective approach makes to “the literature” on the evolution of automated cartography, geographic information systems, computational geography, and related fields is explored. Third, several implications of the retrospective approach for the literature search and review component of theses, dissertations, academic productions, and research proposals, as well as plan, program, and policy evaluation processes in both the private and public sectors are outlined. And fourth, comments are made about applying the AutoCarto Six Retrospective experience to other commemorative events.

Meta-Evaluation: Experiences in an Accelerated Graduate Nurse Education Program.

Most schools of nursing are engaged in some form of program evaluation and recognize the potential benefits in using program evaluation outcomes to influence continuous improvement in program quality. A number of factors exist that may negatively influence program evaluation quality and adversely affect the ability to make sound decisions based on program evaluation outcomes. The potential limitations that threaten program evaluation quality underscore the importance of evaluating the evaluation process itself, also known as meta-evaluation. However, there is an absence of discussion in the nursing literature of the importance of program meta-evaluation. This article seeks to address this gap in the nursing literature and illuminate the need for more schools of nursing to engage in the meta-evaluation process. By introducing 1 model of program meta-evaluation and describing our own endeavors in the program meta-evaluation process, we hope to inspire other schools of nursing to consider using a systematic and formalized process to evaluate their own program evaluation processes to ensure that data obtained from program evaluation are of optimal quality to influence sound, data-driven decisions to promote continued quality and excellence in nursing education programs.

Medical education in Singapore

Allopathic medical education in Singapore extends for more than a century from its simple beginnings. In recent times, changes have been rapid, both in undergraduate and postgraduate specialty medical training. Over the last decade, undergraduate medical education has increased from a single to three medical schools and the postgraduate training has expanded further by incorporating the Accreditation Council for Graduate Medical Education International framework. With these changes, the curricula, assessment systems, as well as teaching and learning approaches, with the use of technology-enhanced learning and program evaluation processes have expanded, largely based on best evidence medical education. To support these initiatives and the recent rapid expansion, most training institutions have incorporated faculty development programs, such as the Centre for Medical Education at the National University of Singapore.

Development of a Logic Model to Guide Evaluations of the ASCA National Model for School Counseling Programs

A logic model was developed based on an analysis of the 2012 American School Counselor Association (ASCA) National Model in order to provide direction for program evaluation initiatives. The logic model identified three outcomes (increased student achievement/gap reduction, increased school counseling program resources, and systemic change and school improvement), seven outputs (student change, parent involvement, teacher competence, school policies and processes, competence of the school counselors, improvements in the school counseling program, and administrator support), six major clusters of activities (direct services, indirect services, school counselor personnel evaluation, program management processes, program evaluation processes and program advocacy) and two inputs (foundational elements and program resources). The identification of these logic model components and linkages among these components was used to identify a number of necessary and important evaluation studies of the ASCA National Model.Keywords: ASCA National Model, school counseling, logic model, program evaluation, evaluation studiesSince its initial publication in 2003, The ASCA National Model: A Framework for School Counseling Programs has had a dramatic impact on the practice of school counseling (American School Counselor Association [ASCA], 2003). Many states have revised their model of school counseling to make it consistent with this model (Martin, Carey, & DeCoster, 2009), and many schools across the country have implemented the ASCA National Model. The ASCA Web site, for example, currently lists over 400 schools from 33 states that have won a Recognized ASCA Model Program (RAMP) award since 2003 as recognition for exemplary implementation of the model (ASCA, 2013).While the ASCA National Model has had a profound impact on the practice of school counseling, very few studies have been published that evaluate the model itself. Evaluation is necessary to determine if the implementation of the model results in the model's anticipated benefits and to determine how the model can be improved. The key studies typically cited (see ASCA, 2005) as supporting the effectiveness of the ASCA National Model (e.g., Lapan, Gysbers, & Petroski, 2001; Lapan, Gysbers, & Sun, 1997) were actually conducted before the model was developed and were designed as evaluations of Comprehensive Developmental Guidance, which is an important precursor and component of the ASCA National Model, but not the model itself.Two recent statewide evaluations of school counseling programs focused on the relationships between the level of implementation of the ASCA National Model and student outcomes. In a statewide evaluation of school counseling programs in Nebraska, Carey, Harrington, Martin, and Hoffman (2012) found that the extent to which a school counseling program had a well-implemented, differentiated delivery system consistent with practices advocated by the ASCA National Model was associated with lower suspension rates, lower discipline incident rates, higher attendance rates, higher math proficiency and higher reading proficiency. These results suggest that model implementation is associated with increased student engagement, fewer disciplinary problems and higher student achievement. In a similar statewide evaluation study in Utah, Carey, Harrington, Martin, and Stevens (2012) found that the extent to which the school counseling program had a programmatic orientation, similar to that advocated in the ASCA National Model, was associated with both higher average ACT scores and a higher number of students taking the ACT. This suggests that model implementation is associated with both increased achievement and a broadening of student interest in college. While these studies suggest that benefits to students are associated with the implementation of the ASCA National Model, additional evaluations are necessary that use stronger (e.g., quasi-experimental and longitudinal) designs and investigate specific components of the model in order to determine their effectiveness or how they can be improved. …

Benefits of curriculum renewal: The Stellenbosch University physiotherapy experience

Background. Driven by a changing healthcare environment, the Division of Physiotherapy (Stellenbosch University) reduced core content and adopted a multimodal approach to teaching and learning. The benefits of curriculum renewal, however, are seldom investigated despite ongoing internal appraisal. Evaluation of the BSc Physiotherapy programme was considered incomplete without determining the worth of the programme. Objectives. To determine whether there was a change in students’ perception of the impact of the programme on personal development; and whether the programme prepared them for community service. Methods. A descriptive comparative desktop analysis was conducted in which the data from the Faculty’s Programme Evaluation process were compared between students enrolled in the old curriculum (2006) and students enrolled in the new curriculum (2011) using pooled data and t -tests to compare responses between the two groups. A level of significance was set at p <0.05. Results. A significant increase in scores was noted for various graduate attributes developed as a result of the programme, such as critical thinking, clinical reasoning, communication and sourcing information ( p <0.01). Similarly, students scored their perceptions related to programme structure significantly higher ( p <0.01). No change was reported regarding students’ ability to maintain a balance between studies and other activities. Scores pertaining to their perception of readiness for community service or professional practice remained the same with both cohorts believing they were well prepared. Conclusion. The renewed format seemed to benefit students greatly in assisting the development of graduate attributes. Students were significantly more satisfied with the structure of the renewed curriculum and – despite extensive changes – the principles-based multimodal approach to teaching and learning was perceived as effective for preparing students for community service. Programmes undertaking curriculum renewal should not only focus on the curriculum content but also develop a variety of learning opportunities to facilitate the development of graduate attributes. The next cycle of evaluation should however reflect on clinical practice.

Developing Leadership for Life

This application brief describes the exploratory assessment of a mentoring program between current students and alumni of a leadership studies minor program. We connect leadership education research and practice in two ways: first, we describe a process of qualitative program evaluation to inform program best practices and improvement. In doing so, we also highlight the value of an alumni mentoring program as a strategic component of leadership education. Our findings demonstrate the mentoring program supported students’ leadership development in the areas of career transition, personal growth, and application to “real-life.” Recommendations are offered for creating formal and informal mentoring opportunities.

Developing Leadership for Life: Outcomes from a Collegiate Student-Alumni Mentoring Program

This application brief describes the exploratory assessment of a mentoring program between current students and alumni of a leadership studies minor program. We connect leadership education research and practice in two ways: first, we describe a process of qualitative program evaluation to inform program best practices and improvement. In doing so, we also highlight the value of an alumni mentoring program as a strategic component of leadership education. Our findings demonstrate the mentoring program supported students’ leadership development in the areas of career transition, personal growth, and application to “real-life.” Recommendations are offered for creating formal and informal mentoring opportunities.

Ecosystemic Needs Assessment for Children with Developmental Coordination Disorder in Elementary School: Multiple Case Studies

ABSTRACTThis study explored the needs of children with developmental coordination disorder (DCD) from an ecosystemic viewpoint as part of a theory-driven program evaluation process. A multiple case study needs assessment was conducted. Participants included ten children with DCD, their parents (n = 12), teachers (n = 9), and service providers (n = 6). Data collection involved semi-structured interviews, validated questionnaires, and a review of the children's records. The results support the relevance of using an ecosystemic model to assess the needs of children with DCD in their life and social contexts. More specifically, the results highlight the need to provide additional services at school, such as occupational therapy and special education, as well as information and training regarding DCD for parents and teachers. The results also point to the relevant variables to consider in an intervention program based on theory-driven evaluations. This study shows how employing an ecosystemic frame of reference provides a better understanding of the needs of children with DCD. Future research should document the ecosystemic profiles and evolution of the needs of children with DCD with a larger sample from diverse socioeconomic backgrounds using a longitudinal study design.

Financial Education Program Evaluation

This study illustrates the process of program evaluation using a logic model. Guided by the Transtheoretical Model of Change and a logic model, this study evaluated the effectiveness of a Retirement and Savings Seminar by measuring participant (n = 54) satisfaction, financial knowledge, financial confidence, and financial behavior change compared to a similar group of 134 non-participants. Participants were very satisfied with the seminar. Their financial knowledge and confidence scores significantly increased from pretest to posttest. Financial knowledge and confidence scores improved more than the comparison group while controlling for group differences in age, income, and pretest scores. Two months later, participants were more likely than the comparison group to have adopted positive financial behaviors as measured by the Financial Preparedness for Retirement Scale. Financial educators can use this study as a model for planning, conducting, and evaluating their programs

Navigating the Seasons of Survivorship in Community Oncology

Nurses have an important role in the development, implementation, and evaluation of cancer survivorship programs. Growing numbers of cancer survivors challenge community oncology practices to incorporate survivorship care according to new standards and guidelines. In response, one community-based oncology clinic created an advanced practice nurse (APN)-led survivorship program using the concept of Seasons of Survival as a guide. Survivorship care, when based on a more expansive definition of survivorship as beginning at the time of diagnosis, encompasses holistic nursing and multidisciplinary care. The APN assesses each patient's concerns and quality of life using a validated measure to tailor survivorship and supportive care. This article reviews the foundation and structure of the program in detail, describes program implementation using case studies, and outlines the program evaluation process and results.

Improved Outcomes Found after Implementing a Systematic Evaluation and Program Improvement Process for Tuberculosis

California's state and local tuberculosis (TB) programs collaborated to develop the Tuberculosis Indicators Project (TIP), a program evaluation and improvement process. In TIP, local and state staff review data, identify program gaps, implement plans to improve local TB program performance, and evaluate outcomes. After 10 years of project implementation, indicator performance changes and patient outcomes were measured. Eighty-seven percent of participating programs showed a performance increase in targeted indicators after three years compared with 57% of comparison groups. Statistically significant performance change was more common in the intervention local health departments (LHDs) than in comparison groups. The most notable performance changes were in the contact investigation and case management indicators. These results indicate that this systematic evaluation and program improvement project was associated with improved LHD TB control performance and may be useful to inform improvement projects in other public health programs.

Making program evaluation activities family-centered: a qualitative study

Purpose: This study explored ways in which program evaluation activities in pediatric rehabilitation settings can become congruent with family-centered service (FCS) philosophy. Methods: Two Canadian pediatric rehabilitation centers participated in this study, which included focus groups with staff members and interviews with parents. Results: Participants identified seven ways in which program evaluation practices could be made congruent with FCS. Suggestions included: (a) the inclusion of a diverse group of program recipients, (b) the use of processes that facilitate family involvement, (c) the recruitment of family champions, (d) the involvement of families in program development, (e) the establishment of evaluations that are relevant to families (f) the development of center-wide statements about family involvement in evaluation and (g) the compensation of families for their active participation in evaluation. Conclusion: For program evaluation practices to be useful and relevant, they should be improved and made consistent with FCS philosophy. Those evaluating pediatric rehabilitation programs need to use approaches and activities that respect the needs, characteristics, cultures and diversity of the program recipients. Such actions will help to improve the quality of care provided, the nature of program evaluation activities, as well as the overall level of FCS in pediatric rehabilitation settings.Implications for RehabilitationBy incorporating family-centered activities into program evaluation processes, clinicians/evaluators can improve the overall level of and adherence to FCS philosophy in their centers, enhance the quality of care provided and, thus, contribute positively to the outcomes of program recipients and their families.Those evaluating pediatric rehabilitation programs need to use evaluation approaches and activities that respect the needs, characteristics, cultures and diversity of the program recipients.By engaging program recipients and their families in program development, they may become more engaged in program evaluation.

Assessing the Impact of Entrepreneurship Education Programs: Challenges and Approaches

Entrepreneurship education programs are increasingly being established and expanded in an effort to equip students with the knowledge and competency necessary to create economic value and jobs. An underlying assumption of these programs is that they create positive outcomes for students; however, the extent and nature of these outcomes have not been well explored in the literature. The purpose of this paper is to provide an overview of contemporary trends in educational evaluation and the challenges specifically associated with the assessment of entrepreneurship education programs. It proposes practical considerations for faculty and administrators involved in developing assessment initiatives for entrepreneurship education programs, including, the importance of reaching consensus on learning outcomes, the use of a stakeholder‐driven approach for setting assessment priorities, and the need to allocate resources to assessment initiatives so they can be sustained long term. It also highlights the value of involving faculty in the program evaluation process and the need to create incentives and opportunities for more assessment‐related research and scholarship within the field.

“Keeping It Real”: An Evaluation Audit of Five Years of Youth-Led Program Evaluation

Youth are increasingly seen as competent in participating in research and program evaluation, two activities previously reserved for adults. This article is a report of the findings from an evaluation audit of Stand Up! Help Out!, a participatory action after-school youth leadership development program for disadvantaged urban youth that utilized youth evaluations to develop a best practices service model. The youths' feedback assisted providers in improving services so that youth engagement in the program was 99% (by comparison with national highs of 79%). Here, the authors describe an important aspect of the process of youth-led program evaluation leading to such high youth engagement: how youth interviewed each other so as to optimize the authenticity of their program evaluations and contributions to program design. Drawing from over 5 years of program evaluation data collected by youth, the authors report on the youths' experiences as informants and coresearchers, consider strategies used to help youth best describe their experiences in the program, and describe implications for other settings looking to incorporate youth-led program evaluation. Youth-led program evaluation has considerable promise for helping service providers make programs more meaningful for disadvantaged youth.

Rethinking programme evaluation in health professions education: beyond ‘did it work?’

For nearly 40 years, outcome-based models have dominated programme evaluation in health professions education. However, there is increasing recognition that these models cannot address the complexities of the health professions context and studies employing alternative evaluation approaches that are appearing in the literature. A similar paradigm shift occurred over 50 years ago in the broader discipline of programme evaluation. Understanding the development of contemporary paradigms within this field provides important insights to support the evolution of programme evaluation in the health professions. In this discussion paper, we review the historical roots of programme evaluation as a discipline, demonstrating parallels with the dominant approach to evaluation in the health professions. In tracing the evolution of contemporary paradigms within this field, we demonstrate how their aim is not only to judge a programme's merit or worth, but also to generate information for curriculum designers seeking to adapt programmes to evolving contexts, and researchers seeking to generate knowledge to inform the work of others. From this evolution, we distil seven essential elements of educational programmes that should be evaluated to achieve the stated goals. Our formulation is not a prescriptive method for conducting programme evaluation; rather, we use these elements as a guide for the development of a holistic 'programme of evaluation' that involves multiple stakeholders, uses a combination of available models and methods, and occurs throughout the life of a programme. Thus, these elements provide a roadmap for the programme evaluation process, which allows evaluators to move beyond asking whether a programme worked, to establishing how it worked, why it worked and what else happened. By engaging in this process, evaluators will generate a sound understanding of the relationships among programmes, the contexts in which they operate, and the outcomes that result from them.

A health equity critique of social marketing: Where interventions have impact but insufficient reach

Health interventions increasingly rely on formative qualitative research and social marketing techniques to effect behavioural change. Few studies, however, incorporate qualitative research into the process of program evaluation to understand both impact and reach: namely, to what extent behaviour change interventions work, for whom, in what contexts, and why. We reflect on the success of a community-based hygiene intervention conducted in the slums of Kathmandu, Nepal, evaluating both maternal behaviour and infant health. We recruited all available mother–infant pairs (n = 88), and allocated them to control and intervention groups. Formative qualitative research on hand-washing practices included structured observations of 75 mothers, 3 focus groups, and 26 in-depth interviews. Our intervention was led by Community Motivators, intensively promoting hand-washing-with-soap at key junctures of food and faeces contamination. The 6-month evaluation period included hand-washing and morbidity rates, participant observation, systematic records of fortnightly community meetings, and follow-up interviews with 12 mothers. While quantitative measures demonstrated improvement in hand-washing rates and a 40% reduction in child diarrhoea, the qualitative data highlighted important equity issues in reaching the ultra-poor. We argue that a social marketing approach is inherently limited: focussing on individual agency, rather than structural conditions constraining behaviour, can unwittingly exacerbate health inequity. This contributes to a prevention paradox whereby those with the greatest need of a health intervention are least likely to benefit, finding hand-washing in the slums to be irrelevant or futile. Thus social marketing is best deployed within a range of interventions that address the structural as well as the behavioural and cognitive drivers of behaviour change. We conclude that critiques of social marketing have not paid sufficient attention to issues of health equity, and demonstrate how this can be addressed with qualitative data, embedded in both the formative and evaluative phases of a health intervention.

THE GRANDE PRAIRIE PACT PROGRAM EVALUATION: DISCREPANCY BETWEEN MODEL EVALUATION PRACTICE AND CONSTRAINED REAL WORLD EVALUATION OF CRIME PREVENTION IN SMALL COMMUNITIES

This article discusses and demonstrates the discrepancies between ideal, theoretical program evaluation processes and real world evaluation practice, which is constrained by numerous and varying factors. The article describes the real world experience of Mount Royal University’s Centre for Criminology and Justice Research researchers in conducting an evaluation of the Police and Crisis Team (PACT) in Grande Prairie, Alberta, including a Social Return on Investment (SROI) analysis. PACT, which partners an Royal Canadian Mounted Police (RCMP) officer with a mental health professional, represents a blend of secondary and tertiary crime prevention and attempts to diminish crime in the community by addressing the risk factors of individuals with mental health concerns (creating trust with individuals, increasing awareness of resources, and decreasing stigmatization in the community). PACT also specifically targets those individuals with mental health issues who are in contact with the law to try to decrease recidivism and increase community safety. Challenges were present in the evaluation due to the time frame, staff turnover, program start-up issues, and confidentiality and sensitivity of the program focus. Despite the challenges, the CCJR team completed an evaluation including a forecast SROI, identifying several successes, challenges, and recommendations for change.

Assessment and evaluation of primary prevention in spinal cord injury.

Although the incidence of spinal cord injury (SCI) is low, the consequences of this disabling condition are extremely significant for the individual, family, and the community. Sequelae occur in the physical, psychosocial, sexual, and financial arenas, making global prevention of SCI crucial. Understanding how to assess and evaluate primary prevention programs is an important competency for SCI professionals. Assessing a program's success requires measuring processes, outcomes, and impact. Effective evaluation can lead future efforts for program design while ensuring accountability for the program itself. The intended impact of primary prevention programs for SCI is to decrease the number of individuals who sustain traumatic injury; many programs have process and outcome goals as well. An understanding of the basic types of evaluation, evaluation design, and the overall process of program evaluation is essential for ensuring that these programs are efficacious. All health care professionals have the opportunity to put prevention at the forefront of their practice. With the current paucity of available data, it is important that clinicians share their program design, their successes, and their failures so that all can benefit and future injury can be prevented.

Evaluation: Theoretical Underpinnings and Web Resources

Librarians may benefit from understanding research theory more fully, looking at the services they offer as a program, and using program evaluation to understand the program value more completely. The process of program evaluation is research, and, as such, it should be situated within theory. This article reviews two of the basic theoretical concepts of research: ontology and epistemology. These concepts work together to help craft the research questions, and the questions drive the selection of methodology. The article also provides a list of web resources useful for understanding research theory in evaluation.

Partners in Change: Bringing People in Recovery into the Process of Evaluating Recovery Oriented Services

Involving people in all aspects of recovery, including the program evaluation process, can not only produce quality outcomes, but can also successfully promote recovery and recovery oriented services (Garverich, Simon, Grudinskas, McDonald, & Grillo, 2008). This article focuses on how involvement in program evaluation promotes recovery by utilizing SAMHSA's Fundamental Components of Recovery. Based on a review of the literature and the experiences at a psychosocial resource center for veterans, the process of involving people in recovery in program evaluation is illustrated with respect to several key recovery components including empowerment, responsibility, hope, peer support, holistic approach, individualized and person-centered care, respect, nonlinear course, strengths-based orientation, and self-direction. Challenges involving people in recovery in the program evaluation process and promoting this process to other staff members are described. Last, lessons learned from the authors' own experiences i...