Evaluation Of Community-based Program Research Articles

No one runs alone: Combining community-based program evaluation with photo elicitation interviewing among people experiencing homelessness

BackgroundHomelessness is associated with poor physical and mental health, but social support may reduce negative health outcomes. A community-academic partnership with Back on My Feet (BoMF), an organization in Baltimore, MD serving people experiencing homelessness, used photo elicitation interviews (PEI) to explore how social interactions within program activities improved mental health among participants. MethodsBetween October 2018 and June 2019, 29 BoMF participants were recruited. Participants were provided digital cameras and photographed what BoMF meant to them. Participants presented photographs in meetings that were audio recorded and transcribed. Thirty-three themes and 44 photographs were generated. Findings were presented at a public photo exhibit. The partnership used content analysis of transcripts to build upon participant-generated themes for peer-review publication. ResultsSocial interaction among BoMF participants and volunteers helped participants form new positive social networks, often for the first time, which made participants feel valued by their peers. This led to increased self-worth, a greater sense of social support and social connectedness, and confidence that participants could overcome future challenges. ConclusionsCombining community-based research with PEI provided a nuanced understanding of the mechanisms through with BoMF’s activities facilitated positive mental health among participants and was a feasible approach to program evaluation.

“Localization” in Fragile Spaces: A Comparative Networks Evaluation of Community-Based Programmes in Pakistan and Afghanistan

ABSTRACT Conventional externally-driven development interventions have shifted towards ‘localized,’ ‘bottom-up,’ participatory approaches, such as Community-Driven Development (CDD). However, structural dynamics of conflict-prone spaces continue to challenge their sustainability. Integrating content analysis, social network analysis (SNA), and qualitative comparisons (archival research and over 100 field interviews), this article examines the efficacy of CDD in Pakistan and Afghanistan. SNA of two World Bank programmes illuminates network properties of power and centralization, corresponding with key insights from fieldwork, including comparisons with Pakistan’s Rural Support Programmes. This mixed-method study reveals critical structural conditions that permit and/or constrain sustainable, locally-owned processes in fragile spaces.

A RE-AIM evaluation of Healthy Together: a family-centred program to support children\u2019s healthy weights

BackgroundHealthy Together (HT) is family-centered program to support healthy eating and physical activity designed for implementation in community organizations serving families who may be experiencing vulnerabilities (e.g., related to low income, isolation, ethnicity, immigrant/refugee status, and/or Indigenous background). The purpose of this study was to conduct an evaluation of HT in a real-world, scale-up phase using the RE-AIM framework.MethodsUsing a cross-sectional, non-comparative design, a community-based program evaluation was conducted in 29 organizations implementing HT as part of their core service programs. Data were collected using questionnaires with program participants and facilitators, and interviews with directors of participating organizations. Quantitative data were analyzed using descriptive statistics and qualitative data were content analyzed.ResultsWith regards to Reach, over 3400 caregivers, children and youth attended community programming that offered HT. Among those attending on the scheduled day for the evaluation, 663 completed the questionnaires. The majority of caregiver respondents (n = 431) were female (92%) and attended with children 0–6 years. Respondents also included children 4–6 years (n = 142) and 7–12 years (n = 65), and youth 13–18 years (n = 25). Effectiveness was demonstrated in reported improvements in physical activity, healthy eating, and strengthened social connections. HT was also widely supported by participants and facilitators. Adoption was influenced by the desire to enrich core service programs for families, HT’s fit within existing programs, organizational commitment, and funding support. Implementation experiences indicated that fidelity to the HT program was generally maintained, with some setting specific adaptations. Maintenance of HT was influenced by financial and non-financial resources within community organizations. Most organizations also introduced new initiatives to extend support for healthy eating and physical activity.ConclusionOur findings indicate improvements in healthy eating and physical activity, and social connections among program participants, as well as efforts by community organizations to create environments to support healthy weights. HT was successfully delivered in “real-world” community settings across multiple contexts and with families with diverse backgrounds. This along with strategies to support program implementation and sustainability indicate that HT provides a model for other public health interventions to promote family health and wellbeing.Trial registrationClincialTrials.gov NCT03550248. Registered May 25, 2018

Changes in weight status, quality of life and behaviours of South Australian primary school children: results from the Obesity Prevention and Lifestyle (OPAL) community intervention program

BackgroundChildhood obesity is a serious public health concern worldwide. Community-based obesity prevention interventions offer promise due to their focus on the broader social, cultural and environmental contexts rather than individual behaviour change and their potential for sustainability and scalability. This paper aims to determine the effectiveness of a South Australian community-based, multi-setting, multi-strategy intervention, OPAL (Obesity Prevention and Lifestyle), in increasing healthy weight prevalence in 9 to 11-year-olds.MethodsA quasi-experimental repeated cross-sectional design was employed. This paper reports on the anthropometric, health-related quality of life (HRQoL) and behaviour outcomes of primary school children (9–11 years) after 2–3 years of intervention delivery. Consenting children from primary schools (20 intervention communities, INT; 20 matched comparison communities, COMP) completed self-report questionnaires on diet, activity and screen time behaviours. HRQoL was measured using the Child Health Utility 9D. Body Mass Index (BMI) z-score and weight status were determined from children’s measured height and weight. A multilevel mixed-effects model, accounting for clustering in schools, was implemented to determine intervention effect. Sequential Bonferroni adjustment was used to allow for multiple comparisons of the secondary outcomes.ResultsAt baseline and final, respectively, 2611 and 1873 children completed questionnaires and 2353 and 1760 had anthropometric measures taken. The prevalence of children with healthy weight did not significantly change over time in INT (OR 1.11, 95%CI 0.92–1.35, p = 0.27) or COMP (OR 0.85, 95%CI 0.68–1.06, p = 0.14). Although changes in the likelihood of obesity, BMI z-score and HRQoL favoured the INT group, the differences were not significant after Bonferroni adjustment. There were also no significant differences between groups at final for behavioural outcomes.ConclusionsOPAL did not have a significant impact on the proportion of 9 to 11-year-olds in the healthy weight range, nor children’s BMI z-score, HRQoL and behaviours. Long-term, flexible community-based program evaluation approaches are required .Trial registrationACTRN12616000477426 (12th April 2016, retrospectively registered).

Outcomes of a Comparison Study into a Group-Based Infant Parenting Programme.

This paper reports on a quantitative evaluation of a group-based programme designed to promote parent-infant attachment and child development. Whilst group-based parenting programmes are recommended for treating and preventing conduct disorder in older children, there is, as yet, little evidence as to whether they have a positive effect on very young children and their carers’. Recent UK Government initiatives to support families and improve parenting skills in the first 2 years of children’s lives have increased the demand for the delivery and evaluation of community-based programmes. Eighty mother–child dyads were recruited from nine areas to intervention (n = 54) and control condition (n = 26). Baseline measures were collected in the children’s home when the infants were on average 3-months-old, and follow-up measures were collected 6 months post-baseline (N = 63). Mothers’ positive play behaviours were independently coded from video recordings taken in the home. Other measures included self-reported maternal confidence and mental well-being, assessed infant development and home environment. Socio-demographic data was collected once at baseline. After controlling for baseline scores, control mothers were observed to be significantly less sensitive during play with their baby at the 6 months follow-up with a significant increase in confidence. No differences were found between the groups on the other measures. This paper provides limited evidence for the effectiveness of the Incredible Years Parents and Babies group-based programme delivered in the first year of life. Further evaluation, particularly with parents at increased risk of poorer outcomes is needed to confirm and extend these results.

Strengths and limitations of a tool for monitoring and evaluating First Peoples' health promotion from an ecological perspective.

BackgroundAn ecological approach to health and health promotion targets individuals and the environmental determinants of their health as a means of more effectively influencing health outcomes. The approach has potential value as a means to more accurately capture the holistic nature of Australian First Peoples’ health programs and the way in which they seek to influence environmental, including social, determinants of health.MethodsWe report several case studies of applying an ecological approach to health program evaluation using a tool developed for application to mainstream public health programs in North America – Richard’s ecological coding procedure.ResultsWe find the ecological approach in general, and the Richard procedure specifically, to have potential for broader use as an approach to reporting and evaluation of health promotion programs. However, our experience applying this tool in academic and community-based program evaluation contexts, conducted in collaboration with First Peoples of Australia, suggests that it would benefit from cultural adaptations that would bring the ecological coding procedure in greater alignment with the worldviews of First Peoples and better identify the aims and strategies of local health promotion programs.ConclusionsEstablishing the cultural validity of the ecological coding procedure is necessary to adequately capture the underlying program activities of community-based health promotion programs designed to benefit First Peoples, and its collaborative implementation with First Peoples supports a human rights approach to health program evaluation.

Shifting the evaluative gaze: Community-based program evaluation in the homeless sector

Homelessness is a growing social issue that is a consequence of structural inequities and contributor to the development of health inequities. Community-based research (CBR) has been proposed as an effective research strategy for addressing health equities and promoting social justice through participatory processes. The purpose of this article is to examine the application of CBR principles and practices in the homeless sector and the implications for the production of knowledge and social change to address homelessness. Drawing on our experiences as researchers and service providers, we reflect on the significant successes and challenges associated with using CBR in the homelessness sector. In our discussion we emphasise insights, challenges and lessons learned from a community-university partnership that focused on an evaluation of a transitional shelter program in a large urban centre where housing is expensive and often unavailable.Keywords: Homelessness, housing, transitional housing, transitional shelter, program evaluation, community-based research

Using an Empowerment Evaluation Approach with Community-based Programs: Reflections from the Front Line

Identifying the most appropriate approach for small-scale program evaluation remains a vexed issue for both the researchers who design and implement evaluations, and for the service provider organisations that seek to ensure that they can use the findings as evidence for further program funding. This article reflects on our experiences and the issues encountered in attempting to undertake useful small-scale, community-based program evaluation, particularly in regional settings, where the evaluation capacity, practices and cultures of organisations are still evolving. Cognisant of increasing funder-fundee tensions arising from more externally controlled social program evaluations and considering varied organisational, program, personnel, funding and broader political needs and influences, we have focused on tailoring evaluation approaches to ensure they are robust, relevant and responsive to the varied organisational contexts in which we are endeavouring to strengthen evaluation capacity. The approach that has emerged over the past eight years fits most closely with an empowerment evaluation framework that, although theoretically well suited, has had to be adapted in order to respect, and work productively with the frequently competing imperatives of the different stakeholders involved.

Monitoring and evaluation of a large-scale community-based program: Recommendations for overcoming barriers to structured implementation

Accountability for the use of limited health resources requires quality information for ongoing monitoring of program effectiveness. We outline the challenges involved in the monitoring and evaluation of a large-scale community nurse home visiting program. Efforts to establish and maintain rigorous assessment were impaired by interrelated difficulties, impacting on the quality of information collected. The consequences and outcomes of these challenges are discussed. Recommendations to facilitate the monitoring and evaluation of community-based programs are made. Successful program monitoring and evaluation, guided by these recommendations, will support the ongoing review of program activities and evidence-based decision making to inform current operation and future planning.

A Community-Based Program Evaluation of Community Competency Trainings

Transgender and gender-nonconforming individuals encounter a multitude of barriers to accessing clinically and culturally competent health care. One strategy to increase the quality and competence of care delivery is workplace trainings. This study describes a community-based program for the evaluation of this type of training. Using a mixed-methods approach, the research team assessed the effectiveness of three competency trainings administered by a local nonprofit organization in the Northwest United States. Quantitative data indicated a significant shift in self-assessed knowledge associated with completion of the training. Qualitative data confirmed this result and revealed a number of important themes about the effect of the trainings on providers and their ability to implement knowledge and skills in practice. Clinical considerations are proposed for providers who seek similar trainings and who aim to increase clinical and cultural competency in delivering care to transgender and gender-nonconforming patients and clients.

Evaluation for community-based programs: The integration of logic models and factor analysis

Purpose To discuss the utility of and value of the use of logic models for program evaluation of community-based programs and more specifically, the integration of logic models and factor analysis to develop and revise a survey as part of an effective evaluation plan. Principal results Diverse stakeholders with varying outlooks used a logic model as a framework to reach agreement on a plan for a state-wide evaluation. This evaluation plan utilized a survey of sixth grade students, administered before and after exposure to a year-long abstinence education program. Components of the logic model were linked to specific survey questions. Exploratory factor analysis was then used to assess whether and how the questions in the survey fit with the constructs of the model; confirmatory factor analysis was used to test the hypothesis that the factors identified in the exploratory analysis were consistently represented in the survey. Major conclusions A logic model is a tool that engages stakeholders to link evaluation instruments more closely to specific program objectives. Thus, stakeholders can more closely assess the extent to which project outcomes have been achieved. In addition, use of factor analysis in the evaluation process can help the stakeholders better understand whether evaluation instruments such as a survey adequately assess program effectiveness. Lastly, a logic model process can help to achieve consensus among diverse stakeholders, by allowing them to focus on objectives that are concrete, measurable, and mutually acceptable.

Evaluation of Community-Based Health Projects: The Healthy Tomorrows Experience

To address the "millennial morbidities," pediatricians must partner with community-based organizations to develop interventions. Little is known about the capacity of the resulting programs for program evaluation or the importance of evaluation in project success and sustainability. The objective of this study was to examine the capacity of community-based health programs to conduct project evaluations and determine the impact of project evaluation on project outcome. Project directors from 149 community-based programs funded from 1989 to 2003 through the Healthy Tomorrows Partnership for Children Program were surveyed regarding their project experience with evaluation and documentation of project outcomes and the current status of their project. Program directors from 123 (83%) programs completed the survey. Despite barriers to the evaluation process, 83% of the respondents indicated that their evaluations produced useful information. Programs that were described by respondents as "well evaluated" were more likely to report that the evaluation was implemented as planned and that the evaluation included outcome measures. Projects were more likely to be sustained in their original form when at least 1 outcome was reported on the survey. Evaluation of community-based programs, although challenging, is beneficial to project success and sustainability. Policy makers and funding agencies should consider ways to encourage community partnerships to incorporate evaluation into their planning process.

Evaluation of community-based programs to increase booster seat use

This manuscript reports the results of an evaluation of two community-based booster seat promotion programs in Michigan; one program focused on a low-income community, while the other focused on a Hispanic community. Each community received funding to develop and implement a booster seat intervention program specific to their community. To determine the effectiveness of each program, direct observation surveys of booster seat use were conducted in each community, as well as in similarly composed comparison communities, before and after program implementation. A process evaluation documented activities and provided additional information for interpreting the results of the direct observation survey. Target age children (4–8 years) were observed traveling in cars, vans/minivans, sport-utility vehicles, and pickup trucks in each community. Baseline booster seat use was 19.0 ± 5.3% and 9.7 ± 2.5% for the low-income and Hispanic program communities, respectively. Post program results showed no significant change for the low-income program community, and a significant increase within the Hispanic program community. The process evaluation revealed challenges for each program and suggestions to overcome those challenges. Findings from the study can be useful to other communities interested in implementing programs to increase the use of booster seats.

Delivering the goods: The importance of screening accuracy for effective community intervention and prevention

AbstractThe utility and feasibility of community intervention and prevention programs depend greatly on the accuracy by which programs' participants are screened from among community dwellers. This is because indiscriminate application of the program to the entire community benefits only a few, while it can even harm the many others. It is therefore necessary to define and screen a target population so the program can be delivered exclusively to them. Consequently, screening accuracy has implications for the plan, design, and evaluation of community‐based programs. In this manuscript indices of screening accuracy are discussed in the context of screening for substance use, and depression. Receiver Operating Characteristic curves, measurement error, as well as design and evaluation of community programs are also discussed in the context of screening accuracy. © 2002 John Wiley & Sons, Inc.

Log-Linear Models for Change in Manifest Categorical Variables

Applied developmental scientists engaged in the evaluation of community-based programs, in tests of therapeutic effectiveness, or in developmental research on special populations (e.g., youth gang members, street children, or refugees) must often conduct individual or contextual analyses that involve multiple categorizations over time. This article discusses models for analysis of change in cross classifications of nominal or ordinal variables. Examples of such models include Clogg, Eliason, and Grego's (1990) change-persistence models and quasi-independence models for nominal or ordinal variables. A definition of trends in nominal level variables is provided. A general model is specified, and two groups of log-linear models are presented as special cases of this model. The first allows researchers to test specific hypotheses concerning, for example, general trends and trends in subtables. The second group extends application of log-linear modeling of mobility tables to rectangular tables. Examples use data from research on change in schizophrenics and from pubertal development.

Etiology and prevention of substance use among Asian American youth.

Among populations identified by the U.S. Bureau of the Census, it is anticipated that the Asian/Pacific Islander (API) population will experience the greatest change between 1990 and 2050 (U.S. Bureau of the Census. [1996, February]. Current population reports. Series P25-1130. Washington, DC: U.S. Government printing office). Limited studies focus on APIs as a racial group and still fewer disaggregate samples to specific ethnic Asian subgroupings. This paper begins with definitions of the API communities, then examines rates of adolescent drug use, risk and protective factors, and preventive intervention effectiveness focused on API communities. The limited epidemiological data suggest that in general, APIs are at a relatively lower risk for drug use than youth from most other ethnic groups. However, the available data also suggest that use may not be as low as generally assumed with rates for alcohol use, smoking, and some illicit drugs being equal to or exceeding those of African Americans and European Americans. Despite the paucity of available data on particular Asian subgroups, the available data demonstrate that there are differences among API subgroups, underscoring the importance of identifying Asian subgroups when studying substance use and when planning prevention and treatment. The limited data examining the etiology of drug use across API subgroups suggests that some of the risk and protective factors derived from majority based research may also be predictors for these populations. These data support the utility of examining the generalizability of existing tested prevention approaches among different API communities. Finally, further efforts should be made to encourage and support the evaluation of community-based programs that already target and deliver services to API youth.

Promising community-level indicators for evaluating cardiovascular health-promotion programs.

Rigorous evaluation of community-based programs can be costly, particularly when a representative sample of all members of the community are surveyed in order to assess the impact of a program on individual health behavior. Community-level indicators (CLIs), which are based on observations of aspects of the community other than those associated with individuals, may serve to supplement individual-level measures in the evaluation of community-based programs or in some cases provide a lower-cost alternative to individual-level measures. Because they are often based on observations of the community environment, CLIs also provide a way of measuring environmental changes--often an intermediate goal of community-based programs. The Centers for Disease Control and Prevention convened a panel of experts knowledgeable about community-based program evaluation and cardiovascular disease (CVD) prevention to develop a list of CLIs, and rate their feasibility, reliability and validity. The indicators developed by the panel covered tobacco use, physical activity, diet and a fourth group that were considered 'cross-cutting' because they related to all three behaviors. The indicators were subdivided into policy and regulation, information, environmental change, and behavioral outcome. For example, policy and regulation indicators included laws and ordinances on tobacco use, policies on physical education, and guidelines for menu and food preparation. These indicators provide a good starting point for communities interested in tracking CVD-related outcomes at the community level.

Zero effects in substance abuse programs. Avoiding false positives and false negatives in the evaluation of community-based programs.

The evaluation of community-based programs poses special design and analysis problems. The present article focuses on two major types of errors that can occur in such evaluations: false positive--incorrectly declaring a program to be effective--and false negatives--incorrectly declaring a program to be ineffective. The evaluation of a national demonstration of community-based programs to reduce substance abuse, Fighting Back, is used to illustrate several approaches to reduce the probability of errors. Both those errors that are affected by the design and those by analytic approaches are considered. Ways to assess multiple outcomes and to match the complexity of the program with design and analytic strategies are proposed. Community trials are complex interventions, and, although they can provide very useful information, their outcomes have to be understood in terms of the constructs they test and the contexts within which they are carried out.

Applying logics of change to the evaluation of community development in health promotion

Evaluation of community-oriented health promotion programs require that professional evaluators clearly distinguish between exogenously and endogenously defined goals and definitions of ‘success’. Acknowledging the different dynamics which underpin community-led change and externally initiated health programs and interventions is essential to this task. It will be argued that the intersection of, and boundaries around, exogenous and endogenous change can best be understood and clarified through the application of ‘logics of rationality’, adapted from social theory. Community activity is characterised by what we have called ‘lifeworld rationality’ community-based health promotion interventions are characterised by ‘formal ratonality’. In addition, we suggest that the value dimensions (‘substantive rationality’) underpinning programs and interventions may be overlooked by policy makers, program planners and professional evaluators under pressure to demonstrate cost-effectiveness and efficiency. Key requirements for successful and appropriate evaluation of community-based programs which are sensitive to the needs and success criteria of communities include a shared understanding by the researchers, program sponsors and community actors of the nature of the changes sought. The co-production of health promotion standards and of indicators to judge the performance of the program or intervention by all stakeholders should be preferred over the trend for establishing community-controlled process evaluations to coexist alongside expert-controlled impact evaluations. It is argued that current approaches to standard setting and indicator development to judge the processes and impacts of interventions are inadequate and several principles for improving their content are given. A locally controlled ethnographic approach to evaluate endogenous community-led change is described in the hope that program planners and evaluators may become more sensitive and receptive to local knowledge. We suggest that engagement with what we have termed the ‘community story’ should be a fundamental requirement for the planning and evaluation of community health programs.

Program evaluation strategies for community-based health promotion programs: perspectives from the cardiovascular disease community research and demonstration studies.

Community-based programs are being widely adopted in the struggle to prevent chronic disease. Program evaluation of community-based programs involves a particular set of problems stemming from the variety of activities being undertaken simultaneously, the multiple intermediate goals of the programs and the rapidity with which the programs evolve. An analysis of the experience of four large community-based cardiovascular disease research and demonstration studies (Stanford Five-City Project, Minnesota Heart Health Program, Pawtucket Heart Health Program and the German Cardiovascular Prevention Project) provides valuable models, methodologies and strategies for planning and conducting evaluations of public health programs or community studies. By comparing and combining their experiences, the four programs have identified eight categories of evaluation for community studies, including formative evaluation, quality assurance, assessment of delivered dose, assessment of received dose, component program impact, intermediate outcomes, community impact and cost analysis. This paper presents information on the strategies by which each of the four programs addressed these evaluation categories.