Process Evaluation Components Research Articles

Protocol for process evaluation of evidence-based care pathways: the case of colorectal cancer surgery.

Care pathways are complex interventions, consisting of multiple 'active ingredients', to structure care processes around patient needs. Numerous studies have reported improved outcomes after implementation of care pathways. The structure-process-outcome framework and the context-mechanism-outcome framework both suggest that outcomes can only be achieved through a certain process within a context or structure. To understand how and why care pathways are effective, understanding of both this process and context is necessary. The aim of this article is to propose a study protocol to evaluate the implementation process of evidence-based care pathways, including the influence of the context. This protocol is explained by applying it to the implementation of a colorectal cancer surgery pathway in an international setting. The Medical Research Council (MRC) guidance on process evaluations for complex interventions is used as the basis for the protocol. The key components of process evaluation are intervention, context, implementation, mechanisms of impact and outcomes. In process evaluations, these components are studied using quantitative and qualitative methods. Among them are patient record analysis, questionnaires, on-site visits and interviews. To guide our methodological choices, the MRC guidance for process evaluations of complex interventions, and published protocols for process evaluations of complex interventions were used. Our protocol is now tailored for the process evaluation of evidence-based care pathways and provides researchers and clinicians methods and tools, as well as a worked example, that can be used to study the process of care pathway implementation. As a result, healthcare professionals will be informed on context factors and implementation processes that can facilitate the implementation of care pathways, improving quality and effectiveness of care processes.

The process of establishing Discovery College in Melbourne

PurposeThe purpose of this paper is to understand and inform the development and implementation of a newly established Discovery College (a youth-focused Recovery College). It also aims to contribute to a broader understanding of the benefits and barriers to establishing Recovery Colleges.Design/methodology/approachThe overall study took a mixed-methods approach to the evaluation of Discovery College, including a qualitative process evaluation component as well as a mixed-methods outcomes evaluation. This paper reports on the findings of the process evaluation, which undertook key informant interviews with a range of stakeholders in the implementation process.FindingsA total of 16 themes emerged from the qualitative data, which were then clustered into four main areas: establishing Discovery College, organisational context, nature of Discovery College and service transformation. Implementation was reported as both feasible and effective. Initial tension between fidelity to the model and a pragmatic approach to action was negotiated during implementation and through an ability of staff to tolerate uncertainty, enabled by the efforts and support of senior service management and college staff.Originality/valueRecovery Colleges co-designed and implemented in youth mental health services are a recent development in the field of mental health care and very little has previously been published regarding the feasibility, effectiveness and acceptability of youth-focused Recovery Colleges. This paper is one of the first to assess the barriers and enablers to the implementation of Discovery College within a clinical youth mental health service.

Process evaluation of two home-based bimanual training programs in children with unilateral cerebral palsy (the COAD-study): protocol for a mixed methods study

BackgroundAs part of the COAD-study two home-based bimanual training programs for young children with unilateral Cerebral Palsy (uCP) have been developed, both consisting of a preparation phase and a home-based training phase. Parents are coached to use either an explicit or implicit motor learning approach while teaching bimanual activities to their child. A process evaluation of these complex interventions is crucial in order to draw accurate conclusions and provide recommendations for implementation in clinical practice and further research. The aim of the process evaluation is to systematically assess fidelity of the home-based training programs, to examine the mechanisms that contribute to their effects on child-related and parent-related outcomes, and to explore the influence of contextual factors.MethodsA mixed methods embedded design is used that emerges from a pragmatism paradigm. The qualitative strand involves a generic qualitative approach. The process evaluation components fidelity (quality), dose delivered (completeness), dose received (exposure and satisfaction), recruitment and context will be investigated. Data collection includes registration of attendance of therapists and remedial educationalists to a course regarding the home-based training programs; a questionnaire to evaluate this course by the instructor; a report form concerning the preparation phase to be completed by the therapist; registration and video analyses of the home-based training; interviews with parents and questionnaires to be filled out by the therapist and remedial educationalist regarding the process of training; and focus groups with therapists and remedial educationalists as well as registration of drop-out rates and reasons, to evaluate the overall home-based training programs. Inductive thematic analysis will be used to analyse qualitative data. Qualitative and quantitative findings are merged through meta-inference.DiscussionSo far, effects of home-based training programs in paediatric rehabilitation have been studied without an extensive process evaluation. The findings of this process evaluation will have implications for clinical practice and further research regarding development and application of home-based bimanual training programs, executed by parents and aimed at improving activity performance and participation of children with uCP.

Motivational interviewing competencies among UK family nurse partnership nurses: a process evaluation component of the building blocks trial.

BackgroundMotivational Interviewing (MI) is a person-centred counselling approach to behaviour change which is increasingly being used in public health settings, either as a stand-alone approach or in combination with other structured programmes of health promotion. One example of this is the Family Nurse Partnership (FNP) a licensed, preventative programme for first time mothers under the age of 20, delivered by specialist family nurses who are additionally trained in MI. The Building Blocks trial was an individually randomised controlled trial comparing effectiveness of Family Nurse Partnership when added to usual care compared to usual care alone within 18 sites in England. The aim of this process evaluation component of the trial is to determine the extent to which Motivational Interviewing skills taught to Family Nurse Partnership nurses were used in their home visits with clients.MethodsBetween July 2010 and November 2011, 92 audio-recordings of nurse-client consultations were collected during the ‘pregnancy’ and ‘infancy’ phases of the FNP programme. They were analysed using The Motivational Interviewing Treatment Integrity (MITI) coding system.ResultsA competent level of overall MI adherent practice according to the MITI criteria for ‘global clinician ratings’ was apparent in over 70 % of the consultations. However, on specific behaviours and the MITI-derived practitioner competency variables, there was a large variation in the percentage of recordings in which “beginner proficiency” levels in MI (as defined by the MITI criteria) was achieved, ranging from 73.9 % for the ‘MI adherent behaviour’ variable in the pregnancy phase to 6.7 % for ‘percentage of questions coded as open’ in the infancy phase.ConclusionsThe results suggest that it is possible to deliver a structured programme in an MI-consistent way. However, some of the behaviours regarded as key to MI practice such as the percentage of questions coded as open can be more difficult to achieve in such a context. This is an important consideration for those involved in designing effective structured interventions with an MI-informed approach and wanting to maintain fidelity to both MI and the structured programme.Trial registrationCurrent Controlled Trials ISRCTN23019866 Registered 20/4/2009.

USING WEIGHTED SCORING RUBRICS IN ENGINEERING ASSESSMENT

When evaluating student work by deducting marks for errors,it is possible to underestimate the importance ofdominant concepts and assign grades at a level that mightnot be in agreement with academic policies. Rubrics facilitateand expedite the marking process but it is importantto examine in detail the parameters and limitations of thisstructured approach to assessment. The structure of thescoring rubric considered in this study promotes the consistencyand validation of the assessment process and discriminatesbetween evaluation components by assigningdifferent weights to dominant and secondary criteria. Theshape of the weight distribution function plays an importantrole in this process. In the proposed rubric structure,an array of performance levels is multiplied by anarray of task components to arrive at a mark and a grade.A uniform weight distribution is easy to develop and utilize,especially for large classes, but it fails to recognizethe importance of dominant components. The proposedapproach allows the incorporation of single and multipledominant criteria modeled by using step or linear distributionfunctions and adjusting the relative value of dominantand secondary components.

Linking implementation process to intervention outcomes in a middle school obesity prevention curriculum, 'Choice, Control and Change'.

This study investigates the link between process evaluation components and the outcomes of a school-based nutrition curriculum intervention, 'Choice, Control and Change'. Ten New York City public middle schools were recruited and randomly assigned into intervention or control condition. The curriculum was to improve sixth to seventh grade students' energy balance related behaviors, based on social cognitive and self-determination theories, and implemented during the 2006-2007 school year (n = 1136). Behaviors and psychosocial variables were measured by self-reported questionnaires. Process components were evaluated with classroom observations, teacher interviews, and a student questionnaire. Using 'Teacher Implementation' (dose delivered) and 'Student Reception' (dose received) process data; intervention group was further categorized into medium- and high-implementation groups. Analysis of covariance revealed that, compared with control group, only high-implementation group showed significant improvement in students' behavior and psychosocial outcomes. Hierarchical linear models showed that 'Teacher Implementation' and 'Student Reception' significantly predicted students' sweetened beverage outcomes (P < 0.05). 'Student Satisfaction' was also greater when these implementation components were higher, and significantly associated with behavior and psychosocial outcomes (P < 0.05). Implementation process influenced the effectiveness of the 'Choice, Control and Change' intervention study. It is important to take into account the process components when interpreting the results of such research.

A protocol for a systematic review of the use of process evaluations in knowledge translation research.

BackgroundExperimental designs for evaluating knowledge translation (KT) interventions for professional behavior change can provide strong estimates of intervention effectiveness but offer limited insight how the intervention worked or not. Furthermore, trials provide little insight into the ways through which interventions lead to behavior change and how they are moderated by different facilitators and barriers. As a result, the ability to generalize the findings from one study to a different context, organization, or clinical problem is severely compromised. Consequently, researchers have started to explore the causal mechanisms in complementary studies (process evaluations) alongside experimental designs for evaluating KT interventions. This study focuses on improving process evaluations by synthesizing current evidence on process evaluations conducted alongside experimental designs for evaluating KT interventions.Methods/DesignA medical research librarian will develop and implement search strategies designed to identify evidence that is relevant to process evaluations in health research. Studies will not be excluded based on design. Included studies must contain a process evaluation component aimed at understanding or evaluating a KT intervention targeting professional behavior change. Two reviewers will perform study selection, quality assessment, and data extraction using standard forms. Disagreements will be resolved through discussion or third party adjudication. Data to be collected include study design, details about data collection approaches and types, theoretical influences, approaches to evaluate intervention dose delivered, intervention dose received, intervention fidelity, intervention reach, data analysis, and study outcomes. This study is not registered with PROSPERO.DiscussionThere is widespread acceptance that the generalizability of quantitative trials of KT interventions would be significantly enhanced to other contexts, health professional groups, and clinical conditions through complementary process evaluations alongside trials. This systematic review will serve as a ‘state of the science’ on methodological approaches to process evaluations and will allow us to: 1) take stock of current research approaches and 2) develop concrete recommendations for knowledge users (e.g., quality consultants and health services researchers) designing future KT process evaluations.Electronic supplementary materialThe online version of this article (doi:10.1186/2046-4053-3-149) contains supplementary material, which is available to authorized users.

A Process Evaluation of an HIV/STI Intervention for Rural African American Youth.

We evaluated the fidelity and implementation of an HIV/AIDS and sexually transmitted infections intervention for rural African American youth. Using a community-based evaluation approach, community partners and researchers monitored four core process-evaluation components: reach, fidelity, dose delivered, and dose received. Researchers collected evaluation data through session observations, facilitator debriefing interviews, a youth focus group, and a satisfaction survey. For reach, more than half of the participants attended the 13 sessions. Participation varied between 62% and 100%. For fidelity, not all sessions were implemented as intended; multiple modifications occurred across sessions. For dose delivered, some lessons were missing materials and content was omitted; facilitators omitted content when there was insufficient time to complete a lesson. For dose received, engagement varied across lessons but youth reported high levels of satisfaction with the intervention. This formative process evaluation enabled us to identify and address multiple challenges to implementation.

Contextualising complementary feeding in a broader framework for stunting prevention.

An estimated 165 million children are stunted due to the combined effects of poor nutrition, repeated infection and inadequate psychosocial stimulation. The complementary feeding period, generally corresponding to age 6-24 months, represents an important period of sensitivity to stunting with lifelong, possibly irrevocable consequences. Interventions to improve complementary feeding practices or the nutritional quality of complementary foods must take into consideration the contextual as well as proximal determinants of stunting. This review presents a conceptual framework that highlights the role of complementary feeding within the layers of contextual and causal factors that lead to stunted growth and development and the resulting short- and long-term consequences. Contextual factors are organized into the following groups: political economy; health and health care systems; education; society and culture; agriculture and food systems; and water, sanitation and environment. We argue that these community and societal conditions underlie infant and young child feeding practices, which are a central pillar to healthy growth and development, and can serve to either impede or enable progress. Effectiveness studies with a strong process evaluation component are needed to identify transdisciplinary solutions. Programme and policy interventions aimed at preventing stunting should be informed by careful assessment of these factors at all levels.

Process Evaluation to Explore Internal and External Validity of the “Act in Case of Depression” Care Program in Nursing Homes

A multidisciplinary, evidence-based care program to improve the management of depression in nursing home residents was implemented and tested using a stepped-wedge design in 23 nursing homes (NHs): "Act in case of Depression" (AiD). Before effect analyses, to evaluate AiD process data on sampling quality (recruitment and randomization, reach) and intervention quality (relevance and feasibility, extent to which AiD was performed), which can be used for understanding internal and external validity. In this article, a model is presented that divides process evaluation data into first- and second-order process data. Qualitative and quantitative data based on personal files of residents, interviews of nursing home professionals, and a research database were analyzed according to the following process evaluation components: sampling quality and intervention quality. Nursing home. The pattern of residents' informed consent rates differed for dementia special care units and somatic units during the study. The nursing home staff was satisfied with the AiD program and reported that the program was feasible and relevant. With the exception of the first screening step (nursing staff members using a short observer-based depression scale), AiD components were not performed fully by NH staff as prescribed in the AiD protocol. Although NH staff found the program relevant and feasible and was satisfied with the program content, individual AiD components may have different feasibility. The results on sampling quality implied that statistical analyses of AiD effectiveness should account for the type of unit, whereas the findings on intervention quality implied that, next to the type of unit, analyses should account for the extent to which individual AiD program components were performed. In general, our first-order process data evaluation confirmed internal and external validity of the AiD trial, and this evaluation enabled further statistical fine tuning. The importance of evaluating the first-order process data before executing statistical effect analyses is thus underlined.

Process evaluation results from the HEALTHY physical education intervention.

Process evaluation is an assessment of the implementation of an intervention. A process evaluation component was embedded in the HEALTHY study, a primary prevention trial for Type 2 diabetes implemented over 3 years in 21 middle schools across the United States. The HEALTHY physical education (PE) intervention aimed at maximizing student engagement in moderate-to-vigorous physical activity through delivery of structured lesson plans by PE teachers. Process evaluation data collected via class observations and interventionist interviews assessed fidelity, dose delivered, implementor participation, dose received and barriers. Process evaluation results indicate a high level of fidelity in implementing HEALTHY PE activities and offering 225 min of PE every 10 school days. Concerning dose delivered, students were active for approximately 33 min of class, representing an average of 61% of the class time. Results also indicate that PE teachers were generally engaged in implementing the HEALTHY PE curriculum. Data on dose received showed that students were highly engaged with the PE intervention; however, student misbehavior was the most common barrier observed during classes. Other barriers included teacher disengagement, large classes, limited gym space and poor classroom management. Findings suggest that the PE intervention was generally implemented and received as intended despite several barriers.

The process evaluation of two interventions aimed at portion size in worksite cafeterias

In a previous study, the effectiveness of introducing a small meal in addition to the existing size and a proportional pricing strategy have been assessed in Dutch worksite cafeterias. To assess the degree of implementation and to inform the design of future interventions, the present study aimed to describe the process evaluation of both interventions. Process evaluation components from Baranowski and Stables, and Rogers (i.e. Recruitment, Maintenance, Context, Resources, Implementation, Exposure, Contamination, and Continued use) were chosen as a theoretical basis. The process evaluation involved qualitative (e.g. structured observations, semi-structured interviews) and quantitative data (e.g. consumer questionnaires) collected from 17 intervention and eight control worksite cafeterias. In all intervention cafeterias, two portion sizes were offered. The pricing instructions were followed in 13 intervention cafeterias. The cafeterias managers indicated that they did not consider offering large and small meals as being complex, risky or time-consuming to implement. Some managers perceived the consumer demand as high, others as (too) low. One year after the study had ended, nine of the intervention cafeterias had continued (at least partly) to follow the protocol. Offering a smaller portion size in addition to the existing size, as well as proportional pricing, was generally implemented as prescribed by the protocol and can be considered promising in terms of continued use. However, additional efforts are needed to make the intervention more effective in motivating consumers to replace their large portion with a small portion.

The Examination of Process Evaluation Use in Church-Based Health Interventions

Churches have been a popular site for the implementation of health promotion interventions. Although the efficacy of church-based health programs have been established, it is unknown which aspects of church-based health promotion drive health behavior change. Process evaluation is a way to increase our understanding of key components of church-based health promotion and to move the field forward. Thus, a systematic review of the utilization of process evaluation in church-based health programs was conducted. Articles from 1990 to 2008 were screened for eligibility, resulting in the analysis of 67 articles. The majority of church-based health programs assessed recruitment (88.1%) and reach (80.6%). About 28.4% assessed dose delivered, and 27.3% measured dose received. Context and fidelity was assessed by 34.3% and 20.9%, respectively, of church-based interventions. Approximately 9% of church-based programs measured fidelity. On average, only three of seven possible components of process evaluation were measured among the studies reviewed. The number of process evaluation components assessed did not differ by program feature (e.g., target population, target health condition, program objective, etc.). Consistency in the conceptualization and measurement of process evaluation may facilitate the implementation of a comprehensive process evaluation effort in church-based and other health promotion interventions.

How to Perform a Preplanned Process Evaluation for Complex Interventions in Geriatric Medicine: Exemplified With the Process Evaluation of a Complex Falls-Prevention Program for Community-Dwelling Frail Older Fallers

Complex interventions are difficult to develop, document, evaluate, and reproduce. Process evaluations aid the interpretation of outcome results by documenting and evaluating each process step in detail. Despite its importance, process evaluations are not embedded in all evaluations of complex interventions. Based on literature, we structured the process evaluation for trials on complex interventions into 3 main components: (1) the success rate of recruitment and quality of the study population, (2) the quality of execution of the complex intervention, and (3) the process of acquisition of the evaluation data. To clarify these process evaluation components and measures, we exemplified them with the preplanned process evaluation of a complex falls-prevention program for community-dwelling frail older fallers and their informal caregivers. The 3 process evaluation components are operationalized, results are presented, and implications discussed. This process evaluation identified several limitations of the intervention and effect study, and resulted in multiple recommendations for improvement of both the intervention as well as the trial. Thus, a good-quality process evaluation gives a detailed description of the most important components of a complex intervention, resulting in an in-depth insight in the actually performed intervention and effect analysis. This allows us to draw the appropriate conclusions on positive or negative trial results, and results in recommendations for implementation, or adjustment of the intervention or effect evaluation, respectively.

Multimethod Strategy for Assessing Program Fidelity: The National Evaluation of the Revised G.R.E.A.T. Program

This study reports the results of the process evaluation component of the Process and Outcome Evaluation of the Gang Resistance Education and Training (G.R.E.A.T.) program. The process evaluation consisted of multiple methods to assess program fidelity: (a) observations of G.R.E.A.T. Officer Trainings (G.O.T); (b) surveys and interviews of G.R.E.A.T.-trained officers and supervisors; (c) surveys of school personnel; and (d) ''on-site,'' direct observations of officers delivering the G.R.E.A.T. program in the study sites. Results illustrate a high level of program fidelity, providing greater confidence in any subsequent outcome results.

Transforming the Organizational Culture of a School of Nursing Through Innovative Program Development

This article illustrates how a grant designed to promote new program development provided a vehicle for organizational transformation. The collaboration surrounding this initiative created a common focus within the school that more effectively channeled its resources and resulted in an unprecedented level of scholarly achievement and recognition. Faculty leveraged the success of this initial grant to procure additional funding for related projects. The importance of partnerships and teamwork were two valuable lessons learned. We believe that our experience is replicable in other schools of nursing interested in organizational transformation. Gibson and Barsade's model of managed change guided the project's implementation and evaluation processes. Recommendations for engaging faculty, gaining support, and developing a collaborative network are discussed in the article, with findings from a stakeholder-focused evaluation demonstrating new program goal achievement as well as the transformative changes that occurred in the organizational culture. A focused, theory-derived program plan, with comprehensive process and outcome evaluation components resulted in a major transformation of one school of nursing. Unanticipated outcomes included renewed synergy among faculty; the development of a preferred vision for the future; scholarly collaboration around a central theme that effectively channeled limited resources and dramatically increased productivity; increased regional and national recognition; and the creation of regional, national, and international partnerships.

The Diffusion of Effective Behavioral Interventions Project: Development, Implementation, and Lessons Learned

Implementation of evidence-based HIV/STD prevention interventions can play an important role in reducing HIV and sexually transmitted diseases. This article describes the development, implementation, and lessons learned of the Diffusion of Effective Behavioral Interventions (DEBI) project, a strategy funded by the Centers for Disease Control and Prevention to diffuse evidence-based, group- and community-level HIV/STD prevention interventions to health departments and community-based organizations nationwide. The article specifically provides an overview of the rationale, description, and theoretical foundation of the project; a review of marketing efforts, including assessment of interests, needs, and capacities relative to the project; a description of project products, their purpose, approach employed to develop them, and their use by implementers; a description of the project's training coordination functions and activities; technical assistance issues; an overview of process and outcome evaluation components; new developments in response to feedback; and a discussion of future directions for DEBI. Project successes and challenges are addressed to inform future efforts to diffuse prevention interventions.

Process evaluations of the 5-a-day projects.

Process evaluation is an important, but infrequently conducted, component of evaluating the impact of health promotion interventions. The process evaluation results from the nine 5-a-Day projects were overviewed. Process evaluation helped explain some of the weaker aspects of program performance, process indicators occasionally declined over time and varied by demographic characteristics, and some process measures were related to mediating variables and program outcomes. Future development of process evaluation must include further development of concepts, more consistent and thorough conduct of process evaluation, appropriate methodological work, and assessment of the relations among the process evaluation components and the program mediators and outcomes. Further development in this area promises refinement in understanding how and why interventions achieve their effects, how best to conduct intervention programs to maximize effects, and enhancement of the internal and external validity of the studies.

Process evaluation in a multisite, primary obesity-prevention trial in American Indian schoolchildren

We describe the development, implementation, and use of the process evaluation component of a multisite, primary obesity prevention trial for American Indian schoolchildren. We describe the development and pilot testing of the instruments, provide some examples of the criteria for instrument selection, and provide examples of how process evaluation results were used to document and refine intervention components. The theoretical and applied framework of the process evaluation was based on diffusion theory, social learning theory, and the desire for triangulation of multiple modes of data collection. The primary objectives of the process evaluation were to systematically document the training process, content, and implementation of 4 components of the intervention. The process evaluation was developed and implemented collaboratively so that it met the needs of both the evaluators and those who would be implementing the intervention components. Process evaluation results revealed that observation and structured interviews provided the most informative data; however, these methods were the most expensive and time consuming and required the highest level of skill to undertake. Although the literature is full of idealism regarding the uses of process evaluation for formative and summative purposes, in reality, many persons are sensitive to having their work evaluated in such an in-depth, context-based manner as is described. For this reason, use of structured, quantitative, highly objective tools may be more effective than qualitative methods, which appear to be more dependent on the skills and biases of the researcher and the context in which they are used.

Reality check. Evaluating a school-based gang prevention model.

The Gang Resistance Education and Training (G.R.E.A.T.) program is a school-based gang prevention initiative developed in 1991 through the collaborative efforts of the Bureau of Alcohol, Tobacco, and Firearms, the Federal Law Enforcement Training Center, and the Phoenix Police Department. Uniformed law enforcement officers, certified as G.R.E.A.T. instructors, teach the 9-week curriculum to middle students. In 1994, the National Institute of Justice funded a national evaluation of the G.R.E.A.T. program. The process evaluation component of this larger study is reported. First, results of on-site observations of the G.R.E.A.T. Officer Training program, including an overview of the training activities, and the authors' assessment of the training process are reported. Second, observations of the implementation of the program by officers at six sites are reported. Of primary concern was whether the program delivered to students was similar to the program taught to the officers during the G.R.E.A.T. Officer Training.