Quality Of Program Delivery Research Articles

The importance of group factors in the delivery of group-based parenting programmes: a process evaluation of Mellow Babies

IntroductionThe role of the group has been largely overlooked within evaluations of group-based parenting programmes. Group contextual factors, including size and level of homogeneity, may impact on essential group processes, such as group identification and cohesion, that are necessary to activate interpersonal change mechanisms and attain programme outcomes. This process evaluation of Mellow Babies, a 14-week attachment-based group parenting programme for mothers of infants aged under 18 months, explores how group context affected mother and practitioner experiences of the programme.MethodsIn-depth interviews were conducted with fourteen mothers and three practitioners from three different Mellow Babies groups. Framework Analysis was employed to analyse data, using groups as cases within the framework matrix while preserving individual participants within each case. This allowed comparisons to be made within and between groups.ResultsFour group contextual factors impacted on the quality of programme delivery: (1) group size; (2) level of group homogeneity; (3) pre-existing relationships; and (4) personalities within the group. These contextual factors affected the hypothesised intervention mechanisms: (1) fluid progression through the stages of group development; (2) a safe, non-judgemental, contained space; (3) social identification with group; (4) group cohesion; and (5) a culture of openness, support and empowerment.DiscussionFindings have implications for future delivery and implementation of group-based parenting programmes, for example, the importance of considering group composition during programme recruitment. Practitioners may also benefit from a stronger focus on group processes and enabling group dynamics within training, supervision, and the programme manual.

Taking Care of Friends: The Implementation Evaluation of a Peer-Focused School Program Using First Aid to Reduce Adolescent Risk-Taking and Injury.

Injury is a leading cause of adolescent deaths, with risk-taking associated with a sizeable proportion of injuries and many of those risks undertaken in the presence of peers or with peers’ knowledge. Novel ways to promote safety are required and using the peer-relationship may be an important mechanism for prevention. This study reports on the implementation evaluation of the Skills for Preventing Injury in Youth (SPIY) program. SPIY is a high-school program designed to reduce injury by encouraging peers to look out for one another and prevent risk-taking, complemented by developing peer helping and first aid skills as well as school connectedness. 152 students and 12 teachers who delivered SPIY participated in separate 30 min focus groups and reported on students’ understanding of peer protective behaviour and the program implementation (adherence, dose, quality of program delivery, and participant responsiveness). Students reported on many approaches to protecting friends and both students and teachers reported they found the program interesting, interactive, and able to be delivered. Peer protection messages were relevant and acceptable to teachers and students in a risk-taking harm reduction program to reduce adolescent injury.

Assessing fidelity of implementation to a technology‐mediated early intervention using process data

AbstractBackgroundProcess data, data generated by a user's interaction with a web‐based application, is an emerging tool in educational research. The current study explores using process data as a measure of implementation fidelity to a randomized control trial (RCT) of the Read It Again Mobile (RIA‐M) curricular supplement.ObjectivesTo determine the extent to which teachers implemented RIA‐M and to assess the utility of using process data in the assessment of fidelity.MethodsThe RCT involved 30 pre‐kindergarten classrooms with a sample of n = 216 students. RIA‐M provides a curricular supplement which teachers may incorporate into classroom instruction and is delivered via a tablet computer. Pre and post literacy assessments are used to determine treatment effect. Process data, produced from teacher interactions with the tablet, and classroom observations are used to assess fidelity.Results and ConclusionsOur findings indicate no difference between treatment and control students in the RCT. Yet, we find that process data provides unique fidelity information concerning treatment exposure, adherence, and quality of program delivery. Specifically, process data indicated that teachers did not demonstrate the same level of fidelity that was captured in classroom observations. This finding provides some evidence for the absence of an intervention effect.Major TakeawaysThe current study improves our understanding of how web‐based interventions may be assessed for implementation fidelity using process data. Further, process data offers a potentially reliable and scalable measure of fidelity for other web‐based educational interventions.

Fidelity of implementation of the building bridges mentoring intervention to prevent violence among youth in low income settings

Violence among youth is a major health and safety burden globally. There is a dearth in the development and evaluation of targeted interventions that addresses the multi-faceted nature of youth violence to ensure effectiveness and replicability. This study aims to systematically evaluate the fidelity of implementation of the Building Bridges Mentoring intervention that focuses on the prevention of youth violence in two low-income communities in South Africa. This study employed a mixed methods concurrent triangulation design. Qualitative data were analysed using deductive thematic analysis, and quantitative data using descriptive statistics and T-tests. The results indicate that for all the intervention components, implementation fidelity (i.e. adherence, exposure, quality of programme delivery, and participant responsiveness) was generally moderate to high, suggesting a ‘good’ implementation of the programme in the real world. Evaluating the fidelity of implementation is vital to obtain a comprehensive insight into whether an intervention was implemented according to its design, and to verify and validate the findings and outcomes, and accord credibility and integrity of the study.

Perceived sustainability of the school-based social and behavior change communication (SBCC) approach on malaria prevention in rural Ethiopia: stakeholders\u2019 perspectives

BackgroundEvidence on what makes the school-engaged social and behavior change communication (SBCC) interventions on malaria prevention more sustainable are limited in literature partly due to its recent emergence. Enrolling the key stakeholders, this study explored the perceived sustainability of the SBCC interventions on malaria prevention through primary school communities in rural Ethiopia.MethodsThe SBCC interventions were implemented from 2017 to 2019 in 75 primary schools and villages in rural Jimma to promote malaria preventive practices. As a part of program evaluation, this study employed a mixed-method to collect qualitative and quantitative data from 205 stakeholders following the end of the program. Data were collected using interview guides and structured questionnaires. The SPSS version 26 and Atlas ti7.1 software were used to analyze the data. Multivariable linear regression modeling was used to identify predictors of the perceived sustainability of the program (SOP).ResultsThe mean score of SOP was 25.93 (SD = 4.32; range 6–30). Multivariable linear regression modeling showed that the perceived risk to malaria (β = 0.150; P = 0.029), self-efficacy (β = 0.192; P = 0.003), and perceived fidelity of implementation (β = 0.292; P = 0.000) and degree of adoption (β = 0.286; P = 0.000) were positively predicted the perceived SOP. The qualitative result identified various barriers and opportunities to sustaining the program that summarized under three themes which include perceptions about the quality of program delivery (e.g inadequate involvement of stakeholders and staffs, concerns over short project life, immature sustainability efforts), school settings (e.g schools’ malaria priority, schools’ climate and quality of coaching) and the outer settings (e.g existing structures in the health and education systems).ConclusionThe study identified key predictive variables such as stakeholders’ perceived risk to malaria, self-efficacy, perceived fidelity of implementation and degree of adoption that could help to improve the sustainment of the school-based SBCC approach on malaria prevention and control. Further longitudinal study should be conducted to examine the rate of decline in program components over time and how improved sustainability would contribute to the effectiveness on malaria preventive behaviors among students.

Implementation fidelity of provider-initiated HIV testing and counseling of tuberculosis patients under the National Tuberculosis Control Program in Kathmandu District of Nepal: an implementation research

BackgroundThere exists low uptake of Human Immunodeficiency Virus (HIV) testing among Tuberculosis (TB) patients through Provider-Initiated HIV Testing and Counseling (PITC) under the national TB control program in Nepal. The degree and quality of program delivery were explored through determining whether the PITC program is currently implemented as intended. This study aimed to assess three major components of the program’s implementation fidelity: adherence to PITC service, exposure, and quality of program delivery in order to optimize and standardize PITC implementation by exploring its barriers and enablers.MethodsThis research used a sequential explanatory mixed method design. Retrospective cross-sectional study of TB patients enrolled in five TB treatment centers of the Kathmandu district from July 1, 2016, to June 30, 2017 was done to assess PITC adherence to Direct Observed Treatment-Short Course (DOTS) protocols. The centers’ TB-DOTS readiness was assessed using the WHO Service Availability and Readiness Assessment checklist. A qualitative study was conducted to explore the barriers and enablers of PITC service implementation.ResultsFrom a total of 643 TB patients registered, 591 (92.1%) patients were offered HIV test counseling. Amongst those, 571 (96.6%) accepted and 523 (91.5%) were tested. Service providers’ HIV knowledge was found to be good although only 2/5 (40%) had participated in PITC training. The key barriers experienced by service providers were: patients feeling offended, stigmatization and lack of human resources in DOTS centers. The main enablers for PITC were national TB program commitment, health workers’ motivation, collaboration between stakeholders and external development partners’ promotion of program implementation.ConclusionIn the selected study sites, PITC services are well integrated into the routine TB control program with a high uptake of HIV testing among registered TB patients. This achievement should be sustained by addressing the identified barriers mainly in the quality of the PITC program delivery.

Criminality, thinking patterns and treatment effects – evaluation of the Swedish cognitive intervention programme ‘new challenges’ targeting adult men with a criminal lifestyle

ABSTRACTThe cognitive intervention programme ‘New Challenges’ targeting adult men with a criminal lifestyle was evaluated in a pilot study. The participants were divided into a cognitive treatment group (n = 32) and a control group (n = 11). In the control group, six participants had no treatment and five participated in 12-step treatment. The participants were measured pre and post using the Psychological Inventory of Criminal Thinking Styles (PICTS), the abridged version of sense of coherence (SOC), Positive and Negative Affect Scale, and Bergström’s quality of programme delivery (QPD). The results of the treatment group showed that criminal thinking patterns dropped significantly from high values to close to normal level. SOC and positive affect increased significantly in the treatment group. Both SOC and positive affect showed positive correlation with QPD. Regarding the possible influence of the 12-step treatment, there was no difference in the control group between participants receiving 12-step treatment and those not receiving treatment. The main conclusion is that the cognitive treatment programme ‘New Challenges’ can contribute to reduced criminal thinking and increased SOC and positive affect, which may prove to be important precursors of reduced criminality.

P169 - Exploring Quality Assessment in Community-Based Nutrition Education

Background (Background, Rationale, Prior Research, and/or Theory): Quality of program delivery is a major component in community-based nutrition education and is also considered “an art and a science”; and while this is associated with positive outcomes, little is known about how the quality of implementation is assessed in community nutrition programs. Padres Preparados, Jóvenes Saludables (PPJS) is a federally funded, longitudinal prevention intervention. We focus on strengthening family-skills of immigrant Latino parents (especially fathers) of adolescents, aimed to promote healthy energy balance-related behaviors. Objective: We explored quality of implementation during a pilot evaluation in spring 2017, using a multi-prong approach: principles-based fidelity, attendance, and participant reaction: interest, relevance, comfort level, and satisfaction. Description: We conducted live observation of a sub-set of sessions with a check-list of “core concepts”. Facilitators answered questions regarding participant engagement and understanding of concepts presented. Participants completed a session evaluation that assessed understanding of core concepts presented (knowledge), and facilitator rapport. We aimed to assess adherence but also other site-specific nuances associated with learning and key outcomes. Evaluation: Of the 13 participating families (fathers), most attended six sessions. In total, parents completed 109 after-session (one-page) evaluation forms. Most (90%) indicated class sessions were somewhat or very useful, somewhat or very interesting, and that participants felt comfortable sharing opinions. Most (94%) of their responses rated facilitators as supportive and as having addressed participants' needs and interests. Conclusions and Implications: This process evaluation findings illustrate quality implementation assessment. Overall facilitators had high levels of facilitation skills. The principle-based fidelity addressed the necessity, not only to capture adherence, but also whether the program has been able to achieve the desired results. It also provided insight to implementation that allows for adaptability to “real life” situations without loss of effectiveness. However, it called for a highly-skilled evaluator and developing the indicators of that can be broadly used in community settings. Funding: USDA.

TRACER STUDY OF THE GRADUATE DEGREE PROGRAMS OF VISAYAS STATE UNIVERSITY

Tracer study is needed to understand how well a university has performed in terms of developing the competencies of its students and in preparing them for the world of work. This study was conducted to determine the employment characteristics and job experiences of the graduates of VSU's graduate degree programs, and their feedback on their educational experiences in the university. Data were gathered by sending online questionnaires to graduates with active email accounts, handing in questionnaires to alumni working in VSU and nearby agencies and institutions, and conducting interviews and focus group discussions with graduates working in some institutions in Bohol. Of the 73 respondents, 78% finished master's degree only at ViSCA/LSU/VSU, while 16% finished both master's and doctorate degrees from the university. Almost all (99%) are employed, 84% have regular or permanent positions, 78% are working in academic institutions/state universities and colleges either as faculty members, researchers, extension workers or administrative staff, 27% occupy supervisory positions, and 64% got promoted after earning their graduate degrees in VSU. A great majority of the respondents (92%) considered the graduate degrees they earned as highly relevant to their current jobs. The top five skills they learned in VSU which they found useful included teaching (78%), communication (69%), critical thinking (63%), problem-solving (55%), and human relation skills (52%). The graduate programs were rated high by the respondents in terms of seven criteria, including relevance of the program to professional requirements (4.65), teaching/learning environment (4.33), teacher-student relationship (4.31), quality of program delivery (4.29), range of courses offered (4.20), library, laboratory and other facilities (4.10), and work placement (4.07). This suggests that the respondents were generally satisfied with the university's graduate curricular offerings. However, they gave some suggestions to further improve the delivery of graduate degree programs by VSU.

Implementation of formative assessment – effects of quality of programme delivery on students’ mathematics achievement and interest

The aim of this study was to contribute to the understanding of the effectiveness of formative assessment interventions by analysing how the quality of programme delivery affects students’ mathematics achievement and interest. Teachers (n = 17) implemented formative assessment in their ninth-grade mathematics classes and provided their students (n = 426) with written process-oriented feedback. Four feedback characteristics (number of feedback comments, specificity, feedback at self level, social reference norm) and two types of embedment of feedback in the instructional context (focus on feedback utilisation, focus on performance evaluation) were evaluated. Multilevel regression analyses revealed no significant effects of feedback characteristics on interest but negative effects of number of feedback comments and specificity on achievement in mathematics. Positive effects on mathematics achievement and interest were found when feedback was embedded in instruction and had emphasis on feedback utilisation. Students’ interest also was affected positively when performance evaluation was stressed.

Factors Associated With Community Health Worker Performance Differ by Task in a Multi-Tasked Setting in Rural Zimbabwe.

Zimbabwe, like most low-income countries, faces health worker shortages. Community health workers (CHWs) bridge this gap by delivering essential health services and nutrition interventions to communities. However, as workloads increase, CHWs' ability to provide quality services may be compromised. We studied influences upon CHWs' performance related to pregnancy surveillance and nutrition and hygiene education in rural Zimbabwe. In the context of a cluster-randomized trial conducted in 2 rural districts between November 2012 and March 2015, 342 government-employed CHWs identified and referred pregnant women for early antenatal care and delivered household-level behavior change lessons about infant feeding and hygiene to more than 5,000 women. In 2013, we conducted a survey among 322 of the CHWs to assess the association between demographic and work characteristics and task performance. Exploratory factor analyses of the Likert-type survey questions produced 8 distinct and reliable constructs of job satisfaction and motivation, supervision, peer support, and feedback (Cronbach α range, 0.68 to 0.92). Pregnancy surveillance performance was assessed from pregnancy referrals, and nutrition and hygiene education performance was assessed by taking the average summative score (range, 5 to 30) of lesson delivery observations completed by a nurse supervisor using a 6-item Likert-type checklist. Poisson and multiple linear regressions were used to test associations between CHW demographic and work characteristics and performance. CHWs who referred more pregnant women were female, unmarried, under 40 years old, from larger households, and of longer tenure. They also perceived work resources to be adequate and received positive feedback from supervisors and the community, but they were less satisfied with remuneration. CHWs with high scores on behavior change lesson delivery were from smaller households, and they received more supportive supervision but less operational supervision. Measures of job satisfaction and motivation were not associated with either task. Among CHWs responsible for multiple tasks in rural Zimbabwe, factors associated with performance of one task were not the same as those associated with performance of another task. Our methods and findings illustrate ways to examine heterogeneity in CHW performance and to identify organizational factors associated with quality of program delivery.

Evaluating the implementation of the Empowering Coaching™ programme

Purpose – The purpose of this paper is to explore the implementation of a theoretically grounded coach education training programme for youth football coaches in Norway, through observational methods. In particular, it focuses on implementation fidelity and programme adaptation, and possible differences between the coach educators (CEs) according to their level of experience. Design/methodology/approach – Implementation fidelity and programme adaptations for seven CEs were explored through the use of a fidelity scale and in-depth qualitative analyses. Participant responsiveness was applied to inform the observational analyses. Findings – Results showed that most of the programme was implemented with moderate to high fidelity, and that it was adapted when delivered. Most of the adaptations seemed to be positive, aligning with the programme’s theoretical foundation and goals. A few negative and neutral adaptations also occurred. The most experienced CEs seemed to deliver the programme with highest fidelity and they also made most positive adaptations. The findings, supported by the participants’ evaluation of the programme delivery, indicate that these CEs also delivered the programme with high quality. Research limitations/implications – In the field of implementation research there is a need for further studies applying observational methods to explore programme effectiveness in relation to implementation fidelity, adaptations made and the quality of programme delivery. Originality/value – This study applies observational methods to evaluate the fidelity of implementation and adaptations made when implementing a coach education programme, and based on these findings quality of delivery is discussed.

Evaluation of a cross-sector community initiative partnership: Delivering a local sport program

Corporate community initiatives (CCI) are often established via cross-sector partnerships with nonprofit agencies to address critical social problems. While there is a growing body of literature exploring the effectiveness and social impact of these partnerships, there is a limited evaluative research on the implementation and execution processes of CCIs. In this paper, we examined the implementation and operational processes in the delivery of a professional sport organization's CCI initiative using program theory evaluation. The findings showed discrepancies between the associate organization and the implementers regarding understanding and fulfilling responsibilities with performing certain aspects (maintaining accurate records and program marketing) of the service delivery protocol. Despite program stakeholders being satisfied overall with the program delivery, contradictions between program stakeholders’ satisfaction in the quality of program delivery was found in critical components (marketing and communications) of the service delivery. We conclude that ongoing evaluations are necessary to pinpoint the catalyst of the discrepancies along with all partners valuing process evaluation in addition to outcome evaluation.

Implementation of a Coaching Program for School Principals

We present a multi-phase coaching model that was implemented to help principals improve their instructional leadership practices. We then discuss a rubric based on this coaching model that we used to evaluate coaches’ implementation of key model phases and to identify principals’ responses to the coaching. After presenting the leadership coaching model, we introduce the implementation rubrics, and then we present contrasting cases from our analyses that illustrate two principals’ varying responses to coaching. We discuss how their coaches differed in two key dimensions of implementation: dose and the quality of program delivery. We conclude with a discussion of how these findings can inform development of future educational leadership coaching programs and guide additional research to evaluate the impact of coaching.

Perspectives on domestic violence: the influence of organisational issues on program outcomes

This paper explores the role of intervention programs for men who commit acts of domestic violence against women and children. It discusses policy, organisational, practice, and research issues in the delivery of contracted-out legally mandated perpetrator programs, drawing on the findings of a collaborative research project conducted between an Australian department of corrective services and an independent group of researchers. This paper, based on a series of in-depth interviews with service providers, victims, and other stakeholders, discusses how the way in which services are organised can influence the nature and quality of program delivery.

Using Online Components to Facilitate Program Implementation: Impact of Technological Enhancements to All Stars on Ease and Quality of Program Delivery

The ease with which programs can be delivered may impact sustainability, fidelity, and ultimately program effectiveness in a dissemination environment. This paper presents results from a study examining the ability of technological enhancements to make it easier for teachers to deliver All Stars, an evidence-based drug prevention program. These enhancements were designed to save time for teachers, produce improvements in quality of delivery, and provide features to increase students' involvement. Results of a randomized field trial revealed that teachers who used the enhancements found it easier to implement key program components compared to facilitators who taught the program as usual. Teachers in the technology-enhanced condition reduced the time required to complete otherwise time-consuming tasks. They also reported the ability to deliver more of the program, although they also reported more modifications and omitted activities within sessions when compared to teachers who delivered the program as usual. Moreover, teachers' attitudes about the program improved after their experience with the enhancements, the vast majority wishing to continue to use them in the future. Results suggest that modest gains in quality of program delivery can be expected when online applications designed to make tasks easier can be adopted by prevention service providers. On the other hand, enhancements that require special skill sets regarding computer-based methods may provide no clear benefit.

Quality of implementation: developing measures crucial to understanding the diffusion of preventive interventions

As prevention programs become disseminated, the most serious threat to effectiveness is maintaining the quality of implementation intended by the developers. This paper proposes a methodology for measuring quality of implementation in school settings and presents data from a pilot study designed to test several of the proposed components. These methods included assessments of adherence, quality of process, the positive or negative valence of adaptations, teachers' attitudes and teachers' understanding of program content. This study was conducted with 11 teachers who had varying degrees of experience who taught Life Skills Training. Observation and interview data were collected during visits to schools. Results suggest that quality of implementation can be measured through observation and interview. Teachers varied in adherence and quality of program delivery. All teachers made adaptations to the program. Experienced teachers were more likely to adhere to the curriculum, deliver it in a way that was more interactive and engaging to students, communicate the goals and objectives better, and make positive adaptations. The field can use these findings as the basis for exploring strategies for measuring and improving quality of implementation.

A review of research on fidelity of implementation: implications for drug abuse prevention in school settings.

To help inform drug abuse prevention research in school settings about the issues surrounding implementation, we conducted a review of the fidelity of implementation research literature spanning a 25-year period. Fidelity has been measured in five ways: (1) adherence, (2) dose, (3) quality of program delivery, (4) participant responsiveness and (5) program differentiation. Definitions and measures of fidelity were found not to be consistent across studies, and new definitions are proposed. While there has been limited research on fidelity of implementation in the social sciences, research in drug abuse prevention provides evidence that poor implementation is likely to result in a loss of program effectiveness. Studies indicate that most teachers do not cover everything in a curriculum, they are likely to teach less over time and training alone is not sufficient to ensure fidelity of implementation. Key elements of high fidelity include teacher training, program characteristics, teacher characteristics and organizational characteristics. The review concludes with a discussion of the tension between fidelity and reinvention/adaptation, and ways of resolving this tension. Recommendations are made for developing a consistent methodology for measuring and analyzing fidelity of implementation. Further, researchers and providers should collaborate to develop ways of introducing flexibility into prevention programs.

Accreditation of offending behaviour programmes in HM Prison Service: ‘What Works’ in practice

Purpose. This study assessed the short‐term impact of offending behaviour programmes in relation to certain key features of programme delivery identified by an accreditation system. The aim was to further inform the debate on ‘What Works’ in practice by establishing whether well‐delivered programmes are more effective.Method. The sample consisted of 5,255 offenders serving custodial sentences in prisons across England and Wales who completed one of two accredited cognitive skills programmes during the financial year 2001/2002. The relationship between a battery of assessment measures, tutor experience, drop‐out rates, audit observations and the quality of programme delivery was explored.Results. A positive short‐term impact was observed on the majority of assessment measures across both programmes. Patterns of change were broadly similar across gender, age, and ethnic groupings. This short‐term impact of the programmes was significantly greater for a group of high‐need prisoners and at those sites where tutors were delivering more frequently. Tutor delivery rates were also found to be related to drop‐out rates and the quality of programme delivery. Attempts to establish a relationship between ratings of tutor performance from video‐monitoring and the short‐term impact of the course were unsuccessful. However, at the programme site level, positive correlations were observed between the quality of delivery and other measures of the site's performance.Conclusion. This study shows that programmes do have a short‐term impact and that this is greater for higher‐need prisoners and at sites where tutors were delivering more frequently. Furthermore, the accreditation system has highlighted key aspects of programme delivery which do appear to influence the short‐term effectiveness of the programmes.

Measuring the real impact of accredited offending behaviour programmes

Purpose. This paper describes an integrated model for the evaluation of offending behaviour programmes in order to assess their real impact.Arguments. To date there has been an over‐reliance on reconviction data as the sole measure of treatment efficacy. Reconviction has a fundamental role to play but cannot be considered in isolation from other empirically related treatment and re‐settlement factors. A rigorous reconviction study requires an adequate comparison group but few papers describe with transparency how this has been achieved. There is also the assumption that participation in treatment renders an individual offender as treated. A treated profile is contingent upon the treatment climate, the quality of programme delivery and how the individual responds to treatment. The influence of life‐style or dynamic risk factors must also be acknowledged in terms of their effect on reconviction.Conclusions. It is recommended that researchers use an integrated approach to the evaluation of accredited offending behaviour programmes. In order to establish a broad balance of treatment outcomes, a shift away from reconviction rates is needed. This paper presents an integrated model for the evaluation of offending behaviour programmes.