Outcome Evaluation Findings Research Articles

Pilot evaluation of the Fiscal Mapping Process for sustainable financing of evidence-based youth mental health treatments: A comparative case study analysis.

Sustained delivery of evidence-based treatments (EBTs) is essential to addressing the public health impacts of youth mental health problems, but is complicated by the limited and fragmented funding available to youth mental health service agencies. Supports are needed that can guide service agencies in accessing sustainable funding for EBTs. We conducted a pilot evaluation of the Fiscal Mapping Process, an Excel-based strategic planning tool that helps service agency leaders identify and coordinate financing strategies for their EBT programs. Pilot testing of the Fiscal Mapping Process was completed with 10 youth mental health service agencies over a 12-month period, using trauma-focused cognitive-behavioral therapy or parent-child interaction therapy programs. Service agency representatives received initial training and monthly coaching in using the tool. We used case study methods to synthesize all available data (surveys, focus groups, coaching notes, document review) and contrast agency experiences to identify key findings through explanation building. Key evaluation findings related to the process and outcomes of using the Fiscal Mapping Process, as well as contextual influences. Process evaluation findings helped clarify the primary use case for the tool and identified the importance-and challenges-of engaging external collaborators. Outcome evaluation findings documented the impacts of the Fiscal Mapping Process on agency-reported sustainment capacities (strategic planning, funding stability), which fully explained reported improvements in outcomes (extent and likelihood)-although these impacts were incremental. Findings on contextual factors documented the influence of environmental and organizational capacities on engagement with the tool and concerns about equitable impacts, but also the view that the process could usefully generalize to other EBTs. Our pilot evaluation of the Fiscal Mapping Process was promising. In future work, we plan to integrate the tool into EBT implementation initiatives and test its impact on long-term sustainment outcomes across various EBTs, while increasing attention to equity considerations.

O26 “These Texts Really Changed My Life”: Outcome Evaluation Findings of a Text Message Intervention for Low-Income Adults

O26 “These Texts Really Changed My Life”: Outcome Evaluation Findings of a Text Message Intervention for Low-Income Adults

Evaluation of Service-Learning in Project WeCan under COVID-19 in a Chinese Context.

Service-learning is a widely adopted educational pedagogy and philosophy. With the support from the Wharf (Holdings) Limited (Group), service-learning was conducted in the “Project WeCan” in Hong Kong. Prior to COVID-19, traditional service-learning was implemented with students learning in the classroom and applying their knowledge and skills to the community through providing direct face-to-face service. With the COVID-19 outbreak in the 2019–20 academic year, school lockdown measures appeared. Students had to learn online and to design and implement service offsite. As the impacts of this rapid shift in paradigm remain unknown, this study examined changes in university students using a pretest–posttest design (n = 124) and perceptions of service-learning experience via the subjective outcome evaluation design (n = 192) under COVID-19. The authors also investigated service recipients’ (n = 56) satisfaction with service activities they participated in during the pandemic. Both objective outcome evaluation and subjective outcome evaluation findings revealed that service providers (university students) and recipients (secondary school students) experienced benefits from the Project. Findings support the benefits of online service-learning in “Project WeCan” even during unprecedented times such as COVID-19.

Condom Availability Programs in Schools: A Review of the Literature.

This review synthesizes findings from the peer-reviewed evaluation literature on condom availability programs (CAPs) in secondary schools. Peer-reviewed evaluation literature indexed in MEDLINE, EMBASE, PsychINFO, ERIC, CINAHL, Sociological Abstracts, SCOPUS, and POPLINE. Manuscripts had to be, written in English, and report evaluation data from a US school-based CAP. Articles were coded independently by 2 authors. Discrepancies were resolved through open discussion. We grouped findings into outcome evaluation and process evaluation findings. Outcome evaluation findings included sexually transmitted infections (STIs), pregnancy rates, condom use, contraception use, sexual risk, and substance use. Process evaluation findings included awareness of CAPs, attitudes toward CAPs, attitudes toward condoms, and receipt of education and instruction. Of the 138 citations reviewed, 12 articles published between 1995 and 2012 met the inclusion criteria, representing 8 programs. Evaluations indicate CAPs yield condom acquisition rates between 23% and 48%, have mixed results related to condom use, and are not associated with increases in sexual and other risk behaviors. One program found CAPs were associated with a decrease in a combined rate of chlamydia and gonorrhea. One program found no association between CAPs and unintended pregnancy. Students' attitudes toward CAPs were favorable and awareness was high. Condom availability programs are accepted by students and can be an appropriate and relevant school-based intervention for teens. Condom availability programs can increase condom use, but more evaluations are needed on CAP impact on rates of HIV, STIs, and unintended pregnancy.

Evaluation of the Project P.A.T.H.S. in Mainland China: Findings Based on Student Diaries

Objectives: Based on 859 student diaries, the present study evaluated a positive youth development program entitled “Tin Ka Ping (TKP) Positive Adolescent Training through Holistic Social programs (P.A.T.H.S.) ” project implemented in mainland China during the 2015–2016 academic year. Method: To understand the perceived effectiveness of the students, the study analyzed quantitative as well as qualitative data derived from the student diaries. Results: The students held very positive views toward the program and the implementers. The narratives of the students also showed that they perceived improvement in the intrapersonal, interpersonal, familial, and societal domains after joining the program. Conclusion: In conjunction with the subjective outcome evaluation findings, the present findings suggest that the “TKP P.A.T.H.S.” project is able to promote holistic development of secondary school students in mainland China.

Profile of Two Second Chance Act Offender Treatment Initiatives: A Research Note

Federal funding efforts have increased the number of reentry programs over the past decade with corresponding evaluations of these initiatives. Reentry programming targets a wide range of offenders though most have focused on medium and high-risk individuals with substance abuse and/or mental health disorders. This research note provides a profile overview of two programs in central Ohio funded by the Second Chance Act and offers a summary of process and outcome evaluation findings from both initiatives. Results from a mixed methods research design suggested that the programs were largely successful in terms of reducing recidivism among participants, though issues related to programmatic fidelity were also identified. While these programs pre-date more current offender intervention efforts, such as the Justice Mental Health Collaboration Program, identified barriers to program success reaffirm the saliency of longstanding problems embedded in and challenged by correctional and police culture.

Perceived effectiveness and satisfaction of a community-based positive youth development program: findings based on high school students

AbstractThis study examined participants’ perceptions of a community-based positive youth development (PYD) program (the Project P.A.T.H.S.) based on the responses of 16,420 junior secondary students who joined the program in 2015. Subjective outcome evaluation approach was adopted to examine the students’ views of program content, program instructors, and program effectiveness. Consistent with previous studies, results showed that students generally perceived the program positively, and positive relationships were found amongst the three domains of evaluation. Multiple regression analyses showed that perceived program content and instructor qualities were significant predictors and could explain 35% of the variance in program effectiveness perceived by the participants. The present findings are basically consistent with previous subjective outcome evaluation findings derived from the school-based and community-based programs of the Project P.A.T.H.S. in Hong Kong.

Preliminary Evaluation of a School-Based Youth Leadership and Prevention Program in Rural Alaska Native Communities.

Youth Leaders Program (YLP) is a health intervention implemented in a rural Alaskan school district, which utilizes natural helpers and peer leaders to increase protective factors such as school engagement and personal/cultural identities, and to reduce risks associated with drug/alcohol abuse, violence, and bullying. Through these means, the program aims to ultimately decrease the disproportionately high rates of indigenous youth suicide in the region. This paper describes process and outcome evaluation findings from the program during the 2013-2014 school year. Data collected include a survey for program participants done at the beginning and end of the study year (n = 61, complete pairs); pre- and post-intervention school data (attendance, GPA, and disciplinary actions) (n = 86); an all-school survey asking students at the participating schools about their experience with YLP and participating youth (n = 764); interviews with program advisors (n = 11) and school principals (n = 2); and focus groups with participating students at all eleven participating schools at the end of the year. Outcomes included increased school attendance (mean attendance increased from 146 to 155 days) and improved academic performance (mean GPA of 8th, 9th, and 10th graders increased from 3.01 to 3.14) of program participants; positive peer reviews of participating student interventions in cases of bullying, depression, and suicidality; and a reported increase in the sense of agency, responsibility, and confidence among participating youth. The YLP appears to improve school climate and increase school and other protective factors for participating students. Recommendations for program implementation in the future and implications for school health will be discussed.

Practice Note: The Irish Drug Treatment Court

Although international countries seek to adopt the American drug court model, each country brings its own culture, philosophy of treatment, and drug use context that affects how each court is implemented. This article reviews the history and context of the Irish Drug Treatment Court. Differences from the American model are considered, such as an emphasis on harm reduction, provision of services predominantly to heroin users, the use of an adult education program as the main intervention, and a varying view of therapeutic jurisprudence. Process and outcome evaluation findings are presented, along with a final discussion related to drug court practice.

Is Subjective Outcome Evaluation Related to Objective Outcome Evaluation? Insights from a Longitudinal Study in Hong Kong

Study ObjectivesEmpirical studies investigating the inter-relationships among subjective outcomes and objective outcomes are sparse in the scientific literature. This study examined the relationship between these 2 forms of outcome evaluation data in Chinese adolescents, utilizing longitudinal data. DesignIn Wave 3 and Wave 4 of a randomized group trial, 2,784 Chinese secondary school students responded to the Chinese Positive Youth Development Scale (CPYDS) assessing objective outcome at pretest (Wave 3) and posttest (Wave 4). Students in the experimental schools also responded to a measure of client satisfaction assessing the subjective outcome of a positive youth development program at posttest (Wave 4). ResultsSubjective outcome evaluation measures (eg, process, program, and effectiveness) at Wave 3 were significantly correlated with CPYDS scores at Wave 4 and difference scores. Subjective outcome evaluation measures also predicted changes in the objective outcome evaluation measures scores across time. ConclusionsThe present findings replicated the previous findings that subjective outcome and objective outcome evaluation findings were intimately related, and subjective outcome measures predicted posttest measures and their changes over time. The present study suggests that subjective outcome evaluation findings may yield useful information regarding program effectiveness in youth work settings.

Factorial Validity of a Subjective Outcome Evaluation Tool for Implementers of a Positive Youth Development Program

Study ObjectivesThis study examined the dimensionality of the subjective outcome evaluation tool assessing the views of program implementers in the context of the Project P.A.T.H.S. in Hong Kong. For illustration purpose, both exploratory and confirmatory factor analyses were performed to examine the factor structure of the instrument. DesignSubjective outcome evaluation findings were collected from 1,170 program implementers who implemented the Grade 7 level program. A validated subjective outcome evaluation scale was used to assess the views of the program implementers. Conceptually, the scale was designed to assess program implementers’ perceptions about program content, implementer qualities, and program effectiveness after completion of the program. ResultsExploratory factor analyses showed that 3 factors were abstracted from the scale and they were stable across 2 random subsamples. Confirmatory factor analyses showed that there was support for a higher-order factor model containing 3 primary factors and 1 second-order factor, and that evidence supporting factorial invariance was found. The 3 subscales were also shown to be reliable with acceptable internal consistency. ConclusionBoth exploratory and confirmatory factor analyses provided support for the factorial validity of the subjective outcome evaluation tool designed for program implementers in the Project P.A.T.H.S. in Hong Kong.

Service leadership education for university students in Hong Kong: qualitative evaluation

AbstractThis study examined the perceptions of students taking a subject entitled “Service Leadership” at The Hong Kong Polytechnic University which was developed using the model proposed by the Hong Kong Institute of Service Leadership and Management (HKI-SLAM). After taking this subject, students were invited to participate in a qualitative study where they used three descriptors and a metaphor to describe their experiences about the subject. Based on the reflections of 50 students, results showed that 96.1% of the descriptors and 90% of the metaphors used by the students were positive. In conjunction with the objective outcome evaluation, subjective outcome evaluation and process evaluation findings, the present study suggests that “Service Leadership” is a subject that can promote holistic development in Chinese university students in Hong Kong.

Post-lecture subjective outcome evaluation of a university subject on leadership and positive youth development in Hong Kong

AbstractPost-lecture subjective outcome evaluation findings of a university subject promoting the leadership qualities and positive youth development of students at The Hong Kong Polytechnic University are reported. Students were invited to respond to a 16-item post-lecture questionnaire gauging their perceptions toward the subject, lecture, and the teachers after each of the 13 lectures. Factor analysis demonstrated that three factors were abstracted from the scale, including subject attributes, lecture attributes, and teacher attributes. Normative findings showed that students had positive perceptions on each of the dimensions throughout the lectures. Multiple regression analyses revealed that subject attributes, lecture attributes, and teacher attributes predicted global evaluation of the lecture and teacher. Based on the subjective outcome evaluation findings, the present study provides further support for the effectiveness of this subject in promoting the holistic development of the students.

The relationship between subjective outcome evaluation and objective outcome evaluation findings: evidence from China

AbstractThe relationships between subjective outcome evaluation and objective subjective evaluation measures were examined in this study. In a positive youth development program (Tin Ka Ping P.A.T.H.S. Project), 1083 high school students responded to objective outcome measures, including the Chinese Positive Youth Development Scale (CPYDS) and measures of thriving and life satisfaction at posttest. The respondents also responded to the Chinese Subjective Outcome Scale (CSOS) at posttest. Results showed that different measures derived from the CSOS (lessons, subject, sharing and benefits) had significant relationships with different measures of the CPYDS, thriving and life satisfaction. The different dimensions of the CSOS scores also predicted different objective outcome measures. The present findings provide additional evidence supporting the linkages between objective outcome and subjective outcome measures.

Helping adolescents with greater psychosocial needs: the extension phase of the project P.A.T.H.S. in Hong Kong

Subjective outcome evaluation data were collected from 24,198 students of 236 schools after completion of the Tier 2 Program in the second year of the Extension Phase of the Project P.A.T.H.S. in Hong Kong. Using schools as units of analysis, results showed that the program, implementers, and effectiveness were perceived in a positive manner. Although subjective outcome evaluation findings differed across participant types, grades, and program types did not show significant differences on the outcome measures. Consistent with previous reports, perceived effectiveness of the program was significantly predicted by student perceptions about the program and program instructor. These findings provide support for the claim that the Tier 2 Program is effective in promoting positive development among adolescents with greater psychosocial needs in the extension phase of the Project P.A.T.H.S. in Hong Kong.

Evaluation of the project P.A.T.H.S. (extension phase) based on the perspective of the program participants

Subjective outcome evaluation data were collected from 236 schools (n=87,943 students) after students had participated in the Tier 1 Program in the second year of the Extension Phase of the Project P.A.T.H.S. in Hong Kong. Using schools as the units of analysis, results showed that the program and implementers were perceived in a positive manner and approximately four-fifths of the participants regarded the program as helpful to them. There were some significant grade differences in the subjective outcome evaluation findings with small effect size. Multiple regression analyses showed that whereas perceived qualities of the program positively predicted perceived effectiveness of the program, perceived qualities of implementers negatively predicted program effectiveness. The present study suggests that irrespective of cohorts at different times, junior secondary school students perceived the program to be beneficial to them. The theoretical and practical implications of the present findings are discussed.

Program implementers’ evaluation of Project P.A.T.H.S. in Hong Kong: a study based on different cohorts

A total of 216 schools participated in the Tier 1 Program (Secondary 1, 2 and 3 levels) of Project P.A.T.H.S. (Positive Adolescent Training through Holistic Social Programmes) in the 2008/09 school year. Based on the subjective outcome evaluation findings collected from students and instructors in each grade, the program implementers wrote down five conclusions in their reports. Utilizing secondary data analysis, the conclusions in the reports were further analyzed. Results showed that most of the conclusions concerning perceptions of the program, instructors and effectiveness of the program were positive in nature. There were also conclusions indicating difficulties encountered and recommendations for improvement. The result of the present study is consistent with previous studies using the same method as well as using other evaluation means for the same grades. The findings suggest that the Tier 1 Program is well received by the stakeholders and the program is effective in promoting the holistic development of adolescents.

Subjective outcome evaluation of the project P.A.T.H.S. In different cohorts of students.

The project P.A.T.H.S. is an indigenously developed positive youth development program in Hong Kong. In the extension phase (2009/2010 school year), subjective outcome evaluation data were collected from 231 schools involving 89,068 participants after completion of the curricula-based Tier 1 Program. With schools as the units of analysis, results showed that participants generally had positive perceptions of the program content and implementers, with over four-fifth of the participants regarded the program as helpful to them. There were some significant grade differences in the subjective outcome evaluation findings, although the related effect size was not strong. Multiple regression analyses revealed that program content and program implementers predicted perceived effectiveness of the program. The present study suggests that irrespective of cohorts, students in the junior secondary years perceived the program to be beneficial to them.

Participants’ Evaluation of the Project P.A.T.H.S.: Are Findings Based on Different Datasets Consistent?

Subjective outcome evaluation findings based on the perspective of the participants of the Project P.A.T.H.S. (Positive Adolescent Training through Holistic Social Programmes) in nine datasets collected from 2005 to 2009 (n = 206, 313 program participants) were examined in this paper. Based on the consolidated data with schools as units, results showed that the participants generally had positive perceptions of the program, implementers, and benefits of the program. More than four-fifths of the participants regarded the program as beneficial to their holistic development. Multiple regression analysis revealed that the perceived qualities of the program and the program implementers predicted perceived effectiveness of the program. Based on the subjective outcome evaluation findings, the present study provides support for the effectiveness of the Tier 1 Program of the Project P.A.T.H.S. in Hong Kong.

Evaluation of the Training Program of the Project P.A.T.H.S.: Findings Based on the Perspective of the Participants from Different Cohorts

Subjective outcome evaluation findings based on the perspective of the participants participating in a 3-day training program of the Project P.A.T.H.S. are reported in this paper. The findings were based on the data collected from the training workshops conducted between 2005 and 2009 (N = 4.167). Results showed that the respondents had good and positive perceptions of the training program and found it very valuable, particularly with respect to training instructors and familiarization with the project. Besides, the training participants were able to acquire attitude, knowledge and skills that are conducive to the successful implementation of the program. Based on the subjective outcome evaluation findings, it is concluded that the training program was effective in helping the participants to acquire the necessary knowledge, attitudes and skills in implementing the program.