Measuring the Effectiveness of the Urban Youth Connection: The Case for Dose-Response Modeling to Demonstrate the Impact of an Adolescent Substance Abuse Prevention Program

Abstract

Introduction. This paper presents methods and findings from an outcome evaluation of an adolescent drug prevention program serving predominantly Hispanic and African American students in an inner-city public middle school and secondary school in Boston. The program, entitled the Urban Youth Connection, consists of individual, pair and group counseling to high risk students identified by school personnel. The study design and statistical analysis techniques are presented as a model approach to estimating the impact of adolescent drug prevention programs to maximize scientific rigor without compromising the service delivery mission of the program. The objectives of the intervention are to prevent and reduce 30-day use of alcohol, tobacco and other drugs, and to reduce risk and improve resiliency factors associated with healthy behaviors: self-esteem, social coping, depression, interpersonal violence, school involvement and aspirations, school attendance, and academic performance. Methods. A pre-test, post-test comparison group design was used to identify changes over time in behavioral, psychosocial, and academic outcomes. Stratified analysis was used to adjust for baseline differences in outcomes of interest between the treatment and comparison groups. A dose-response relationship was measured between amount of services received and self-reported behavioral outcomes, academic performance, and school attendance, by means of correlational and regression analyses. Self-reported behavioral outcomes were measured through surveys administered in both participating schools in the fall and spring of 1993-1994. Academic performance and school attendance was abstracted from student report cards at the end of the 1993-1994 school year. Service records, self-administered questionnaires, and grade and attendance reports were linked in an integrated computerized management information system. Results. Benefits of increased program exposure were indicated for intermediate outcomes of depression, social coping, school performance, especially for high risk students in the secondary school. Reduction of tobacco, beer and marijuana use among treatment students in the secondary school is suggested from this study. The dose-response model which associated amount of program exposure with self-reported outcomes and academic performance and attendance indicated that the intervention had the greatest effect on academic indicators among high school students who were identified as high risk at baseline. Discussion. The Urban Youth Connection shows promise of having an impact on intermediate outcomes of interest for high risk youth in middle school and high school settings. Behavioral indicators related to substance use or abuse are not impacted by the intervention as measured by a smaller subset of paired pre/post self-reported surveys among treatment and comparison group students. Early findings from the pre-test/post-test comparison are limited by small sample sizes. Trends suggest that program success may be demonstrated with increasing sample size as the program continues. Cohort studies in an action setting can lead to insignificant findings due to lower statistical power as a result of small sample sizes. Dose-response modeling can be a useful technique to make causal inferences to estimate program impact and can have more statistical power since this analytic approach can yield larger sample sizes useful for analysis. Dose-response modeling requires linkage of service information to client outcomes. Either approach requires stratification to adjust for variability within both the treatment group and the comparison group that result from application of quasi-experimental study designs for evaluating outcomes of adolescent drug prevention programs.