Lagged Versus Difference Score Regression: An Example From a Community-Based Educational Seminar Evaluation

Abstract

Listen

Translate article

Objectives. To demonstrate the use of evaluation data in determining the effectiveness of a series of community-based educational seminars designed to educate substance abuse treatment counselors on the latest science regarding the effects of alcohol abuse on diseases of the liver and pancreas; in the process, to highlight the use of difference scores versus lagged regression as alternative approaches to conducting such evaluations. Methods. Data were collected from 338 seminar participants during 4 half-day seminars in 2007, 2009, 2010, and 2014. Pre- and post-seminar data were collected immediately before and after the seminar, approximately 3 hours apart. Survey questions included demographics and a series of multiple-choice knowledge questions that were identical at pre- and posttest. Results. Analyses showed that, on average, participants improved their knowledge by 19.7 percentage points from 0.44 to 0.63, or 44.7%. Factors associated with learning using difference scores revealed that only survey wave (time) was associated with learning. In contrast, lagged regression results showed that in addition to wave, participants who reported attending the seminar because they were required to for their job learned less, whereas those attending because they wanted to learn the material learned more. Education was also significantly associated with knowledge change. Conclusions. This study illustrates that relying on difference score analyses to estimate programmatic effects and changes in knowledge for educational programs can lead to faulty conclusions whereas lagged regression is a better approach to assess program impact.