Training Novices To Evaluate Physical Activity Promotion Material Quality: Results Of A Pilot Study

Abstract

The credibility and usability of lay physical activity promotion material are a persistent problem (Thomas & Cardinal, 2020, TJACM). These quality issues deter the material in promoting health literacy, a major predictor of prevention-oriented health behaviors (e.g., regular exercise, U.S. National Action Plan to Improve Health Literacy). Few studies, however, have evaluated the quality of lay material over time (Thomas et al., 2018, Quest). PURPOSE: In order to conduct a repeated-measures study of the quality of physical activity promotion web articles for lay adults, one experienced researcher (JDT) trained undergraduates on how to use the Suitability Assessment of Materials (SAM) protocol (Doak et al., 1996). METHODS: An adapted version of the SAM protocol was used (Thomas & Cardinal, 2020, Quest). Three undergraduates were part of the pilot project (ENT, SAL, CNC; Feb.-Aug. 2020). Each was introduced to the protocol and practiced its methods using six web articles that were written in English and not used in the longitudinal study. Coding by ENT was compared to JDT for analytic purposes; ENT was pre-assigned to evaluate material for the longitudinal study. Per Krippendorff’s alpha and intraclass coefficient measures, ENT showed acceptable between/within coder agreement during the training phase. Jul.-Aug. 2020, ENT then tested his skills further using a random subset of 16 unique web articles that were part of the longitudinal study. RESULTS: ENT between-coder agreement ranged from Good/Substantial (both .68) to Excellent/Almost Perfect (both .86) across the SAM’s main categories. Overall agreement was Excellent/Substantial (range: .76-.77). Within-coder agreement was Excellent/Almost Perfect across all main categories (both ≥ .85). CONCLUSION: This pilot study demonstrates that a novice coder can learn to code material with a high degree of fidelity. This contrasts with speculation that the SAM protocol may be too subjective for good coder agreement. Debriefing revealed key insights: (a) four “rules” that helped the novice coder achieve fidelity (e.g., review notes often) and (b) developer “bad habits” that limit the educational quality of material (e.g., jargon-filled advice). We will discuss our results and practical lessons based on post-hoc text profiles of material sampled in our pilot study.