Interrater reliability of the Patient Education Materials Assessment Tool (PEMAT)

Abstract

ObjectiveTo assess the interrater reliability (IRR) and usability of the Patient Education Materials Assessment Tool (PEMAT) and the relationship between PEMAT scores and readability levels. MethodsOne hundred ten materials (80 print, 30 audiovisual) were evaluated, each by two raters, using the PEMAT. IRR was calculated using Gwet’s AC1 and summarized across items in each PEMAT domain (understandability and actionability) and by material type. A survey was conducted to solicit raters’ experience using the PEMAT. Readability of each material was assessed using the SMOG Index. ResultsThe median IRR was 0.92 for understandability and 0.93 for actionability across all relevant items, indicating good IRR. Eight PEMAT items had Gwet’s AC1 values less than 0.81. PEMAT and SMOG Index scores were inversely correlated, with a Spearman’s rho of −0.20 (p=0.081) for understandability and −0.15 (p=0.194) for actionability. While 92% of raters agreed the PEMAT was easy to use, survey results suggested specific items for clarification. ConclusionWhile the PEMAT demonstrates moderate to excellent IRR overall, amendments to items with lower IRR may increase the usefulness of the tool. Practice implicationsThe PEMAT is a useful supplement to reading level alone in the assessment of educational materials.