Development of the Patient Education Materials Assessment Tool (PEMAT): a new measure of understandability and actionability for print and audiovisual patient information.

OBJECTIVE

To develop a reliable and valid instrument to assess the understandability and actionability of print and audiovisual materials.

METHODS

We compiled items from existing instruments/guides that the expert panel assessed for face/content validity. We completed four rounds of reliability testing, and produced evidence of construct validity with consumers and readability assessments.

RESULTS

The experts deemed the PEMAT items face/content valid. Four rounds of reliability testing and refinement were conducted using raters untrained on the PEMAT. Agreement improved across rounds. The final PEMAT showed moderate agreement per Kappa (Average K=0.57) and strong agreement per Gwet's AC1 (Average=0.74). Internal consistency was strong (α=0.71; Average Item-Total Correlation=0.62). For construct validation with consumers (n=47), we found significant differences between actionable and poorly-actionable materials in comprehension scores (76% vs. 63%, p<0.05) and ratings (8.9 vs. 7.7, p<0.05). For understandability, there was a significant difference for only one of two topics on consumer numeric scores. For actionability, there were significant positive correlations between PEMAT scores and consumer-testing results, but no relationship for understandability. There were, however, strong, negative correlations between grade-level and both consumer-testing results and PEMAT scores.

CONCLUSIONS

The PEMAT demonstrated strong internal consistency, reliability, and evidence of construct validity.

PRACTICE IMPLICATIONS

The PEMAT can help professionals judge the quality of materials (available at: http://www.ahrq.gov/pemat).