Issen Laurel, Woodcock Thomas, McNicholas Christopher, Lennox Laura, Reed Julie E
Imperial College London, National Institute for Health Research Collaboration for Leadership in Applied Health Research and Care Northwest London, Chelsea and Westminster Hospital, 4th Floor, Lift Bank D, 369 Fulham Road, London, UK.
Int J Qual Health Care. 2018 Aug 1;30(7):508-513. doi: 10.1093/intqhc/mzy063.
Despite criticisms that many quality improvement (QI) initiatives fail due to incomplete programme theory, there is no defined way to evaluate how programme theory has been articulated. The objective of this research was to develop, and assess the usability and reliability of scoring criteria to evaluate programme theory diagrams.
Criteria development was informed by published literature and QI experts. Inter-rater reliability was tested between two evaluators. About 63 programme theory diagrams (42 driver diagrams and 21 action-effect diagrams) were reviewed to establish whether the criteria could support comparative analysis of different approaches to constructing diagrams.
Components of the scoring criteria include: assessment of overall aim, logical overview, clarity of components, cause-effect relationships, evidence and measurement. Independent reviewers had 78% inter-rater reliability. Scoring enabled direct comparison of different approaches to developing programme theory; action-effect diagrams were found to have had a statistically significant but moderate improvement in programme theory quality over driver diagrams; no significant differences were observed based on the setting in which driver diagrams were developed.
The scoring criteria summarise the necessary components of programme theory that are thought to contribute to successful QI projects. The viability of the scoring criteria for practical application was demonstrated. Future uses include assessment of individual programme theory diagrams and comparison of different approaches (e.g. methodological, teaching or other QI support) to produce programme theory. The criteria can be used as a tool to guide the production of better programme theory diagrams, and also highlights where additional support for QI teams could be needed.
尽管有人批评许多质量改进（QI）举措因项目理论不完善而失败，但目前尚无确定的方法来评估项目理论的阐述方式。本研究的目的是开发用于评估项目理论图的评分标准，并评估其可用性和可靠性。
根据已发表的文献和QI专家的意见制定标准。在两名评估者之间测试了评分者间信度。审查了约63个项目理论图（42个驱动因素图和21个行动-效果图），以确定这些标准是否能够支持对构建图的不同方法进行比较分析。
评分标准的组成部分包括：总体目标评估、逻辑概述、组成部分的清晰度、因果关系、证据和测量。独立评审员的评分者间信度为78%。评分使得能够直接比较开发项目理论的不同方法；发现行动-效果图在项目理论质量上比驱动因素图有统计学上显著但适度的提高；根据驱动因素图的开发背景未观察到显著差异。
评分标准总结了项目理论中被认为有助于成功实施QI项目的必要组成部分。证明了评分标准在实际应用中的可行性。未来的用途包括评估单个项目理论图以及比较生成项目理论的不同方法（例如方法学、教学或其他QI支持）。这些标准可以用作指导生成更好的项目理论图的工具，还可以突出显示QI团队可能需要额外支持的地方。
