Department of Family Medicine, University of Ottawa, Ottawa, Ontario, Canada.
Ann Fam Med. 2011 Jan-Feb;9(1):57-62. doi: 10.1370/afm.1195.
We assessed interrater reliability (IRR) of chart abstractors within a randomized trial of cardiovascular care in primary care. We report our findings, and outline issues and provide recommendations related to determining sample size, frequency of verification, and minimum thresholds for 2 measures of IRR: the κ statistic and percent agreement.
We designed a data quality monitoring procedure having 4 parts: use of standardized protocols and forms, extensive training, continuous monitoring of IRR, and a quality improvement feedback mechanism. Four abstractors checked a 5% sample of charts at 3 time points for a predefined set of indicators of the quality of care. We set our quality threshold for IRR at a κ of 0.75, a percent agreement of 95%, or both.
Abstractors reabstracted a sample of charts in 16 of 27 primary care practices, checking a total of 132 charts with 38 indicators per chart. The overall κ across all items was 0.91 (95% confidence interval, 0.90-0.92) and the overall percent agreement was 94.3%, signifying excellent agreement between abstractors. We gave feedback to the abstractors to highlight items that had a κ of less than 0.70 or a percent agreement less than 95%. No practice had to have its charts abstracted again because of poor quality.
A 5% sampling of charts for quality control using IRR analysis yielded κ and agreement levels that met or exceeded our quality thresholds. Using 3 time points during the chart audit phase allows for early quality control as well as ongoing quality monitoring. Our results can be used as a guide and benchmark for other medical chart review studies in primary care.
我们评估了心血管初级保健随机试验中图表摘要者的组内可靠性（IRR）。我们报告了我们的发现，并概述了与确定样本量、验证频率以及两种 IRR 度量的最小阈值相关的问题和建议：κ统计量和百分比一致性。
我们设计了一个数据质量监测程序，包括 4 个部分：使用标准化的协议和表格、广泛的培训、持续监测 IRR 以及质量改进反馈机制。4 名摘要者在 3 个时间点检查了 5%的图表样本，以检查护理质量的一组预定指标。我们将 IRR 的质量阈值设定为 κ 值为 0.75、百分比一致性为 95%或两者兼而有之。
摘要者在 27 个初级保健实践中的 16 个重新检查了图表样本，总共检查了 132 份图表，每份图表有 38 个指标。所有项目的总体κ值为 0.91（95%置信区间，0.90-0.92），总体百分比一致性为 94.3%，表明摘要者之间具有极好的一致性。我们向摘要者提供反馈，以突出那些κ值低于 0.70 或百分比一致性低于 95%的项目。没有一个实践因为质量差而必须重新检查图表。
使用 IRR 分析对质量控制进行 5%的图表抽样，产生了符合或超过我们质量阈值的κ和一致性水平。在图表审核阶段使用 3 个时间点可以实现早期质量控制以及持续的质量监测。我们的结果可以作为其他初级保健医疗图表审查研究的指南和基准。
