Hripcsak G, Kuperman G J, Friedman C, Heitjan D F
Columbia University, New York, New York, USA.
J Am Med Inform Assoc. 1999 Mar-Apr;6(2):143-50. doi: 10.1136/jamia.1999.0060143.
To assess the reliability of a reference standard for an information extraction task.
Twenty-four physician raters from two sites and two specialties judged whether clinical conditions were present based on reading chest radiograph reports.
Variance components, generalizability (reliability) coefficients, and the number of expert raters needed to generate a reliable reference standard were estimated.
Per-rater reliability averaged across conditions was 0.80 (95% CI, 0.79-0.81). Reliability for the nine individual conditions varied from 0.67 to 0.97, with central line presence and pneumothorax the most reliable, and pleural effusion (excluding CHF) and pneumonia the least reliable. One to two raters were needed to achieve a reliability of 0.70, and six raters, on average, were required to achieve a reliability of 0.95. This was far more reliable than a previously published per-rater reliability of 0.19 for a more complex task. Differences between sites were attributable to changes to the condition definitions.
In these evaluations, physician raters were able to judge very reliably the presence of clinical conditions based on text reports. Once the reliability of a specific rater is confirmed, it would be possible for that rater to create a reference standard reliable enough to assess aggregate measures on a system. Six raters would be needed to create a reference standard sufficient to assess a system on a case-by-case basis. These results should help evaluators design future information extraction studies for natural language processors and other knowledge-based systems.
评估信息提取任务中参考标准的可靠性。
来自两个地点和两个专业的24名医生评分者根据阅读胸部X光片报告判断是否存在临床病症。
估计方差成分、泛化（可靠性）系数以及生成可靠参考标准所需的专家评分者数量。
各条件下评分者的平均可靠性为0.80（95%可信区间，0.79 - 0.81）。九种个体病症的可靠性从0.67到0.97不等，中心静脉置管存在和气胸最为可靠，胸腔积液（不包括心力衰竭）和肺炎最不可靠。需要一到两名评分者才能达到0.70的可靠性，平均需要六名评分者才能达到0.95的可靠性。这比之前发表的一项更复杂任务中评分者的可靠性0.19要可靠得多。不同地点之间的差异归因于病症定义的变化。
在这些评估中，医生评分者能够根据文本报告非常可靠地判断临床病症的存在。一旦确认了特定评分者的可靠性，该评分者就有可能创建一个足够可靠的参考标准，以评估系统的总体指标。需要六名评分者来创建一个足以逐案评估系统的参考标准。这些结果应有助于评估者为自然语言处理器和其他基于知识的系统设计未来的信息提取研究。
