The University of Queensland, School of Population Health, Queensland, Australia.
Int J Epidemiol. 2011 Oct;40(5):1308-13. doi: 10.1093/ije/dyr109. Epub 2011 Jul 6.
Currently, we are not aware of a method to assess graphically on one simple plot agreement between more than two observers making continuous measurements on the same subjects.
We aimed to develop a simple graphical method to assess agreement between multiple observers using continuous measurements. The Bland-Altman graphical method for assessing agreement between two observers using continuous measures was modified and extended to accommodate multiple observers. Mathematical formulae are derived and real data examples used to illustrate the proposed method.
The examples show that the proposed graphical method of assessing agreement provides clinically useful information. This information includes estimates of the limits of agreement with the mean and a visual means for determining these limits over the range of measurements. In a data example that included five readers' measurements of 40 lung lesions, the intra-class correlation (ICC) was 0.84 indicating readers can reliably measure the lesions. However, the estimated limits of agreement with the mean were -1.1 to 1.1 cm implying that the readers' measurements can plausibly differ from the mean estimated tumour size by more than 1 cm. This is a clinically significant difference according to the study authors. In addition, a plot of the limits of agreement with the mean by mean tumour size shows heterogeneous agreement presumably due to the varying degrees of definition at the edge of the lesions.
The proposed graphical method of assessing agreement can be used alongside other measures such as ICC for reporting on reproducibility in studies of multiple observers making continuous measurements.
目前，我们尚不清楚有一种方法可以在一个简单的图表上评估两个以上观察者对同一组连续测量结果的图形一致性。
我们旨在开发一种简单的图形方法，以评估多个观察者使用连续测量结果的一致性。对用于评估两个观察者使用连续测量一致性的 Bland-Altman 图形方法进行了修改和扩展，以适应多个观察者。推导出了数学公式，并使用真实数据示例来说明所提出的方法。
示例表明，所提出的评估一致性的图形方法提供了有用的临床信息。这些信息包括对均值的一致性的界限的估计值，以及在测量范围内确定这些界限的直观方法。在包括五个读者对 40 个肺病变的测量数据的示例中，组内相关系数（ICC）为 0.84，表明读者可以可靠地测量病变。然而，与均值的估计一致性界限为-1.1 至 1.1cm，这意味着读者的测量结果与估计的平均肿瘤大小可能相差超过 1cm。根据研究作者的说法，这是一个具有临床意义的差异。此外，通过平均肿瘤大小绘制一致性界限的图表明存在异质性一致性，这可能是由于病变边缘的定义程度不同。
所提出的评估一致性的图形方法可以与 ICC 等其他指标一起用于报告多个观察者进行连续测量的重复性研究。
