VNU-HCM, 106101An Giang University, Vietnam.
Mathematical Sciences, School of Science, 5376RMIT University, Australia.
Stat Methods Med Res. 2021 Oct;30(10):2329-2351. doi: 10.1177/09622802211037068. Epub 2021 Aug 27.
Inter-rater agreement measures are used to estimate the degree of agreement between two or more assessors. When the agreement table is ordinal, different weight functions that incorporate row and column scores are used along with the agreement measures. The selection of row and column scores is effectual on the estimated degree of agreement. The weighted measures are prone to the anomalies frequently seen in agreement tables such as unbalanced table structures or grey zones due to the assessment behaviour of the raters. In this study, Bayesian approaches for the estimation of inter-rater agreement measures are proposed. The Bayesian approaches make it possible to include prior information on the assessment behaviour of the raters in the analysis and impose order restrictions on the row and column scores. In this way, we improve the accuracy of the agreement measures and mitigate the impact of the anomalies in the estimation of the strength of agreement between the raters. The elicitation of prior distributions is described theoretically and practically for the Bayesian estimation of five agreement measures with three different weights using an agreement table having two grey zones. A Monte Carlo simulation study is conducted to assess the classification accuracy of the Bayesian and classical approaches for the considered agreement measures for a given level of agreement. Recommendations for the selection of the highest performing agreement measure and weight combination are made in the breakdown of the table structure and sample size.
组内一致性评估用于评估两个或多个评估者之间的一致性程度。当一致性表格为有序时，需要结合行和列分数使用不同的权重函数以及一致性评估方法。行和列分数的选择会影响估计的一致性程度。加权评估方法容易受到评估者评估行为引起的异常情况的影响，例如不平衡表格结构或灰色区域。在这项研究中，提出了用于估计组内一致性评估方法的贝叶斯方法。贝叶斯方法可以在分析中包含评估者评估行为的先验信息，并对行和列分数施加顺序限制。通过这种方式，我们可以提高一致性评估方法的准确性，并减轻评估者之间一致性强度估计中异常情况的影响。理论和实践上都描述了先验分布的启发式方法，用于使用具有两个灰色区域的一致性表格对五种具有三种不同权重的一致性评估方法进行贝叶斯估计。进行了蒙特卡罗模拟研究，以评估在给定一致性水平下，考虑的一致性评估方法的贝叶斯和经典方法的分类准确性。根据表格结构和样本大小的细分，提出了选择表现最佳的一致性评估方法和权重组合的建议。
