Department of Medical Microbiology and Infection Prevention, University Medical Centre Utrecht, P.O. Box 85500, 3508 GA, Utrecht, The Netherlands.
Julius Centre for Health Sciences and Primary Care, University Medical Centre Utrecht, Utrecht, The Netherlands.
Antimicrob Resist Infect Control. 2022 Jan 21;11(1):10. doi: 10.1186/s13756-022-01050-w.
Surveillance is the cornerstone of surgical site infection prevention programs. The validity of the data collection and awareness of vulnerability to inter-rater variation is crucial for correct interpretation and use of surveillance data. The aim of this study was to investigate the reliability and validity of surgical site infection (SSI) surveillance after colorectal surgery in the Netherlands.
In this multicentre prospective observational study, seven Dutch hospitals performed SSI surveillance after colorectal surgeries performed in 2018 and/or 2019. When executing the surveillance, a local case assessment was performed to calculate the overall percentage agreement between raters within hospitals. Additionally, two case-vignette assessments were performed to estimate intra-rater and inter-rater reliability by calculating a weighted Cohen's Kappa and Fleiss' Kappa coefficient. To estimate the validity, answers of the two case-vignettes questionnaires were compared with the answers of an external medical panel.
1111 colorectal surgeries were included in this study with an overall SSI incidence of 8.8% (n = 98). From the local case assessment it was estimated that the overall percent agreement between raters within a hospital was good (mean 95%, range 90-100%). The Cohen's Kappa estimated for the intra-rater reliability of case-vignette review varied from 0.73 to 1.00, indicating substantial to perfect agreement. The inter-rater reliability within hospitals showed more variation, with Kappa estimates ranging between 0.61 and 0.94. In total, 87.9% of the answers given by the raters were in accordance with the medical panel.
This study showed that raters were consistent in their SSI-ascertainment (good reliability), but improvements can be made regarding the accuracy (moderate validity). Accuracy of surveillance may be improved by providing regular training, adapting definitions to reduce subjectivity, and by supporting surveillance through automation.
监测是手术部位感染预防计划的基石。数据收集的有效性和对评分者间变异性的易感性的认识，对于正确解释和使用监测数据至关重要。本研究旨在调查荷兰结直肠手术后手术部位感染（SSI）监测的可靠性和有效性。
在这项多中心前瞻性观察研究中，7 家荷兰医院对 2018 年和/或 2019 年进行的结直肠手术后进行了 SSI 监测。在执行监测时，进行了局部病例评估，以计算医院内评分者之间的总体百分比一致性。此外，还进行了两次病例模拟评估，通过计算加权 Cohen's Kappa 和 Fleiss' Kappa 系数来估计评分者内和评分者间的可靠性。为了估计有效性，两个病例模拟评估问卷的答案与外部医学小组的答案进行了比较。
本研究共纳入 1111 例结直肠手术，SSI 总发生率为 8.8%（n=98）。从局部病例评估中估计，医院内评分者之间的总体百分比一致性良好（平均 95%，范围 90-100%）。病例模拟审查的内评分者可靠性的 Cohen's Kappa 估计值从 0.73 到 1.00，表明存在实质性到完美的一致性。医院内的评分者间可靠性差异较大，Kappa 估计值在 0.61 到 0.94 之间。总的来说，87.9%的评分者的答案与医学小组一致。
本研究表明，评分者在 SSI 确定方面具有一致性（可靠性良好），但在准确性方面（有效性中等）可以进行改进。通过定期培训、调整定义以减少主观性以及通过自动化支持监测，可以提高监测的准确性。
