School of Public Health, Curtin University, Perth, Australia.
Department of Epidemiology and Preventive Medicine, Monash University, Melbourne, Australia.
Ann Work Expo Health. 2019 Jan 7;63(1):45-53. doi: 10.1093/annweh/wxy084.
To estimate the inter-rater agreement between exposure assessment to asthmagens in current jobs by algorithms based on task-based questionnaires (OccIDEAS) and by experts.
Participants in a cross-sectional national survey of exposure to asthmagens (AWES-Asthma) were randomly split into two subcohorts of equal size. Subcohort 1 was used to determine the most common asthmagen groups and occupational groups. From subcohort 2, a random sample of 200 participants was drawn and current occupational exposure (yes/no) was assessed in these by OccIDEAS and by two experts independently and then as a consensus. Inter-rater agreement was estimated using Cohen's Kappa coefficient. The null hypothesis was set at 0.4, because both the experts and the automatic algorithm assessed the exposure using the same task-based questionnaires and therefore an agreement better than by chance would be expected.
The Kappa coefficients for the agreement between the experts and the algorithm-based assessments ranged from 0.37 to 1, while the agreement between the two experts ranged from 0.29 to 0.94, depending on the agent being assessed. After discussion by both experts the Kappa coefficients for the consensus decision and OccIDEAS were significantly larger than 0.4 for 7 of the 10 asthmagen groups, while overall the inter-rater agreement was greater than by chance (P < 0.0001).
The web-based application OccIDEAS is an appropriate tool for automated assessment of current exposure to asthmagens (yes/no), and requires less time-consuming work by highly-qualified research personnel than the traditional expert-based method. Further, it can learn and reuse expert determinations in future studies.
评估基于任务问卷（OccIDEAS）的算法和专家对当前工作中变应原暴露评估的评分者间一致性。
参与变应原暴露横断面全国调查（AWES-Asthma）的参与者被随机分为两个大小相等的亚组。亚组 1 用于确定最常见的变应原组和职业组。从亚组 2 中抽取 200 名随机参与者，然后由 OccIDEAS 和两位专家独立评估这些参与者当前的职业暴露（是/否），然后达成共识。使用 Cohen's Kappa 系数评估评分者间一致性。零假设设定为 0.4，因为专家和自动算法都使用相同的基于任务的问卷评估暴露情况，因此预计会有比偶然更好的一致性。
专家和基于算法评估之间的一致性 Kappa 系数范围为 0.37 至 1，而两位专家之间的一致性 Kappa 系数范围为 0.29 至 0.94，具体取决于评估的变应原。经过两位专家的讨论，共识决策和 OccIDEAS 的 Kappa 系数对于 10 个变应原组中的 7 个显著大于 0.4，而总体而言，评分者间的一致性大于偶然（P < 0.0001）。
基于网络的应用程序 OccIDEAS 是一种自动评估当前变应原暴露（是/否）的合适工具，与传统的专家方法相比，它需要更少的高素质研究人员的耗时工作。此外，它可以在未来的研究中学习和重复使用专家的决定。
