Department Population Health Sciences, Institute for Risk Assessment Sciences, Utrecht University, Utrecht, The Netherlands.
Occupational and Environmental Epidemiology Branch, Division of Cancer Epidemiology and Genetics, National Cancer Institute, Bethesda, MD, USA.
Ann Work Expo Health. 2023 Jun 6;67(5):663-672. doi: 10.1093/annweh/wxad002.
Automatic job coding tools were developed to reduce the laborious task of manually assigning job codes based on free-text job descriptions in census and survey data sources, including large occupational health studies. The objective of this study is to provide a case study of comparative performance of job coding and JEM (Job-Exposure Matrix)-assigned exposures agreement using existing coding tools.
We compared three automatic job coding tools [AUTONOC, CASCOT (Computer-Assisted Structured Coding Tool), and LabourR], which were selected based on availability, coding of English free-text into coding systems closely related to the 1988 version of the International Standard Classification of Occupations (ISCO-88), and capability to perform batch coding. We used manually coded job histories from the AsiaLymph case-control study that were translated into English prior to auto-coding to assess their performance. We applied two general population JEMs to assess agreement at exposure level. Percent agreement and PABAK (Prevalence-Adjusted Bias-Adjusted Kappa) were used to compare the agreement of results from manual coders and automatic coding tools.
The coding per cent agreement among the three tools ranged from 17.7 to 26.0% for exact matches at the most detailed 4-digit ISCO-88 level. The agreement was better at a more general level of job coding (e.g. 43.8-58.1% in 1-digit ISCO-88), and in exposure assignments (median values of PABAK coefficient ranging 0.69-0.78 across 12 JEM-assigned exposures). Based on our testing data, CASCOT was found to outperform others in terms of better agreement in both job coding (26% 4-digit agreement) and exposure assignment (median kappa 0.61).
In this study, we observed that agreement on job coding was generally low for the three tools but noted a higher degree of agreement in assigned exposures. The results indicate the need for study-specific evaluations prior to their automatic use in general population studies, as well as improvements in the evaluated automatic coding tools.
为了减少根据人口普查和调查数据来源（包括大型职业健康研究）中的自由文本工作描述手动分配工作代码的繁琐任务，开发了自动工作代码工具。本研究的目的是提供一个案例研究，比较使用现有的编码工具对编码和 JEM（职业暴露矩阵）分配的暴露的一致性。
我们比较了三种自动工作编码工具[AUTONOC、CASCOT（计算机辅助结构化编码工具）和 LabourR]，选择这些工具是基于可用性、将英语自由文本编码为与 1988 年版国际职业分类（ISCO-88）密切相关的编码系统，以及批量编码的能力。我们使用从亚洲淋巴病例对照研究中手动编码的工作历史记录，在自动编码之前将其翻译成英文，以评估其性能。我们应用了两个一般人群 JEM 来评估暴露水平的一致性。百分一致率和 PABAK（调整偏倚的调整后的 Kappa）用于比较手动编码者和自动编码工具的结果一致性。
三种工具的编码百分一致率在最详细的 4 位 ISCO-88 级别上的精确匹配为 17.7%至 26.0%。在更一般的工作编码级别（例如，在 1 位 ISCO-88 中为 43.8%-58.1%）和暴露分配（12 个 JEM 分配的暴露中 PABAK 系数的中位数范围为 0.69-0.78）方面，一致性更好。根据我们的测试数据，CASCOT 在工作编码（26%的 4 位一致率）和暴露分配（中位数kappa 为 0.61）方面的一致性更好。
在这项研究中，我们观察到三种工具的工作编码一致性通常较低，但注意到分配的暴露一致性较高。结果表明，在将自动工具用于一般人群研究之前，需要针对具体研究进行评估，并改进评估的自动编码工具。
