Swansea University Medical School, Swansea University, Swansea, Wales, UK.
Neurology Department, Swansea Bay University Health Board, Swansea, Wales, UK.
J Biomed Semantics. 2024 Sep 15;15(1):17. doi: 10.1186/s13326-024-00316-z.
Natural language processing (NLP) is increasingly being used to extract structured information from unstructured text to assist clinical decision-making and aid healthcare research. The availability of expert-annotated documents for the development and validation of NLP applications is limited. We created synthetic clinical documents to address this, and to validate the Extraction of Epilepsy Clinical Text version 2 (ExECTv2) NLP pipeline.
We created 200 synthetic clinic letters based on hospital outpatient consultations with epilepsy specialists. The letters were double annotated by trained clinicians and researchers according to agreed guidelines. We used the annotation tool, Markup, with an epilepsy concept list based on the Unified Medical Language System ontology. All annotations were reviewed, and a gold standard set of annotations was agreed and used to validate the performance of ExECTv2.
The overall inter-annotator agreement (IAA) between the two sets of annotations produced a per item F1 score of 0.73. Validating ExECTv2 using the gold standard gave an overall F1 score of 0.87 per item, and 0.90 per letter.
The synthetic letters, annotations, and annotation guidelines have been made freely available. To our knowledge, this is the first publicly available set of annotated epilepsy clinic letters and guidelines that can be used for NLP researchers with minimum epilepsy knowledge. The IAA results show that clinical text annotation tasks are difficult and require a gold standard to be arranged by researcher consensus. The results for ExECTv2, our automated epilepsy NLP pipeline, extracted detailed epilepsy information from unstructured epilepsy letters with more accuracy than human annotators, further confirming the utility of NLP for clinical and research applications.
自然语言处理（NLP）越来越多地被用于从非结构化文本中提取结构化信息，以辅助临床决策和帮助医疗保健研究。用于开发和验证 NLP 应用程序的专家注释文档的可用性有限。我们创建了合成临床文档来解决这个问题，并验证了癫痫临床文本提取版本 2（ExECTv2）NLP 管道。
我们根据癫痫专家的医院门诊咨询创建了 200 封合成门诊信函。这些信件由经过培训的临床医生和研究人员根据商定的指南进行双重注释。我们使用基于统一医学语言系统本体的癫痫概念列表的注释工具 Markup。所有注释都进行了审查，并确定了一个黄金标准注释集，用于验证 ExECTv2 的性能。
两组注释产生的总体注释者间一致性（IAA）每个项目的 F1 得分为 0.73。使用黄金标准验证 ExECTv2 的总体 F1 得分为每个项目 0.87，每个信函 0.90。
合成信函、注释和注释指南已免费提供。据我们所知，这是第一个可供具有最少癫痫知识的 NLP 研究人员使用的公共可用的带注释的癫痫诊所信函和指南集。IAA 结果表明，临床文本注释任务具有挑战性，需要通过研究人员共识来安排黄金标准。我们的自动癫痫 NLP 管道 ExECTv2 从非结构化的癫痫信函中提取详细的癫痫信息的准确率高于人工注释者，进一步证实了 NLP 在临床和研究应用中的实用性。
