Gu Huanying Helen, Hripcsak George, Chen Yan, Morrey C Paul, Elhanan Gai, Cimino James, Geller James, Perl Yehoshua
SHRP, University of Medicine and Dentistry of New Jersey.
AMIA Annu Symp Proc. 2007 Oct 11;2007:294-8.
The UMLS is a terminological system that integrates many source terminologies. Each concept in the UMLS is assigned one or more semantic types from the Semantic Network, an upper level ontology for biomedicine. Due to the complexity of the UMLS, errors exist in the semantic type assignments. Finding assignment errors may unearth modeling errors. Even with sophisticated tools, discovering assignment errors requires manual review. In this paper we describe the evaluation of an auditing project of UMLS semantic type assignments. We studied the performance of the auditors who reviewed potential errors. We found that four auditors, interacting according to a multi-step protocol, identified a high rate of errors (one or more errors in 81% of concepts studied) and that results were sufficiently reliable (0.67 to 0.70) for the two most common types of errors. However, reliability was low for each individual auditor, suggesting that review of potential errors is resource-intensive.
统一医学语言系统（UMLS）是一个整合了许多源术语的术语系统。UMLS中的每个概念都被赋予语义网络中的一种或多种语义类型，语义网络是生物医学的一个上层本体。由于UMLS的复杂性，语义类型分配中存在错误。发现分配错误可能会揭示建模错误。即使有复杂的工具，发现分配错误也需要人工审核。在本文中，我们描述了对UMLS语义类型分配审核项目的评估。我们研究了审核潜在错误的审核员的表现。我们发现，四名审核员按照多步骤协议进行交互，识别出了很高比例的错误（在所研究的概念中有81%存在一个或多个错误），并且对于两种最常见的错误类型，结果具有足够的可靠性（0.67至0.70）。然而，每个审核员的可靠性都很低，这表明对潜在错误的审核需要大量资源。
