Department of Computing Science, University of Aberdeen, Aberdeen AB24 3UE, UK.
Artif Intell Med. 2012 Jun;55(2):71-86. doi: 10.1016/j.artmed.2012.03.001. Epub 2012 Apr 4.
The work reported here focuses on developing novel techniques which enable an expert to detect inconsistencies in 2 (or more) perspectives that the expert might have on the same (classification) task. The high level task which the experts (physicians) had set themselves was to classify, on a 5-point severity scale (A-E), the hourly reports produced by an intensive care unit's patient management system.
The INSIGHT system has been developed to support domain experts exploring, and removing inconsistencies in their conceptualization of a task. We report here a study of intensive care physicians reconciling 2 perspectives on their patients. The 2 perspectives provided to INSIGHT were an annotated set of patient records where the expert had selected the appropriate category to describe that snapshot of the patient, and a set of rules which are able to classify the various time points on the same 5-point scale. Inconsistencies between these 2 perspectives are displayed as a confusion matrix; moreover INSIGHT then allows the expert to revise both the annotated datasets (correcting data errors, or changing the assigned categories) and the actual rule-set.
Each of the 3 experts achieved a very high degree of consensus (~97%) between his refined knowledge sources (i.e., annotated hourly patient records and the rule-set). We then had the experts produce a common rule-set and then refine their several sets of annotations against it; this again resulted in inter-expert agreements of ~97%. The resulting rule-set can then be used in applications with considerable confidence.
This study has shown that under some circumstances, it is possible for domain experts to achieve a high degree of correlation between 2 perspectives of the same task. The experts agreed that the immediate feedback provided by INSIGHT was a significant contribution to this successful outcome.
本研究旨在开发新的技术，使专家能够检测到同一（分类）任务中专家可能具有的两个（或更多）视角之间的不一致。专家（医生）设定的高级任务是使用 5 点严重程度量表（A-E）对重症监护病房患者管理系统生成的每小时报告进行分类。
INSIGHT 系统旨在支持领域专家探索和消除其对任务概念化的不一致。我们在此报告了一项关于重症监护医师协调其患者的两个视角的研究。向 INSIGHT 提供的两个视角是一组经过注释的患者记录，专家在其中选择了适当的类别来描述患者的那一时刻快照，以及一组能够对同一 5 点量表上的各个时间点进行分类的规则。这两个视角之间的不一致以混淆矩阵的形式显示；此外，INSIGHT 还允许专家修订注释数据集（纠正数据错误，或更改分配的类别）和实际规则集。
三位专家中的每一位都实现了他的精炼知识源（即，经过注释的每小时患者记录和规则集）之间非常高的一致性（~97%）。然后，我们让专家生成一个共同的规则集，然后根据该规则集对他们的几个注释集进行细化；这再次导致专家之间的一致性约为 97%。由此产生的规则集可以在应用程序中使用，具有相当的可信度。
这项研究表明，在某些情况下，领域专家有可能在同一任务的两个视角之间实现高度相关性。专家们一致认为，INSIGHT 提供的即时反馈对这一成功结果做出了重大贡献。
