Quaglini S, Stefanelli M, Barosi G, Berzuini A
Dipartimento di Informatica e Sistemistica, Università di Pavia, Italy.
Comput Biomed Res. 1988 Aug;21(4):307-23. doi: 10.1016/0010-4809(88)90046-8.
This paper reports the results of an evaluation study of the current level of performance given by ANEMIA, a knowledge-based consultation system addressing the clinical problem of managing anemic patients. ANEMIA was developed on a mainframe using the AI programming scheme EXPERT and then translated into a version running on a personal computer. At present the system is able to provide assistance in the diagnosis and management of 65 disease entities. After extensive local testing of accuracy, completeness, and consistency of the knowledge base included into ANEMIA, we designed a study to evaluate whether the system is able to appropriately mirror also the reasoning of well-known hematologists other than those who provided the knowledge. We were also interested in testing whether there were conflicting opinions among hematologists. Thus, we designed a validation study in which ANEMIA's performance could be compared with that of six hematologists and the interexpert consensus evaluated. ANEMIA's overall performance was judged acceptable in 87% (26/30) of the cases, while expert evaluators agreed with their colleagues in 90% (27/30) of them. A low interexpert consensus was found: considering the ratings given by different hematologists to the same ANEMIA performance, complete agreement occurred only 47% of the time.
本文报告了一项评估研究的结果，该研究针对基于知识的咨询系统ANEMIA处理贫血患者临床问题的当前性能水平。ANEMIA是在大型机上使用人工智能编程方案EXPERT开发的，然后被翻译成在个人计算机上运行的版本。目前，该系统能够为65种疾病实体的诊断和管理提供帮助。在对ANEMIA知识库的准确性、完整性和一致性进行广泛的本地测试之后，我们设计了一项研究，以评估该系统是否能够恰当地反映除提供知识的血液学家之外的其他知名血液学家的推理过程。我们还对测试血液学家之间是否存在意见冲突感兴趣。因此，我们设计了一项验证研究，在该研究中可以将ANEMIA的性能与六位血液学家的性能进行比较，并评估专家间的共识。在87%（26/30）的病例中，ANEMIA的总体性能被判定为可接受，而专家评估者在90%（27/30）的病例中与同事意见一致。发现专家间的共识较低：考虑不同血液学家对相同ANEMIA性能给出的评分，完全一致仅出现47%的时间。
