Sarker Abeed, Mollá Diego, Paris Cécile
Department of Computing, Macquarie University, Sydney, NSW 2109, Australia.
Department of Computing, Macquarie University, Sydney, NSW 2109, Australia.
Artif Intell Med. 2015 Jun;64(2):89-103. doi: 10.1016/j.artmed.2015.04.001. Epub 2015 Apr 22.
Evidence-based medicine practice requires practitioners to obtain the best available medical evidence, and appraise the quality of the evidence when making clinical decisions. Primarily due to the plethora of electronically available data from the medical literature, the manual appraisal of the quality of evidence is a time-consuming process. We present a fully automatic approach for predicting the quality of medical evidence in order to aid practitioners at point-of-care.
Our approach extracts relevant information from medical article abstracts and utilises data from a specialised corpus to apply supervised machine learning for the prediction of the quality grades. Following an in-depth analysis of the usefulness of features (e.g., publication types of articles), they are extracted from the text via rule-based approaches and from the meta-data associated with the articles, and then applied in the supervised classification model. We propose the use of a highly scalable and portable approach using a sequence of high precision classifiers, and introduce a simple evaluation metric called average error distance (AED) that simplifies the comparison of systems. We also perform elaborate human evaluations to compare the performance of our system against human judgments.
We test and evaluate our approaches on a publicly available, specialised, annotated corpus containing 1132 evidence-based recommendations. Our rule-based approach performs exceptionally well at the automatic extraction of publication types of articles, with F-scores of up to 0.99 for high-quality publication types. For evidence quality classification, our approach obtains an accuracy of 63.84% and an AED of 0.271. The human evaluations show that the performance of our system, in terms of AED and accuracy, is comparable to the performance of humans on the same data.
The experiments suggest that our structured text classification framework achieves evaluation results comparable to those of human performance. Our overall classification approach and evaluation technique are also highly portable and can be used for various evidence grading scales.
循证医学实践要求从业者获取最佳可用医学证据，并在做出临床决策时评估证据质量。主要由于医学文献中有大量电子可用数据，人工评估证据质量是一个耗时的过程。我们提出一种全自动方法来预测医学证据质量，以帮助临床护理点的从业者。
我们的方法从医学文章摘要中提取相关信息，并利用来自专门语料库的数据应用监督机器学习来预测质量等级。在深入分析特征（如文章的出版类型）的有用性之后，通过基于规则的方法从文本以及与文章相关的元数据中提取这些特征，然后将其应用于监督分类模型。我们建议使用一系列高精度分类器的高度可扩展且便携的方法，并引入一种称为平均误差距离（AED）的简单评估指标，该指标简化了系统比较。我们还进行了详细的人工评估，以将我们系统的性能与人工判断进行比较。
我们在一个包含1132条循证医学推荐的公开可用、专门注释的语料库上测试和评估我们的方法。我们基于规则的方法在自动提取文章出版类型方面表现出色，高质量出版类型的F值高达0.99。对于证据质量分类，我们的方法获得了63.84%的准确率和0.271的平均误差距离。人工评估表明，就平均误差距离和准确率而言，我们系统的性能与人类在相同数据上的性能相当。
实验表明，我们的结构化文本分类框架取得了与人类性能相当的评估结果。我们的整体分类方法和评估技术也具有高度的便携性，可用于各种证据分级量表。
