Cardiovascular Center, National Taiwan University Hospital Yun-Lin Branch, Dou-Liou City, Taiwan.
J Eval Clin Pract. 2011 Aug;17(4):832-8. doi: 10.1111/j.1365-2753.2011.01712.x. Epub 2011 Jun 26.
Critical appraisal, one of the most crucial steps in the practice of evidence-based medicine, is expertise-dependent and time-consuming. The objective of this study was to develop and evaluate an automated text-mining system that could determine the evidence level provided by a medical article.
A text processor was designed and built to interpret the abstracts of medical literature. The system extracted information about: (1) the impact factor of the journal; (2) study design; (3) human subject involvement; (4) number of subjects; (5) P-value; and (6) confidence intervals. We used a classification tree algorithm (C4.5) to create a decision tree using supervised classification. Each article was categorized into evidence level A, B or C, and the output was compared to that determined by domain experts (the reference standard).
We used a corpus of 3180 cardiovascular disease original research articles, of which 1108 were previously assigned evidence level A, 1705 level B and 367 level C by domain experts. The abstracts were analysed by our automated system and an evidence level was assigned. The algorithm accurately classified 85% of the articles. The agreement between computer and domain experts was substantial (κ-value: 0.78). Cross-validation showed consistent results across repeated tests.
The automated engine accurately classified the evidence level. Misclassification might have resulted from incomplete information retrieval and inaccurate data extraction. Further efforts will focus on assessing relevance and using additional study design features to refine evidence level classification.
循证医学实践中最重要的步骤之一是批判性评价，它依赖于专业知识且耗时。本研究旨在开发和评估一种自动文本挖掘系统，以确定医学文章提供的证据水平。
设计并构建了一个文本处理器来解释医学文献的摘要。该系统提取了有关以下信息：（1）期刊的影响因子；（2）研究设计；（3）是否涉及人体研究对象；（4）研究对象数量；（5）P 值；以及（6）置信区间。我们使用分类树算法（C4.5）创建了一个决策树，采用有监督分类方法。将每个文章分类为证据水平 A、B 或 C，输出结果与领域专家（参考标准）确定的结果进行比较。
我们使用了 3180 篇心血管疾病原始研究文章的语料库，其中 1108 篇先前被领域专家分配为证据水平 A，1705 篇为 B，367 篇为 C。我们的自动系统分析了摘要并分配了证据水平。该算法准确地分类了 85%的文章。计算机与领域专家之间的一致性很高（κ 值：0.78）。交叉验证显示了重复测试的一致结果。
自动引擎准确地对证据水平进行了分类。分类错误可能是由于信息检索不完整和数据提取不准确造成的。进一步的努力将集中在评估相关性和使用其他研究设计特征来改进证据水平分类上。
