Kanakia Anshul, Wang Kuansan, Dong Yuxiao, Xie Boya, Lo Kyle, Shen Zhihong, Wang Lucy Lu, Huang Chiyuan, Eide Darrin, Kohlmeier Sebastian, Wu Chieh-Han
Microsoft Research, Redmond, WA, United States.
Allen Institute for Artificial Intelligence, Seattle, WA, United States.
Front Res Metr Anal. 2020 Nov 23;5:596624. doi: 10.3389/frma.2020.596624. eCollection 2020.
On the behest of the Office of Science and Technology Policy in the White House, six institutions, including ours, have created an open research dataset called COVID-19 Research Dataset (CORD-19) to facilitate the development of question-answering systems that can assist researchers in finding relevant research on COVID-19. As of May 27, 2020, CORD-19 includes more than 100,000 open access publications from major publishers and PubMed as well as preprint articles deposited into medRxiv, bioRxiv, and arXiv. Recent years, however, have also seen question-answering and other machine learning systems exhibit harmful behaviors to humans due to biases in the training data. It is imperative and only ethical for modern scientists to be vigilant in inspecting and be prepared to mitigate the potential biases when working with any datasets. This article describes a framework to examine biases in scientific document collections like CORD-19 by comparing their properties with those derived from the citation behaviors of the entire scientific community. In total, three expanded sets are created for the analyses: 1) the enclosure set CORD-19E composed of CORD-19 articles and their references and citations, mirroring the methodology used in the renowned "A Century of Physics" analysis; 2) the full closure graph CORD-19C that recursively includes references starting with CORD-19; and 3) the inflection closure CORD-19I, that is, a much smaller subset of CORD-19C but already appropriate for statistical analysis based on the theory of the scale-free nature of the citation network. Taken together, all these expanded datasets show much smoother trends when used to analyze global COVID-19 research. The results suggest that while CORD-19 exhibits a strong tilt toward recent and topically focused articles, the knowledge being explored to attack the pandemic encompasses a much longer time span and is very interdisciplinary. A question-answering system with such expanded scope of knowledge may perform better in understanding the literature and answering related questions. However, while CORD-19 appears to have topical coverage biases compared to the expanded sets, the collaboration patterns, especially in terms of team sizes and geographical distributions, are captured very well already in CORD-19 as the raw statistics and trends agree with those from larger datasets.
应白宫科学技术政策办公室的要求，包括我们机构在内的六个机构创建了一个名为“COVID-19研究数据集（CORD-19）”的开放研究数据集，以促进问答系统的开发，该系统可以帮助研究人员查找有关COVID-19的相关研究。截至2020年5月27日，CORD-19包含来自主要出版商和PubMed的超过100,000篇开放获取出版物，以及存入medRxiv、bioRxiv和arXiv的预印本文章。然而，近年来，由于训练数据中的偏差，问答系统和其他机器学习系统也出现了对人类有害的行为。对于现代科学家来说，在使用任何数据集时保持警惕，检查并准备减轻潜在偏差是必要且符合道德规范的。本文描述了一个框架，通过将科学文献集合（如CORD-19）的属性与从整个科学界的引用行为中得出的属性进行比较，来检查其中的偏差。总共创建了三个扩展集用于分析：1）封闭集CORD-19E，由CORD-19文章及其参考文献和引用组成，反映了著名的“物理学百年”分析中使用的方法；2）完全封闭图CORD-19C，它递归地包含以CORD-19开头的参考文献；3）拐点封闭集CORD-19I，即CORD-19C的一个小得多的子集，但已适合基于引用网络的无标度性质理论进行统计分析。综合来看，所有这些扩展数据集在用于分析全球COVID-19研究时显示出更平滑的趋势。结果表明，虽然CORD-19对近期和主题聚焦的文章有强烈倾向，但用于应对大流行所探索的知识涵盖了更长的时间跨度且非常跨学科。一个具有如此扩展知识范围的问答系统在理解文献和回答相关问题方面可能表现得更好。然而，虽然与扩展集相比，CORD-19似乎存在主题覆盖偏差，但协作模式，特别是在团队规模和地理分布方面，在CORD-19中已经很好地体现出来了，因为原始统计数据和趋势与来自更大数据集的数据一致。
