Australian Institute for Machine Learning, North Terrace, Adelaide, Australia; School of Public Health, University of Adelaide, North Terrace, Adelaide 5000, Australia; Royal Adelaide Hospital, North Terrace, Adelaide, Australia.
Acad Radiol. 2020 Jan;27(1):106-112. doi: 10.1016/j.acra.2019.10.006. Epub 2019 Nov 6.
Medical artificial intelligence systems are dependent on well characterized large-scale datasets. Recently released public datasets have been of great interest to the field, but pose specific challenges due to the disconnect they cause between data generation and data usage, potentially limiting the utility of these datasets.
We visually explore two large public datasets, to determine how accurate the provided labels are and whether other subtle problems exist. The ChestXray14 dataset contains 112,120 frontal chest films, and the Musculoskeletal Radiology (MURA) dataset contains 40,561 upper limb radiographs. A subset of around 700 images from both datasets was reviewed by a board-certified radiologist, and the quality of the original labels was determined.
The ChestXray14 labels did not accurately reflect the visual content of the images, with positive predictive values mostly between 10% and 30% lower than the values presented in the original documentation. There were other significant problems, with examples of hidden stratification and label disambiguation failure. The MURA labels were more accurate, but the original normal/abnormal labels were inaccurate for the subset of cases with degenerative joint disease, with a sensitivity of 60% and a specificity of 82%.
Visual inspection of images is a necessary component of understanding large image datasets. We recommend that teams producing public datasets should perform this important quality control procedure and include a thorough description of their findings, along with an explanation of the data generating procedures and labeling rules, in the documentation for their datasets.
医学人工智能系统依赖于具有良好特征的大规模数据集。最近发布的公共数据集引起了该领域的极大兴趣，但由于它们在数据生成和数据使用之间造成的脱节，这些数据集可能会限制其用途，因此带来了一些特殊的挑战。
我们直观地研究了两个大型公共数据集，以确定提供的标签的准确性以及是否存在其他细微问题。ChestXray14 数据集包含 112120 张胸部正位片，Musculoskeletal Radiology (MURA) 数据集包含 40561 张上肢 X 光片。两个数据集的大约 700 张图像子集由一名经过董事会认证的放射科医生进行了审查，并确定了原始标签的质量。
ChestXray14 标签没有准确反映图像的视觉内容，阳性预测值比原始文档中呈现的值低 10%至 30%左右。还存在其他重大问题，包括隐藏分层和标签歧义失败的例子。MURA 标签更准确，但原始的正常/异常标签对于有退行性关节疾病的病例子集是不准确的，其敏感性为 60%，特异性为 82%。
对图像进行直观检查是理解大型图像数据集的必要组成部分。我们建议制作公共数据集的团队应执行此重要的质量控制程序，并在其数据集的文档中包括对其发现的全面描述，以及对数据生成过程和标记规则的解释。
