University of Maryland, Computer Science Department, Iribe Center for Computer Science and Engineering, College Park, Maryland, USA.
Division of Imaging, Diagnostics, and Software Reliability (DIDSR), OSEL, CDRH, FDA, Silver Spring, Maryland, USA.
Med Phys. 2023 Jul;50(7):4255-4268. doi: 10.1002/mp.16219. Epub 2023 Jan 27.
PURPOSE: Machine learning algorithms are best trained with large quantities of accurately annotated samples. While natural scene images can often be labeled relatively cheaply and at large scale, obtaining accurate annotations for medical images is both time consuming and expensive. In this study, we propose a cooperative labeling method that allows us to make use of weakly annotated medical imaging data for the training of a machine learning algorithm. As most clinically produced data are weakly-annotated - produced for use by humans rather than machines and lacking information machine learning depends upon - this approach allows us to incorporate a wider range of clinical data and thereby increase the training set size. METHODS: Our pseudo-labeling method consists of multiple stages. In the first stage, a previously established network is trained using a limited number of samples with high-quality expert-produced annotations. This network is used to generate annotations for a separate larger dataset that contains only weakly annotated scans. In the second stage, by cross-checking the two types of annotations against each other, we obtain higher-fidelity annotations. In the third stage, we extract training data from the weakly annotated scans, and combine it with the fully annotated data, producing a larger training dataset. We use this larger dataset to develop a computer-aided detection (CADe) system for nodule detection in chest CT. RESULTS: We evaluated the proposed approach by presenting the network with different numbers of expert-annotated scans in training and then testing the CADe using an independent expert-annotated dataset. We demonstrate that when availability of expert annotations is severely limited, the inclusion of weakly-labeled data leads to a 5% improvement in the competitive performance metric (CPM), defined as the average of sensitivities at different false-positive rates. CONCLUSIONS: Our proposed approach can effectively merge a weakly-annotated dataset with a small, well-annotated dataset for algorithm training. This approach can help enlarge limited training data by leveraging the large amount of weakly labeled data typically generated in clinical image interpretation.
目的:机器学习算法最好使用大量准确标注的样本进行训练。虽然自然场景图像通常可以相对廉价且大规模地进行标注,但获取医学图像的准确标注既费时又昂贵。在本研究中,我们提出了一种合作标注方法,允许我们利用弱标注的医学成像数据来训练机器学习算法。由于大多数临床产生的数据都是弱标注的 - 为人类而非机器生成,并且缺乏机器学习所依赖的信息 - 这种方法使我们能够纳入更广泛的临床数据,从而增加训练集的规模。
方法:我们的伪标注方法由多个阶段组成。在第一阶段,使用数量有限的高质量专家生成标注样本对先前建立的网络进行训练。该网络用于对仅包含弱标注扫描的单独较大数据集进行标注。在第二阶段,通过相互交叉检查两种类型的标注,我们获得更准确的标注。在第三阶段,我们从弱标注扫描中提取训练数据,并将其与完全标注的数据相结合,生成更大的训练数据集。我们使用这个更大的数据集来开发胸部 CT 中结节检测的计算机辅助检测 (CADe) 系统。
结果:我们通过在训练中向网络提供不同数量的专家标注扫描,然后使用独立的专家标注数据集来测试 CADe,评估了所提出的方法。我们证明,当专家标注的可用性严重受限时,纳入弱标注数据可将竞争性能指标 (CPM) 提高 5%,CPM 定义为不同假阳性率下的灵敏度平均值。
结论:我们提出的方法可以有效地将弱标注数据集与小的、标注良好的数据集合并,用于算法训练。这种方法可以通过利用临床图像解释中通常生成的大量弱标注数据来帮助扩大有限的训练数据。
IEEE J Biomed Health Inform. 2022-8
Comput Methods Programs Biomed. 2024-6
Med Image Anal. 2024-8
Sensors (Basel). 2022-8-13
J Med Imaging (Bellingham). 2023-9