IEEE Trans Med Imaging. 2019 May;38(5):1284-1294. doi: 10.1109/TMI.2018.2883237. Epub 2018 Nov 26.
Crowdsourcing in pathology has been performed on tasks that are assumed to be manageable by nonexperts. Demand remains high for annotations of more complex elements in digital microscopic images, such as anatomical structures. Therefore, this paper investigates conditions to enable crowdsourced annotations of high-level image objects, a complex task considered to require expert knowledge. Seventy six medical students without specific domain knowledge who voluntarily participated in three experiments solved two relevant annotation tasks on histopathological images: 1) labeling of images showing tissue regions and 2) delineation of morphologically defined image objects. We focus on methods to ensure sufficient annotation quality including several tests on the required number of participants and on the correlation of participants' performance between tasks. In a set up simulating annotation of images with limited ground truth, we validated the feasibility of a confidence score using full ground truth. For this, we computed a majority vote using weighting factors based on individual assessment of contributors against scattered gold standard annotated by pathologists. In conclusion, we provide guidance for task design and quality control to enable a crowdsourced approach to obtain accurate annotations required in the era of digital pathology.
病理众包已应用于非专业人员可完成的任务。对数字显微镜图像中更复杂元素（如解剖结构）的注释需求仍然很高。因此，本文研究了实现高水平图像对象众包注释的条件，这是一项被认为需要专业知识的复杂任务。76 名没有特定领域知识的医学生自愿参加了三项实验，他们对组织学图像完成了两项相关的注释任务：1）标记显示组织区域的图像，2）描绘形态定义的图像对象。我们专注于确保足够注释质量的方法，包括对所需参与者数量的多项测试，以及参与者在任务之间表现的相关性测试。在一个模拟使用有限真实数据进行图像注释的设置中，我们验证了使用完整真实数据的置信度评分的可行性。为此，我们使用基于对贡献者的个体评估的加权因子，计算多数投票，这些贡献者是由病理学家进行分散的黄金标准注释的。总之，我们为任务设计和质量控制提供了指导，以实现数字病理学时代所需的准确注释的众包方法。
