Howard Hughes Medical Institute, Ashburn, VA, United States.
Front Neural Circuits. 2018 Nov 13;12:102. doi: 10.3389/fncir.2018.00102. eCollection 2018.
Automatic image segmentation is critical to scale up electron microscope (EM) connectome reconstruction. To this end, segmentation competitions, such as CREMI and SNEMI, exist to help researchers evaluate segmentation algorithms with the goal of improving them. Because generating ground truth is time-consuming, these competitions often fail to capture the challenges in segmenting larger datasets required in connectomics. More generally, the common metrics for EM image segmentation do not emphasize impact on downstream analysis and are often not very useful for isolating problem areas in the segmentation. For example, they do not capture connectivity information and often over-rate the quality of a segmentation as we demonstrate later. To address these issues, we introduce a novel strategy to enable evaluation of segmentation at large scales both in a supervised setting, where ground truth is available, or an unsupervised setting. To achieve this, we first introduce new metrics more closely aligned with the use of segmentation in downstream analysis and reconstruction. In particular, these include synapse connectivity and completeness metrics that provide both meaningful and intuitive interpretations of segmentation quality as it relates to the preservation of neuron connectivity. Also, we propose measures of segmentation correctness and completeness with respect to the percentage of "orphan" fragments and the concentrations of self-loops formed by segmentation failures, which are helpful in analysis and can be computed without ground truth. The introduction of new metrics intended to be used for practical applications involving large datasets necessitates a scalable software ecosystem, which is a critical contribution of this paper. To this end, we introduce a scalable, flexible software framework that enables integration of several different metrics and provides mechanisms to evaluate and debug differences between segmentations. We also introduce visualization software to help users to consume the various metrics collected. We evaluate our framework on two relatively large public groundtruth datasets providing novel insights on example segmentations.
自动图像分割对于扩大电子显微镜 (EM) 连接组重建至关重要。为此，存在分割竞赛，如 CREMI 和 SNEMI，以帮助研究人员评估分割算法，目的是改进它们。由于生成地面实况很耗时，这些竞赛往往无法捕捉到在连接组学中需要分割更大数据集的挑战。更一般地说，用于 EM 图像分割的常用指标不强调对下游分析的影响，并且通常对于隔离分割中的问题区域不是非常有用。例如，它们不捕获连接信息，并且经常像我们稍后演示的那样高估分割的质量。为了解决这些问题，我们引入了一种新策略，以便在有地面实况的监督设置或无监督设置下，在大规模上评估分割。为此，我们首先引入了与下游分析和重建中分割使用更紧密相关的新指标。特别是，这些指标包括突触连接性和完整性指标，它们提供了与神经元连接保持相关的分割质量的有意义和直观的解释。此外，我们提出了针对“孤儿”片段的百分比和由分割失败形成的自循环浓度的分割正确性和完整性度量，这些度量有助于分析并且可以在没有地面实况的情况下计算。引入旨在用于涉及大数据集的实际应用的新指标需要可扩展的软件生态系统，这是本文的一个关键贡献。为此，我们引入了一个可扩展的、灵活的软件框架，该框架能够集成几种不同的指标，并提供评估和调试分割之间差异的机制。我们还引入了可视化软件来帮助用户使用收集到的各种指标。我们在两个相对较大的公共地面实况数据集上评估了我们的框架，为示例分割提供了新的见解。
