Gao Naiyu, Shan Yanhu, Zhao Xin, Huang Kaiqi
IEEE Trans Image Process. 2021;30:6013-6023. doi: 10.1109/TIP.2021.3090522. Epub 2021 Jul 1.
Panoptic segmentation (PS) is a complex scene understanding task that requires providing high-quality segmentation for both thing objects and stuff regions. Previous methods handle these two classes with semantic and instance segmentation modules separately, following with heuristic fusion or additional modules to resolve the conflicts between the two outputs. This work simplifies this pipeline of PS by consistently modeling the two classes with a novel PS framework, which extends a detection model with an extra module to predict category- and instance-aware pixel embedding (CIAE). CIAE is a novel pixel-wise embedding feature that encodes both semantic-classification and instance-distinction information. At the inference process, PS results are simply derived by assigning each pixel to a detected instance or a stuff class according to the learned embedding. Our method not only demonstrates fast inference speed but also the first one-stage method to achieve comparable performance to two-stage methods on the challenging COCO benchmark.
全景分割（PS）是一项复杂的场景理解任务，需要为事物对象和材质区域提供高质量的分割。以前的方法分别使用语义分割和实例分割模块来处理这两类，随后采用启发式融合或额外的模块来解决两个输出之间的冲突。这项工作通过用一个新颖的PS框架对这两类进行一致建模来简化PS的流程，该框架通过一个额外的模块扩展了一个检测模型，以预测类别和实例感知像素嵌入（CIAE）。CIAE是一种新颖的逐像素嵌入特征，它对语义分类和实例区分信息进行编码。在推理过程中，PS结果只需根据学习到的嵌入将每个像素分配给一个检测到的实例或一个材质类别即可简单得出。我们的方法不仅展示了快速的推理速度，而且是第一个在具有挑战性的COCO基准上达到与两阶段方法相当性能的单阶段方法。
