Li Xiangtai, Xu Shilin, Yang Yibo, Yuan Haobo, Cheng Guangliang, Tong Yunhai, Lin Zhouchen, Yang Ming-Hsuan, Tao Dacheng
IEEE Trans Pattern Anal Mach Intell. 2024 Dec;46(12):11087-11103. doi: 10.1109/TPAMI.2024.3453916. Epub 2024 Nov 6.
Panoptic Part Segmentation (PPS) unifies panoptic and part segmentation into one task. Previous works utilize separate approaches to handle things, stuff, and part predictions without shared computation and task association. We aim to unify these tasks at the architectural level, designing the first end-to-end unified framework, Panoptic-PartFormer. Moreover, we find the previous metric PartPQ biases to PQ. To handle both issues, we first design a meta-architecture that decouples part features and things/stuff features, respectively. We model things, stuff, and parts as object queries and directly learn to optimize all three forms of prediction as a unified mask prediction and classification problem. We term our model as Panoptic-PartFormer. Second, we propose a new metric Part-Whole Quality (PWQ), better to measure this task from pixel-region and part-whole perspectives. It also decouples the errors for part segmentation and panoptic segmentation. Third, inspired by Mask2Former, based on our meta-architecture, we propose Panoptic-PartFormer++ and design a new part-whole cross-attention scheme to boost part segmentation qualities further. We design a new part-whole interaction method using masked cross attention. Finally, extensive ablation studies and analysis demonstrate the effectiveness of both Panoptic-PartFormer and Panoptic-PartFormer++. Compared with previous Panoptic-PartFormer, our Panoptic-PartFormer++ achieves 2% PartPQ and 3% PWQ improvements on the Cityscapes PPS dataset and 5% PartPQ on the Pascal Context PPS dataset. On both datasets, Panoptic-PartFormer++ achieves new state-of-the-art results. Our models can serve as a strong baseline and aid future research in PPS.
全景部分分割（PPS）将全景分割和部分分割统一为一个任务。以往的工作采用单独的方法来处理事物、材质和部分预测，没有共享计算和任务关联。我们旨在在架构层面统一这些任务，设计了首个端到端统一框架全景部分Former（Panoptic-PartFormer）。此外，我们发现先前的度量标准部分PQ质量（PartPQ）偏向于全景质量（PQ）。为了解决这两个问题，我们首先设计了一种元架构，分别解耦部分特征与事物/材质特征。我们将事物、材质和部分建模为对象查询，并直接学习将所有三种预测形式优化为统一的掩码预测和分类问题。我们将我们的模型称为全景部分Former。其次，我们提出了一种新的度量标准部分-整体质量（PWQ），能更好地从像素区域和部分-整体角度来衡量此任务。它还解耦了部分分割和全景分割的误差。第三，受Mask2Former启发，基于我们的元架构，我们提出了全景部分Former++，并设计了一种新的部分-整体交叉注意力方案以进一步提升部分分割质量。我们使用掩码交叉注意力设计了一种新的部分-整体交互方法。最后，广泛的消融研究和分析证明了全景部分Former和全景部分Former++的有效性。与之前的全景部分Former相比，我们的全景部分Former++在Cityscapes PPS数据集上实现了部分PQ质量提升2%、PWQ提升3%，在Pascal Context PPS数据集上实现了部分PQ质量提升5%。在这两个数据集上，全景部分Former++都取得了新的最优结果。我们的模型可以作为一个强大的基线，并有助于未来在PPS方面的研究。
