IEEE Trans Pattern Anal Mach Intell. 2023 Jun;45(6):7035-7049. doi: 10.1109/TPAMI.2020.3001940. Epub 2023 May 5.
In this work, we consider transferring the structure information from large networks to compact ones for dense prediction tasks in computer vision. Previous knowledge distillation strategies used for dense prediction tasks often directly borrow the distillation scheme for image classification and perform knowledge distillation for each pixel separately, leading to sub-optimal performance. Here we propose to distill structured knowledge from large networks to compact networks, taking into account the fact that dense prediction is a structured prediction problem. Specifically, we study two structured distillation schemes: i) pair-wise distillation that distills the pair-wise similarities by building a static graph; and ii) holistic distillation that uses adversarial training to distill holistic knowledge. The effectiveness of our knowledge distillation approaches is demonstrated by experiments on three dense prediction tasks: semantic segmentation, depth estimation and object detection. Code is available at https://git.io/StructKD.
在这项工作中，我们考虑将结构信息从大型网络转移到紧凑网络，以用于计算机视觉中的密集预测任务。以前用于密集预测任务的知识蒸馏策略通常直接借用图像分类的蒸馏方案，并分别对每个像素进行知识蒸馏，导致性能不理想。在这里，我们提出从大型网络中提取结构化知识到紧凑网络，考虑到密集预测是一个结构化预测问题的事实。具体来说，我们研究了两种结构化的蒸馏方案：i）对蒸馏，通过构建静态图来提取对之间的相似性；ii）整体蒸馏，使用对抗训练来提取整体知识。我们的知识蒸馏方法的有效性通过在三个密集预测任务上的实验得到了验证：语义分割、深度估计和目标检测。代码可在 https://git.io/StructKD 获得。
