Vandenhende Simon, Georgoulis Stamatios, Van Gansbeke Wouter, Proesmans Marc, Dai Dengxin, Van Gool Luc
IEEE Trans Pattern Anal Mach Intell. 2022 Jul;44(7):3614-3633. doi: 10.1109/TPAMI.2021.3054719. Epub 2022 Jun 3.
With the advent of deep learning, many dense prediction tasks, i.e., tasks that produce pixel-level predictions, have seen significant performance improvements. The typical approach is to learn these tasks in isolation, that is, a separate neural network is trained for each individual task. Yet, recent multi-task learning (MTL) techniques have shown promising results w.r.t. performance, computations and/or memory footprint, by jointly tackling multiple tasks through a learned shared representation. In this survey, we provide a well-rounded view on state-of-the-art deep learning approaches for MTL in computer vision, explicitly emphasizing on dense prediction tasks. Our contributions concern the following. First, we consider MTL from a network architecture point-of-view. We include an extensive overview and discuss the advantages/disadvantages of recent popular MTL models. Second, we examine various optimization methods to tackle the joint learning of multiple tasks. We summarize the qualitative elements of these works and explore their commonalities and differences. Finally, we provide an extensive experimental evaluation across a variety of dense prediction benchmarks to examine the pros and cons of the different methods, including both architectural and optimization based strategies.
随着深度学习的出现，许多密集预测任务，即产生像素级预测的任务，在性能上有了显著提升。典型的方法是孤立地学习这些任务，也就是说，为每个单独的任务训练一个单独的神经网络。然而，最近的多任务学习（MTL）技术通过学习共享表示联合处理多个任务，在性能、计算和/或内存占用方面显示出了有前景的结果。在本次综述中，我们全面介绍了计算机视觉中用于多任务学习的最新深度学习方法，特别强调密集预测任务。我们的贡献如下。首先，我们从网络架构的角度考虑多任务学习。我们进行了广泛的概述，并讨论了最近流行的多任务学习模型的优缺点。其次，我们研究了各种优化方法来处理多个任务的联合学习。我们总结了这些工作的定性要素，并探讨它们的共性和差异。最后，我们在各种密集预测基准上进行了广泛的实验评估，以检验不同方法的优缺点，包括基于架构和优化的策略。
