Graduate School of Artificial Intelligence, Seoul National University, Seoul, Republic of Korea.
Department of Electrical and Computer Engineering, Seoul National University, Seoul, Republic of Korea.
PLoS One. 2022 Mar 18;17(3):e0265456. doi: 10.1371/journal.pone.0265456. eCollection 2022.
In reinforcement learning, reward-driven feature learning directly from high-dimensional images faces two challenges: sample-efficiency for solving control tasks and generalization to unseen observations. In prior works, these issues have been addressed through learning representation from pixel inputs. However, their representation faced the limitations of being vulnerable to the high diversity inherent in environments or not taking the characteristics for solving control tasks. To attenuate these phenomena, we propose the novel contrastive representation method, Action-Driven Auxiliary Task (ADAT), which forces a representation to concentrate on essential features for deciding actions and ignore control-irrelevant details. In the augmented state-action dictionary of ADAT, the agent learns representation to maximize agreement between observations sharing the same actions. The proposed method significantly outperforms model-free and model-based algorithms in the Atari and OpenAI ProcGen, widely used benchmarks for sample-efficiency and generalization.
在强化学习中，直接从高维图像中进行奖励驱动的特征学习面临两个挑战：解决控制任务的样本效率和对未见观测的泛化能力。在之前的工作中，这些问题已经通过从像素输入中学习表示来解决。然而，它们的表示存在易受环境固有多样性影响或不考虑解决控制任务的特点的局限性。为了减轻这些现象，我们提出了新颖的对比表示方法，即动作驱动辅助任务 (ADAT)，它迫使表示集中在决定动作的基本特征上，忽略与控制无关的细节。在 ADAT 的增强状态-动作字典中，代理学习表示，以最大化具有相同动作的观察之间的一致性。所提出的方法在 Atari 和 OpenAI ProcGen 等广泛使用的样本效率和泛化基准中，显著优于无模型和基于模型的算法。
