Liu Xin, Chen Yaran, Chen Guixing, Li Haoran, Zhao Dongbin
IEEE Trans Cybern. 2025 May;55(5):2234-2247. doi: 10.1109/TCYB.2025.3548821. Epub 2025 Apr 23.
Unsupervised skill discovery seeks to acquire different useful skills without extrinsic reward via unsupervised reinforcement learning (RL), with the discovered skills efficiently adapting to multiple downstream tasks in various ways. However, recent advanced skill discovery methods struggle to well balance state exploration and skill diversity, particularly when the potential skills are rich and hard to discern. In this article, we propose contrastive dynamic skill discovery (ComSD) which generates diverse and exploratory unsupervised skills through a novel intrinsic incentive, named contrastive dynamic reward. It contains a particle-based exploration reward to make agents access far-reaching states for exploratory skill acquisition, and a novel contrastive diversity reward to promote the discriminability between different skills. Moreover, a novel dynamic weighting mechanism between the above two rewards is proposed to balance state exploration and skill diversity, which further enhances the quality of the discovered skills. Extensive experiments and analysis demonstrate that ComSD can generate diverse behaviors at different exploratory levels for multijoint robots, enabling state-of-the-art adaptation performance on challenging downstream tasks. It can also discover distinguishable and far-reaching exploration skills in the challenging tree-like 2-D maze.
无监督技能发现旨在通过无监督强化学习（RL）在没有外部奖励的情况下获取不同的有用技能，所发现的技能能够以各种方式有效地适应多个下游任务。然而，最近的先进技能发现方法难以很好地平衡状态探索和技能多样性，特别是当潜在技能丰富且难以辨别时。在本文中，我们提出了对比动态技能发现（ComSD），它通过一种名为对比动态奖励的新型内在激励来生成多样化和探索性的无监督技能。它包含一个基于粒子的探索奖励，以使智能体访问深远状态以获取探索性技能，以及一个新型的对比多样性奖励，以促进不同技能之间的可辨别性。此外，还提出了上述两种奖励之间的新型动态加权机制，以平衡状态探索和技能多样性，这进一步提高了所发现技能的质量。广泛的实验和分析表明，ComSD可以为多关节机器人在不同探索水平上生成多样化行为，在具有挑战性的下游任务上实现了领先的适应性能。它还可以在具有挑战性的树状二维迷宫中发现可区分且深远的探索技能。
