Li Yao, Wang YuHui, Tan XiaoYang
School of Computer and Information Technology, Shanxi University, China.
Center of Excellence in GenAI, King Abdullah University of Science and Technology, Saudi Arabia.
Neural Netw. 2025 Jan;181:106825. doi: 10.1016/j.neunet.2024.106825. Epub 2024 Oct 28.
Goal-conditioned reinforcement learning is widely used in robot control, manipulating the robot to accomplish specific tasks by maximizing accumulated rewards. However, the useful reward signal is only received when the desired goal is reached, leading to the issue of sparse rewards and affecting the efficiency of policy learning. In this paper, we propose a method to generate highly valued subgoals for efficient goal-conditioned policy learning, enabling the development of smart home robots or automatic pilots in our daily life. The highly valued subgoals are conditioned on the context of the specific tasks and characterized by suitable complexity for efficient goal-conditioned action value learning. The context variable captures the latent representation of the particular tasks, allowing for efficient subgoal generation. Additionally, the goal-conditioned action values regularized by the self-adaptive ranges generate subgoals with suitable complexity. Compared to Hindsight Experience Replay that uniformly samples subgoals from visited trajectories, our method generates the subgoals based on the context of tasks with suitable difficulty for efficient policy training. Experimental results show that our method achieves stable performance in robotic environments compared to baseline methods.
目标条件强化学习在机器人控制中被广泛应用，通过最大化累积奖励来操纵机器人完成特定任务。然而，只有在达到期望目标时才会收到有用的奖励信号，这导致了奖励稀疏的问题，并影响了策略学习的效率。在本文中，我们提出了一种方法来生成高价值子目标，以实现高效的目标条件策略学习，从而推动智能家居机器人或自动驾驶仪在我们日常生活中的发展。高价值子目标取决于特定任务的上下文，并具有适合高效目标条件动作值学习的复杂度。上下文变量捕获特定任务的潜在表示，从而实现高效的子目标生成。此外，通过自适应范围进行正则化的目标条件动作值会生成具有适当复杂度的子目标。与从已访问轨迹中均匀采样子目标的事后经验回放相比，我们的方法基于具有适当难度的任务上下文来生成子目标，以实现高效的策略训练。实验结果表明，与基线方法相比，我们的方法在机器人环境中实现了稳定的性能。
