Yang Qisen, Wang Shenzhi, Zhang Qihang, Huang Gao, Song Shiji
IEEE Trans Neural Netw Learn Syst. 2024 Nov;35(11):16288-16300. doi: 10.1109/TNNLS.2023.3293508. Epub 2024 Oct 29.
Offline reinforcement learning (RL) optimizes the policy on a previously collected dataset without any interactions with the environment, yet usually suffers from the distributional shift problem. To mitigate this issue, a typical solution is to impose a policy constraint on a policy improvement objective. However, existing methods generally adopt a "one-size-fits-all" practice, i.e., keeping only a single improvement-constraint balance for all the samples in a mini-batch or even the entire offline dataset. In this work, we argue that different samples should be treated with different policy constraint intensities. Based on this idea, a novel plug-in approach named guided offline RL (GORL) is proposed. GORL employs a guiding network, along with only a few expert demonstrations, to adaptively determine the relative importance of the policy improvement and policy constraint for every sample. We theoretically prove that the guidance provided by our method is rational and near-optimal. Extensive experiments on various environments suggest that GORL can be easily installed on most offline RL algorithms with statistically significant performance improvements.
离线强化学习（RL）在先前收集的数据集上优化策略，无需与环境进行任何交互，但通常会遇到分布偏移问题。为了缓解这个问题，一种典型的解决方案是在策略改进目标上施加策略约束。然而，现有方法通常采用“一刀切”的做法，即在一个小批次甚至整个离线数据集中对所有样本只保持单一的改进-约束平衡。在这项工作中，我们认为不同的样本应该用不同的策略约束强度来处理。基于这一想法，我们提出了一种名为引导式离线强化学习（GORL）的新颖插件式方法。GORL使用一个引导网络，以及仅少量的专家示范，来自适应地确定每个样本的策略改进和策略约束的相对重要性。我们从理论上证明了我们方法提供的引导是合理且近乎最优的。在各种环境下的大量实验表明，GORL可以很容易地安装在大多数离线强化学习算法上，并在性能上有统计学意义的显著提升。
