Feng Mingxiao, Yang Yaodong, Zhou Wengang, Li Houqiang
CAS Key Laboratory of GIPAS, University of Science and Technology of China, Hefei, China; Institute of Artificial Intelligence, Hefei Comprehensive National Science Center, Hefei, China.
Peking University, Beijing, China.
Neural Netw. 2025 Apr;184:107081. doi: 10.1016/j.neunet.2024.107081. Epub 2024 Dec 31.
In MARL (Multi-Agent Reinforcement Learning), the trial-and-error learning paradigm based on multiple agents requires massive interactions to produce training samples, significantly increasing both the training cost and difficulty. Therefore, enhancing data efficiency is a core issue in MARL. However, in the context of MARL, agent partially observed information leads to a lack of consideration for agent interactions and coordination from an ego perspective under the world model, which becomes the main obstacle to improving the data efficiency of current proposed MARL methods. To address this, motivated by the success of learning a world model in RL and cognitive science, we devise a world-model-driven learning paradigm enabling agents to gain a more holistic representation of individual observation of the environment. Specifically, we present the Transition-Informed Multi-Agent Representations (TIMAR) framework, which leverages the joint transition model, i.e., a surrogate world model that captures the dynamics of the multi-agent system, to learn effective representations among agents through a self-supervised learning objective. This objective encourages consistency between predicted and actual future observations, allowing the model to learn without explicit labels. TIMAR incorporates an auxiliary module to predict future transitions based on sequential observations and actions, allowing agents to infer the latent state of the system and consider the influences of others. Unlike traditional MARL approaches that primarily focus on efficient policy improvement, TIMAR is designed to learn a useful semantic representation from high-dimensional observations. This enables the used MARL algorithm built on these representations to achieve improvements in data efficiency. Experimental evaluation of TIMAR in various MARL environments demonstrates its significantly improved performance and data efficiency compared to strong baselines such as MAPPO, HAPPO, finetuned QMIX, MAT, and MA2CL. In addition, we found TIMAR can also improve the generalization of the Transformer-based MARL algorithm such as MAT.
在多智能体强化学习（MARL）中，基于多个智能体的试错学习范式需要大量交互来生成训练样本，这显著增加了训练成本和难度。因此，提高数据效率是MARL中的一个核心问题。然而，在MARL的背景下，智能体的部分观测信息导致在世界模型下从自我视角缺乏对智能体交互与协调的考虑，这成为提高当前所提出的MARL方法数据效率的主要障碍。为了解决这个问题，受强化学习和认知科学中学习世界模型成功的启发，我们设计了一种由世界模型驱动的学习范式，使智能体能够获得对环境个体观测的更整体表示。具体而言，我们提出了转换信息多智能体表示（TIMAR）框架，该框架利用联合转换模型，即一个捕捉多智能体系统动态的替代世界模型，通过自监督学习目标来学习智能体之间的有效表示。这个目标鼓励预测的未来观测与实际未来观测之间的一致性，使模型能够在没有明确标签的情况下进行学习。TIMAR纳入了一个辅助模块，用于根据顺序观测和动作预测未来转换，使智能体能够推断系统的潜在状态并考虑其他智能体的影响。与主要专注于高效策略改进的传统MARL方法不同，TIMAR旨在从高维观测中学习有用的语义表示。这使得基于这些表示构建的MARL算法能够在数据效率方面取得提升。在各种MARL环境中对TIMAR进行的实验评估表明，与MAPPO、HAPPO、微调后的QMIX、MAT和MA2CL等强大基线相比，其性能和数据效率有显著提高。此外，我们发现TIMAR还可以提高基于Transformer的MARL算法（如MAT）的泛化能力。
