Xiao Dan, Tan Ah-Hwee
School of Computer Engineering, Nanyang Technological University, Singapore 639798, Singapore.
IEEE Trans Syst Man Cybern B Cybern. 2007 Dec;37(6):1567-80. doi: 10.1109/tsmcb.2007.907040.
Temporal-Difference-Fusion Architecture for Learning, Cognition, and Navigation (TD-FALCON) is a generalization of adaptive resonance theory (a class of self-organizing neural networks) that incorporates TD methods for real-time reinforcement learning. In this paper, we investigate how a team of TD-FALCON networks may cooperate to learn and function in a dynamic multiagent environment based on minefield navigation and a predator/prey pursuit tasks. Experiments on the navigation task demonstrate that TD-FALCON agent teams are able to adapt and function well in a multiagent environment without an explicit mechanism of collaboration. In comparison, traditional Q-learning agents using gradient-descent-based feedforward neural networks, trained with the standard backpropagation and the resilient-propagation (RPROP) algorithms, produce a significantly poorer level of performance. For the predator/prey pursuit task, we experiment with various cooperative strategies and find that a combination of a high-level compressed state representation and a hybrid reward function produces the best results. Using the same cooperative strategy, the TD-FALCON team also outperforms the RPROP-based reinforcement learners in terms of both task completion rate and learning efficiency.
用于学习、认知和导航的时间差分融合架构（TD-FALCON）是自适应共振理论（一类自组织神经网络）的推广，它结合了用于实时强化学习的时间差分方法。在本文中，我们研究了一组TD-FALCON网络如何基于雷场导航和捕食者/猎物追捕任务在动态多智能体环境中进行协作学习和运行。在导航任务上的实验表明，TD-FALCON智能体团队能够在没有明确协作机制的多智能体环境中自适应并良好运行。相比之下，使用基于梯度下降的前馈神经网络、通过标准反向传播和弹性传播（RPROP）算法训练的传统Q学习智能体，其性能水平要差得多。对于捕食者/猎物追捕任务，我们试验了各种协作策略，发现高级压缩状态表示和混合奖励函数的组合产生了最佳结果。使用相同的协作策略，TD-FALCON团队在任务完成率和学习效率方面也优于基于RPROP的强化学习者。
