Zhang Zhengquan, Xu Feng
Key Laboratory of Information Science of Electromagnetic Waves, Fudan University, Shanghai, P.R.C.
Neural Comput. 2024 Apr 23;36(5):963-1021. doi: 10.1162/neco_a_01642.
The free energy principle and its corollary, the active inference framework, serve as theoretical foundations in the domain of neuroscience, explaining the genesis of intelligent behavior. This principle states that the processes of perception, learning, and decision making-within an agent-are all driven by the objective of "minimizing free energy," evincing the following behaviors: learning and employing a generative model of the environment to interpret observations, thereby achieving perception, and selecting actions to maintain a stable preferred state and minimize the uncertainty about the environment, thereby achieving decision making. This fundamental principle can be used to explain how the brain processes perceptual information, learns about the environment, and selects actions. Two pivotal tenets are that the agent employs a generative model for perception and planning and that interaction with the world (and other agents) enhances the performance of the generative model and augments perception. With the evolution of control theory and deep learning tools, agents based on the FEP have been instantiated in various ways across different domains, guiding the design of a multitude of generative models and decision-making algorithms. This letter first introduces the basic concepts of the FEP, followed by its historical development and connections with other theories of intelligence, and then delves into the specific application of the FEP to perception and decision making, encompassing both low-dimensional simple situations and high-dimensional complex situations. It compares the FEP with model-based reinforcement learning to show that the FEP provides a better objective function. We illustrate this using numerical studies of Dreamer3 by adding expected information gain into the standard objective function. In a complementary fashion, existing reinforcement learning, and deep learning algorithms can also help implement the FEP-based agents. Finally, we discuss the various capabilities that agents need to possess in complex environments and state that the FEP can aid agents in acquiring these capabilities.
自由能原理及其推论——主动推理框架，是神经科学领域的理论基础，用于解释智能行为的起源。该原理指出，在一个智能体中，感知、学习和决策过程均由“最小化自由能”这一目标驱动，表现为以下行为：学习并运用环境的生成模型来解释观测结果，从而实现感知；选择行动以维持稳定的偏好状态，并最小化关于环境的不确定性，从而实现决策。这一基本原理可用于解释大脑如何处理感知信息、了解环境并选择行动。两个关键原则是，智能体采用生成模型进行感知和规划，以及与世界（和其他智能体）的交互会提升生成模型的性能并增强感知。随着控制理论和深度学习工具的发展，基于自由能原理的智能体已在不同领域以各种方式得以实现，指导了众多生成模型和决策算法的设计。本文首先介绍自由能原理的基本概念，接着阐述其历史发展以及与其他智能理论的联系，然后深入探讨自由能原理在感知和决策方面的具体应用，涵盖低维简单情形和高维复杂情形。将自由能原理与基于模型的强化学习进行比较，以表明自由能原理提供了更好的目标函数。我们通过在标准目标函数中添加预期信息增益，利用Dreamer3的数值研究对此进行说明。以互补的方式，现有的强化学习和深度学习算法也有助于实现基于自由能原理的智能体。最后，我们讨论了智能体在复杂环境中需要具备的各种能力，并指出自由能原理可帮助智能体获得这些能力。
