Wellcome Centre for Human Neuroimaging, UCL Queen Square Institute of Neurology, London WC1N 3AR, U.K.
Wellcome Centre for Human Neuroimaging, UCL Queen Square Institute of Neurology, London WC1N 3AR, U.K., and Department of Mathematics, Imperial College London, U.K.
Neural Comput. 2021 Mar;33(3):713-763. doi: 10.1162/neco_a_01351.
Active inference offers a first principle account of sentient behavior, from which special and important cases-for example, reinforcement learning, active learning, Bayes optimal inference, Bayes optimal design-can be derived. Active inference finesses the exploitation-exploration dilemma in relation to prior preferences by placing information gain on the same footing as reward or value. In brief, active inference replaces value functions with functionals of (Bayesian) beliefs, in the form of an expected (variational) free energy. In this letter, we consider a sophisticated kind of active inference using a recursive form of expected free energy. Sophistication describes the degree to which an agent has beliefs about beliefs. We consider agents with beliefs about the counterfactual consequences of action for states of affairs beliefs about those latent states. In other words, we move from simply considering beliefs about "what would happen if I did that" to "what I would what would happen if I did that." The recursive form of the free energy functional effectively implements a deep tree search over actions and outcomes in the future. Crucially, this search is over sequences of belief states as opposed to states per se. We illustrate the competence of this scheme using numerical simulations of deep decision problems.
主动推断为有感知的行为提供了一个基本原理的解释,从中可以推导出特殊而重要的情况,例如强化学习、主动学习、贝叶斯最优推断、贝叶斯最优设计。主动推断通过将信息增益与奖励或价值放在同一水平线上,巧妙地解决了先验偏好中的开发-探索困境。简而言之,主动推断用(贝叶斯)信念的泛函代替了价值函数,其形式为期望(变分)自由能。在这封信中,我们考虑了一种使用预期自由能递归形式的复杂主动推断。复杂程度描述了一个代理对自身信念的了解程度。我们考虑对行动对事态的反事实后果有信念的代理,以及对那些潜在状态的信念。换句话说,我们从简单地考虑“如果我那样做会发生什么”转变为“我会怎样考虑如果我那样做会发生什么”。自由能泛函的递归形式有效地在未来的行动和结果上执行了深度树搜索。至关重要的是,这种搜索是针对信念状态序列而不是状态本身。我们使用深度决策问题的数值模拟来说明该方案的能力。