Borgstede Matthias
Foundations of Education, University of Bamberg, Bamberg, Germany.
Front Psychol. 2021 Nov 19;12:684544. doi: 10.3389/fpsyg.2021.684544. eCollection 2021.
Several authors have proposed that mechanisms of adaptive behavior, and reinforcement learning in particular, can be explained by an innate tendency of individuals to seek information about the local environment. In this article, I argue that these approaches adhere to an essentialist view of learning that avoids the question why information seeking should be favorable in the first place. I propose a selectionist account of adaptive behavior that explains why individuals behave as if they had a tendency to seek information without resorting to essentialist explanations. I develop my argument using a formal selectionist framework for adaptive behavior, the multilevel model of behavioral selection (MLBS). The MLBS has been introduced recently as a formal theory of behavioral selection that links reinforcement learning to natural selection within a single unified model. I show that the MLBS implies an average gain in information about the availability of reinforcement. Formally, this means that behavior reaches an equilibrium state, if and only if the Fisher information of the conditional probability of reinforcement is maximized. This coincides with a reduction in the randomness of the expected environmental feedback as captured by the information theoretic concept of expected surprise (i.e., entropy). In contrast to existing attempts to link adaptive behavior to information theoretic concepts (e.g., the ), neither information gain nor surprise minimization is treated as a first principle. Instead, the result is formally deduced from the MLBS and therefore constitutes a mathematical property of the more general principle of behavioral selection. Thus, if reinforcement learning is understood as a selection process, there is no need to assume an active agent with an innate tendency to seek information or minimize surprise. Instead, information gain and surprise minimization emerge naturally because it lies in the very nature of selection to produce order from randomness.
几位作者提出，适应性行为机制，尤其是强化学习，可以用个体寻求有关当地环境信息的先天倾向来解释。在本文中，我认为这些方法坚持一种本质主义的学习观点，回避了为什么寻求信息首先应该是有利的这个问题。我提出了一种关于适应性行为的选择主义解释，它解释了为什么个体的行为就好像他们有一种寻求信息的倾向，而无需诉诸本质主义的解释。我使用一个关于适应性行为的形式化选择主义框架——行为选择的多层次模型（MLBS）来展开我的论证。MLBS最近被引入作为一种行为选择的形式化理论，它在一个统一的模型中将强化学习与自然选择联系起来。我表明MLBS意味着在强化可用性信息方面的平均增益。形式上，这意味着当且仅当强化条件概率的费希尔信息最大化时，行为达到平衡状态。这与预期意外（即熵）的信息论概念所捕捉到的预期环境反馈的随机性降低相吻合。与将适应性行为与信息论概念联系起来的现有尝试（例如， ）不同，信息增益和意外最小化都没有被视为第一原理。相反，这个结果是从MLBS形式推导出来的，因此构成了行为选择更一般原则的一个数学性质。因此，如果强化学习被理解为一个选择过程，就无需假设一个具有寻求信息或最小化意外先天倾向的主动主体。相反，信息增益和意外最小化自然出现，因为从随机性中产生秩序正是选择的本质所在。
