Joel Daphna, Niv Yael, Ruppin Eytan
Department of Psychology, Tel-Aviv University, Ramat-Aviv, Israel.
Neural Netw. 2002 Jun-Jul;15(4-6):535-47. doi: 10.1016/s0893-6080(02)00047-3.
A large number of computational models of information processing in the basal ganglia have been developed in recent years. Prominent in these are actor-critic models of basal ganglia functioning, which build on the strong resemblance between dopamine neuron activity and the temporal difference prediction error signal in the critic, and between dopamine-dependent long-term synaptic plasticity in the striatum and learning guided by a prediction error signal in the actor. We selectively review several actor-critic models of the basal ganglia with an emphasis on two important aspects: the way in which models of the critic reproduce the temporal dynamics of dopamine firing, and the extent to which models of the actor take into account known basal ganglia anatomy and physiology. To complement the efforts to relate basal ganglia mechanisms to reinforcement learning (RL), we introduce an alternative approach to modeling a critic network, which uses Evolutionary Computation techniques to 'evolve' an optimal RL mechanism, and relate the evolved mechanism to the basic model of the critic. We conclude our discussion of models of the critic by a critical discussion of the anatomical plausibility of implementations of a critic in basal ganglia circuitry, and conclude that such implementations build on assumptions that are inconsistent with the known anatomy of the basal ganglia. We return to the actor component of the actor-critic model, which is usually modeled at the striatal level with very little detail. We describe an alternative model of the basal ganglia which takes into account several important, and previously neglected, anatomical and physiological characteristics of basal ganglia-thalamocortical connectivity and suggests that the basal ganglia performs reinforcement-biased dimensionality reduction of cortical inputs. We further suggest that since such selective encoding may bias the representation at the level of the frontal cortex towards the selection of rewarded plans and actions, the reinforcement-driven dimensionality reduction framework may serve as a basis for basal ganglia actor models. We conclude with a short discussion of the dual role of the dopamine signal in RL and in behavioral switching.
近年来，已经开发了大量关于基底神经节信息处理的计算模型。其中突出的是基底神经节功能的行动者-评判者模型，该模型基于多巴胺神经元活动与评判者中的时间差预测误差信号之间的强烈相似性，以及纹状体中多巴胺依赖的长期突触可塑性与行动者中由预测误差信号引导的学习之间的相似性。我们选择性地回顾了几个基底神经节的行动者-评判者模型，重点关注两个重要方面：评判者模型再现多巴胺发放时间动态的方式，以及行动者模型考虑已知基底神经节解剖学和生理学的程度。为了补充将基底神经节机制与强化学习（RL）联系起来的努力，我们引入了一种对评判网络进行建模的替代方法，该方法使用进化计算技术来“进化”出一种最优的RL机制，并将进化出的机制与评判者的基本模型联系起来。我们通过对基底神经节电路中评判者实现的解剖学合理性进行批判性讨论来结束对评判者模型的讨论，并得出结论，这种实现基于与基底神经节已知解剖学不一致的假设。我们回到行动者-评判者模型的行动者部分，该部分通常在纹状体水平上建模，细节很少。我们描述了一种基底神经节的替代模型，该模型考虑了基底神经节-丘脑皮质连接的几个重要且先前被忽视的解剖学和生理学特征，并表明基底神经节对皮质输入进行强化偏向的降维。我们进一步表明，由于这种选择性编码可能会使额叶皮质水平的表征偏向于选择有奖励的计划和行动，强化驱动的降维框架可能作为基底神经节行动者模型的基础。我们最后简要讨论了多巴胺信号在RL和行为切换中的双重作用。
