Tanaka Saori C, Samejima Kazuyuki, Okada Go, Ueda Kazutaka, Okamoto Yasumasa, Yamawaki Shigeto, Doya Kenji
Department of Bioinformatics and Genomics, Nara Institute of Science and Technology, Japan.
Neural Netw. 2006 Oct;19(8):1233-41. doi: 10.1016/j.neunet.2006.05.039. Epub 2006 Sep 18.
In learning goal-directed behaviors, an agent has to consider not only the reward given at each state but also the consequences of dynamic state transitions associated with action selection. To understand brain mechanisms for action learning under predictable and unpredictable environmental dynamics, we measured brain activities by functional magnetic resonance imaging (fMRI) during a Markov decision task with predictable and unpredictable state transitions. Whereas the striatum and orbitofrontal cortex (OFC) were significantly activated both under predictable and unpredictable state transition rules, the dorsolateral prefrontal cortex (DLPFC) was more strongly activated under predictable than under unpredictable state transition rules. We then modelled subjects' choice behaviours using a reinforcement learning model and a Bayesian estimation framework and found that the subjects took larger temporal discount factors under predictable state transition rules. Model-based analysis of fMRI data revealed different engagement of striatum in reward prediction under different state transition dynamics. The ventral striatum was involved in reward prediction under both unpredictable and predictable state transition rules, although the dorsal striatum was dominantly involved in reward prediction under predictable rules. These results suggest different learning systems in the cortico-striatum loops depending on the dynamics of the environment: the OFC-ventral striatum loop is involved in action learning based on the present state, while the DLPFC-dorsal striatum loop is involved in action learning based on predictable future states.
在学习目标导向行为时，智能体不仅要考虑每个状态下给出的奖励，还要考虑与动作选择相关的动态状态转换的后果。为了理解在可预测和不可预测的环境动态下动作学习的大脑机制，我们在一个具有可预测和不可预测状态转换的马尔可夫决策任务中，通过功能磁共振成像（fMRI）测量大脑活动。虽然在可预测和不可预测的状态转换规则下，纹状体和眶额皮质（OFC）均被显著激活，但在可预测的状态转换规则下，背外侧前额叶皮质（DLPFC）的激活程度比不可预测的状态转换规则下更强。然后，我们使用强化学习模型和贝叶斯估计框架对受试者的选择行为进行建模，发现受试者在可预测的状态转换规则下采用了更大的时间折扣因子。基于模型的fMRI数据分析揭示了在不同状态转换动态下纹状体在奖励预测中的不同参与情况。腹侧纹状体在不可预测和可预测的状态转换规则下均参与奖励预测，尽管背侧纹状体在可预测规则下主要参与奖励预测。这些结果表明，根据环境动态，皮质-纹状体回路中存在不同的学习系统：OFC-腹侧纹状体回路参与基于当前状态的动作学习，而DLPFC-背侧纹状体回路参与基于可预测未来状态的动作学习。
