基于模型的多巴胺预测。
Model-based predictions for dopamine.
机构信息
Princeton Neuroscience Institute & Department of Psychology, Princeton University, Princeton, NJ 08540, United States.
Princeton Neuroscience Institute & Department of Psychology, Princeton University, Princeton, NJ 08540, United States; National Institute on Drug Abuse, Baltimore, MD 21224, United States; School of Psychology, University of New South Wales, Australia.
出版信息
Curr Opin Neurobiol. 2018 Apr;49:1-7. doi: 10.1016/j.conb.2017.10.006. Epub 2017 Oct 31.
Abstract
Phasic dopamine responses are thought to encode a prediction-error signal consistent with model-free reinforcement learning theories. However, a number of recent findings highlight the influence of model-based computations on dopamine responses, and suggest that dopamine prediction errors reflect more dimensions of an expected outcome than scalar reward value. Here, we review a selection of these recent results and discuss the implications and complications of model-based predictions for computational theories of dopamine and learning.
摘要
相位多巴胺反应被认为编码了与无模型强化学习理论一致的预测误差信号。然而，最近的一些发现强调了基于模型计算对多巴胺反应的影响，并表明多巴胺预测误差反映了预期结果的多个维度，而不仅仅是标量奖励值。在这里，我们回顾了其中的一些最新结果，并讨论了基于模型的预测对多巴胺和学习的计算理论的影响和复杂性。
相似文献
1
Model-based predictions for dopamine.基于模型的多巴胺预测。
Curr Opin Neurobiol. 2018 Apr;49:1-7. doi: 10.1016/j.conb.2017.10.006. Epub 2017 Oct 31.
2
A neural network model with dopamine-like reinforcement signal that learns a spatial delayed response task.一种具有类似多巴胺强化信号的神经网络模型，用于学习空间延迟反应任务。
Neuroscience. 1999;91(3):871-90. doi: 10.1016/s0306-4522(98)00697-6.
3
Evaluation of the hypothesis that phasic dopamine constitutes a cached-value signal.对阶段性多巴胺构成一种缓存值信号这一假说的评估。
Neurobiol Learn Mem. 2018 Sep;153(Pt B):131-136. doi: 10.1016/j.nlm.2017.12.002. Epub 2017 Dec 18.
4
Dopamine errors drive excitatory and inhibitory components of backward conditioning in an outcome-specific manner.多巴胺错误以特定于结果的方式驱动反向条件作用的兴奋性和抑制性成分。
Curr Biol. 2022 Jul 25;32(14):3210-3218.e3. doi: 10.1016/j.cub.2022.06.035. Epub 2022 Jun 24.
5
How we learn to make decisions: rapid propagation of reinforcement learning prediction errors in humans.我们如何学习做决策：强化学习预测错误在人类中的快速传播。
J Cogn Neurosci. 2014 Mar;26(3):635-44. doi: 10.1162/jocn_a_00509. Epub 2013 Oct 29.
6
Predictive reward signal of dopamine neurons.多巴胺神经元的预测性奖励信号。
J Neurophysiol. 1998 Jul;80(1):1-27. doi: 10.1152/jn.1998.80.1.1.
7
The role of learning-related dopamine signals in addiction vulnerability.与学习相关的多巴胺信号在成瘾易感性中的作用。
Prog Brain Res. 2014;211:31-77. doi: 10.1016/B978-0-444-63425-2.00003-9.
8
A distributional code for value in dopamine-based reinforcement learning.多巴胺基强化学习中的价值分布代码。
Nature. 2020 Jan;577(7792):671-675. doi: 10.1038/s41586-019-1924-6. Epub 2020 Jan 15.
9
Optogenetic Blockade of Dopamine Transients Prevents Learning Induced by Changes in Reward Features.光遗传学阻断多巴胺瞬变可防止因奖励特征变化引起的学习。
Curr Biol. 2017 Nov 20;27(22):3480-3486.e3. doi: 10.1016/j.cub.2017.09.049. Epub 2017 Nov 2.
10
Deficient reinforcement learning in medial frontal cortex as a model of dopamine-related motivational deficits in ADHD.中前额叶皮层的强化学习不足可作为 ADHD 中与多巴胺相关的动机缺陷的模型。
Neural Netw. 2013 Oct;46:199-209. doi: 10.1016/j.neunet.2013.05.008. Epub 2013 May 21.
引用本文的文献
1
Social inequity disrupts reward-based learning.社会不平等会扰乱基于奖励的学习。
Commun Psychol. 2025 Aug 16;3(1):125. doi: 10.1038/s44271-025-00300-y.
2
Striatal Gradient in Value-Decay Explains Regional Differences in Dopamine Patterns and Reinforcement Learning Computations.纹状体价值衰减梯度解释了多巴胺模式和强化学习计算中的区域差异。
J Neurosci. 2025 Jul 18. doi: 10.1523/JNEUROSCI.0170-25.2025.
3
Striatal dopamine signals errors in prediction across different informational domains.纹状体多巴胺信号在不同信息领域中预测误差。
Sci Adv. 2025 Jul 11;11(28):eadq9684. doi: 10.1126/sciadv.adq9684. Epub 2025 Jul 9.
4
Modulation of Dopamine Neurons Alters Behavior and Event Encoding in the Nucleus Accumbens during Pavlovian Conditioning.在巴甫洛夫条件反射过程中，多巴胺神经元的调节会改变伏隔核中的行为和事件编码。
J Neurosci. 2025 Jun 25;45(26):e0061252025. doi: 10.1523/JNEUROSCI.0061-25.2025.
5
Developing algorithmic psychiatry via multi-level spanning computational models.通过多层次跨尺度计算模型发展算法精神病学。
Cell Rep Med. 2025 May 20;6(5):102094. doi: 10.1016/j.xcrm.2025.102094. Epub 2025 Apr 28.
6
Prospective contingency explains behavior and dopamine signals during associative learning.前瞻性偶然性解释了联想学习过程中的行为和多巴胺信号。
Nat Neurosci. 2025 Mar 18. doi: 10.1038/s41593-025-01915-4.
7
Devaluing memories of reward: a case for dopamine.贬低奖励记忆：多巴胺的一个实例
Commun Biol. 2025 Feb 3;8(1):161. doi: 10.1038/s42003-024-07440-7.
8
Contextual cues facilitate dynamic value encoding in the mesolimbic dopamine system.情境线索有助于中脑边缘多巴胺系统中的动态价值编码。
Curr Biol. 2025 Feb 24;35(4):746-760.e5. doi: 10.1016/j.cub.2024.12.031. Epub 2025 Jan 23.
9
The curious case of dopaminergic prediction errors and learning associative information beyond value.多巴胺能预测误差与学习价值以外的关联信息的奇妙案例。
Nat Rev Neurosci. 2025 Mar;26(3):169-178. doi: 10.1038/s41583-024-00898-8. Epub 2025 Jan 8.
10
Dopaminergic responses to identity prediction errors depend differently on the orbitofrontal cortex and hippocampus.多巴胺能对身份预测误差的反应在眶额皮质和海马体上的依赖方式有所不同。
bioRxiv. 2024 Dec 17:2024.12.11.628003. doi: 10.1101/2024.12.11.628003.
本文引用的文献
1
Dopamine, Inference, and Uncertainty.多巴胺、推理与不确定性。
Neural Comput. 2017 Dec;29(12):3311-3326. doi: 10.1162/neco_a_01023. Epub 2017 Sep 28.
2
Dopamine Neurons Respond to Errors in the Prediction of Sensory Features of Expected Rewards.多巴胺神经元对预期奖励的感觉特征预测错误做出反应。
Neuron. 2017 Sep 13;95(6):1395-1405.e3. doi: 10.1016/j.neuron.2017.08.025.
3
Ensembles in medial and lateral orbitofrontal cortex construct cognitive maps emphasizing different features of the behavioral landscape.内侧和外侧眶额皮质中的集合构建了强调行为场景不同特征的认知地图。
Behav Neurosci. 2017 Jun;131(3):201-212. doi: 10.1037/bne0000195.
4
Dopamine transients are sufficient and necessary for acquisition of model-based associations.多巴胺瞬变对于基于模型的联想学习而言既是充分的也是必要的。
Nat Neurosci. 2017 May;20(5):735-742. doi: 10.1038/nn.4538. Epub 2017 Apr 3.
5
Midbrain Dopamine Neurons Signal Belief in Choice Accuracy during a Perceptual Decision.中脑多巴胺神经元在知觉决策中对选择准确性的置信度进行信号传递。
Curr Biol. 2017 Mar 20;27(6):821-832. doi: 10.1016/j.cub.2017.02.026. Epub 2017 Mar 9.
6
Dopamine reward prediction errors reflect hidden-state inference across time.多巴胺奖励预测误差反映了跨时间的隐藏状态推理。
Nat Neurosci. 2017 Apr;20(4):581-589. doi: 10.1038/nn.4520. Epub 2017 Mar 6.
7
Dynamic Interaction between Reinforcement Learning and Attention in Multidimensional Environments.多维环境中强化学习与注意力之间的动态交互
Neuron. 2017 Jan 18;93(2):451-463. doi: 10.1016/j.neuron.2016.12.040.
8
Opposite initialization to novel cues in dopamine signaling in ventral and posterior striatum in mice.小鼠腹侧和背侧纹状体中多巴胺信号对新线索的相反初始化。
Elife. 2017 Jan 5;6:e21886. doi: 10.7554/eLife.21886.
9
Midbrain dopamine neurons control judgment of time.中脑多巴胺神经元控制时间判断。
Science. 2016 Dec 9;354(6317):1273-1277. doi: 10.1126/science.aah5234.
10
Temporal Specificity of Reward Prediction Errors Signaled by Putative Dopamine Neurons in Rat VTA Depends on Ventral Striatum.大鼠腹侧被盖区中假定多巴胺能神经元发出的奖励预测误差的时间特异性取决于腹侧纹状体。
Neuron. 2016 Jul 6;91(1):182-93. doi: 10.1016/j.neuron.2016.05.015. Epub 2016 Jun 9.
Zotero 插件