Treuting Robert Louis, Banaie Boroujeni Kianoush, Gerrity Charles Grimes, Neumann Adam, Tiesinga Paul, Womelsdorf Thilo
Department of Biomedical Engineering, Vanderbilt University, Nashville, TN 37240, USA; Vanderbilt Brain Institute, Nashville, TN 372404, USA.
Department of Psychology, Vanderbilt University, Nashville, TN 37240, USA; Princeton Neuroscience Institute, Princeton University, Princeton, NJ 08540, USA.
Neuron. 2025 Aug 20;113(16):2708-2723.e7. doi: 10.1016/j.neuron.2025.05.018. Epub 2025 Jun 10.
Reinforcement learning can benefit from adaptive strategies that adjust exploration-exploitation levels, leverage working memory, or guide attention toward relevant information. We tested how the anterior cingulate cortex (ACC) and the striatum support these processes during learning of feature-based attention at varying feature uncertainty and motivational saliency. Brief, gaze-contingent electrical stimulation affected adaptive reinforcement learning in ACC and the striatum at high feature uncertainty, but in opposite ways. ACC stimulation impaired learning, while striatum stimulation improved learning. Modeling showed that ACC stimulation impaired optimizing exploration and use of prediction errors to reduce uncertainty, while striatum stimulation improved the updating of value expectations. These findings were consistent with neuronal selectivity. In ACC, neurons tracked error history and fired more strongly during more uncertain choices, while in the striatum, neurons fired more strongly during more certain, higher-value choices. These results show that the ACC and the striatum optimize the guidance of exploration toward reward-relevant objects during periods of uncertainty.
强化学习可以从调整探索-利用水平、利用工作记忆或引导注意力指向相关信息的自适应策略中受益。我们测试了前扣带皮层（ACC）和纹状体在不同特征不确定性和动机显著性的基于特征的注意力学习过程中如何支持这些过程。短暂的、基于注视的电刺激在高特征不确定性下影响了ACC和纹状体中的自适应强化学习，但方式相反。ACC刺激损害学习，而纹状体刺激改善学习。建模表明，ACC刺激损害了优化探索和利用预测误差以减少不确定性，而纹状体刺激改善了价值期望的更新。这些发现与神经元选择性一致。在ACC中，神经元跟踪误差历史，在更不确定的选择期间放电更强，而在纹状体中，神经元在更确定、更高价值的选择期间放电更强。这些结果表明，ACC和纹状体在不确定性期间优化了对与奖励相关对象的探索引导。
