Striatal Gradient in Value-Decay Explains Regional Differences in Dopamine Patterns and Reinforcement Learning Computations.

Dopamine has been suggested to encode reward-prediction-error (RPE) in reinforcement learning (RL) theory, but also shown to exhibit heterogeneous patterns depending on regions and conditions: some exhibiting ramping response to predictable reward while others only responding to reward-predicting cue. It remains elusive how these heterogeneities relate to various RL algorithms proposed to be employed by animals/humans, such as RL under predictive state representation, hierarchical RL, and distributional RL. Here we demonstrate that these relationships can be coherently explained by incorporating the decay of learned values (value-decay), implementable by the decay of dopamine-dependent plastic changes in the synaptic strengths. First, we show that value-decay causes ramping RPE under certain state representations but not under others. This accounted for the observed gradual fading of dopamine ramping across repeated reward navigation, attributed to the gradual formation of predictive state representations. It also explained the cue-type and inter-trial-interval-dependent temporal patterns of dopamine. Next, we constructed a hierarchical RL model composed of two coupled systems-one with value-decay and one without. The model accounted for distinct patterns of neuronal activity in parallel striatal-dopamine circuits and their proposed roles in flexible learning and stable habit formation. Lastly, we examined two distinct algorithms of distributional RL with and without value-decay. These algorithms explained how distinct dopamine patterns across striatal regions relate to the reported differences in the strength of distributional coding. These results suggest that within-striatum differences-specifically, a medial-to-lateral gradient in value or synaptic decay-tune regional RL computations by generating distinct patterns of dopamine/RPE signals. Dopamine had been considered to universally represent reward-prediction-error for simple reinforcement learning (RL). However, recent studies revealed that dopamine in fact exhibits various patterns depending on regions and conditions. Simultaneously, it has been shown that animals' value learning cannot be always described by simple RL but rather described by more sophisticated algorithms, namely, RL under particular state representations, hierarchical RL, and distributional RL. A major remaining question is mechanistically how various patterns of dopamine are generated and how they achieve various RL computations in different regions and conditions. We present a novel coherent answer to this, in which the key is regional difference/gradient in the degree of the decay of dopamine-dependent plastic changes in the cortico-striatal synapses that store values.