Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Review Article
  • Published:

Computational roles for dopamine in behavioural control

Nature volume 431, pages 760–767 (2004)Cite this article

Abstract

Neuromodulators such as dopamine have a central role in cognitive disorders. In the past decade, biological findings on dopamine function have been infused with concepts taken from computational theories of reinforcement learning. These more abstract approaches have now been applied to describe the biological algorithms at play in our brains when we form value judgements and make choices. The application of such quantitative models has opened up new fields, ripe for attack by young synthesizers and theoreticians.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Figure 1: TD prediction-error signal encoded in dopamine neuron firing.
Figure 2: Equating incentive salience with the actor–critic model.
Figure 3: Scaled responses to a monetary reward in the ventral striatum.
Figure 4: Detecting actor and critic signals in the human brain using fMRI.
Figure 5: The flow and transformation of signals carried by the dopaminergic system.

Similar content being viewed by others

References

  1. Sutton, R. S. & Barto, A. G. Reinforcement learning (MIT, Cambridge, Massachusetts, 1998).

    Google Scholar 

  2. Montague, P. R., Dayan, P. & Sejnowski, T. J. A framework for mesencephalic dopamine systems based on predictive Hebbian learning. J. Neurosci. 16, 1936–1947 (1996).

    Article  CAS  Google Scholar 

  3. Schultz, W., Dayan, P. & Montague, P. R. A neural substrate of prediction and reward. Science 275, 1593–1599 (1997).

    Article  CAS  Google Scholar 

  4. Friston, K. J., Tononi, G., Reeke, G. N., Sporns, O. & Edelman, G. M. Value-dependent selection in the brain: simulation in a synthetic neural model. Neuroscience 59, 229–243 (1994).

    Article  CAS  Google Scholar 

  5. Houk, J. C., Adams, J. L. & Barto, A. G. in Models of Information Processing in the Basal Ganglia (eds Houk, J. C. Davis, J. L. & Beiser, D. G.) Ch. 13, 249–270 (MIT, Cambridge, Massachusetts, 1995).

    Google Scholar 

  6. Skinner, B. F. Behaviorism at fifty. Science 140, 951–958 (1963).

    Article  ADS  CAS  Google Scholar 

  7. Sutton, R. S. Learning to predict by the methods of temporal difference. Mach. Learn. 3, 9–44 (1988).

    Google Scholar 

  8. Doya, K. Metalearning and neuromodulation. Neural Netw. 15, 495–506 (2002).

    Article  Google Scholar 

  9. Dayan, P. & Abbott, L. F. Theoretical Neuroscience Ch. 9, 331–358 (MIT, Cambridge, Massachusetts, 2001).

    MATH  Google Scholar 

  10. Rescorla, R. A. & Wagner A. R. in Classical Conditioning 2: Current Research and Theory (eds Black, A. H. & Prokasy, W. F.) 64–69 (Appleton Century-Crofts, New York, 1972).

    Google Scholar 

  11. Bertsekas, D. P. & Tsitsiklis, J. N. in Neuro-Dynamic Programming (Athena Scientific, Belmont, Massachusetts, 1996).

    MATH  Google Scholar 

  12. Schultz W., Apicella, P. & Ljungberg, T. Responses of monkey dopamine neurons to reward and conditioned stimuli during successive steps of learning a delayed response task. J. Neurosci. 13, 900–913 (1993).

    Article  CAS  Google Scholar 

  13. Hollerman, J. R. & Schultz, W. Dopamine neurons report an error in the temporal prediction of reward during learning. Nature Neurosci. 1, 304–309 (1998).

    Article  CAS  Google Scholar 

  14. Schultz, W. Predictive reward signal of dopamine neurons. J. Neurophysiol. 80, 1–27 (1998).

    Article  CAS  Google Scholar 

  15. Waelti, P., Dickinson, A. & Schultz, W. Dopamine responses comply with basic assumptions of formal learning theory. Nature 412, 43–48 (2001).

    Article  ADS  CAS  Google Scholar 

  16. Bayer, H. M. & Glimcher, P. W. Subjective estimates of objective rewards: using economic discounting to link behavior and brain. Soc. Neurosci. Abstr. 28, 358.6 (2002).

    Google Scholar 

  17. Berridge, K. C. & Robinson, T. E. What is the role of dopamine in reward: hedonic impact, reward learning, or incentive salience? Brain Res. Rev. 28, 309–369 (1998).

    Article  CAS  Google Scholar 

  18. Everitt, B. J. et al. Associative processes in addiction and reward: the role of amygdala-ventral striatal subsystems. Ann. NY Acad. Sci. 877, 412–438 (1999).

    Article  ADS  CAS  Google Scholar 

  19. Ikemoto, S. & Panksepp, J. The role of nucleus accumbens dopamine in motivated behavior: a unifying interpretation with special reference to reward-seeking. Brain Res. Rev. 31, 6–41 (1999).

    Article  CAS  Google Scholar 

  20. Di Chiara, G. & Imperato, A. Drugs abused by humans preferentially increase synaptic dopamine concentrations in the mesolimbic system of freely moving rats. Proc. Natl Acad. Sci. USA 85, 5274–5278 (1988).

    Article  ADS  CAS  Google Scholar 

  21. Berke, J. D. & Hyman, S. E. Addiction, dopamine, and the molecular mechanisms of memory. Neuron 25, 515–532 (2000).

    Article  CAS  Google Scholar 

  22. Ikemoto, S. & Panksepp, J. Dissociations between appetitive and consummatory responses by pharmacological manipulations of reward-relevant brain regions. Behav. Neurosci. 110, 331–345 (1996).

    Article  CAS  Google Scholar 

  23. Salamone, J. D. & Correa, M. Motivational views of reinforcement: implications for understanding the behavioral functions of nucleus accumbens dopamine. Behav. Brain Res. 137, 3–25 (2002).

    Article  CAS  Google Scholar 

  24. Redgrave, P., Prescott, T. J. & Gurney, K. Is the short-latency dopamine response too short to signal reward error? Trends Neurosci. 22, 146–151 (1999).

    Article  CAS  Google Scholar 

  25. Egelman, D. M., Person, C., Montague, P. R. A computational role for dopamine delivery in human decision-making. J. Cogn. Neurosci. 10, 623–630 (1998).

    Article  CAS  Google Scholar 

  26. McClure, S. M., Daw, N. & Montague, P. R. A computational substrate for incentive salience. Trends Neurosci. 26, 423–428 (2003).

    Article  CAS  Google Scholar 

  27. Balleine, B. W. & Dickinson, A. The effect of lesions of the insular cortex on instrumental conditioning: evidence for a role in incentive memory. Neurosci. 20, 8954–8964 (2000).

    Article  CAS  Google Scholar 

  28. Berridge, K. C. in The Psychology of Learning and Motivation: Advances in Research and Theory Vol. 40 (ed. Medin, D. L.) 223–278 (Academic, San Diego, 2001).

    Google Scholar 

  29. Dayan, P. & Balleine, B. W. Reward, motivation and reinforcement learning. Neuron 36, 285–298 (2002).

    Article  CAS  Google Scholar 

  30. Berns, G. S., McClure, S. M., Pagnoni, G. & Montague, P. R. Predictability modulates human brain response to reward. J. Neurosci. 21, 2793–2798 (2001).

    Article  CAS  Google Scholar 

  31. O'Doherty, J. P., Deichmann, R., Critchley, H. D. & Dolan, R. J. Neural responses during anticipation of a primary taste reward. Neuron 33, 815–826 (2002).

    Article  CAS  Google Scholar 

  32. O'Doherty, J. P., Dayan, P., Friston, K., Critchley, H. & Dolan, R. J. Temporal difference models and reward related learning in the human brain. Neuron 38, 329–337 (2003).

    Article  CAS  Google Scholar 

  33. McClure, S. M., Berns, G. S. & Montague, P. R. Temporal prediction errors in a passive learning task activate human striatum. Neuron 38, 339–346 (2003).

    Article  CAS  Google Scholar 

  34. O'Doherty, J. P. et al. Dissociable roles of ventral and dorsal striatum in instrumental conditioning. Science 304, 452–454 (2004).

    Article  ADS  CAS  Google Scholar 

  35. Aharon, I. et al. Beautiful faces have variable reward value: fMRI and behavioral evidence. Neuron 32, 537–551 (2001).

    Article  CAS  Google Scholar 

  36. Breiter, H. C. et al. Acute effects of cocaine on human brain activity and emotion. Neuron 19, 591–611 (1997).

    Article  CAS  Google Scholar 

  37. Breiter, H. C., Aharon, I., Kahneman, D., Dale, A. & Shizgal, P. Functional imaging of neural responses to expectancy and experience of monetary gains and losses. Neuron 30, 619–639 (2001).

    Article  CAS  Google Scholar 

  38. Knutson, B., Westdorp, A., Kaiser, E. & Hommer, D. fMRI visualization of brain activity during a monetary incentive delay task. Neuroimage 12, 20–27 (2000).

    Article  CAS  Google Scholar 

  39. Knutson, B., Adams, C. M., Fong, G. W. & Hommer, D. J. Anticipation of increasing monetary reward selectively recruits nucleus accumbens. J. Neurosci. 15, 1–5 (2001).

    Google Scholar 

  40. Thut, G. et al. Activation of the human brain by monetary reward. Neuroreport 8, 1225–1228 (1997).

    Article  CAS  Google Scholar 

  41. Delgado, M. R., Nystrom, L. E., Fissel, C., Noll, D. C. & Fiez, J. A. Tracking the hemodynamic responses to reward and punishment in the striatum. J. Neurophysiol. 84, 3072–3077 (2000).

    Article  CAS  Google Scholar 

  42. Elliott, R., Friston, K. J. & Dolan, R. J. Dissociable neural responses in human reward systems. J. Neurosci. 20, 6159–6165 (2000).

    Article  CAS  Google Scholar 

  43. Montague, P. R. & Berns, G. S. Neural economics and the biological substrates of valuation. Neuron 36, 265–284 (2002).

    Article  CAS  Google Scholar 

  44. Pagnoni, G., Zink, C. F., Montague, P. R. & Berns, G. S. Activity in human ventral striatum locked to errors of reward prediction. Nature Neurosci. 5, 97–98 (2002).

    Article  CAS  Google Scholar 

  45. Gehring, W. J., Goss, B., Coles, M. G. H., Meyer, D. E. & Donchin, E. A neural system for error detection and compensation. Psychol. Sci. 4, 385–390 (1993).

    Article  Google Scholar 

  46. Falkenstein, M., Hohnsbein, J. & Hoormann, J. in Perspectives of Event-Related Potentials Research (eds Karmos, G. et al.) 287–296 (Elsevier Science, Amsterdam, 1994).

    Google Scholar 

  47. Gehring, W. J. & Willoughby, A. R. The medial frontal cortex and the rapid processing of monetary gains and losses. Science 295, 2279–2282 (2002).

    Article  ADS  CAS  Google Scholar 

  48. Ullsperger, M. & von Cramon, D. Y. Error monitoring using external feedback: specific roles of the habenular complex, the reward system, and the cingulate motor area revealed by functional magnetic resonance imaging. J. Neurosci. 23, 4308–4314 (2003).

    Article  CAS  Google Scholar 

  49. Nieuwenhuis, S., Yeung, N., Holroyd, C. B., Schurger, A. & Cohen, J. D. Sensitivity of electrophysiological activity from medial frontal cortex to utilitarian and performance feedback. Cereb. Cort. 14, 741–747 (2004).

    Article  Google Scholar 

  50. Holroyd, C. B. & Coles, M. G. The neural basis of human error processing: reinforcement learning, dopamine, and the error-related negativity. Psychol. Rev. 109, 679–709 (2002).

    Article  Google Scholar 

  51. Holroyd, C. B., Nieuwenhuis, S., Yeung, N. & Cohen, J. D. Errors in reward prediction are reflected in the event-related brain potential. Neuroreport 14, 2481–2484 (2003).

    Article  Google Scholar 

  52. Holroyd, C. B., Larsen, J. T. & Cohen, J. D. Context dependence of the event-related brain potential associated with reward and punishment. Psychophysiol. 41, 245–253 (2004).

    Article  Google Scholar 

  53. Holroyd, C. B. et al. Dorsal anterior cingulate cortex shows fMRI response to internal and external error signals. Nature Neurosci. 7, 497–498 (2004).

    Article  CAS  Google Scholar 

  54. Miller, E. K. & Cohen, J. D. An integrative theory of prefrontal cortex function. Annu. Rev. of Neurosci. 24, 167–202 (2001).

    Article  CAS  Google Scholar 

  55. 'Reilly, R. C., Braver, T. S. & Cohen, J. D. in Models of Working Memory: Mechanisms of Active Maintenance and Executive Control (eds Miyake, A. & Shah, P.) Ch. 11, 375–411 (Cambridge Univ. Press, New York, 1999).

    Book  Google Scholar 

  56. Miller, E. K., Li, L. & Desimone, R. A neural mechanism for working and recognition memory in inferior temporal cortex. Science 254, 1377–1379 (1991).

    Article  ADS  CAS  Google Scholar 

  57. Miller, E. K., Erickson, C. A. & Desimone, R. Neural mechanisms of visual working memory in prefrontal cortex of the macaque. J. Neurosci. 16, 5154–5167 (1996).

    Article  CAS  Google Scholar 

  58. Duncan, J. Disorganization of behavior after frontal lobe damage. Cog. Neuropsychol. 3, 271–290 (1986).

    Article  Google Scholar 

  59. Shallice, T. in From Neuropsychology to Mental Structure (Cambridge Univ. Press, Cambridge, 1988).

    Book  Google Scholar 

  60. Koechlin, E., Ody, C. & Kouneiher, F. The architecture of cognitive control in the human prefrontal cortex. Science 302, 1181–1185 (2003).

    Article  ADS  CAS  Google Scholar 

  61. Stuss, D. T. & Knight, R. T. Principles of Frontal Lobe Function (Oxford Univ. Press, New York, 2002).

    Book  Google Scholar 

  62. Braver, T. S. & Cohen, J. D. in Attention and Performance XVIII; Control of Cognitive Processes (eds Monsell, S. & Driver, J.) 713–737 (MIT, Cambridge, Massachusetts, 2000).

    Google Scholar 

  63. Daw, N. D., Kakade, S. & Dayan, P. Opponent interactions between serotonin and dopamine. Neural Netw. 15, 603–616 (2002).

    Article  Google Scholar 

  64. O'Reilly, R. C., Noelle, D. C., Braver, T. S. & Cohen, J. D. Prefrontal cortex and dynamic categorization tasks: representational organization and neuromodulatory control. Cereb. Cort. 12, 246–257 (2002).

    Article  Google Scholar 

  65. Rougier, N. P. & O'Reilly, R. C. Learning representations in a gated prefrontal cortex model of dynamic task switching. Trends Cogn. Sci. 26, 503–520 (2002).

    Google Scholar 

  66. Wise, R. A. & Bozarth, M. A. A psychomotor stimulant theory of addiction. Psychol. Rev. 94, 469–492 (1987).

    Article  CAS  Google Scholar 

  67. Hyman, S. E. & Malenka, R. C. Addiction and the brain: the neurobiology of compulsion and its persistence. Nature Rev. Neurosci. 2, 695–703 (2001).

    Article  CAS  Google Scholar 

  68. Potenza, M. N. et al. Gambling urges in pathological gambling: a functional magnetic resonance imaging study. Arch. Gen. Psych. 60, 828–836 (2003).

    Article  Google Scholar 

  69. Cohen, B. Dopamine receptors and antipsychotic drugs. Mclean Hosp. J. 6, 95–115 (1981).

    ADS  Google Scholar 

  70. Weinberger, D. R. Implications of normal brain development for the pathogenesis of schizophrenia. Arch. Gen. Psych. 44, 660–669 (1987).

    Article  CAS  Google Scholar 

  71. Servan-Schreiber, D., Printz, H. & Cohen, J. D. A network model of catecholamine effects: gain, signal-to-noise ratio and behavior. Science 249, 892–895 (1990).

    Article  ADS  CAS  Google Scholar 

  72. Montague, P. R. et al. Dynamic gain control of dopamine delivery in freely moving animals. J. Neurosci. 24, 1754–1759 (2004).

    Article  CAS  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Neuroscience, Baylor College of Medicine, 1 Baylor Plaza, Houston, 77030, Texas, USA

    P. Read Montague

  2. Menninger Department of Psychiatry and Behavioral Sciences, Baylor College of Medicine, 1 Baylor Plaza, Houston, 77030, Texas, USA

    P. Read Montague

  3. Harvard University, Cambridge, 02138, Massachusetts, USA

    Steven E. Hyman

  4. Department of Psychiatry, University of Pittsburgh, Princeton, 08544, New Jersey, USA

    Jonathan D. Cohen

  5. Department of Psychology, Center for the Study of Brain, Mind & Behavior, Green Hall, Princeton University, Princeton, 08544, New Jersey, USA

    Jonathan D. Cohen

Authors
  1. P. Read Montague
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Steven E. Hyman
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jonathan D. Cohen
    View author publications

    You can also search for this author in PubMed Google Scholar

Ethics declarations

Competing interests

The authors declare no competing financial interests.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Montague, P., Hyman, S. & Cohen, J. Computational roles for dopamine in behavioural control. Nature 431, 760–767 (2004). https://doi.org/10.1038/nature03015

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/nature03015

This article is cited by

Comments

By submitting a comment you agree to abide by our Terms and Community Guidelines. If you find something abusive or that does not comply with our terms or guidelines please flag it as inappropriate.

Search

Advanced search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing