Abstract
In simple probabilistic learning environments, the informational value of corrective feedback gradually declines over time. This is because prediction errors persist despite learners acquiring the contingencies between stimuli and outcomes. An adaptive solution to the problem of unavoidable prediction error is to discount feedback from the learning environment. We provide novel neural evidence of feedback discounting using a combination of behavioral modeling and electroencephalography (EEG). Participants completed a probabilistic categorization task while EEG activity was recorded. We used a model-based analysis of choice behavior to identify individuals that did and did not discount feedback. We then contrasted changes in the feedback-related negativity (FRN) for these two groups. For individuals who did not discount feedback, we observed learning-related reductions in the FRN that reflected incremental changes in choice behavior. By contrast, for individuals who discounted feedback, we found that the FRN was effectively eliminated due to the rapid onset of feedback discounting. The use of a feedback discounting strategy was linked to superior performance on the task, highlighting the adaptive nature of discounting when trial-to-trial outcomes are variable, but the long-term contingencies relating cues and outcomes are stable.
Similar content being viewed by others
Notes
Throughout this article, we follow convention and use the term “amplitude” as a shorthand to describe the signed voltage difference relative to some reference level.
References
Arbel, Y., & Wu, H. (2016). A neurophysiological examination of quality of learning in a feedback-based learning task. Neuropsychologia, 93, 13–20.
Bellebaum, C., & Daum, I. (2008). Learning-related changes in reward expectancy are reflected in the feedback-related negativity. European Journal of Neuroscience, 27, 1823–1835.
Bellebaum, C., Kobza, S., Thiele, S., & Daum, I. (2010). It was not MY fault: event-related brain potentials in active and observational learning from feedback. Cerebral Cortex, 20, 2874–2883.
Blair, M., & Homa, D. L. (2005). Integrating novel dimensions to eliminate category exceptions: when more is less. Journal of Experimental Psychology: Learning, Memory, & Cognition, 31, 258–271.
Bland, A. R., & Schaefer, A. (2011). Electrophysiological correlates of decision making under varying levels of uncertainty. Brain Research, 1417, 55–66.
Bode, S., Bennett, D., Stahl, J., & Murawski, C. (2014). Distributed patterns of event-related potentials predict subsequent ratings of abstract stimulus attributes. PLoS One, 9, e109070.
Brainard, D. H. (1997). The psychophysics toolbox. Spatial Vision, 10, 433–436.
Castellan, N. J. (1973). Multiple-cue probability learning with irrelevant cues. Organizational Behavior & Human Performance, 9, 16–29.
Chase, H. W., Swainson, R., Durham, L., Benham, L., & Cools, R. (2011). Feedback-related negativity codes prediction error but not behavioral adjustment during probabilistic reversal learning. Journal of Cognitive Neuroscience, 23, 936–946.
Cohen, M. X., Elger, C. E., & Ranganath, C. (2007). Reward expectation modulates feedback-related negativity and EEG spectra. NeuroImage, 35(2), 968–978.
Cohen, M. X., Wilmes, K. A., & van de Vijver, I. (2011). Cortical electrophysiological network dynamics of feedback learning. Trends in Cognitive Sciences, 15, 558–566.
Craig, S., & Lewandowsky, S. (2012). Whichever way you choose to categorize, working memory helps you learn. Quarterly Journal of Experimental Psychology, 65, 439–464.
Craig, S., & Lewandowsky, S. (2013). Working memory supports inference learning just like classification learning. Quarterly Journal of Experimental Psychology, 66, 1493–1503.
Craig, S., Lewandowsky, S., & Little, D. R. (2011). Error discounting in probabilistic category learning. Journal of Experimental Psychology: Learning, Memory, & Cognition, 37, 673–687.
Delorme, A., & Makeig, S. (2004). EEGLAB: an open source toolbox for analysis of single-trial EEG dynamics including independent component analysis. Journal of Neuroscience Methods, 134, 9–21.
Edgell, S. E. (1983). Delayed exposure to configural information in nonmetric multiple-cue probability learning. Organizational Behavior & Human Decision Processes, 32, 55–65.
Edgell, S. E., & Morrissey, J. M. (1987). Delayed exposure to additional relevant information in nonmetric multiple-cue probability learning. Organizational Behavior & Human Decision Processes, 40, 22–38.
Edwards, W. (1961). Probability learning in 1000 trials. Journal of Experimental Psychology, 62, 385–394.
Eppinger, B., Kray, J., Mock, B., & Mecklinger, A. (2008). Better or worse than expected? Aging, learning, and the ERN. Neuropsychologia, 46, 521–539.
Erickson, M. A., & Kruschke, J. K. (1998). Rules and exemplars in category learning. Journal of Experimental Psychology: General, 127, 107–140.
Falkenstein, M., Hohnsbein, J., Hoormann, J., & Blanke, L. (1991). Effects of crossmodal divided attention on late ERP components. II. Error processing in choice reaction tasks. Electroencephalography & Clinical Neurophysiology, 78, 447–455.
Forstmann, B. U., Wagenmakers, E. J., Eichele, T., Brown, S., & Serences, J. T. (2011). Reciprocal relations between cognitive neuroscience and formal cognitive models: opposites attract? Trends in Cognitive Sciences, 15, 272–279.
Friedman, D., & Massaro, D. W. (1998). Understanding variability in binary and continuous choice. Psychonomic Bulletin & Review, 5, 370–389.
Gehring, W. J., & Willoughby, A. R. (2002). The medial frontal cortex and the rapid processing of monetary gains and losses. Science, 295, 2279–2282.
Gehring, W. J., Goss, B., Coles, M. G. H., Meyer, D. E., & Donchin, E. (1993). A neural system for error detection and compensation. Psychological Science, 4, 385–390.
Hajcak, G., Moser, J. S., Holroyd, C. B., & Simons, R. F. (2006). The feedback-related negativity reflects the binary evaluation of good versus bad outcomes. Biological Psychology, 71, 148–154.
Hajcak, G., Moser, J. S., Holroyd, C. B., & Simons, R. F. (2007). It’s worse than you thought: the feedback negativity and violations of reward prediction in gambling tasks. Psychophysiology, 44, 905–912.
Holroyd, C. B., & Coles, M. G. H. (2002). The neural basis of human error processing: reinforcement learning, dopamine, and the error-related negativity. Psychological Review, 109, 679–709.
Holroyd, C. B., Krigolson, O. E., Baker, R., Lee, S., & Gibson, J. (2009). When is an error not a prediction error? An electrophysiological investigation. Cognitive, Affective, & Behavioral Neuroscience, 9, 59–70.
Ichikawa, N., Siegle, G. J., Dombrovski, A., & Ohira, H. (2010). Subjective and model-estimated reward prediction: association with the feedback-related negativity (FRN) and reward prediction error in a reinforcement learning task. International Journal of Psychophysiology, 78, 273–283.
Kalish, M. L., Newell, B. R., & Dunn, J. C. (2017). More is generally better: higher working memory capacity does not impair perceptual category learning. Journal of Experimental Psychology: Learning, Memory, & Cognition, 43, 503–514.
Kamin, L. J. (1969). Predictability, surprise, attention, and conditioning. In R. M. Church & B. A. Campbell (Eds.), Punishment and aversive behavior (pp. 279–296). New York: Appleton-Century-Crofts.
Krigolson, O. E., Pierce, L. J., Holroyd, C. B., & Tanaka, J. W. (2009). Learning to become an expert: reinforcement learning and the acquisition of perceptual expertise. Journal of Cognitive Neuroscience, 21, 1834–1841.
Kruschke, J. K., & Johansen, M. K. (1999). A model of probabilistic category learning. Journal of Experimental Psychology: Learning, Memory, & Cognition, 25, 1083–1119.
Lagnado, D. A., Newell, B. R., Kahan, S., & Shanks, D. R. (2006). Insight and strategy in multiple-cue learning. Journal of Experimental Psychology: General, 135, 162–183.
Lewandowsky, S. (2011). Working memory capacity and categorization: individual differences and modeling. Journal of Experimental Psychology: Learning, Memory & Cognition, 37, 720–738.
Lewandowsky, S., Yang, L.-X., Newell, B. R., & Kalish, M. L. (2012). Working memory does not dissociate between different perceptual categorization tasks. Journal of Experimental Psychology: Learning, Memory, & Cognition, 38, 881–904.
Little, D. R., & Lewandowsky, S. (2012). Multiple-cue probability learning. In N. Seel (Ed.), Encyclopedia of the learning sciences (pp. 2386–2388). New York: Springer.
Little, J. L., & McDaniel, M. A. (2015). Individual differences in category learning: memorization versus rule abstraction. Memory & Cognition, 43, 283–297.
Lopez-Calderon, J., & Luck, S. J. (2014). ERPLAB: an open-source toolbox for the analysis of event-related potentials. Frontiers in Human Neuroscience, 8, 1–14.
Love, B. C., & Gureckis, T. M. (2007). Models in search of a brain. Cognitive, Affective, & Behavioral Neuroscience, 7, 90–108.
Luce, R. D. (1963). Detection and recognition. In R. D. Luce, R. R. Bush, & E. Galanter (Eds.), Handbook of mathematical psychology (pp. 103–189). New York: Wiley.
Luck, S. J. (2005). An introduction to the event-related potential technique. Cambridge: MIT Press.
Luft, C. D. B. (2014). Learning from feedback: the neural mechanisms of feedback processing facilitating better performance. Behavioral Brain Research, 261, 356–368.
Luque, D., López, F. J., Marco-Pallares, J., Càmara, E., & Rodríguez-Fornells, A. (2012). Feedback-related brain potential activity complies with basic assumptions of associative learning theory. Journal of Cognitive Neuroscience, 24, 794–808.
McDaniel, M. A., Cahill, M. J., Robbins, M., & Wiener, C. (2014). Individual differences in learning and transfer: stable tendencies for learning exemplars versus abstracting rules. Journal of Experimental Psychology: General, 143, 668–693.
Miltner, W. H. R., Braun, C. H., & Coles, M. G. H. (1997). Event-related brain potentials following incorrect feedback in a time-estimation task: evidence for a “generic” neural system for error detection. Journal of Cognitive Neuroscience, 9, 788–798.
Navarro, D. J., & Newell, B. R. (2014). Information versus reward in a changing world. In P. Bello, M. Guarini, M. McShane, & B. Scassellati (Eds.), Proceedings of the 36th annual conference of the cognitive science society (pp. 1054–1059).
Navarro, D. J., Newell, B. R., & Schulze, C. (2016). Learning and choosing in an uncertain world: an investigation of the explore-exploit dilemma in static and dynamic environments. Cognitive Psychology, 85, 43–77.
Nieuwenhuis, S., Holroyd, C. B., Mol, N., & Coles, M. G. H. (2004a). Reinforcement-related brain potentials from medial frontal cortex: origins and functional significance. Neuroscience & Biobehavioral Reviews, 28, 441–448.
Nieuwenhuis, S., Yeung, N., Holroyd, C. B., Schurger, A., & Cohen, J. D. (2004b). Sensitivity of electrophysiological activity from medial frontal cortex to utilitarian and performance feedback. Cerebral Cortex, 14, 741–747.
Pelli, D. G. (1997). The VideoToolbox software for visual psychophysics: transforming numbers into movies. Spatial Vision, 10, 437–442.
Pietschmann, M., Simon, K., Endrass, T., & Kathmann, N. (2008). Changes of performance monitoring with learning in older and younger adults. Psychophysiology, 45, 559–568.
Potts, G. F., Martin, L. E., Kamp, S.-M., & Donchin, E. (2011). Neural response to action and reward prediction errors: comparing the error-related negativity to behavioral errors and the feedback-related negativity to reward prediction violations. Psychophysiology, 48, 218–228.
Rakow, T., Newell, B. R., & Zougkou, K. (2010). The role of working memory in information acquisition and decision making: lessons from the binary prediction task. Quarterly Journal of Experimental Psychology, 63, 1335–1360.
Rescorla, R. A., & Wagner, A. R. (1972). A theory of Pavlovian conditioning: variations in the effectiveness of reinforcement and non-reinforcement. In A. H. Black & W. F. Prokasy (Eds.), Classical conditioning II: current research and theory (pp. 64–99). NewYork: Appleton-Century-Crofts.
Sailer, U., Fischmeister, F. P. S., & Bauer, H. (2010). Effects of learning on feedback-related brain potentials in a decision-making task. Brain Research, 1342, 85–93.
Sambrook, T. D., & Goslin, J. (2015). A neural reward prediction error revealed by a meta-analysis of ERPs using great grand averages. Psychological Bulletin, 141, 213–235.
Schiffer, A.-M., Siletti, K., Waszak, F., & Yeung, N. (2017). Adaptive behaviour and feedback processing integrate experience and instruction in reinforcement learning. NeuroImage, 146, 626–641.
Schulze, C., & Newell, B. R. (2016). Taking the easy way out? Increasing implementation effort reduces probability maximizing under cognitive load. Memory & Cognition, 44, 806–818.
Sewell, D. K., & Lewandowsky, S. (2012). Attention and working memory capacity: insights from blocking, highlighting, and knowledge restructuring. Journal of Experimental Psychology: General, 141, 444–469.
Shanks, D. R., Tunney, R. J., & McCarthy, J. D. (2002). A re-examination of probability matching and rational choice. Journal of Behavioral Decision Making, 15, 233–250.
Stahl, J. (2010). Error detection and the use of internal and external error indicators: an investigation of the first-indicator hypothesis. International Journal of Psychophysiology, 77, 43–52.
Takeda, Y., Yamanaka, K., & Yamamoto, Y. (2008). Temporal decomposition of EEG during a simple reaction time task into stimulus- and response-locked components. NeuroImage, 39, 742–754.
Walsh, M. M., & Anderson, J. R. (2011). Modulation of the feedback-related negativity by instruction and experience. Proceedings of the National Academy of Science, 108, 19048–19053.
Walsh, M. M., & Anderson, J. R. (2012). Learning from experience: event-related potential correlates of reward processing, neural adaptation, and behavioral choice. Neuroscience & Biobehavioral Reviews, 36, 1870–1884.
Yeung, N., & Sanfey, A. G. (2004). Independent coding of reward magnitude and valence in the human brain. Journal of Neuroscience, 24, 6258–6264.
Acknowledgements
We thank Kashmira Daruwalla and Maggie Webb for assistance during data collection.
Funding
This research was supported by Australian Research Council Discovery Early Career Researcher Awards to David Sewell (DE140100772) and Stefan Bode (DE140100350).
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
The study was approved by the Human Research Ethics Committee at the University of Melbourne and was conducted in accordance with the Declaration of Helsinki.
Rights and permissions
About this article
Cite this article
Sewell, D.K., Warren, H.A., Rosenblatt, D. et al. Feedback Discounting in Probabilistic Categorization: Converging Evidence from EEG and Cognitive Modeling. Comput Brain Behav 1, 165–183 (2018). https://doi.org/10.1007/s42113-018-0012-6
Published:
Issue Date:
DOI: https://doi.org/10.1007/s42113-018-0012-6