An 'unexpected' outcome: many parts of our brain, not just a few areas, learn and adapt from unanticipated results

or, technically, 
Expectation modulates neural representations of valence throughout the human brain [See the original abstract on PubMed]

Authors: Ashwin G. Ramayya, Isaac Pedisich, Michael J. Kahana

Brief prepared by: Ryan G. Natan
Brief approved by: Shivon Robinson
Section Chief: Isaac Perron
Date posted: March 24, 2017 

Brief in Brief (TL;DR)

What do we know: We try to make choices that will lead to the best outcomes, but when results are different than expected, we learn from our experience. Research has shown that the frontal cortex, the brain region in control of decision making, detects these unexpected outcomes. 

What don’t we know: The frontal cortex uses our experiences—expected or unexpected—to learn, thus shaping our future decisions. However, recent research has shown that many more brain regions are capable of detecting expected negative and positive outcomes. Therefore, it is possible that brain regions beyond the frontal cortex could also detect unexpected outcomes. 

What this study shows: The scientists found that many parts of the cerebral cortex respond to unexpected outcomes, which is surprising because these brain regions are not directly involved in decision making. This study suggests that these brain regions also play a different role in learning from experience. 

What we can do in the future because of this study: Next, we can test if these brain regions actually contribute to learning from experience, and whether these different brain regions learn differently. This would help us figure out how the brain breaks down information in order to learn. 

Why you should care: In many forms of mental illness, people lose the ability to learn from their experience, leading to poor decision making. In order to figure out how to help these patients, it is important to understand how healthy individuals learn. Studies like these help us figure out what parts of the brain might be malfunctioning during mental illness, and point us to brain areas that need treatment.

Brief for Non-Neuroscientists

We typically choose behaviors based on what we expect will lead to the best outcomes. Reinforcement learning occurs when we encounter an unexpected outcome; if we expect a large reward but instead receive punishment or trivial reward, we use this information to update our beliefs and shape our future behavior. While human subjects learn to play a game involving high- and low-probability outcomes, the unexpected outcomes lead to strong activity in brain regions that control decision making, such as the frontal cortex. In other words, frontal cortex 'encodes' unexpected outcomes and uses this information to guide reinforcement learning. 

However, a recent experiment demonstrated that expected rewards and punishments lead to strong activity in many regions throughout the brain, not just the frontal cortex, opening the possibility that other brain regions are also involved in reinforcement learning. To test if they also encode unexpected outcomes, Ramayya et al. recorded electrical signals from the cortex of epilepsy patients, who already have an array of electrodes directly contacting their brain for medical treatment, while they learned a probabilistic choice game. Similar to the previous findings, 19 out of 21 cortical regions across the brain encoded expected rewards or penalties. More interestingly, 9 of the regions encoded unexpected outcomes, including regions of the cortex not thought to be directly related to decision making. Specifically, regions within the occipital, parietal and temporal lobes (classically considered important for vision, sensation, and language/memory, respectively) showed heightened brain activity during unexpected outcomes in this task. This experiment will shape future work in the field because it shows that many more brain regions may be involved in reinforcement learning than previously thought.

Brief for Neuroscientists

We typically choose behaviors based on what we expect will lead to the best outcomes, and reinforcement learning supports our ability to improve these choices. During unexpected outcomes, reward prediction error (i.e., the difference between the predicted result and the actual result) represent the strongest driver of reinforcement learning. While subjects perform active learning tasks, functional MRI (fMRI) and single neuron studies show that frontal cortex and other brain regions associated with executive control appear to encode reward prediction error. These results point to exclusive control of reinforcement learning by the executive control network. However, recent fMRI studies demonstrate that expected reward and punishment signals are present throughout the brain, raising the possibility that non-executive control regions may support reinforcement learning by encoding reward prediction error. To test this, Ramayya et al. used intracranial electroencephalography to measure gamma-band oscillatory activity (70-200 Hz) in epilepsy patients while they learned a probabilistic two-alternative forced-choice task. Supporting previous findings, 50% of the 4,306 recording sites and 19 out of 21 cortical regions encoded expected rewards or penalties. Confirming the authors' hypothesis, 10% of recording sites and 9 brain regions encoded reward prediction error, including regions in the occipital, parietal, and temporal lobes. Further, the strength of the reward prediction error signals strongly correlated with the subjects' task performance. These results demonstrate that learning related signals are distributed widely across the brain, beyond the executive control network. This study will shape future work in the field by widening the brain regions studied toward understanding reinforcement learning.