How does the brain transform our memories during sleep?

About the author: Liz Siefert

Liz is a 4th year PhD candidate working with Dr. Anna Schapiro and Dr. Brett Foster. She is interested in how our memories change and shift overtime, and the role of arousal states (wake, sleep) in these changes.

or technically,

Memory reactivation during sleep does not act holistically on object memory

[See Original Abstract on Pubmed]

Authors of the study: Elizabeth M. Siefert, Sindhuja Uppuluri, Jianing Mu, Marlie C. Tandoc, James W. Antony, Anna C. Schapiro

Whether you’re a human, dog, or fruit fly, we all sleep. There’s no doubt that sleep is essential, but why we sleep is still hotly debated. Many ideas have centered around the observation that sleep is important for strengthening key memories and putting them into storage. Neuroscientists believe this happens during sleep when the brain replays memories to strengthen them, a process called memory reactivation. Supporting this idea, numerous studies have shown that people tend to do better if they take even a short nap between learning something and being tested on it.

NGG student Elizabeth Siefert was fascinated by the process of memory reactivation during sleep, but wanted to better understand what impact it has on our memories. She noticed that sleep can’t just be improving memory overall, because that’s not how memory works. “As we go about our lives, our memories don’t always continue to improve,” says Siefert. “Often, they get worse, or they change in different ways. So really, across time, memory isn’t just improving, but it’s transforming.” Following on these observations, Siefert designed an experiment to understand whether sleep simply improves memory or rather has the power to transform it by strengthening some aspects of a memory while allowing others to fade.

Studying the sleeping brain

To probe the nature of memory transformations during sleep, Siefert and her team needed a memory test that allowed them to assess different aspects of memory. “Memories have lots of different features, so we wanted to know if memory reactivation has the power to act on different features of our memories in different ways,” says Siefert. She did this by asking participants to learn the identities of several satellites belonging to three groups. The satellites were created by mixing different parts so that the team could control the different features of a memory (Figure 1). Some shared features appeared on multiple satellites within a group, but never on satellites in other groups. Other unique features were specific to just one satellite, allowing the participants to identify it by name. This allowed the team to look at how representations of the individual versus the shared features were transformed in memory during sleep. Siefert asked the participants to learn the satellite names and groups, had them take a nap, and then tested how well they remembered the unique versus shared satellite features.

Just because participants took a nap doesn’t mean they were necessarily going to replay their memories of the satellites. That’s why Siefert stepped in to nudge their sleeping brains to replay the memories she was interested in. She did this using a method called targeted memory reactivation (TMR). During TMR, experimenters measure a participant’s brain activity while they sleep and look for moments when the sleeping brain is most likely to replay a memory. When those moments are identified, the experimenter plays a cue to remind them of a recent memory and encourage the brain to replay that specific memory. The cues are played softly enough that they don’t wake up the participant, and participants don’t know that the cues are being played, so any impacts of TMR are unconscious. “The best method we know in humans for studying memory reactivation in sleep is TMR,” says Siefert. “Our sleeping brain is already prioritizing certain information, and what we’re doing with TMR is biasing the brain to prioritize what we want it to.” In Siefert’s experiment, she played the name of some of the satellites the participant had just learned to encourage reactivation of those memories.

Figure 1. Satellites used in the memory task. Satellites belonged to three groups: alpha, beta, or gamma. The purple and orange boxes highlight the shared and unique features of the satellite volar. The purple boxes show volar’s features that are shared with other satellites in the alpha category, while the orange boxes show the features that are unique to volar.

Clarifying the relationship between sleep and memory

With all the pieces in place, Siefert was ready to ask how memory replay during sleep impacted different features of memory. Before the nap, participants tended to do a better job learning the satellites’ unique features compared to their shared features. In other words, they were better able to identify that a particular feature belonged to an individual satellite than that a particular feature was shared by satellites of a certain group. After a nap with TMR, that divide only widened. Siefert found that TMR increased participants’ memory for unique features while decreasing their memory for shared features. For satellites whose names were not used as cues during sleep, there was no impact on memory. This demonstrates that the effects were likely due to the reactivation encouraged by TMR.  “That showed us that memory reactivation during sleep, it’s not just wholistically improving our memory, it has the power to act on specific features of our memory in very specific ways,” says Siefert. “It [can] even impact different features of our memory in different ways, such as improving some even at the cost of others.”

Why were certain features strengthened while others got weaker with reactivation? One possibility is that the cues used to trigger replay during sleep were the satellite names. This may have encouraged the brain to prioritize information about the individuals’ identity rather than their group membership. Future studies could use the group names as a cue instead and see if that nudges the brain to prioritize shared over unique features. Another possibility is the fact that people already tended to learn the unique features better before the nap. That prioritization may have carried over into their sleeping brains. “Your learning strategy and goals before sleep might bias what type of information the sleeping brain wants to reactivate,” says Siefert. “It’s possible that it was those learning strategies that led that information to be benefitted more by sleep.” Importantly, no matter what the explanation, it’s not the case that unique features will always be remembered over shared features following a night of sleep. Instead, a complex combination of learning goals and strategies likely shape exactly how memories are transformed during sleep.

Unbeknownst to the participants, Siefert was also testing how the way she presented the cues during sleep impacted memory. Sometimes she played the same satellite name over and over in a block, and other times she played satellite names intermixed with each other. Importantly, she always played the satellite names the same number of times during sleep, only changing in what order she played them. While both methods led to improved memory for unique over shared features, repeating the same satellite name many times in a row was more effective in transforming memory than playing the names in a random order. Siefert suggests that this may be evidence that reactivating memories in blocks helped the brain differentiate things in memory more than random reactivations.

The lab is already extending their results to understand more about the relationship between sleep and memory. Specifically, they’re interested in how sleep may help us take new information and incorporate it into older memories. “A study that we’ve run in the lab since [mine] is trying to understand what it looks like to learn new information that is aligned with things that you know from the past and how that new information can become integrated into older memories without totally overwhelming those old memories,” says Siefert. It seems that we’re only scratching the surface of what TMR can teach us about the sleeping brain’s relationship to memory.

What does this mean for everyday life?

After learning about these results, you might wonder whether students should give up on wakeful studying and just play their textbooks while they sleep. Unfortunately, it’s still not that easy. “Because [memory reactivation] doesn’t just improve memory and has this transformation component, if you took this device home and used it to play a textbook it’s not clear to me whether it would improve or hurt your memory of that information,” says Siefert. “You might need specific learning goals, you would need to think carefully about when you are delivering the cue, and you need to consider lots of other things.” It may be possible to design a system that could help a student study in their sleep, but we still need to learn a lot more about how TMR works before that will be possible.

Despite its complicated nature, some scientists are optimistic that they may be able to bring TMR to your bedroom. Rather than recording brain activity to target specific moments for reactivation, they aim to use audio recordings of movement during sleep to target the longer periods of sleep when they think replay naturally occurs. “For me in the lab it’s really important to target specific moments so I really know what my cues are doing, but in the real world that might be less important,” says Siefert. TMR is already being used in some clinical settings to help stroke patients relearn how to move their bodies, and Siefert suggested that it could one day be helpful for things like language learning if you’ve already learned some of the basics. For those who may be worried about TMR being used for brain washing or unconscious influence, Siefert says we aren’t capable of that now and may never be. “Your brain is already doing things and we’re just biasing it to little quick moments,” says Siefert. “We aren’t at the point where we can totally change the way that you’re thinking about something.”

Even in the absence of a TMR system on your nightstand, one takeaway is clear: sleep is important. “We don’t have a good understanding of why we’re remembering and forgetting certain things but knowing that the brain is selecting information means that that selection is probably important, and that selection is clearly happening during sleep,” says Siefert. “That means you should be getting good sleep, because you want to allow your brain time to process the information.” So next time you’re faced with the decision to stay up a little later studying for a test or prepping for a meeting, remember that choosing to sleep may be even more important than putting in that extra half hour of work.

About the brief writer: Catrina Hacker

Catrina Hacker is a PhD candidate working in Dr. Nicole Rust’s Lab. She is broadly interested in the neural correlates of cognitive processes and is currently studying how we remember what we see.

Learn more about the team’s memory reactivation study in the original paper.

How you find what you’re looking for.

or technically,

Signals in inferotemporal and perirhinal cortex suggest an untangling of visual target information.

[See Original Abstract on Pubmed]

Authors of the study: Marino Pagan, Luke S Urban, Margot P Wohl, Nicole C Rust

Finding Your Keys
You are late and the Uber is already outside. Where are your wallet and keys? You scan the nearest table. A dirty coffee cup, excessive CVS coupons, and at last, you see your wallet and keys, poking out from under a shirt. While this process might seem effortless, quickly finding what you are looking for in a crowded scene -- a process called “visual search” -- is an ability that even sophisticated computer programs have trouble with ¹. Marino Pagan in Nicole Rust’s lab at the University of Pennsylvania spent his PhD studying exactly how our brainsThe brain is an organ that serves as the center of the nervous system in all vertebrate and most invertebrate animals. can quickly and flexibly find what we’re looking for out of everything we are looking at.

However, this question has been difficult for scientists to answer. Before Marino performed his experiments, it was unknown which part of the brainThe brain is an organ that serves as the center of the nervous system in all vertebrate and most invertebrate animals. was responsible for visual search. In other words, scientists hadn’t yet been able to identify neuronsA nerve cell that uses electrical and chemical signals to send information to other cells including other neurons and muscles that specifically respond when what you are looking for matches what you are looking at. Additionally, it was unknown how any brainThe brain is an organ that serves as the center of the nervous system in all vertebrate and most invertebrate animals. area would combine the information about what you are looking for and what you are looking at. How do scientists go about answering where and how the brainThe brain is an organ that serves as the center of the nervous system in all vertebrate and most invertebrate animals. identifies different visual search “targets?” Before we tell you the results, we’re going to break down the approach Marino took to answering these questions.

Where in the BrainThe brain is an organ that serves as the center of the nervous system in all vertebrate and most invertebrate animals.?
Let’s first examine the question of where -- where are the neuronsA nerve cell that uses electrical and chemical signals to send information to other cells including other neurons and muscles in the brainThe brain is an organ that serves as the center of the nervous system in all vertebrate and most invertebrate animals. that are responsible for identifying visual search targets? We can answer this question by guessing what the activity of a brainThe brain is an organ that serves as the center of the nervous system in all vertebrate and most invertebrate animals. area that identifies search targets would look like and then looking for brainThe brain is an organ that serves as the center of the nervous system in all vertebrate and most invertebrate animals. areas with activity that matches our hypothesis. Like we mentioned before, neuronsA nerve cell that uses electrical and chemical signals to send information to other cells including other neurons and muscles in this area would respond or “turn on” when what you are looking at matches what you are looking for. In our earlier example about getting to your Uber, we would guess that a visual search area would turn on when you were looking at your wallet, or your keys, but not the coffee cup. However, in a different situation--let’s say making coffee--what you are looking for is different. This time, the visual search area would respond when you look at the coffee cup, but not the other items. Essentially, the visual search area should turn on when you are looking at the item you were searching for.

How?
Now let’s examine how the brainThe brain is an organ that serves as the center of the nervous system in all vertebrate and most invertebrate animals. figures out whether what you’re looking at matches what you’re looking for. All information in the brainThe brain is an organ that serves as the center of the nervous system in all vertebrate and most invertebrate animals. is represented as different patterns of neuronsA nerve cell that uses electrical and chemical signals to send information to other cells including other neurons and muscles turning on - or “firing.” For example, a pattern in which all neuronsA nerve cell that uses electrical and chemical signals to send information to other cells including other neurons and muscles fire could mean something different than a pattern in which only half the neuronsA nerve cell that uses electrical and chemical signals to send information to other cells including other neurons and muscles fire. In order for the brainThe brain is an organ that serves as the center of the nervous system in all vertebrate and most invertebrate animals. to determine whether what you’re looking for matches what you’re looking at, the pattern of neuronA nerve cell that uses electrical and chemical signals to send information to other cells including other neurons and muscles firing in the visual search area must be different for matches versus non-matches. When your brainThe brain is an organ that serves as the center of the nervous system in all vertebrate and most invertebrate animals. can differentiate - or separate - the neuronA nerve cell that uses electrical and chemical signals to send information to other cells including other neurons and muscles firing patterns for matches and non-matches, you will be able to distinguish between the two categories in the real world! So our question of how now becomes more specific - how does the brainThe brain is an organ that serves as the center of the nervous system in all vertebrate and most invertebrate animals. separate patterns of firing for matches and non-matches? It turns out, there are many ways that the brainThe brain is an organ that serves as the center of the nervous system in all vertebrate and most invertebrate animals. can separate firing patterns! Learning which ones the brainThe brain is an organ that serves as the center of the nervous system in all vertebrate and most invertebrate animals. uses not only teaches us about how our brainsThe brain is an organ that serves as the center of the nervous system in all vertebrate and most invertebrate animals. work, but can also help us build better computer algorithms to perform search tasks- not just for finding keys on a table, but for, say, identifying faces in a crowd.

Although the brainThe brain is an organ that serves as the center of the nervous system in all vertebrate and most invertebrate animals. has a lot of neuronsA nerve cell that uses electrical and chemical signals to send information to other cells including other neurons and muscles, we’re going to think about different ways to separate firing patterns by pretending there are only two neuronsA nerve cell that uses electrical and chemical signals to send information to other cells including other neurons and muscles in the brainThe brain is an organ that serves as the center of the nervous system in all vertebrate and most invertebrate animals. area responsible for visual search. In this situation, one way for the brainThe brain is an organ that serves as the center of the nervous system in all vertebrate and most invertebrate animals. to determine whether a firing pattern says “match” or “no match” is to make a simple rule that divides the patterns into two groups. One example of a rule could be “If neuronA nerve cell that uses electrical and chemical signals to send information to other cells including other neurons and muscles 1 is firing more than neuronA nerve cell that uses electrical and chemical signals to send information to other cells including other neurons and muscles 2, you are looking at a match. Otherwise, you are not”. This rule is shown in Fig 1, on the left. Notice how the rule makes it easy to draw a straight line that perfectly puts all matches on one side of the line, and all non-matches on the other. This type of neural firing is said to be linearly separable. It is linear something that can be represented as a straight line on a graph, and directly proportional changes in two related quantities because a simple, straight line can separate the two categories. This way of separating firing patterns is both very reliable and very easy for the brainThe brain is an organ that serves as the center of the nervous system in all vertebrate and most invertebrate animals. to do! Other ways of separating might require many more complicated rules (an example of this is shown in Fig 1, right). Therefore, linearly separable neural firing is good candidate for how the brainThe brain is an organ that serves as the center of the nervous system in all vertebrate and most invertebrate animals. might distinguish between search targets versus other objects.

With all this in mind, Marino Pagan and his PhD advisor Nicole Rust could make hypotheses about how the brainThe brain is an organ that serves as the center of the nervous system in all vertebrate and most invertebrate animals. identifies different visual search “targets” and which area of the brainThe brain is an organ that serves as the center of the nervous system in all vertebrate and most invertebrate animals. is responsible for this: 1) the brainThe brain is an organ that serves as the center of the nervous system in all vertebrate and most invertebrate animals.’s “visual search area” would turn on when what you are looking for matches what you are looking at, and 2) neuronsA nerve cell that uses electrical and chemical signals to send information to other cells including other neurons and muscles in this visual search area will have separable firing patterns for matches vs. non-matches. To test these hypotheses, they did one experiment to recreate the process we go through when looking for our keys.

Figure 1

Results
First, Marino trained monkeys to recognize specific, everyday objects (i.e. keys or wallet) in a sequence of images interspersed with other objects (i.e. the coffee cup). They then looked for (1) where in the brainThe brain is an organ that serves as the center of the nervous system in all vertebrate and most invertebrate animals. there was activity specific to targets and (2) how this region separated its firing patterns for matches and non-matches (i.e. were they linearly separable). They narrowed their search to two regions of the brainThe brain is an organ that serves as the center of the nervous system in all vertebrate and most invertebrate animals.: the inferior temporal lobe and the perirhinal cortex. The inferior temporal lobe is a part of the visual system and is thought to be the first place that memory (i.e. what you are looking for) and visual information (i.e. what you are looking at) are combined². The perirhinal cortex receives information from the inferior temporal lobe and is necessary for good performance on visual search tasks³.

Marino first asked whether either brainThe brain is an organ that serves as the center of the nervous system in all vertebrate and most invertebrate animals. region had activity patterns that were selective for search targets. They found this selectivity to be much stronger in the perirhinal cortex than the inferior temporal lobe, suggesting that the where of visual search is primarily the perirhinal cortex (PRH).

To address how this selective activity arose, Marino then asked if neural firing in response to targets was more linearly separable in one brainThe brain is an organ that serves as the center of the nervous system in all vertebrate and most invertebrate animals. region compared to the other. After looking over many neuronsA nerve cell that uses electrical and chemical signals to send information to other cells including other neurons and muscles, they found that it was much easier to draw a simple line that separated targets from non-targets (similar to Fig 1, left) in perirhinal cortex compared to inferior temporal lobe. Together, Marino’s findings suggest that the perirhinal cortex codes information of the location of the search target, separated from other objects using , linearsomething that can be represented as a straight line on a graph, and directly proportional changes in two related quantities separability.

There are still many exciting questions to answer. What is the inferior temporal lobe doing to combine memory and visual information? How is the brainThe brain is an organ that serves as the center of the nervous system in all vertebrate and most invertebrate animals. cell activity different in the inferior temporal lobe compared to perirhinal cortex? Do those differences contribute to the linearsomething that can be represented as a straight line on a graph, and directly proportional changes in two related quantities separability we see in the perirhinal cortex? Marino found that the answer was a bit more complicated. He found that the inferior temporal cortex may use non-linearsomething that can be represented as a straight line on a graph, and directly proportional changes in two related quantities separability, where flexible curves can separate visual and remembered information instead of rigid lines . The inferior temporal cortex then sends this information separable by flexible curves to the perirhinal cortex, which then may transform the information to again be separated by a line.

Conclusion
Like most problems in science, one experiment cannot fully and conclusively reveal everything there is to know about how we “search” with our eyes. However, the work of Marino Pagan and his mentor Nicole Rust takes important steps closer to this understanding, and adds valuable new information about where and how search targets are identified in the brainThe brain is an organ that serves as the center of the nervous system in all vertebrate and most invertebrate animals.. Not only does their work shine light on previously mysterious ways the brainThe brain is an organ that serves as the center of the nervous system in all vertebrate and most invertebrate animals. supports everyday actions like visual search but it also provides a foundation to engineer computers that can scan and find objects as quickly and flexibly as we do.

Figure 2

About the brief writer: Jeni Stiso

Jeni is a PhD Candidate in Dani Bassett’s lab. Jeni is interested in cognitive and computational neuroscience. She is interested in how changes in the electrical activity of the brain help people learn things.

Citations:

1. https://medium.com/deep-dimension/an-analysis-on-computer-vision-problems-6c68d56030c3

2. Chelazzi, L., & Desimone, R. (1993). A neural basis for visual search in IT. Nature. 363, pages 345–347.

3. Meunier, M., Bachevalier, J., Mishkin, M. & Murray, E.A. Effects on visual recognition of combined and separate ablations of the entorhinal and perirhinal cortex in rhesus monkeys. J. Neurosci. 13, 5418–5432 (1993).

To learn more about how the brain helps us quickly identify what we’re looking for, check out the full paper here.