How you find what you’re looking for.
or technically,
Signals in inferotemporal and perirhinal cortex suggest an untangling of visual target information.
[See Original Abstract on Pubmed]
Authors of the study: Marino Pagan, Luke S Urban, Margot P Wohl, Nicole C Rust
Finding Your Keys
You are late and the Uber is already outside. Where are your wallet and keys? You scan the nearest table. A dirty coffee cup, excessive CVS coupons, and at last, you see your wallet and keys, poking out from under a shirt. While this process might seem effortless, quickly finding what you are looking for in a crowded scene -- a process called “visual search” -- is an ability that even sophisticated computer programs have trouble with 1. Marino Pagan in Nicole Rust’s lab at the University of Pennsylvania spent his PhD studying exactly how our brainsThe brain is an organ that serves as the center of the nervous system in all vertebrate and most invertebrate animals. can quickly and flexibly find what we’re looking for out of everything we are looking at.
However, this question has been difficult for scientists to answer. Before Marino performed his experiments, it was unknown which part of the brainThe brain is an organ that serves as the center of the nervous system in all vertebrate and most invertebrate animals. was responsible for visual search. In other words, scientists hadn’t yet been able to identify neuronsA nerve cell that uses electrical and chemical signals to send information to other cells including other neurons and muscles that specifically respond when what you are looking for matches what you are looking at. Additionally, it was unknown how any brainThe brain is an organ that serves as the center of the nervous system in all vertebrate and most invertebrate animals. area would combine the information about what you are looking for and what you are looking at. How do scientists go about answering where and how the brainThe brain is an organ that serves as the center of the nervous system in all vertebrate and most invertebrate animals. identifies different visual search “targets?” Before we tell you the results, we’re going to break down the approach Marino took to answering these questions.
Where in the BrainThe brain is an organ that serves as the center of the nervous system in all vertebrate and most invertebrate animals.?
Let’s first examine the question of where -- where are the neuronsA nerve cell that uses electrical and chemical signals to send information to other cells including other neurons and muscles in the brainThe brain is an organ that serves as the center of the nervous system in all vertebrate and most invertebrate animals. that are responsible for identifying visual search targets? We can answer this question by guessing what the activity of a brainThe brain is an organ that serves as the center of the nervous system in all vertebrate and most invertebrate animals. area that identifies search targets would look like and then looking for brainThe brain is an organ that serves as the center of the nervous system in all vertebrate and most invertebrate animals. areas with activity that matches our hypothesis. Like we mentioned before, neuronsA nerve cell that uses electrical and chemical signals to send information to other cells including other neurons and muscles in this area would respond or “turn on” when what you are looking at matches what you are looking for. In our earlier example about getting to your Uber, we would guess that a visual search area would turn on when you were looking at your wallet, or your keys, but not the coffee cup. However, in a different situation--let’s say making coffee--what you are looking for is different. This time, the visual search area would respond when you look at the coffee cup, but not the other items. Essentially, the visual search area should turn on when you are looking at the item you were searching for.
How?
Now let’s examine how the brainThe brain is an organ that serves as the center of the nervous system in all vertebrate and most invertebrate animals. figures out whether what you’re looking at matches what you’re looking for. All information in the brainThe brain is an organ that serves as the center of the nervous system in all vertebrate and most invertebrate animals. is represented as different patterns of neuronsA nerve cell that uses electrical and chemical signals to send information to other cells including other neurons and muscles turning on - or “firing.” For example, a pattern in which all neuronsA nerve cell that uses electrical and chemical signals to send information to other cells including other neurons and muscles fire could mean something different than a pattern in which only half the neuronsA nerve cell that uses electrical and chemical signals to send information to other cells including other neurons and muscles fire. In order for the brainThe brain is an organ that serves as the center of the nervous system in all vertebrate and most invertebrate animals. to determine whether what you’re looking for matches what you’re looking at, the pattern of neuronA nerve cell that uses electrical and chemical signals to send information to other cells including other neurons and muscles firing in the visual search area must be different for matches versus non-matches. When your brainThe brain is an organ that serves as the center of the nervous system in all vertebrate and most invertebrate animals. can differentiate - or separate - the neuronA nerve cell that uses electrical and chemical signals to send information to other cells including other neurons and muscles firing patterns for matches and non-matches, you will be able to distinguish between the two categories in the real world! So our question of how now becomes more specific - how does the brainThe brain is an organ that serves as the center of the nervous system in all vertebrate and most invertebrate animals. separate patterns of firing for matches and non-matches? It turns out, there are many ways that the brainThe brain is an organ that serves as the center of the nervous system in all vertebrate and most invertebrate animals. can separate firing patterns! Learning which ones the brainThe brain is an organ that serves as the center of the nervous system in all vertebrate and most invertebrate animals. uses not only teaches us about how our brainsThe brain is an organ that serves as the center of the nervous system in all vertebrate and most invertebrate animals. work, but can also help us build better computer algorithms to perform search tasks- not just for finding keys on a table, but for, say, identifying faces in a crowd.
Although the brainThe brain is an organ that serves as the center of the nervous system in all vertebrate and most invertebrate animals. has a lot of neuronsA nerve cell that uses electrical and chemical signals to send information to other cells including other neurons and muscles, we’re going to think about different ways to separate firing patterns by pretending there are only two neuronsA nerve cell that uses electrical and chemical signals to send information to other cells including other neurons and muscles in the brainThe brain is an organ that serves as the center of the nervous system in all vertebrate and most invertebrate animals. area responsible for visual search. In this situation, one way for the brainThe brain is an organ that serves as the center of the nervous system in all vertebrate and most invertebrate animals. to determine whether a firing pattern says “match” or “no match” is to make a simple rule that divides the patterns into two groups. One example of a rule could be “If neuronA nerve cell that uses electrical and chemical signals to send information to other cells including other neurons and muscles 1 is firing more than neuronA nerve cell that uses electrical and chemical signals to send information to other cells including other neurons and muscles 2, you are looking at a match. Otherwise, you are not”. This rule is shown in Fig 1, on the left. Notice how the rule makes it easy to draw a straight line that perfectly puts all matches on one side of the line, and all non-matches on the other. This type of neural firing is said to be linearly separable. It is linear something that can be represented as a straight line on a graph, and directly proportional changes in two related quantities because a simple, straight line can separate the two categories. This way of separating firing patterns is both very reliable and very easy for the brainThe brain is an organ that serves as the center of the nervous system in all vertebrate and most invertebrate animals. to do! Other ways of separating might require many more complicated rules (an example of this is shown in Fig 1, right). Therefore, linearly separable neural firing is good candidate for how the brainThe brain is an organ that serves as the center of the nervous system in all vertebrate and most invertebrate animals. might distinguish between search targets versus other objects.
With all this in mind, Marino Pagan and his PhD advisor Nicole Rust could make hypotheses about how the brainThe brain is an organ that serves as the center of the nervous system in all vertebrate and most invertebrate animals. identifies different visual search “targets” and which area of the brainThe brain is an organ that serves as the center of the nervous system in all vertebrate and most invertebrate animals. is responsible for this: 1) the brainThe brain is an organ that serves as the center of the nervous system in all vertebrate and most invertebrate animals.’s “visual search area” would turn on when what you are looking for matches what you are looking at, and 2) neuronsA nerve cell that uses electrical and chemical signals to send information to other cells including other neurons and muscles in this visual search area will have separable firing patterns for matches vs. non-matches. To test these hypotheses, they did one experiment to recreate the process we go through when looking for our keys.
You are late and the Uber is already outside. Where are your wallet and keys? You scan the nearest table. A dirty coffee cup, excessive CVS coupons, and at last, you see your wallet and keys, poking out from under a shirt. While this process might seem effortless, quickly finding what you are looking for in a crowded scene -- a process called “visual search” -- is an ability that even sophisticated computer programs have trouble with 1. Marino Pagan in Nicole Rust’s lab at the University of Pennsylvania spent his PhD studying exactly how our brainsThe brain is an organ that serves as the center of the nervous system in all vertebrate and most invertebrate animals. can quickly and flexibly find what we’re looking for out of everything we are looking at.
However, this question has been difficult for scientists to answer. Before Marino performed his experiments, it was unknown which part of the brainThe brain is an organ that serves as the center of the nervous system in all vertebrate and most invertebrate animals. was responsible for visual search. In other words, scientists hadn’t yet been able to identify neuronsA nerve cell that uses electrical and chemical signals to send information to other cells including other neurons and muscles that specifically respond when what you are looking for matches what you are looking at. Additionally, it was unknown how any brainThe brain is an organ that serves as the center of the nervous system in all vertebrate and most invertebrate animals. area would combine the information about what you are looking for and what you are looking at. How do scientists go about answering where and how the brainThe brain is an organ that serves as the center of the nervous system in all vertebrate and most invertebrate animals. identifies different visual search “targets?” Before we tell you the results, we’re going to break down the approach Marino took to answering these questions.
Where in the BrainThe brain is an organ that serves as the center of the nervous system in all vertebrate and most invertebrate animals.?
Let’s first examine the question of where -- where are the neuronsA nerve cell that uses electrical and chemical signals to send information to other cells including other neurons and muscles in the brainThe brain is an organ that serves as the center of the nervous system in all vertebrate and most invertebrate animals. that are responsible for identifying visual search targets? We can answer this question by guessing what the activity of a brainThe brain is an organ that serves as the center of the nervous system in all vertebrate and most invertebrate animals. area that identifies search targets would look like and then looking for brainThe brain is an organ that serves as the center of the nervous system in all vertebrate and most invertebrate animals. areas with activity that matches our hypothesis. Like we mentioned before, neuronsA nerve cell that uses electrical and chemical signals to send information to other cells including other neurons and muscles in this area would respond or “turn on” when what you are looking at matches what you are looking for. In our earlier example about getting to your Uber, we would guess that a visual search area would turn on when you were looking at your wallet, or your keys, but not the coffee cup. However, in a different situation--let’s say making coffee--what you are looking for is different. This time, the visual search area would respond when you look at the coffee cup, but not the other items. Essentially, the visual search area should turn on when you are looking at the item you were searching for.
How?
Now let’s examine how the brainThe brain is an organ that serves as the center of the nervous system in all vertebrate and most invertebrate animals. figures out whether what you’re looking at matches what you’re looking for. All information in the brainThe brain is an organ that serves as the center of the nervous system in all vertebrate and most invertebrate animals. is represented as different patterns of neuronsA nerve cell that uses electrical and chemical signals to send information to other cells including other neurons and muscles turning on - or “firing.” For example, a pattern in which all neuronsA nerve cell that uses electrical and chemical signals to send information to other cells including other neurons and muscles fire could mean something different than a pattern in which only half the neuronsA nerve cell that uses electrical and chemical signals to send information to other cells including other neurons and muscles fire. In order for the brainThe brain is an organ that serves as the center of the nervous system in all vertebrate and most invertebrate animals. to determine whether what you’re looking for matches what you’re looking at, the pattern of neuronA nerve cell that uses electrical and chemical signals to send information to other cells including other neurons and muscles firing in the visual search area must be different for matches versus non-matches. When your brainThe brain is an organ that serves as the center of the nervous system in all vertebrate and most invertebrate animals. can differentiate - or separate - the neuronA nerve cell that uses electrical and chemical signals to send information to other cells including other neurons and muscles firing patterns for matches and non-matches, you will be able to distinguish between the two categories in the real world! So our question of how now becomes more specific - how does the brainThe brain is an organ that serves as the center of the nervous system in all vertebrate and most invertebrate animals. separate patterns of firing for matches and non-matches? It turns out, there are many ways that the brainThe brain is an organ that serves as the center of the nervous system in all vertebrate and most invertebrate animals. can separate firing patterns! Learning which ones the brainThe brain is an organ that serves as the center of the nervous system in all vertebrate and most invertebrate animals. uses not only teaches us about how our brainsThe brain is an organ that serves as the center of the nervous system in all vertebrate and most invertebrate animals. work, but can also help us build better computer algorithms to perform search tasks- not just for finding keys on a table, but for, say, identifying faces in a crowd.
Although the brainThe brain is an organ that serves as the center of the nervous system in all vertebrate and most invertebrate animals. has a lot of neuronsA nerve cell that uses electrical and chemical signals to send information to other cells including other neurons and muscles, we’re going to think about different ways to separate firing patterns by pretending there are only two neuronsA nerve cell that uses electrical and chemical signals to send information to other cells including other neurons and muscles in the brainThe brain is an organ that serves as the center of the nervous system in all vertebrate and most invertebrate animals. area responsible for visual search. In this situation, one way for the brainThe brain is an organ that serves as the center of the nervous system in all vertebrate and most invertebrate animals. to determine whether a firing pattern says “match” or “no match” is to make a simple rule that divides the patterns into two groups. One example of a rule could be “If neuronA nerve cell that uses electrical and chemical signals to send information to other cells including other neurons and muscles 1 is firing more than neuronA nerve cell that uses electrical and chemical signals to send information to other cells including other neurons and muscles 2, you are looking at a match. Otherwise, you are not”. This rule is shown in Fig 1, on the left. Notice how the rule makes it easy to draw a straight line that perfectly puts all matches on one side of the line, and all non-matches on the other. This type of neural firing is said to be linearly separable. It is linear something that can be represented as a straight line on a graph, and directly proportional changes in two related quantities because a simple, straight line can separate the two categories. This way of separating firing patterns is both very reliable and very easy for the brainThe brain is an organ that serves as the center of the nervous system in all vertebrate and most invertebrate animals. to do! Other ways of separating might require many more complicated rules (an example of this is shown in Fig 1, right). Therefore, linearly separable neural firing is good candidate for how the brainThe brain is an organ that serves as the center of the nervous system in all vertebrate and most invertebrate animals. might distinguish between search targets versus other objects.
With all this in mind, Marino Pagan and his PhD advisor Nicole Rust could make hypotheses about how the brainThe brain is an organ that serves as the center of the nervous system in all vertebrate and most invertebrate animals. identifies different visual search “targets” and which area of the brainThe brain is an organ that serves as the center of the nervous system in all vertebrate and most invertebrate animals. is responsible for this: 1) the brainThe brain is an organ that serves as the center of the nervous system in all vertebrate and most invertebrate animals.’s “visual search area” would turn on when what you are looking for matches what you are looking at, and 2) neuronsA nerve cell that uses electrical and chemical signals to send information to other cells including other neurons and muscles in this visual search area will have separable firing patterns for matches vs. non-matches. To test these hypotheses, they did one experiment to recreate the process we go through when looking for our keys.
Figure 1
Results
First, Marino trained monkeys to recognize specific, everyday objects (i.e. keys or wallet) in a sequence of images interspersed with other objects (i.e. the coffee cup). They then looked for (1) where in the brainThe brain is an organ that serves as the center of the nervous system in all vertebrate and most invertebrate animals. there was activity specific to targets and (2) how this region separated its firing patterns for matches and non-matches (i.e. were they linearly separable). They narrowed their search to two regions of the brainThe brain is an organ that serves as the center of the nervous system in all vertebrate and most invertebrate animals.: the inferior temporal lobe and the perirhinal cortex. The inferior temporal lobe is a part of the visual system and is thought to be the first place that memory (i.e. what you are looking for) and visual information (i.e. what you are looking at) are combined2. The perirhinal cortex receives information from the inferior temporal lobe and is necessary for good performance on visual search tasks3.
Marino first asked whether either brainThe brain is an organ that serves as the center of the nervous system in all vertebrate and most invertebrate animals. region had activity patterns that were selective for search targets. They found this selectivity to be much stronger in the perirhinal cortex than the inferior temporal lobe, suggesting that the where of visual search is primarily the perirhinal cortex (PRH).
To address how this selective activity arose, Marino then asked if neural firing in response to targets was more linearly separable in one brainThe brain is an organ that serves as the center of the nervous system in all vertebrate and most invertebrate animals. region compared to the other. After looking over many neuronsA nerve cell that uses electrical and chemical signals to send information to other cells including other neurons and muscles, they found that it was much easier to draw a simple line that separated targets from non-targets (similar to Fig 1, left) in perirhinal cortex compared to inferior temporal lobe. Together, Marino’s findings suggest that the perirhinal cortex codes information of the location of the search target, separated from other objects using , linearsomething that can be represented as a straight line on a graph, and directly proportional changes in two related quantities separability.
There are still many exciting questions to answer. What is the inferior temporal lobe doing to combine memory and visual information? How is the brainThe brain is an organ that serves as the center of the nervous system in all vertebrate and most invertebrate animals. cell activity different in the inferior temporal lobe compared to perirhinal cortex? Do those differences contribute to the linearsomething that can be represented as a straight line on a graph, and directly proportional changes in two related quantities separability we see in the perirhinal cortex? Marino found that the answer was a bit more complicated. He found that the inferior temporal cortex may use non-linearsomething that can be represented as a straight line on a graph, and directly proportional changes in two related quantities separability, where flexible curves can separate visual and remembered information instead of rigid lines . The inferior temporal cortex then sends this information separable by flexible curves to the perirhinal cortex, which then may transform the information to again be separated by a line.
Conclusion
Like most problems in science, one experiment cannot fully and conclusively reveal everything there is to know about how we “search” with our eyes. However, the work of Marino Pagan and his mentor Nicole Rust takes important steps closer to this understanding, and adds valuable new information about where and how search targets are identified in the brainThe brain is an organ that serves as the center of the nervous system in all vertebrate and most invertebrate animals.. Not only does their work shine light on previously mysterious ways the brainThe brain is an organ that serves as the center of the nervous system in all vertebrate and most invertebrate animals. supports everyday actions like visual search but it also provides a foundation to engineer computers that can scan and find objects as quickly and flexibly as we do.
First, Marino trained monkeys to recognize specific, everyday objects (i.e. keys or wallet) in a sequence of images interspersed with other objects (i.e. the coffee cup). They then looked for (1) where in the brainThe brain is an organ that serves as the center of the nervous system in all vertebrate and most invertebrate animals. there was activity specific to targets and (2) how this region separated its firing patterns for matches and non-matches (i.e. were they linearly separable). They narrowed their search to two regions of the brainThe brain is an organ that serves as the center of the nervous system in all vertebrate and most invertebrate animals.: the inferior temporal lobe and the perirhinal cortex. The inferior temporal lobe is a part of the visual system and is thought to be the first place that memory (i.e. what you are looking for) and visual information (i.e. what you are looking at) are combined2. The perirhinal cortex receives information from the inferior temporal lobe and is necessary for good performance on visual search tasks3.
Marino first asked whether either brainThe brain is an organ that serves as the center of the nervous system in all vertebrate and most invertebrate animals. region had activity patterns that were selective for search targets. They found this selectivity to be much stronger in the perirhinal cortex than the inferior temporal lobe, suggesting that the where of visual search is primarily the perirhinal cortex (PRH).
To address how this selective activity arose, Marino then asked if neural firing in response to targets was more linearly separable in one brainThe brain is an organ that serves as the center of the nervous system in all vertebrate and most invertebrate animals. region compared to the other. After looking over many neuronsA nerve cell that uses electrical and chemical signals to send information to other cells including other neurons and muscles, they found that it was much easier to draw a simple line that separated targets from non-targets (similar to Fig 1, left) in perirhinal cortex compared to inferior temporal lobe. Together, Marino’s findings suggest that the perirhinal cortex codes information of the location of the search target, separated from other objects using , linearsomething that can be represented as a straight line on a graph, and directly proportional changes in two related quantities separability.
There are still many exciting questions to answer. What is the inferior temporal lobe doing to combine memory and visual information? How is the brainThe brain is an organ that serves as the center of the nervous system in all vertebrate and most invertebrate animals. cell activity different in the inferior temporal lobe compared to perirhinal cortex? Do those differences contribute to the linearsomething that can be represented as a straight line on a graph, and directly proportional changes in two related quantities separability we see in the perirhinal cortex? Marino found that the answer was a bit more complicated. He found that the inferior temporal cortex may use non-linearsomething that can be represented as a straight line on a graph, and directly proportional changes in two related quantities separability, where flexible curves can separate visual and remembered information instead of rigid lines . The inferior temporal cortex then sends this information separable by flexible curves to the perirhinal cortex, which then may transform the information to again be separated by a line.
Conclusion
Like most problems in science, one experiment cannot fully and conclusively reveal everything there is to know about how we “search” with our eyes. However, the work of Marino Pagan and his mentor Nicole Rust takes important steps closer to this understanding, and adds valuable new information about where and how search targets are identified in the brainThe brain is an organ that serves as the center of the nervous system in all vertebrate and most invertebrate animals.. Not only does their work shine light on previously mysterious ways the brainThe brain is an organ that serves as the center of the nervous system in all vertebrate and most invertebrate animals. supports everyday actions like visual search but it also provides a foundation to engineer computers that can scan and find objects as quickly and flexibly as we do.
Figure 2
About the brief writer: Jeni Stiso
Jeni is a PhD Candidate in Dani Bassett’s lab. Jeni is interested in cognitive and computational neuroscience. She is interested in how changes in the electrical activity of the brain help people learn things.
Citations:
https://medium.com/deep-dimension/an-analysis-on-computer-vision-problems-6c68d56030c3
Chelazzi, L., & Desimone, R. (1993). A neural basis for visual search in IT. Nature. 363, pages 345–347.
Meunier, M., Bachevalier, J., Mishkin, M. & Murray, E.A. Effects on visual recognition of combined and separate ablations of the entorhinal and perirhinal cortex in rhesus monkeys. J. Neurosci. 13, 5418–5432 (1993).