94

94 THE BRAIN FROM INSIDE OUT Chapter 3. Perception from Action 55

cortical pyramidal cells is exactly in this range, and it determines their integra- but also exert autonomic e ects (e.g., change heart rate) and control endocrine
tion ability.22 Discharges of upstream neurons within this window can success- function (e.g., milk production). Furthermore, thought and imagination can
fully trigger an action potential in the downstream reader neuron. Generating also be conceived as actions (Chapters 5 and 9).6 us, the interaction of the
spike responses in reader neurons is the main reason for a cell assembly to perceptual and action systems is a complex problem.
come together. erefore, from the point of view of a single reader neuron, all Most details of the world around us pass without our notice. Stimuli become
neurons whose spiking activity contributes to its own spike can be regarded as salient or meaningful through learning and become percepts. But then if per-
a meaningful assembly. Other upstream neurons that re outside this critical ceptual awareness is understood as an interpretation of sensation by the brain,
time window (i.e., nonsynchronously) can only be part of another assembly. it is an active process. Physiologically, awareness requires that the signal’s trace
us, by monitoring spiking activity of reader neurons, one can objectively be distributed in many brain structures and linger in neuronal networks for
determine whether the upstream neurons are part of the same assembly and some time.7 is fact is interesting because initiating movement also has a vo-
serve the same goal (the discharge of the reader neuron) or belong to di erent litional component; we are aware of our voluntary actions, as opposed to re ex
assemblies (Figure 4.2). Members of an assembly can project individually to movements and automatic, well-learned actions, such as walking. erefore,
hundreds or thousands of other neurons. erefore, each neuron can be part there is terrain between sensory inputs and motor outputs, a hypothetical cen-
of many assemblies, determined largely by its number of targets. ese other tral (or “top level”) processor. is poorly understood but o en speculated
potential constellations become an assembly when they activate another reader about ground is usually referred to by the name of homunculus, “decision-
neuron. maker” or volition.8
In my de nition, principal cells do not need to be anatomically connected In the outside-in framework, sensory information is distributed to higher
to each other to form an assembly. It is also inconsequential whether the up- order cortical areas and, consecutively, funneled to the motor areas. e al-
stream assembly member neurons are neighbors or reside in multiple corners leged brain route from sensation to motor response is o en referred to as the
of the brain. For example, a coactive group of thalamic neurons that lead to “perception–action loop.” Decisions are made somewhere in the unexplained
the discharge of their target cortical neurons is a meaningful cell assembly. territory between sensory inputs and motor outputs. ere is no need for ana-
However, by Hebb’s de nition, these neurons would not be regarded as an as- tomical connections going in the opposite direction.
sembly because they are not connected to each other by excitatory synapses.
In contrast, viewed from the reader neuron de nition of the cell assembly, the
synchronous activity of these thalamic neurons within the membrane time 6. at thought is an action (see also Llinás, 2002) is, of course, a very di erent idea from the
outside-in view that our thoughts are the result of the synthesis of sensory inputs.
constant of their downstream cortical neurons renders them an e ective as-
sembly of neurons. erefore, this de nition of the cell assembly is closer to 7. Many experiments suggest that conscious recognition requires the recruitment of a large
number of neurons in an extended, distributed complex brain circuit. According to Libet
the concept of the population vector than to Hebb’s de nition. However, if
(2005), this is longer than 0.5 second, which he calls “mind time,” supporting a sensory input–
the presynaptic neurons are also connected, their synchronous discharge may conscious decision–action arc model. See criticism of this requirement in Goodale et al. (1986).
strengthen their synaptic communication, as predicted by Hebb’s plasticity
8. e little man (or rather a “little rat” in most experiments) is of course only a metaphor
for an assumed brain mechanism or, alternatively, the soul. e origin of this homunculus
thinking is religion and folk psychology. e soul is o en interpreted as an alternative version
of the self. Sometime, such as during dreams, the soul is believed to leave the body and even
empower it with supernatural abilities. A er death, soul and body divorce, and the soul leaves
22. e membrane time constant (τ) is the product of the resistance rm and capacitance cm behind the helpless dead body for good. Because the soul has superb interpreting powers, if
of the membrane (τ = r m cm) and represents a duration within which the membrane poten- anything similar exists in the brain, it should have similar interpretive powers. e yin-yang
tial relaxes to approximately 37% (= e-1) of its initial value a er a voltage step. Neurons with dichotomy of Confucianism re ects a similar yet very di erent general idea: the yin derives
longer τ can integrate postsynaptic potentials for longer time periods (Johnston and Wu, from heaven, whereas yang is from the Earth. Everything is a product of yin and yang. ey
1995; Koch et al., 1996). e time constant can vary depending on the state of the network in cannot be divorced from each other. Many, like, Daniel Dennett, liken the homunculus literally
which the neuron is embedded (Destexhe et al., 2003). During intense synaptic activity, such to a person watching a movie projected from the retina and other modalities, in what he calls a
as hippocampal sharp wave ripples, the neuron’s input impedance decreases, accompanied by “Cartesian eater” where it all comes together; this is in reference to Descartes’s dualist view
a shortened time window of input integration, corresponding to the fast ripple waves (6–7 ms; of the mind (Dennett, 1991). e homunculus is o en used as a fallacy argument for the in -
Buzsáki, 2015). nite regress of logic.
56

56 THE BRAIN FROM INSIDE OUT Chapter 4. Neuronal Assembly 93

I have already suggested that the brain’s primary goal is to generate actions (a) (b)
and evaluate their consequences with the help of its sensors. 9 Combining the his-
torical perspective of cognitive neuroscience (Chapter 1) and the experiments
presented later, I  will conclude that the correct order is “action–perception”
cycle. First, let’s examine experiments that lead to my conclusion.

(c)
PHI PHENOMENA AND PASSIVE OBSERVERS

Occasionally, our eyes (actually our brains) play tricks on us. Among the best-
known illusions is apparent motion or the phi phenomenon. In its simplest ver-
sion, two small balls at two discrete locations on the screen alternate over a
short interval (say 60 ms). All humans, and most likely other mammals, see
a single ball moving back and forth between the two positions, even though
there is no movement whatsoever on the screen.10 If the two balls have di erent Figure 4.2. Cell assembly: the fundamental unit of neural syntax. A: A raster plot
of a subset of hippocampal pyramidal cells that were active during a 1-sec period of
colors in addition to motion, we also experience an abrupt color change in the
spatial exploration on an open eld, ordered by their temporal relationship. Each line
middle of the ball’s illusory passage between the two locations. is is very in- corresponds to a neuron, and each tick is an action potential in that neuron. Top four
teresting because the color change is felt 25 ms or so before the second color ellipsoids indicate a repeatedly active cell assembly, which alternates with another one
is actually ashed.11 How can the perceptual e ect occur before its physical (bottom). Bottom trace shows local eld potential dominated by theta and nested gamma
cause? Phi phenomena have kept philosophers and psychologists busy over the oscillatory waves. Vertical lines indicate troughs of theta waves. B: Recording sites from
past century o ering numerous explanations. ey disagree on details, such the hippocampal CA1 pyramidal layer in the rat. C: Spike timing of a randomly chosen
as whether the mind generates the movement by lling in the missing pieces member neuron of a cell assembly is predictable from peer activity. Distribution of time
or the intervening apparent motion is produced retrospectively and somehow scales at which peer activity optimally improved spike time prediction of a given cell. e
median optimal time scale is 23 ms (vertical line).
Modi ed with permission from Harris et al. (2003).

reasoned that members of the assembly should work together within a meas-
9. Ho man’s interface model of perception (Ho man, 1998; Ho man et  al., 2005)  also urable time window (Figure 4.2). Like members of an orchestra, neurons in a
emphasizes that we do not simply passively view the world, but also act on it. We can interact circuit can e ectively time their actions relative to others.20 So we tried to de-
with the environment not because we perceive its objective reality but because the world has termine the time window within which neurons can best predict the timing of
su cient regularity that allows for the prediction of our actions on it. However, Ho man states
each other’s spikes. By varying the analysis window experimentally, we found
that “having a perceptual experience does not require motor movements” (Ho man et  al.,
2005). I agree that this may be the case in an already movement-calibrated brain, yet I maintain that the best prediction of the spike timing of single hippocampal neurons
that without action calibration at some stage of life perception does not arise. from the activity of their peers was when the time window varied between 10
10. e phi illusion was discovered by the German psychologist Max Wertheimer (1912). For and 30 ms. 21
an extended discussion, see Kolers and von Grünau (1976) or Dennett and Kinsbourne (1992). is is an important time window in neurophysiology because many physio-
Many contemporary neuroscience researchers use illusions and subjective feedback about them logical variables share it. First and foremost, the membrane time constant (τ) of
as tools to probe neuronal responses (Dehaene and Changeux, 2011; Koch, 2004). e title of
Giulio Tononi’s excellent and entertaining book on consciousness-related topics is Phi (2012).
20. is project was led by Ken Harris et al. (2003), then a postdoc in my lab. See Truccolo
11. ere are many interesting practical and theoretical implications of the phi phenomena.
et al. (2010).
In a movie theater, we do not perceive a rapid ipping of still frames at 24 frames per second.
Instead, we have the illusion of continuous movement without interruptions. Neon light 21. Jensen and Lisman (1996, 2000); Harris et al. (2003); Harris (2005); Kelemen and Fenton
advertisements also use the same principle to create motion perception. (2010); Lansner (2009).
92

92 THE BRAIN FROM INSIDE OUT Chapter 3. Perception from Action 57

it receives to respond one way or another. e reader mechanism17 can be a projected backward in time, but nearly all of the theories call upon conscious-
muscle, a single neuron, groups of neurons, a machine, or even a human ob- ness as explanans, the thing that explains, rather than the phenomenon to be
server who interprets the meaning of the inputs. To be meaningful, the same explained.
constellation of the assembly should always lead to a similar action, whether Phi phenomena, as well as ambiguous pictures, Rorschach patches, and other
it is at the level of a single neuron, neuronal groups, muscles, or hormonal re- illusions, are favorite territory of consciousness researchers. is research program
lease. e assembly has to have a consistent consequence. is action outcome o en focuses on perception, asking, for example, how the brain can extract and
is what makes any neuronal coalition into a meaningful assembly.18 bind sensory features to produce a uni ed object. But this program stops short
Reading the impact of a cell assembly requires a temporal integration mech- of explaining the perception–action cycle12 because it says nothing about action.
anism. Neurons come together in time; that is, they synchronize to achieve is neglect creates a huge gap between perception and action that requires lling
an action that is not possible for single members alone. Whether events are it with mental constructs such as volition and decision-making. is separation
synchronous or not can be determined only from the point of view of an ob- largely explains why research on perception and action have taken independent
server.19 By extension, I suggest that a neuronal assembly can only be de ned paths in neuroscience.
from the perspective of a neuronal reader mechanism. If action is a prerequisite of perception, as I claim, then there should be no per-
ception without action. You can show that this is indeed the case by xing your
gaze steadily on a stationary target. No matter what surrounds the gaze target,
Reader Mechanism– Defined Cell Assembly even if pink elephants are running around, it will all disappear a er just a few
seconds. is never-failing demonstration is known as the Troxler e ect.13
All the preceding discussion is based on reasoning and speculation. However, Another way to probe the primacy of action is to get rid of its consequences
the only way forward for science is by quantitative measurements; in this case, altogether. One such manipulation is called retinal stabilization. In an early,
examination of the temporal relationships of spikes among putative assembly ingenious instantiation, a suction cup with a mirror on it was attached to the
members. By the 1990s, my group and other laboratories worked out methods cornea. (Today, we would call it a contact lens.) When a picture or movie is
to record su ciently large ensembles of neurons so that we could address a projected onto the mirror, you can no longer investigate the scene with your
critical question: What determines the precise timing of a neuron’s spikes? We eye movements because the image on the retina moves with your eyes, in the
hypothesized, as did Hebb, that the recorded neurons take part in di erent same direction and at the same speed and amplitude. As you can guess by now,
assemblies, but not all assembly members are active on each occasion. We also without the brain’s ability to obtain a “second opinion” by eye movement inves-
tigation, vision ceases. 14

17. Synonyms for “reader” can be terms like “observer,” “classi er,” “integrator,” or “actuator,”
depending on whether one’s background is physics, biology, computational science, or engi- 12. For a discussion of the involvement of higher order cortical areas, see Quintana and Fuster
neering. In the present de nition, they all refer to the same thing. (1999).
18. At the most complex level, such “caused” e ects may be plans, memories, decisions, or 13. Named a er Ignaz Troxler, a Swiss physician who rst demonstrated the e ect (Troxler
thoughts (Chapter 13). Berkeley’s dictum “Esse est percipi” (“To be is to be perceived”) was a et al., 1804). We must x our gaze to inspect the details of a scene, but if we xate a while, it
response to British empiricist philosophy. Berkeley famously asked “if a tree falls in a forest fades from our brain. See it yourself: https:// en.wikipedia.org/wiki/Troxler%27s_fading. One
and no one is around to hear it, does it make a sound?” (Berkeley, 1710/2010). Galileo Galilei, explanation of the Troxler e ect and related illusions is neuronal “adaptation” in the retina
the most prominent champion of Aristotelian logic, posed a similar question: “if ears, tongues, or further up in the visual system, without the actual identi cation of the cellular and circuit
and noses be taken away, the number, shape, and motion of bodies would remain, but not their mechanisms of the adaptation (Martinez-Conde et al., 2004). e moment you move your eyes,
tastes, sounds, and odors. e latter, external to the living creature, I believe to be nothing but sight returns. Zen Buddhism e ectively exploits this feature of the visual system for meditation.
mere names” (1623/1954). Perhaps nothing is more foreign to contemporary neuroscience than In the famous zen Rock Garden (Karesansui) in the Ryoanji Temple in Kyoto, the meditator can
Berkeley’s subjective idealism, yet the idea that in the absence of a “reader-actuator‟ meaning focus on one of the een rocks arising from at white sand and make the rest of the (visual)
does not exist well resonates with an engineering approach to brain function. Although the met- world disappear. It works well; I tried it.
aphor of the reader/interpreter/actuator admittedly has an element of spookiness, it captures
14. e most popular citation for the stabilization experiments is Yarbus’s book (1967). Alfred
the key features of control theory of dynamical systems: a goal or desired output.
Yarbus was a Soviet psychologist who recorded eye movements in humans while they viewed
19. is is a classic problem in relativity theory (see Chapter 10). objects and scenes. He showed that the eye movements reliably track the observer’s interest in
58

58 THE BRAIN FROM INSIDE OUT Chapter 4. Neuronal Assembly 91

ACTION SPEED LIMITS P ERCEPTION SPEED experiments in the learning, behaving animal as opposed to subjective rea-
soning. However, even when the best tools are available, experiments must be
Instead of asking why the brain can be fooled by arti cial or rare patterns, per- guided by a hypothesis that can be rejected or retained with some con dence.
haps it is more productive to ask why the brain reliably produces illusions.15 As
autoshaped behaviors are not mistakes (Chapter 1), illusions are not mistakes
either. ey are due to the brain’s ability to e ciently extract important features CELL ASSEMBLIES FOR A PURPOSE
from garbage. Some distinctions are of paramount importance, whereas we fail
to observe most of the enormous variability in the environment.16 Perception In my view, a fundamental problem with Hebb’s “representational concept” is
is not the veridical representation of the objective world even in case of vision, that it presupposes that the same inputs always mobilize the same set of neurons
to which sense the human brain devoted a very large part of its real estate. Even because this framework suggests that objects in the world should correspond
the most sophisticated observer, without the knowledge that the color balls in to neuronal responses in the brain. We can call it the “neuronal correlate of
the phi illusion do not move, cannot tell whether the brain is faithfully fol- x” approach. According to the representational outside-in framework, the best
lowing the motion of a ball or whether two stationary balls are being ashed in strategy by which to understand mechanisms of perception and to identify
a rapid sequence. ere is no way to distinguish the two possibilities because input-representing cell assemblies is to present various stimuli to the brain and
brain mechanisms to track such fast changes consciously do not exist. examine the spatiotemporal distribution of the evoked neuronal responses.16
So why not have a faster perceptual system? If the primary goal of the However, as discussed in Chapters 1 and 3, it is a questionable strategy to iden-
brain were to perceive and process, it would be odd for evolution to compro- tify the neuronal correlates of human-invented mental constructs on the pre-
mise speed. e answer may lie in the mechanics of our body. Conditionally sumption that they must have clear boundaries that correspond to their neural
accepting my conjecture that the brain’s fundamental goal is to produce action, representation. Correlating spiking with external stimulus features is informa-
there is no reason to invest in speed if the muscle system that it controls is slow. tive to the experimenter, but such correlation is not available to neurons in
e contraction speed of vertebrate muscles is determined by the properties of the brain (Chapter  1). If cell assemblies are meaningful events for neuronal
myosin, a contractile protein that is largely preserved across all mammals inde- computation, they should have predictable consequences for their downstream
pendent of body size. erefore, small and large brains must deal with problems partners. Finally, because the brain is also active during sleep, cell assemblies
at the same temporal scale. In the phi illusion, jerky or saccadic movements of are expected to be as important during sleep as in the waking state (Chapter 8).
the eye are needed to refocus the color vision region (fovea) of the retina from e outside-in, representational framework is mute about the existence and
function of cell assemblies during o ine modes of brain operation.
In Chapter 3, I already expressed doubt that the representational strategy can
the details of the objects. In the case of the human face, the investigation of eyes, mouth, cheek, objectively identify cell assemblies because the brain’s fundamental priority is
or nose regions depends on what emotions are being identi ed. See also Riggs and Ratli
not to faithfully “represent” the surrounding world but to simulate practically
(1952) and Ditchburn and Ginsborg (1952). ere is a resurgence of interest in eye-tracking
techniques today, perhaps due to the recognition that they provide useful information about useful aspects of it based on prior experience and select the most advantageous
the “mind’s opinion and intentions.” Wearable systems can identify one’s interest in particular action in the current situation. From this perspective, an objective de nition of
details of scenes and advertisements with high precision. Every major technology-oriented the cell assembly requires two related key conditions: a reader classi er and a
company, from Sony to Google, o ers such devices. temporal frame. Let me elaborate on this bold statement a bit. Inspired by the
15. Visual areas V1 and MT may be involved in phi phenomena as they respond to real move- population vector concept, I suggest that a cell assembly can only be de ned
ment and apparent movement (e.g., induced by alternating ashing of stationary stimuli). from the perspective of downstream “reader” mechanisms because the biolog-
Neuronal spiking in these cortical regions varies systematically with spatial and temporal ical relevance of a particular constellation of active neurons (i.e., a presumed
properties of the stimuli that closely parallel the psychophysically de ned limits of apparent
cell assembly or assembly sequence) can only be judged from its consequences.
motion in human participants (Newsome et al., 1986).
In my world, the term “reader” refers to a mechanism that can use the inputs
16. Visual artists exploit illusory perceptual experiences for aesthetic goals. Victor Vasarely,
who was born in my hometown of Pécs, Hungary, and other painters create impossible three-
dimensional objects on planar surfaces. e impossibility becomes apparent only a er carefully 16. Engel et al. (2001); Hebb (1949); James (1890); Milner (1996); von der Malsburg (1994);
scanning the continuity of the lines. Or think of another painter: Maurits Escher. Hubel and Wiesel (1962); Rieke et al. (1997).
90

90 THE BRAIN FROM INSIDE OUT Chapter 3. Perception from Action 59

computer cursors or robotic arms via thoughts; that is, by the voluntary pro- one ball to the other. ese ballistic eye movements occur three to four times a
duction of spikes in the appropriate sets of neurons.14 second on average and are the fastest movements in the human body (100–300
degrees/s or 20–30 ms during reading).17 e skeletal muscles are considerably
slower than those that move the eyeballs. Nobody can move his ngers faster
Relationship Between Population Vector and Cell Assembly than 20 taps per second, not even Franz Liszt.18 Even if an imagined super brain
could perceive, decide, and send a movement command in less than a mil-
ere is a clear parallel between cell assemblies and population vectors. lisecond, it would have little bene t for the muscles, which must temporally
However, in contrast to the mathematically de ned population vector, the cell coordinate their actions over a sluggish tens to hundreds of milliseconds. From
assembly concept is only loosely described as representing “things” through this evolutionary perspective, we have one more reason to believe that percep-
excitatory connections. In line with the prevailing framework of cognitive psy- tion is secondary to action.
chology, the assembly concept has primarily remained a sensory input-driven
model of brain activity. e strength of Hebb’s hypothesis is its generalizability,
but this is also its major weakness. e idea that changing constellations of Perception– Action or Action– Perception?
neuronal ring patterns underlie our cognitive capacity is not di erent from
the layman’s view and is hardly falsi able since there is no alternative hypo- e clever skeptic, of course, can argue that there is no reason to prioritize the
thesis. What makes Hebb’s concept distinct and speci c are his suggestions “chicken and egg” two-way problem of action–perception because it is a cycle
that cell assemblies are formed by experience-driven synaptic changes and that anyway. An o en-used argument in favor of the perception–action cause–e ect
unique assemblies with assumed boundaries can represent separate objects and direction is the existence of simple re exes, such as the patellar re ex. When a
thoughts. neurologist strikes the patellar tendon with a re ex hammer, it stretches the
However, the absence of a rigorous de nition raises many questions. Who muscle spindle receptors in the quadriceps muscle. e muscle receptors are
has more cell assemblies, Albert Einstein, William Shakespeare, you, or me? If peripheral axon processes of the neurons in the dorsal root ganglion of the
cell assemblies are created by training, as is typically the case in arti cial neu- spinal cord, and they convey the stretch-induced change to the ventral horn
ronal networks, does the brain start with no assemblies and accumulate them neurons of the cord, which in turn send their axons to innervate the same quad-
over the years? How would the brain’s dynamic landscape look without any riceps muscle. Every time the neurologist strikes the tendon, the monosynaptic
experience? What happens to cell assemblies during sleep? How big is a cell as- response is activated and a kick is observed. At the same time, another axon
sembly? As the average path length in the brain is only a few synapses,15 every branch of the neuron in the dorsal root ganglion sends the same spike message
input-triggered ignition would spread to the entire brain through the excita- to another neuron in the spinal cord which, in turn, conveys the information
tory connections. Because Hebb’s original assembly concept does not involve to the thalamus. In line with the Bell-Magendie rule, so the argument goes,
inhibition, it is not clear what mechanisms would mark the boundaries among the sensorimotor re ex must be the evolutionary precursor of perception. But
the neuronal groups. Ideally, these questions would be addressed through who is doing the sensing? e neuron in the ganglion, the ventral horn motor

14. Using the concept of the population vector, the pioneering Miguel Nicolelis of Duke 17. Myosin is a contractile protein found in skeletal muscles. e contraction of human my-
University and John Donoghue of Brown University have dreamed up a spectacular chapter of osin is only twofold slower than the contraction of myosin in the rat, which is more than
translational neuroscience by creating brain–machine interface devices that can read intentions 100-fold smaller (Szent-Györgyi, 1951). For eye-movement speed in humans, see Fischer and
and translate them into the movements of arti cial actuators (Nicolelis and Lebedev, 2009; Ramsperger (1984). Muscles are the second cell type in the body that can rapidly change their
Hochberg et al., 2012). Georgopoulos was a student of the legendary neurophysiologist Vernon membrane potentials (and generate action potentials). Several key molecules in muscles have
Mountcastle, who coined the term “cortical columns” (Mountcastle, 1957). In turn, a prom- similar functions in neurons. Both muscles and neurons can generate energy by both aerobic
inent student of Georgopoulos, Andrew Schwartz, now at Pittsburgh University, perfected and anaerobic metabolism (e.g., Buzsáki et al., 2007).
population vector analysis for a more e cient two-dimensional control of a robotic arm in
18. In skeletal muscles with sensory feedback, fast a erent processing is performed by two
a tetraplegic patient (Collinger et  al., 2013). e brain– machine interface eld is a rapidly
large side appendages of the thalamocortical systems: the olivocerebellar system and the basal
moving translational application of the cell assembly coding ideas.
ganglia. In his best-selling book (I of the Vortex; 2002), Rodolfo Llinás eloquently describes the
15. Sporns (2010). evolutionary reasons that the brain should lag behind the body’s speed.
60

60 THE BRAIN FROM INSIDE OUT Chapter 4. Neuronal Assembly 89

neuron, or the thalamus? What if some disease destroys the motor neurons but (a) (b)
leaves the sensory pathways intact from the muscles and skin to the spinal cord,
thalamus, and the rest of the brain. Of course, no stretch re ex will occur. But
the issue is whether the patient will sense the strike of the hammer. I vote for a
no answer because the strike-induced neuronal activation has no meaning to
the brain since it has not been grounded.
Let’s put my argument into a broader evolutionary perspective: an animal
that can only act has a higher probability of survival than an animal that can
only perceive. Sensing, or being aware of sensing, has no utility whatsoever
unless the organism can act. On the other hand, movement can be useful
even without any sensory information. In seawater, where our ancient animal
(c) (d)
ancestors lived, with abundant food around, rhythmic movement is su cient
to feed a simple creature. Once a movement control mechanism is in place—
and only a er this step—it makes sense to develop sensors that more e ciently
guide movements toward food or shelter to improve the chances of survival.
is argument is supported by another observation that the nervous system
and muscles emerged together in the jelly sh. 19 Only through doing something
do animals have some purpose of being.
From the perspective of evolution, therefore, there is a fundamental distinc-
tion between “perception–action” and “action–perception.” If perception were
the primary objective of brain design, many of our perceptual systems would
appear to be built poorly. Movement, on the other hand, has to have useful Figure 4.1. Population vector coding of reach movements. A: Monkey reaches to one of
consequences to call it an “action.” Disconnecting its consequences on sensing eight possible reach directions. B: Trajectory of arm movements. C: Spiking activity of an
the surrounding world breaks the action–perception loop and renders the example neuron. Note very di erent ring rates in di erent reach directions. D: Accurate
system useless. coding of eight movement directions by the population vector computed from the
Such interruption of the cycle can occur naturally. During rapid eye move- discharge rates of many neurons recorded in the motor cortex. Each one of the eight
ment (REM) sleep, we temporally lose our body, as our muscles cease to be clusters consists of the weighted preferred directions of the neurons (solid lines) and the
calculated neuronal population vector (dotted arrows). e arrows in the center diagram
controlled by the nervous system. When we wake, we immediately get our
indicate the direction of the instructed movement.
body back. Occasionally, these brain functions get de-coordinated in time, and Figure courtesy of Apostolos Georgopoulos. 
we wake up a few seconds before muscle control is reinstated. is terrifying
feeling is known as sleep paralysis. e brain is confused because stimuli im-
pinge on our sensors, but they make no sense and induce no meaning since simultaneous recording of many neurons, improved versions of the population
we cannot act on them. Demonic voices are heard, grotesque incubi dance vector method have been used to decode movement direction in real time. It
on our chest, and we feel threatened. We are not dreaming but fully alert. became an indispensable tool in so-called brain–machine interface experiments,
ese hallucinations, like the illusions we discussed earlier, are an inevitable allowing both monkeys and permanently paralyzed human patients to control

19. Llinás (2002) points out an even more striking example of the primacy of action in the
tunicate, a marine invertebrate animal. In their larval stage, tunicates resemble a tadpole and the movement is in the neuron’s tuning direction. However, this assumption turned out to
move around happily looking for a home base. As soon as they nd a suitable place to settle, be incorrect as the contribution of individual neurons is strongly skewed (Chapter 12). For
they metamorphose into a barrel-like sedentary adult form. In this sessile existence, there is no a contemporary approach to the population coding of arm movement, see Churchland et al.
need for movement, so tunicates digest most of their brains. (2012).
8

88 THE BRAIN FROM INSIDE OUT Chapter 3. Perception from Action 61

assembly hypothesis, neuroscientists tacitly assumed that the recorded single consequence of brain-body-environment interactions.20 We connect to the
neurons had several unrecorded partners and that their mean collective beha- world not through our sensors (although they are essential) but through our
vior was more reliable than the information available from a single neuron.12 actions. is is the only way that sensation/perception can become “grounded”
To test the true collective behavior of neurons and prove or dispute Hebb’s as- to the real world as experience (Chapter  1). e distance between two trees
sembly idea, researchers had to wait until simultaneous recording of large num- and two mountain peaks may appear identical on the retina. It is only through
bers of neurons became possible. walking and moving one’s eyes that such distinctions can be learned by the
brain. Again, I reiterate that the brain’s main function is not veridical percep-
tion and representation the objective world with its mostly meaningless details,
Population Vector but to learn from the consequences of the brain’s actions about those aspects of
the environment that matter for particular goals, such as reduction of hunger.
Apostolos Georgopoulos and his colleagues at the University of Minnesota Recent anatomical and physiological experiments provide abundant support
were interested in the neuronal control of movement direction. ey trained a in favor of the critical role of action in perceptual computation. Primary sen-
monkey to move its arm to one of eight possible targets and observed a striking sory areas, and especially areas designated as “higher order sensory,” receive
relationship between the discharge activity of single neurons and the direction numerous inputs from brain regions de ned as “motor.” In fact, beyond cortical
of the monkey’s arm movement. Many neurons in the motor cortex had a pre- areas which receive direct, thalamus-relayed inputs from the senses and which
ferred direction of reach. at is, they red action potentials maximally when send axons to the spinal cord, all other areas show very strong and widespread
the monkey’s hand moved in a particular direction, less so when it moved to- interconnections.21 Neurophysiological experiments in rodents document that
ward a neighboring target, and not at all when the hand moved opposite to more than half of the variability of neuronal activity across the entire cortex, in-
the preferred direction (Figure 4.1). Although the experimenters recorded only cluding those we refer to as primary sensory, can be explained by self-generated
one neuron at a time, they collected the activity of many neurons, one a er the action parameters such as body motion, whisker movements, orofacial
other, and analyzed them as if they had been recorded simultaneously. is movements, and pupil diameter changes. Such strong action-modulation of
cavalier simpli cation was possible because the monkey’s motor task was very brain activity belittles responses to sensory stimuli. e involvement of wide
stereotypical. brain areas in action suggests that nearly every cortical neuron receives two
To assess the contribution of all neurons to any given action, Georgopoulos types of a erents, conveying both action and sensory inputs, although in dif-
formulated a population vector hypothesis. In such a vector, the contributions ferent proportions.22
of neurons with di erent preferences are summed to produce a nal movement
command. Each neuron res the most spikes when the arm moves toward its
preferred target, but neurons with nearby preferred directions can also sup-
port the same direction less strongly, so the nal vote is calculated by vecto- 20. I was scared to death during my rst sleep-paralysis experience. But safe recoveries from
rial summation of preferred directions of individual neurons weighted by their the momentary early morning attacks allowed me to internalize the experience (as discussed in
Chapter 5), and they no longer bother me. In other words, I “grounded” these experiences by
ring rates. By examining the ring rates of many direction-tuned neurons in assigning a neutral meaning to them. You may have had similar adventures; nightmares occur
a given time window, the population vector model can precisely describe the in about one-third of the adult population (Ohayon et al., 1999) though usually during non-
resulting movement direction. Moreover, when the monkeys were required to REM sleep, which is di erent from sleep paralysis. Horri c experiences in rare cases of anes-
translate locations of visual stimuli into spatially shi ed reach targets, the pop- thesia arise from the same mechanism: the brain can (mis)interpret sensory inputs without its
ability to act. Anxiety and panic attacks may involve similar mechanisms due to our inability or
ulation vector accurately predicted mental rotations.13 In recent years, using
perceived inability to change a situation and to be in charge.
21. Petreanu et al. (2012); Economo et al. (2016); Chen et al. (2018); Han et al. (2018); Harris
12. A similar sentiment was also expressed by von Neumann (1958), who suggested that mul- et al. (2018).
tiple neurons redundantly work together and that their joined activity corresponds to a bit of
22. Stringer et al. (2018) concluded that “an understanding of the function of sensory cortex
information.
will likely be impossible without measuring and understanding its relation to ongoing behavior,
13. Georgopoulos et al. (1986, 1989). An implicit prediction of the population vector idea is beyond easily-characterized measures such as running and pupil dilation.” See also Musall et al.
that neurons are relatively equal in their contribution to command muscle movements when (2018).
62

62 THE BRAIN FROM INSIDE OUT Chapter 4. Neuronal Assembly 87

SHORT- CIRCUITING enlarges them if they already exist) in contact with the soma of the second cell.”
THE ACTION– PERC EPTION CYCLE Or, put more simply, “neurons that re together wire together.”9
Hebb’s de nition of a cell assembly relies on the structural and physiological
When we shout loudly near someone’s ears, she may have di culty hearing for connectivity of neurons. Speci cally, its members are connected by excitatory
a few seconds. In contrast, we can resume normal conversation right a er our synapses created or strengthened by experience. Once a cell assembly is formed,
own shouting is heard by our ears. We are protected in that case by multiple activation of a small group of members can reactivate its entire spatiotemporal
mechanisms in our auditory system, as the Greeks speculated long ago. Today, signature. e assembly and plasticity concepts of Hebb’s hypothesis de ned
we have a name for it: corollary discharge. As the name implies, this ancillary a program in theoretical neuroscience that has prevailed to the current day.
activity occurs simultaneously with the action output. e action-initiating Based on this hypothesis, cognitive psychology established a grand and com-
circuits of the brain send action potentials not only to the downstream motor prehensive research project to link psychological and physiological processes.10
pathways but also synchronously to other areas in the brain. is secondary It opened the path to tracking down the physiological mechanisms of classi -
activity provides a feedback-reporting mechanism for self-organized action. “I cation and categorization.11
am the agent that brought about change in the sensors.” While the concept of the cell assembly has proved useful, Hebb’s de nition
A particular sensory modality may be activated when a stimulus unexpectedly made its alleged substrate virtually unidenti able. In addition, tracking down
occurs or when the animal is actively searching for the stimulus or expecting it. the signatures of cell assemblies in physiological experiments with the available
ese two types of stimuli evoke di erent responses, like the feelings when you single-neuron method turned out to be a formidable task. Investigators noted
touch or are touched. An unexpected stimulus can induce either a generalized early on the large trial-to-trial variability of neuronal ring patterns. Even the
response, such as a startle reaction in response to a loud noise, or an orienting “best” neuron, which robustly responded to a stimulus on some trials, red
response.23 e orienting response is the brain’s active attempt to learn more one lonely spike or remained completely silent on other trials. Under the cell
about an unexpected event.
As discussed in Chapter 1, neurons cannot interpret the relevance of the sig-
9. e original sentence is “neurons wire together if they re together” (Löwel and Singer,
nals conveyed by sensory inputs because they cannot ground their response 1992), although the phrase has been most widely popularized Carla Schatz. For demonstration
without some independent veri cation. For neuronal networks to interpret the of Hebb’s rule, known also as “spike timing-dependent plasticity” (or STDP) see Magee and
world, they need two types of information for comparison. e extra informa- Johnston (1997); Markram et al. (1997); Bi and Poo (1998). Prior to Hebb, the timing rule was
tion can be supplied by the movement-induced changes of the brain’s sensors. already well-known from Pavlovian conditioning. e conditional signal (CS) always must pre-
cede the unconditional signal (US), and both the CS and US should occur with similar prob-
Only by comparing two signals, one of which is grounded by movement or
ability to produce successful associations. Reversing the contingency actually makes the CS a
previous knowledge (which was also generated by action at some point in life), predictor of US at a lower than chance probability. Before the in vitro experiments on STDP,
can the brain gure out what happened out there. is distinction is easy to Levy and Steward (1983) had already demonstrated the paramount importance of timing
demonstrate by closing one eye and moving the open eyeball from the side in anesthetized rats by associating the contralateral (weak) and ipsilateral (strong, teacher)
entorhinal input-evoked local eld potential (LFP) responses in the dentate gyrus.
10. e concepts of the cell assembly and spike timing-dependent plasticity have become the
guiding rules in many forms of arti cial neural networks. Due to connectedness, activity in a
23. e term “orienting re ex” was coined by Ivan Pavlov and studied most extensively by
few neurons tends to activate all members of an assembly. As a result, the pattern as a whole
Evgeny Sokolov (1960, 1963). Sokolov noted that if a novel signal is not followed by reward,
becomes “auto-associated” and xed to represent a particular item. e most popular model
the animal quickly learns to ignore it, a process he called habituation. Interestingly, Sokolov
based on these principles is the Hop eld attractor network (Hop eld, 1982). Activity in the
used the same comparator circuit as used in the corollary discharge model (Figure 3.1) to de-
Hop eld network varies with time and can jump or move slowly from one stable state (called
scribe how the brain (particularly the hippocampus) can detect whether something is novel or
the “attractor”) to the next. e jumps of the activity bump can be considered to be Hebb’s
not. is is perhaps not surprising. When an eye saccade occurs, the brain makes a prediction
phase sequence. e number of items (memories) that can be stored with tolerable interference
about the expected content of the scene a er the saccade based on the current state of its sen-
scales with the number of neurons.
sory knowledge (a Bayesian prior; Chapter 13). If nothing new happens, no sensory change is
registered. However, if a novel thing appears during the saccade, there is a mismatch between 11. According to Gerald Edelman, the most fundamental brain operations are integration and
expected and actual inputs (i.e., if the prior model is erroneous, the model is updated). e pro- segregation (Edelman, 1987; Tononi et al., 1994). Other antonym pairs, such as parsing versus
cess of updating can be called perception. Behaviorally, a mismatch between the predicted and grouping, di erentiation versus generalization, and pattern separation versus pattern comple-
detected states of the sensors will trigger an action: the orienting response. tion, are also frequently used.
86

86 THE BRAIN FROM INSIDE OUT Chapter 3. Perception from Action 63

hypothetical neuronal assemblies with assumed discrete boundaries could rep- with your nger. e entire visual eld will appear to move at the pace of your
resent objects or even abstract entities of thought. Coupling two assemblies nger’s movement. In contrast, when you move your head back and forth or
could serve as a neuronal substrate for associations, as activation of one as- scan the world with your eyes, producing a similar pattern of stimulation on
sembly could then trigger another assembly. Flexibly linking several assemblies the retina, the world appears stationary even if you blink or make saccadic eye
into a ring could underlie short-term memory, as reverberation of the ac- movements. Critically, if your eye muscles were paralyzed for some reason, and
tivity could persist a er the triggering stimulus had vanished. e chaining you intended to make an eye movement to the right, then you would observe a
of such hypothetical cell assemblies, in turn, could support complex cognitive shi of the visual eld in the same direction. 24 What would be a proactive brain
processes, such as memory recall, thinking, and planning, assuming that some mechanism to compensate for such confusing shi s?
internal process can indeed generate such sequences (Chapter 6).7
Hebb also realized that a second rule is necessary to exibly connect
assemblies and reinforce these connections so that they can support long- The Proactive Corollary Discharge Mechanism
term associations. He suggested a general way that synaptic connections be-
tween pairs of neurons could be modi ed by appropriate timing of their action Distinguishing the real and perceived worlds has entertained both philosophers
potentials. Modi cation of many such connections could form a new memory and artists prior to neuroscience. Perhaps the Italian painter Filippo Brunelleschi
or an “engram.”8 A historical paradox is that his second rule, the Hebbian plas- was the rst to test it experimentally. On a wooden panel about twelve inches
ticity rule or simply Hebb’s rule, an idea that he borrowed from others, is what square, he painted the Baptistery of Florence in its surroundings. In the center, he
earned him a prominent place in the neuroscience hall of fame. In his own made a hole and invited the viewer to look through the back of the painting while
words, “ e general idea is an old one, that any two cells or systems of cells that he held a mirror. Brunelleschi carefully positioned the painting (i.e., its re ec-
are repeatedly active at the same time will tend to become ‘associated,’ so that tion in the mirror) so that the real baptistery and the painting lined up perfectly.
activity in one facilitates activity in the other.” Hebb’s one-sentence rule became rough the mirror, the real world looked more two-dimensional and the painting
perhaps the most o en cited quote in neuroscience, “When one cell repeatedly more three-dimensional, thus the viewer could not distinguish easily between the
assists in ring another, the axon of the rst cell develops synaptic knobs (or two mirror re ections. His goal was to demonstrate that only by moving the head
back and forth can reality and image be distinguished, making and unmaking the
viewer’s own world. It is the viewer’s voluntary action of head movement in this
artistic peep-show that can distinguish real change from apparent change.25
Erich von Holst and Horst Mittelstaedt, working on sh and invertebrates
7. Hebb’s ideas shaped the thinking of numerous researchers who incorporated his ideas in var- in Germany, and Roger Sperry, studying the optokinetic re ex in humans at
ious ways into their brain theories; for example, Miller (1956); Marr (1971); Braitenberg (1971);
the California Institute of Technology, went further than Brunelleschi. ey
John (1972); Shaw et al. (1985); Damasio (1989); Abeles (1991); Churchland and Sejnowski
(1992); Edelman (1987); Wickelgren (1999); Pulvermüller (2003); McGregor (1993); Miller wondered whether a motor command exits the brain unnoticed or leaves a trace
(1996); Milner (1996); Kelso (1995); Mesulam (1998); Laurent (1999); Varela et  al. (2001); behind. 26 ey came to the conclusion that there must be brain mechanisms
Yuste et al. (2005); Harris (2005); Buzsáki (2010) and brain models, e.g., Willshaw et al. (1969); that separate self-generated stimulation from externally induced stimulation
Palm and Aertsen (1986); Hop eld (1982); Amit (1988); Bienenstock (1994); Wennekers et al.
(2003). e term “cell assembly” evolved over years, and one might argue that it no longer
24. is is exactly what patients with paralysis of the eye muscles see. Herr Professor
resembles what Hebb originally intended. In my discussion, I use Hebb’s original de nition.
Kornmüller (1931) famously anaesthetized his own eye muscles to investigate what happens
8. e idea of the “engram,” used to de ne a hypothetical group of neurons representing when the volitional act is blocked. He reported an apparent shi of the visual eld, which could
memories, is closely related to the concept of cell assembly. e term “engram” was introduced not be distinguished from a similar magnitude of true shi of visual input.
by German psychologist Richard Semon (1859–1918; see Schachter, 2001)  and popularized
25. As described in Wootton (2015, p. 165).
by Karl Lashley (1930), who made a lifelong search to identify the engram. Lashley’s “failure”
to localize it could be explained by the lack of a good de nition or its widely distributed na- 26. e paper by von Holst and Horst Mittelstaedt (1950) is an extraordinary read. In thirty
ture. e Nobel Laurate Susumu Tonegawa at MIT has recently revived the term “engram” and pages, the entire history of motor control and sensory physiology is exposed, many experiments
performed a series of experiments with optogenetical methods to erase and implant memories from ies to sh are presented, and the ndings are summarized in quantitative models. e
into the hippocampus of mice (Tonegawa et al., 2015). For a review of the history and present authors alert the reader that the brain of even the simplest creatures consists of more than
status of the engram, see Schacter (2001), Josselyn et al. (2015). “simply a set of connecting cables between receptors and muscles!” ey not only demonstrated
64

64 THE BRAIN FROM INSIDE OUT Chapter 4. Neuronal Assembly 85

of the brain’s sensors. is hypothetical mechanism assumes that a copy of disband themselves when not needed emerged. is is the cell assembly or neu-
the motor command signal leaves an image of itself somewhere in the brain. ronal ensemble hypothesis.
Sperry called it “corollary discharge,” while the German duo referred to it as
“rea erenz,” both of which likened it to the negative of a photograph. In turn,
the stimulation from the sensor during the movement is the positive print that, CELL ASSEMBLY
when united with the negative, makes the image disappear. Experiments by
Sperry, von Holst, and Mittelstaedt were among the rst to demonstrate that e concept of neuronal assembly is one of those things that all neuroscientists
action directly in uences sensation. Neurons giving rise to an eventual motor talk about, weave into their important ndings, and use to explain complex
command send a copy to neurons dealing with the sensor-supplied signals. observations—even the workings of the mind—yet it lacks a rigorous de ni-
ese latter neurons (call them comparators) can compare what comes in and tion. As a result, scientists de ne cell assembly di erently. e concept is most
what goes out from the brain27 (Figure 3.1). o en associated with Donald O. Hebb, who coined the term in his classic book
e corollary discharge (i.e., the mechanisms that allows the sensory system e Organization of Behavior.5 Hebb recognized that a single neuron cannot af-
to be informed about an action command) is an invention of the brain. No such fect its targets reliably and suggested that a discrete, physically interconnected
internal rea erentation exists from the ventral horn motoneurons to the sen- group of spiking neurons (the cell assembly) is the unit that can represent a
sory side of the spinal cord. So Betz’s intuition is only partially right. ere is distinct percept, cognitive entity, or concept. An assembly of neurons does not
indeed a geometric separation of the posterior “sensory” and anterior “motor” live in isolation but communicates e ectively with other assemblies. Because
areas in the brain. However, their way of communication is very di erent from of the assumed strong interconnectivity of the assembly members, the activa-
that of the spinal cord. e spinal cord is a modular system, with each module tion of a su cient number of them can activate the entire assembly, a pro-
utilizing pretty much the same algorithm and intersegmental coordination. It is cess described in early texts as “ignition” of the assembly. Hebb needed both a
mostly about re exes and locomotion supported by a large number of neurons. large group of interconnected neurons and a trigger because he was in search
of “representations” of the outside world and wanted to understand how one
signal leads to another.
that an insect can discriminate between its own movements and movements in its environ-
ment but were con dent enough to generalize their ndings: “the rea erence principle applies
e assembly idea likely grew out of the Berliner Gestalt psychology at the
throughout the CNS [central nervous system] from the lowest phenomena (internal and ex- beginning of the past century, which de ned the hypothetical brain substrates
ternal control of the limbs, relations of di erent parts of the body to each other) to the highest of perception on the principles of proximity, similarity, and good continua-
(orientation in space, perception, illusions).” Compare this with Sperry’s description “[ e] tion.6 Hebb’s cell-assembly concept could provide brain-based explanations,
kinetic component may arise centrally as part of the excitation pattern of the overt movement.
at least in principle, for many experimental psychological observations. ese
us, any excitation pattern that normally results in a movement that will cause a displace-
ment of the visual image in the retina may have a corollary discharge into the visual centers to
compensate for the retinal displacement. is implies an anticipatory adjustment in the visual 5. I  am Donald Hebb’s scienti c “grandson.” Cornelius (Case) Vanderwolf was one of his
centers speci c for each movement with regard to its direction and speed . . . with the retinal few students and also my postdoctoral advisor (http://neurotree.org/neurotree/). Despite our
eld rotated 180 degrees, any such anticipatory adjustment would be in diametric disharmony “family” relationship, it is fair to point out that Hebb was not the only person to conceive the
with the retinal input, and would therefore cause accentuation rather than cancellation of the assembly or ensemble idea. Similar suggestions were put forward by James (1890), Sherrington
illusory movement” (Sperry, 1950, p. 488). e essential features of corollary discharge are also (1942), Nikolai Bernstein (1947 in Russian; English translation, 1967), and Konorski (1948).
the foundation of predictive coding (Rao and Ballard, 1999; Kilner et  al., 2007)  and motor But perhaps the rst credit should go to Yves Delage (1919): “every modi cation engraved in
control theory (Shumway-Cook and Woollacott, 1995; Wolpert et  al., 1995; Kawato, 1999; the neuron’s vibratory mode as a result of its co-action with others leaves a trace that is more
Grush, 2004). or less permanent in the vibratory mode resulting from its hereditary structures and from the
e ects of its previous coactions. us, its current vibratory mode re ects the entire history
27. e corollary discharge mechanism represents a basic pattern of loop organization of the
of its previous participations in diverse representations” (Translation by Frégnac et al., 2010).
brain. Output patterns inform input analyzer circuits. We will discuss extensions of this fun-
Delage thus had already combined the advantages of oscillations and cell groupings, and his
damental mechanism in multiple chapters. e comparator circuit has been long viewed as
description also relates to Hebb’s second major concept (i.e., the spike timing-based plasticity
a fundamental mechanism of many brain functions (Sokolov, 1960, 1963). MacKay’s (1956)
rule). Sub sole nihil novi est.
epistemological automaton compares actual and expected sensory inputs. Excitatory recur-
rent feedback circuits might be viewed as an internalized version of the corollary discharge 6. e Gestalt idea of psychology has served as an important and recurring counterpoint to the
mechanism. single-neuron “doctrine” of physiology (Barlow, 1972).
84

84 THE BRAIN FROM INSIDE OUT Chapter 3. Perception from Action 65

to probe their responses with sounds and odorants. Instead, we use visual
patterns to probe activity patterns in visual areas. However, as we move deeper
into the brain, the idea of a relevant stimulus becomes increasingly more
ambiguous. Furthermore, when we study more complex behaviors, such as
emotions or memory, the correlational approach needs to be complemented
by perturbational methods. Finding the best correlate of the activity of a single
neuron and declaring that it responds exclusively or even primarily to, for ex-
ample, the orientation or color of a visual stimulus, is a heroic venture because
the same neurons should be tested over and over, while making sure that the
brain state of the animal does not change in the meanwhile. Moreover, nding
the physiologically relevant “tuning” response of the neuron may fail if we do
not already know its job at least approximately. For example, a neuron may
respond relatively selectively to a particular orientation of a black vertical bar
even though its main contribution is to color identi cation. e outside-in ap-
proach can provide a tremendous amount of useful information, but it is inher-
ently handicapped because a neuron can be probed only in a few conditions in
Figure 3.1. Schematics of the corollary discharge mechanism. Motor command signal
a given experiment.
is sent from the motor areas to eye muscles (e erent signal). At the same time, corollary
Although the spiking activity of a single neuron can o er only limited infor- discharge (dashed arrows) is also sent to comparator mechanisms in the sensory system.
mation, we could perhaps get an average picture of its physiological function by e comparator performs a subtraction or divisive normalization on the external
repeating the same experiment again and again, collecting more neurons under (exa erent) signal determined by the corollary discharge. In addition, the magnitude
the same experimental conditions, and “pretending” that all neurons were re- of the rea erent signal from the tension sensor in the muscle can exert a delayed a ect
corded simultaneously. is idea is similar to asking each player of an orchestra on the sensory signal. Projections from motor cortex to sensory cortical areas are a
to play his or her part separately and then combine the individual pieces into common architectural feature in all mammals.
a whole. is is the rationale of the single-neuron doctrine.3 A reasonable ar-
gument against the single-neuron framework is that, in a exible brain circuit, In fact, the volume of the spinal cord in the elephant is larger than the brain
neurons can be repurposed for multiple jobs, like a neuroscientist who can of a macaque monkey. So the “smartness” of the brain is due not only to large
also function as a parent, a tennis partner, or a handyman, depending on the numbers of neurons and synaptic connections. What also matters is how those
circumstances.4 More importantly, with single neurons recorded one at a time, connections are organized. e corollary recurrent loop from the output to the
one cannot examine how neurons in uence each other. A jazz song pieced to- input is such a key motif of the cortex. I will return to recurrency, positive feed-
gether from the music played by isolated members will likely not sound right. back, and corollary discharge in several chapters to argue that these mechanisms
Based on these considerations, theoreticians began to question that the single are the most important grounding processes of cognitive operations.
neuron is the fundamental unit of computation in the brain. Instead, the idea
that a group of neurons that can align themselves for a particular purpose and
Two Types of Corollary Discharge Mechanisms

3. e classic paper in this eld of research is by Barlow (1972). ere are at least two ways to distinguish the sensory stimulation produced by
4. For example, the same neurons in the entorhinal cortex can wear many hats. ey may be one’s own movement from real motion in the surrounding world. e rst one
classi ed as positional, speed, or head-directional neurons. Although a minority of neurons can be called adaptive ltering, an example of which can be found in crickets.
can appear as specialists, the majority have mixed properties. Instead of individual functional
classes, all neurons may have multiplexed properties (Hardcastle et al., 2017). Particular be-
Have you ever tried to catch a chirping cricket? Crickets are wildly loud, but
havioral correlates, designated by the experimenter, may in fact correspond to tails of wide when you move toward them, they detect your motion and become quiet. For
distributions of response features (Chapter 12). this reason, farmers in rural Japan used crickets as sentinels to protect their
6

66 THE BRAIN FROM INSIDE OUT

households. ey put them in small cages where they sang during the dark
hours of the night, except when they sensed something moving. e ensuing si-
lence then served to alert the householder.28 Crickets do not have ears, but they
are equipped with a pair of tympanal organs on their legs. is super-sensitive 4
device can detect vibrating air molecules (sound), which provides their primary
alarm mechanism to avoid predators. In addition to listening, male crickets also
Neuronal Assembly
“sing”; that is, they have a le and scraper body part on their wings that makes
a loud chirping noise. eir 100-decibel chirps are several-fold louder than the
sound of your footsteps, yet the cricket can be fully alert while chirping. Its
The Fundamental Unit of Communication
brain counteracts the massive in ux of auditory, proprioceptive, and mechano-
receptive information during sound production by inhibiting its central audi-
tory neurons with a corollary discharge synchronously with sound production.
Remarkably, in the cricket’s brain, the mechanisms responsible for this forward
signaling are accomplished by a single inhibitory neuron. When this corol-
lary discharge neuron is active, it selectively inhibits the response of auditory
neurons associated with self-generated chirps. A subtraction or division/nor-
malization process (Chapter 11) in these auditory neurons removes the loud E pluribus unum. (Out of many, one.)
self-generated vibration signal yet maintains the cricket’s ability to detect much —G S U S A
quieter external stimuli.29 United we stand, divided we fall.
Distinguishing real changes in the environment from apparent sensory —T S K
changes due to one’s own movement can be also accomplished through a second
mechanism that could be called “time division” because corollary discharge and Success is determined by the collective.
signal detection occur at di erent times. For example, when you blink or pro- —L B
duce saccadic eye movements, the input from the world to the retina disappears
or gets blurred, but we do not notice that. An ideal observer could “see” every-
thing around itself all the time, but in the real world the “seeing” is a compro-

T
mise between spatial resolution and speed. A  y’s eye and a digital camera both he outside-in framework o ered an explicit recipe to neuroscience for
have a certain number of pixels to work with. e camera typically scans the how to explore the brain:  present various stimuli and monitor brain
target pixel by pixel, and its e ective resolution is determined by the number responses. To study learning, we should register the relationship between
the stimulus and neuronal response at various stages of learning. Similarly, to
study motor behavior, we should establish a reliable relationship between neu-
28. As told by Dethier (1987). His essay is worth reading for both scienti c insight and the ronal activity and movement patterns. In principle, anything that the experi-
aesthetic beauty of the narrative. menter can conceive, including complex terms that describe various aspects of
29. In a remarkable series of studies, Poulet and Hedwig (2006) used simultaneous intracel- cognition (check out Figure 1.1 again), can be correlated with neuronal activity.
lular recordings of auditory neurons in the prothoracic ganglion, where primary auditory sig- Recording neurons, one at a time, was one of the rst popular techniques
nals are processed, while searching for the corollary discharge in the mesothoracic ganglion,
in the arsenal of neurophysiology.2 e goal for the experimenter is to nd
which houses part of the chirp pattern-generating network. e dendrites and cell body of the
inhibitory corollary discharge neuron are surrounded by the chirp pattern generator neurons, the relevant stimulus that can induce reliable neuronal ring. For example,
which excite it, and its extremely large axon tree forms numerous synapses with the auditory when recording from the ganglion cells of the retina, it would not make sense
neuropil in the prothoracic ganglion. e neuron generates bursts of spikes in synchrony with
each chirp but does not respond to external sound. Its destruction has no impact on chirp pro-
duction, so its sole job description is to provide corollary discharge to the auditory neurons. To 1. Barabási (2018).
date, this is the clearest mechanistic description of such a mechanism. 2. Hubel (1957); Evarts (1964).
82

Chapter 3. Perception from Action 67

of pixels the processor handles in a given amount of time. e compound eye

of the y has approximately 3,000 pixels, and, by ying fast (perhaps y times
faster than we walk, relative to body size), the y generates an image shi on
the retina, called optic ow, that scans a large part of its environment. e y’s
erratic-looking zig-zag ights are actually stop-go saccadic patterns that ac-
tively shape its visual input by enhancing depth resolution and distinguishing
gures from the background.
Other creatures scan the world di erently. Frogs lack eye muscles, but respi-
ration moves their eyes rhythmically. e preying mantis moves its entire head
in saccadic scans. Spiders, with their body anchored to their net, protrude their
eyes and rotate them like miniature periscopes to view the world. Vertebrates
with superior brain processing power exploit both spatial resolution and scan-
ning strategies. e eyes of primates, including humans, have many cones in
the foveal region for accurate vision and many more rods surrounding the
fovea to detect motion. Primate eyes can move either slowly (explorative) or
quickly (saccadic). Like spiders, we can stay stationary yet monitor a very large
part of our surroundings as long as our brain moves our eyes. During saccadic
eye movements, the visual scene does not get blurred. Instead, it is suppressed
momentarily by postulated corollary discharge. e visual eld appears to be at
one place or another, without experiencing the in-between scan. You can check
this by looking into a mirror. You see yourself continuously and do not notice
your saccadic eye movements. e suppression mechanism alerts visual pro-
cessing areas in the brain that the forthcoming disruption is the result of your
brain’s command, not a change in the visual world.
In the primate brain, the neuronal hardware of this forward signaling is a bit
more complicated than in the cricket, but the principle is the same. Neurons
in the frontal eye eld area of the neocortex are good candidates for the job
because they receive both visual and corollary discharge information about
eye saccades from the medial dorsal nucleus of the thalamus. A er comparing
the visual input with the grounding information about eye movement, frontal
eye eld neurons inform their targets whether the stimulus remained stable
or moved, the strength and timing of their corollary discharge response, and
the magnitude of stimulus translation if the scene moved during the saccade.
Inactivation of the action path, for example by infusing local anesthetic in the
thalamic medial dorsal nucleus to abolish spiking activity locally, will result in
a failure to inform the visual neurons about the necessary grounding informa-
tion. As a result, frontal eye eld neurons supply nonsense information to their
targets. Under such conditions, the experimental monkey reports that the visual
scene jumps with each saccade just as humans feel a er eye muscle paralysis.
In further support for the role of the frontal eye eld, experiments in monkeys
show that microstimulation of this area enhances the responsiveness of visual
68

68 THE BRAIN FROM INSIDE OUT Chapter 3. Perception from Action 81

cortical neurons with spatially overlapping receptive elds.30 e just-discussed why movements of the “other body” are inevitably sensed by both brains. It is
experiments and observations indicate that the self-motion–generating circuits because the physical constraints of the skeletal system have enabled the incor-
provide a prediction signal for the sensory system so that the expected signal poration of a body extension. e other body becomes part of the body scheme
can be compared with the one conveyed to the brain by the sensors. as a result of action-induced joint experience.
If action is so critical for perception, it should play a signi cant role in cog-
nition as well. at this is indeed the case, I will discuss in Chapter 5. However,
The Illusion of Continuous Vision a prerequisite of that discussion is the understanding of the collective behavior
of neurons, the topic of the next chapter.
Sampling of the visual world is not continuous because it is interrupted by
saccades, with each saccade causing a loss of up to 10% in sampling time.
However, this loss has important advantages. First, blurred vision is prevented SUMMARY
because during the saccade the visual input is suppressed. Second, corollary
discharge is an important timing signal that helps coordinate neuronal activity e short message of this long chapter is that perception is an action-based
beyond the visual system. ird, suppressed spiking in several types of visual process, an exploration initiated by the brain. is message is fundamentally
neurons during the saccade allows them to replenish their resources, for ex- di erent from the representational view, fueled by the outside-in empiricist phi-
ample by restituting dendritic sodium and calcium ion channels inactivated losophy, which inevitably poses the question: What comes between perception
during intense spiking. As a result, a er the saccade, the visual system is tran- and action? e homunculus with its decision-making power—and, I suspect,
siently more sensitive to stimulation, a considerable gain. the problem of consciousness as well—are unavoidable logical consequences of
Action in the form of muscle activity (again!) may be the evolutionary or- the separation of perception from action.
igin of intermittent sampling of sensory inputs. e muscle’s fastest reaction is I promote the alternative view: things and events in the world can acquire
a twitch followed by a refractory period, during which it cannot be activated. meaning only through brain-initiated actions. In this process, the brain does not
is mechanical constraint may explain why sensors adapted to such a regime. represent the world in its numerous and largely irrelevant details but extracts
Regardless of its origin, intermittent sampling is advantageous for transmitting those aspects that have become relevant to the organism by exploration. us
information. First, intermittency introduces a way to chunk the information, the brain builds a simpli ed, customized model of the world by encoding
similar to spaces between words. us, every neuronal message can have a be- the relationships of events to each other. ese aspects of model building are
ginning and stop code (Chapter 6). Second, intermittency simpli es how infor- uniquely di erent from brain to brain.
mation processing can be coordinated across brain areas because it introduces e critical physiological mechanism that grounds the sensory input to make
the means of generating a clear time reference frame that can be shared by all it an experience is “corollary discharge”: a reference copy of a motor command
mechanisms involved. Blind people discovered the supremacy of intermittent sent to a comparator circuit from the action-initiating brain areas. is com-
parator mechanism allows the brain to examine the relationship between a true
30. Neurons in the parietal cortex (particularly in the lateral intraparietal area) can also func- change in the sensory input and a change due to self-initiated movement of the
tion as corollary discharge comparators and drive neurons in the frontal eye eld. Other brain sensors. e same corollary discharge mechanism also serves active sensing,
regions, including the lateral geniculate body, superior colliculus, and even early visual areas the process by which sensory receptors can be most e ciently utilized to sense
V1 to V4 appear to be involved. In addition to suppression of retinal inputs, many interesting the environment.
things happen in the brain around the time of saccadic eye movements, including compres-
sion of space and time and displacement of part of the world depending on the direction of
the saccade (see Chapter  10). e magno- and parvocellular pathways in the early stages of
the visual system are a ected di erentially. It is mainly the magnocellular pathway of the lat-
eral geniculate body that is suppressed by saccades and informs the motion centers. In addi-
tion to corollary mechanisms, visual motion caused by the eye movement itself can contribute
to masking vision. Duhamel et al. (1992); Umeno and Goldberg (1997); Sommer and Wurtz
(2006); Crapse and Sommer (2012); Moore et al. (2003). For reviews, see Ross et al. (2001);
Crapse and Sommer (2008).
80

80 THE BRAIN FROM INSIDE OUT Chapter 3. Perception from Action 69

touches on the two hands are synchronized, the participant soon feels that she sensory sampling by realizing that forms constructed from raised dots were
is touching her own hand. e illusory touch, in turn, will activate the ventral more easily discriminated by the ngertip than continuous raised lines.31
premotor cortex, intraparietal cortex, and cerebellum, indicating that the illu- Sensing text with ngers also begins with movement.
sion re ects the detection of congruent multisensory signals from one’s own
body, as revealed in imaging experiments.52
ese illusions derive from the lack of grounding by action. If the participant PERC EPTION AND ACTIVE SENSING
moves her hand and arm while the rubber hand is being touched, then no il-
lusion arises. Furthermore, the participant loses the sense of ownership of the Not everyone has the luxury of saccadic eye movements. A  research partici-
arm instantaneously when the experimenter moves it. An obvious direct test of pant named AI was unable to move her eyes since birth because of brosis in
the action–perception framework would be to deprive the brain entirely of its her eye muscles. Yet she had no complaints about her vision. She could read
motor and autonomic outputs and examine whether sensation could be evoked and write, although her reading speed was slow. She accomplished this feat,
in a brain that had never experienced touch in conjunction with action. Short compensating for her lack of eye movements, by making head saccades. ese
of such an action-deprivation experiment, we could ask how the brain would saw-tooth movements to her right had an average duration of 200 ms. At the
handle an unusual body. For example, what if two brains shared one body? end of the line, her head jerked back to the le side of the page to read the
Nature has provided several such examples. next line. During picture viewing, AI’s head movements scanned the important
details of the picture, similarly to the way we deploy eye saccades to extract in-
teresting features from a scene (Figure 3.2). By actively moving her head (and
Two Brains Control One Body consequently her eyes), AI could enhance her visual sensing ability.32
Active sensing refers to a brain-initiated search as opposed to the response to
Abby and Brittany are conjoined twins who live in Minnesota. ey were an expected event. 33 Perhaps another word, “observation,” equally well captures
charming children, and today they are self-con dent adults. e twins have the same process. In the real world, stimuli are not given to the brain. It has to
separate heads and spinal cords, a joint chest with two hearts, two hands, and acquire them. e sensitivity of sensors depends, in part, on the e ectors that
two legs. Each brain and spinal cord innervates one hand and one leg. e sense can move them and maximize their e cacy. It is a bit like the relationship be-
of touch and limb control are restricted to the half of the body that each brain tween a camera and its user. e camera takes the photo, but what is captured
innervates. Yet Abby and Brittany can perform many movement patterns that depends on the action of the user. Unexpected stimuli, when they occur inde-
require bilateral hand and leg coordination, such as walking, running, swim- pendently of movement, trigger search behavior and optimize the sensors. e
ming, ball games, or driving a car. ey have di erent tastes and even di erent physiological foundation of such active sensing is also corollary discharge.
career goals.53 Because their brains have separate somatosensory innervation Active sensing occurs in two forms: the entire body can translocate (e.g., the
of parts of their shared body, many observers have wondered how they e ec- y’s solution for vision), or sensors are moved locally (e.g., the spider’s solution
tively coordinate their sensations to perform the synchronous or alternating
movements needed for many simple or more complex activities. However, if 31. Louis Braille, who was blind from childhood, went on to create the Braille tactile writing
action teaches perception, as I suggested in this chapter, then we can explain system, a sort of parallel Morse code.
32. Gilchrist et al. (1997).
33. Ehud Ahissar at the Weizmann Institute refers to active sensing as “closed-loop” sensing,
52. Botvinick (2004); Ehrsson et al. (2005).
in contrast to the perception–action semi-loop, in which the interaction between the sen-
53. ere are numerous YouTube sites on these twins; visit https:// www.yahoo.com/tv/ sory organ and the environment is the rst step (Ahissar and Assa, 2016). James J.  Gibson’s
conjoined-twins--abby---brittany--get-their-own-reality-show- -video-.html. Conjoined twins, a ordance theory can be considered as a forerunner of active sensing. According to his theory,
when they are respectfully approached, could be a rich source of unique information on brain- not only is the world perceived in terms of object shapes and spatial relationships, but these
body questions. Among these is the important question of whether a feeling of anxiety is a things also o er intuitive possibilities for action (a ordances). e environment “o ers the an-
body-induced e ect on the brain. For example, researchers could show anxiogenic pictures imal, what it provides or furnishes, either for good or ill” (Gibson, 1979). While Gibson tried to
to one twin and can ask the other twin how she feels. Many other brain-body-brain questions eliminate the decision- maker between perception and action, he still thought that perception
could be explored in conjoined twins. drives action.
70

70 THE BRAIN FROM INSIDE OUT Chapter 3. Perception from Action 79

between movement and spindles does not completely vanish. e reader may
have experienced occasional dramatic whole-body jerks just before sleep. Such
movements trigger a sleep spindle, as in premature babies.

ACTION AFFECTS BODY MAP S

From my perspective, the term “somatosensory” cortex is a misnomer of the

outside-in framework as it not only senses but also simulates the body.49 is
is most clearly demonstrated by the continued representation of missing body
parts. e ownership of phantom limbs and the painful feelings that arises
from “them” a er their loss or amputation is a well-studied clinical problem. If
the sensory cortex were truly sensory and fully dependent on external inputs,
phantom limbs would not be constructed mentally.50 Another striking example
Figure 3.2. Head movements of patient AI with paralyzed eyes (head saccades) and eye
of body simulation by the somatosensory system is false body part illusions.
movements from a control subject (eye saccades) during text reading recorded using
a head-mounted search coil. Although AI is considerably slower overall, and her head
In the laboratory, a research participant is seated with her le arm resting on a
stability is not as good as that of the control subject, her saccade strategy is the same as small table, hidden from her view by a screen, and a life-sized rubber model of
that of the normal subject. Right, AI's head movements while viewing a picture. a hand and arm is placed on the table directly in front of the participant. e
Redrawn a er Gilchrist et al. (1997). participant is asked to x her eyes on the arti cial hand while the experimenter
visibly touches the rubber hand with a paintbrush and, at the same time and
unknown to the participant, also touches her hidden hand with another paint-
for vision), but the purpose is the same. e two strategies can be exploited by brush. If the real and rubber arms are touched multiple times in synchrony, the
the same animal. For example, when a hunting dog is taking a bird scent sample average participant voluntarily accepts the rubber hand as her own. is simple
on the ground, it sni s the grass rhythmically, but to chase the same scent in the procedure is su cient to produce a feeling of ownership of a foreign body and
air, it runs fast with its nose up to create a scent ow. In either case, the physi- incorporate it into the body scheme.51 In another version of the experiment,
ological foundation to make sense of such active sensing behavior is corollary the experimenter moves the participant’s le index nger so that it touches the
discharge. right rubber hand on the knuckle of the index nger, and, at the same time, the
experimenter touches the participant’s right index nger on the knuckle. If the

Mechanisms of Active Sensing
49. Michael Brecht expanded on these ideas and formulated a body model theory of the somat-
osensory cortex (Brecht, 2017). He suggests that layer 4 of somatosensory cortex mirrors the
Researchers understand the neuronal mechanisms underlying certain forms of entire body and not just sensory a erents. is body model is continuously updated by layer 6
active sensing. Some animals can both sense and generate electric elds. e to layer 4 inputs and compares to an avatar rather than a mere sensory map.
weakly electric sh (Eigenmannia) actively probes its environment with elec- 50. Ramachandran et al. (1995). Similarly, blind people can imagine and dream about visual
tricity. It has a specialized structure called the electric organ in its tail, which scenes. Conversely, patients with bilateral damage to the primary visual cortex can still re-
emits electric elds. It also has electroreceptors called tuberous receptors in its spond to some visual stimuli and navigate around visual objects, even though they report
seeing nothing and having no memory of the objects. is phenomenon is known as blindsight
skin that sense electric elds for object detection (electrolocation) and commu- (Cowey, 2010; see it yourself here:  https://www.youtube.com/watch?v=GwGmWqX0MnM.
nication with other electric sh (electrocommunication) in muddy waters. Monkeys with primary visual area lesions behave similarly; Cowey and Stoerig, 1995). ese
In electrolocation, the sh’s brain analyzes the frequency and phase di erence observations illustrate that “sensory” cortices do not simply serve to veridically “represent” the
of its emitted sinusoid electric eld and the version returned from objects. When external world.
two sh are near each other, the frequency di erence of the elds emitted by each 51. See a demonstration here: https:// www.youtube.com/watch?v=sxwn1w7MJvk.
78

78 THE BRAIN FROM INSIDE OUT Chapter 3. Perception from Action 71

are meaningful teaching patterns to the somatosensory thalamus and cortex is determined by the combination of the two sh’s discharge patterns. us, the
because these are the combinations that will be used later in life. Random and problem of sorting out self-generated electric eld from elds produced by other
independent induction of cortical activity by the 600 or so skeletal muscles sh is the same as that of touching something versus being touched—only the
that move the mammalian body would be of very little biological relevance. medium is electrical, not mechanical. e solution of the brain is also similar: a
Furthermore, sending organized inputs to the brain cannot be achieved through circuit analogous to the corollary discharge signal comparator. e ring pattern
merely a genetic blueprint because the metric relations among the body parts in the electric organ is controlled by motor neurons that are driven by the motor
change rapidly as the body grows. command nucleus. e axons of the neurons from the command nucleus bifur-
How can such dumb “training” from muscle twitch combinations contribute cate, with one pathway going to the medullary relay nucleus whose axons pro-
to the formation of the body map? In the newborn rat up, every twitch and limb ject along the spinal cord and excite electromotoneurons that generate the electric
jerk induces a “spindle-shaped” oscillatory pattern in the somatosensory cortex organ discharges. e other collateral pathway innervates the bulbar area, which
lasting for a few hundred milliseconds. In the adult animal, neurophysiologists contains the corollary discharge comparator circuit (Figure 3.1). is circuit is
would call such a pattern a thalamocortical “sleep spindle” because such patterns analogous, albeit di erent in its details, to the corollary circuit of the cricket.34
occur in adults only during non-REM sleep. Both in the pup and in a prema- Echolocation, also called bio-sonar, is another well-known active sense that
turely born human baby, these are the rst organized cortical patterns.47 When works by emitting sound or ultrasound calls into the environment and analyzing
long-range corticocortical connections form a er birth in the rat, the spindle the returned echoes to locate objects and prey. Bats and marine mammals are
oscillation can serve to bind together neuronal groups that are coactivated in the animals best known for their sonar sensing. Whales use a series of clicks
the sensory cortical areas as a result of the simultaneous movement in neigh- during their dives. e precision of bat echolocation is good enough to identify
boring agonistic muscles. Likewise, muscles with an antagonistic movement mosquitoes even when both the insect and the bat are ying. e sound local-
relationship in the body (e.g., the biceps and triceps of the upper arm) will in- ization mechanisms in the brainstem act by detecting the time di erences of
duce consistent activity–silence relationships in their sensory cortex and create sounds between the two ears. e self-vocalized sounds are strongly attenuated
an inhibitory relationship between the respective neuronal groups. A er the between the cochlear nerve and the inferior colliculus, early stations of the au-
long-range corticocortical connections are in place a week or so a er birth ditory system, as well as in the auditory cortex.35
and the body map is formed, the spindles become con ned to non-REM sleep. e neuronal mechanism used in echolocation is similar to our ability to use
en the pups can use their body map to respond to local touch and use the self-generated sounds to tell whether we are in a small or large room. Normally,
proprioceptive information from the muscles and muscle tendons for e ort- we do not use this option much because we have the luxury of vision. However,
less ambulation.48 us, the initially meaningless, action-induced feedback this ability can be enhanced if needed. Some blind individuals can replace their
from sensors transduces the spatial layout of the body into temporal spiking eyesight by making loud footsteps, tapping a cane, or producing mouth clicks
relationships among neurons in the brain. is developmental process is how and listening to the returning echoes. ese active sensing methods are good
the brain acquires knowledge of the body it controls or, more appropriately, enough for crossing roads, even riding a bicycle or rollerblading. Some excep-
cooperates with. us, a dumb teacher (i.e., the stochastically occurring move- tional blind individuals can identify an object’s distance, size, and texture, such
ment patterns) can increase the brain’s smartness about its owner’s body land- as the di erence between a metal fence and a wooden fence.36
scape. Once the body scheme is built, the relationship between spindles and
muscle jerks disappears. e spindles become “internalized” (Chapter 5) and
continue to occur during sleep as a self-organized pattern. Yet the relationship
34. Walter Heiligenberg’s book is a masterful combination of ethology, electrophysiology,
and modeling on electric sensing in Eigenmannia used for jamming avoidance (Heiligenberg,
1991). For an update, see Chacron (2007).
47. Similar to early spindles, spontaneous retinal waves trigger activity in every part of the
visual system during early developmental stages (Katz and Shatz, 1996). 35. Suga and Schlegel (1972); Hechavarría (2013).
48. Khazipov et  al. (2004); Buzsáki (2006). e early spindles in premature human babies 36. Echolocation is a trainable skill that can provide blind people with self-reliance in their
are called delta- brush oscillations, and the alternating pattern of silence and activity is known daily life. Ben Underwood and Daniel Kish, two Californians who were blind from child-
as trace alternans. As in rat pups, muscle twitches induce delta brushes at 29 to 31 weeks of hood, are the best-known individuals to use these sonar techniques (Kremer, 2012; Kolarik
postconceptional age in preterm neonates and in utero (Milh et al., 2007). et al., 2017).
72

72 THE BRAIN FROM INSIDE OUT Chapter 3. Perception from Action 77

In imaging studies, blind research participants show increased activity in be taken as evidence in favor of the sensory-to-motor sequence of perception.
visual cortical areas, and to some degree in the mid-temporal region, when However, such perception occurs in an already “calibrated” brain. Without ac-
they actively localize objects based on their acquired sonar methods. Such ac- tively sensing stimuli, without investigating them at some point in life, stimuli
tivation is strong in individuals who became blind early in childhood, weaker cannot not acquire a meaning to the brain. Once meaning (i.e., signi cance of a
in people who lost their eyesight later in life, and absent in normal control stimulus or event for the animal) emerges, it can be placed in a memory store.
participants. Surprisingly, no di erences are detected in the auditory cortex. We can say it becomes internalized, and the internalized pattern can function
ese observations imply that the visual cortex, deprived of its natural input, as a grounding mechanism for further perceptual interpretations. I will expand
can be trained to adopt a novel function, such as echolocation.37 To succeed, on these issues in Chapter 5.
self-initiated action is a precondition.

BODY TEACHING BRAIN
Olfaction
Although the brain has no a priori clues about what its sensors are sensing or
Smelling provides an elegant example of active sensing in mammals. In olfac- what its e ectors are e ecting (Chapter 1), the developing brain does not start
tory research, every investigator knows that whatever questions they study, the from scratch; it bene ts enormously from inherited and early programs. But
phase of respiration should be taken into account because respiratory move- one-size- ts-all blueprints are not adequate to do the job because bodies come
ment modulates perception. e goal of sni ng is to optimally sample the in di erent shapes and forms. e genetic blueprint is simply a “protomap.” In
odorant and route it to the olfactory receptors. Sni ng is a rhythmic pattern the newborn rat, the spinothalamic tract and thalamocortical somatosensory
(4–10 sni s per second) that generates turbulence in the nose so that odorants, pathways are already in place. is map thus has some correspondences with
which would only slowly di use during regular breathing, come in direct con- body parts. However, cross-talk among the body part representations is limited
tact with the receptor. As a result of sni ng, the sampling becomes intermit- at this early stage because the corticocortical connections are just beginning
tent: e ective during inhalation and reduced or absent during exhalation. But to grow a er birth. How do the growing axons “know” which neurons to in-
that is not all. e entire olfactory system gets informed ahead of time about nervate and which ones to avoid? e proximity relationships help but are not
the expected arrival of the odorant by a corollary discharge signal, initiated by su cient.
exhalation (or more likely due to relaxation of the inspiration motor action). At the output end, the ventral horn motoneurons in the spinal cord, already
While the exact paths and mechanisms of this signaling are not well under- wired to the muscles, begin to generate irregular, uncoordinated movements.
stood, ample physiological evidence shows that the entire system, from the ol- ese seemingly aimless movements in newborn rodents are the same as fetal
factory receptor neurons to the olfactory (piriform) cortex, generates rhythmic movements or “baby kicks” observed in later stages of pregnancy in humans.
patterns in phase with the sni cycles. From the olfactory receptor neurons Every expectant mother and physician knows that such kicks are important
to the mitral cells (the main output of the olfactory bulb), all neurons show aspects of the normal development of the fetus. However, the biological utility
large-amplitude membrane voltage oscillations and synchronous spiking to the of the kicks and their service to the brain have been clari ed only recently. As
sni cycle. Pyramidal neurons in the olfactory cortex, where the signi cance you will see later, each kick helps the brain to learn about the physics of the
and meaning of the odor information are likely extracted, also show strong body it controls.
inhalation-coupled spiking output. Stretch sensors in muscles and tendons report the contractile state of the
One may argue that the mechanical e ect of active sni ng, inducing turbu- muscle to the spinal cord and eventually to the somatosensory cortex. In ad-
lence in the nose, is su cient to enhance odorant detection and identi cation dition, twitches of skeletal muscles increase the probability that the skin over
without any need to inform neurons in the olfactory system about the phase of the muscle will touch another pup in the nest or touch the wall of the womb in
respiration. To counter that argument, researchers injected odorants into the case of human fetuses. Due to the physical constraints of the bones and joints,
only a limited fraction of muscle movement combinations ever occurs, out of
the potentially very large number of possibilities that could result from un-
37. De Volder (1999); aler et al. (2011). restrained combinations of muscle activity All these movement combinations
76

76 THE BRAIN FROM INSIDE OUT Chapter 3. Perception from Action 73

communication between them.44 But recent discoveries in rodents paint a dif- bloodstream at low concentrations in human volunteers, thus removing the
ferent picture. Rodents have an elaborate whisker system adapted for tactile air turbulence e ect. Participants still detected the odorant more reliably when
exploration, especially in the dark. Whiskers palpate objects just like ngers they sni ed, demonstrating that, even in the absence of the odorant in the nose,
do. ere are multiple loops between the input and output paths, which in- the motor action still provides a gain. 38 is nding supports the claim that
clude short brainstem connections and longer thalamic, cortical, and cerebellar feedback from action to the sensory system enhances perception.
paths. Importantly, ample anatomical evidence shows that the tra c between
sensory and motor areas is bidirectional even in the cortex. Signals of whisker
motion reach the somatosensory area and can serve as corollary discharge. Vision as a Searchlight
ese signals enable neurons in this area to integrate the movements and
touches of multiple whiskers over time, key components of object identi cation Corollary discharge from saccadic eye movements serves as a signal to inform
and navigation by active touch. In addition, the somatosensory area of whisker the visual system whether the world or the eyeball is moving. e same circuit
representation (called the barrel cortex) forms an equally direct and prominent is involved in visual search, which is another name for active sensing. Vision or
motor control pathway. Activation of neurons in the “somatosensory” cortex is seeing is not a passive camera-like function but an active scanning of the envi-
as e ective in controlling whisker movements as stimulation of the anatomi- ronment.39 High-acuity vision in humans (i.e., what allows you to read text in
cally designated motor cortex. Direct anatomical projections from the somato- this book) is possible only with the cone receptors of the foveal region, corre-
sensory cortex to the spinal cord are present not only in rodents but in monkeys sponding to the central ve degrees of the eld of vision. One degree of visual
as well, suggesting that similar mechanisms are at work in di erent species.45 angle is roughly the width of your index nger held at arm’s length. If we x our
A classical demonstration for the critical role of action in perception is eyes on some part of a picture to keep the high-acuity receptors in place, we can
the tactile-vision sensory substitution experiment by Paul Bach-y-Rita and see that the surrounding region becomes blurry. e e ect is like pointing a
colleagues at the University of Wisconsin, which allowed blind people to “see” ashlight toward part of the environment in darkness. To see what is out there,
using an array of electrodes placed on the tongue and stimulating the sensory we must move the ashlight back and forth. During the day as well, we are
terminals with weak electrical pulses. Blind participants were able to partially taking in our surroundings in snippets, but because of the frequent scanning
gain some visual function conveyed to the tongue by the output of a video eye movements, our visual system’s memory creates the illusion that we see the
camera. A crucial component of the successful outcome in each case was the entire visual scene with high acuity and at once.
participants’ ability to control of the camera. Under passive tongue “viewing” Even when we x our eyes on a spot, eye movements assist vision via fre-
conditions, only tickling the tongue was felt but vision-like sensation did not quent “microsaccades,” tiny jumps of the eye that move the scene across a few
occur in the participants.46 dozen photoreceptors. ese movements ensure that our eyes do not xate on
Of course, you may counter my arguments by simple introspection and a completely stationary picture. Microsaccades increase visual performance by
say that you can sit completely immobile and yet perfectly process the sen- enhancing the spiking activity of neurons in the visual cortex. When macaque
sory ow. A  touch on your hand or a bug ying in your visual eld can be
detected with no muscular e ort whatsoever. erefore, such examples can
38. e authors (Bocca et al., 1965) interpreted their observations showing that the mechanical
stimulation of the epithelium by sni ng lowered the threshold for detection of blood-borne
molecules. A  contemporary demonstration of bypassing the olfactory mucosa is by Dmitri
44. Neurosurgeon Wilder G.  Pen eld, working with neurologist Herbert Jasper at the
Rinberg and colleagues (Smear et al., 2013). ey optogenetically stimulated a single glomer-
Montreal Neurological Institute, electrically stimulated the brain of awake patients with elec-
ulus, the functional unit of olfactory processing, and showed that this “implanted odorant” was
trical probes and observed their motor responses or obtained their verbal feedback. e stimu-
su cient for odor perception by simultaneously relaying identity, intensity, and timing signals.
lation technique allowed them to create crude sensory and motor maps of the cortex (cartoon
representations of the body surface, or homunculi). ese somewhat misleading anatomically 39. I have o en wondered where visual science would stand today if early visual researchers
discrete maps are still used today, practically unaltered. had been engineers or control theorists. Instead of taking the reductionist path of xing heads
and eyes, what if they had developed methods for detecting the position of the eyes in relation-
45. Ferezou et al. (2007); Mátyás et al. (2010); Hatsopoulos and Suminski (2011); Huber et al.
ship to the environment in freely behaving animals and observed the mechanisms of seeing in
(2012); O’Connor et al. (2013); McElvain et al. (2017).
action? It is notable that David Hubel focused his interest on the signi cance of eye movements
46. Bach-y-Rita et al. (1969). in vision toward the end of his career (Martinez-Conde et al., 2004).
74

74 THE BRAIN FROM INSIDE OUT Chapter 3. Perception from Action 75

monkeys xate on a target, and an optimally oriented line is centered over a Distinguishing between our own speech and a recording of it is a piece of cake
cell’s so-called receptive eld40 in the primary visual cortex, spiking activity because the corollary discharge mechanism provides an intimate relationship
increases a er microsaccades. Importantly, such microsaccade-induced, tran- between sound production and hearing. Each uttered sound leads to the next
siently enhanced neuronal excitability is present in every visual area examined action, and every sound is preceded by action preparation. Sound utterances
by di erent research groups and even in the lateral geniculate body, a tha- adjust the sound-controlling muscles and help to achieve the desired outcome.
lamic structure that transmits the retinal information to the primary visual As in other sensory systems, the auditory regions of the cortex also receive
cortex. A  great number of neurons are a ected by eye movements, as might direct projections from motor areas, providing a substrate for conveying corol-
be expected because visual areas occupy a large part of the brain.41 In other lary discharge signals. A copy of the motor command from the premotor cor-
words, movements of the eye improve the visual performance of the sensory tical region is sent to the auditory cortical areas, where it strongly suppresses
system and assist with the maintenance of perception during visual xation. both spontaneous and tone-evoked synaptic activity during a wide variety of
Saccades thus can be considered “visual sni s,” which provide a sensory gain natural movements. is is achieved by a so-called feed-forward inhibitory
(Chapter 11). mechanism, which entails the activation of inhibitory interneurons in the au-
In terms of visual function, there is no real di erence between regular and ditory cortex and the consequent transient inhibition of the spiking activity of
microsaccades. During eye xation, the saccades are smaller (“micro”), while pyramidal neurons. As a result, sound-evoked cortical responses are selectively
during “voluntary” visual search, they are larger. Both types of saccades are suppressed during movement.42 In addition to cortical mechanisms, the mus-
initiated in the superior colliculus in the midbrain. e widespread anatom- cles of the middle ear, which regulate the sound transmission from the eardrum
ical projections from this structure may explain why AI could e ectively com- to the cochlea, perform a similar preemptive action. eir activation by the
pensate for her loss of eye muscles by inducing saccade-like head movements movement command strongly attenuates the sound pressure before it can af-
(Figure 3.2). is observation emphasizes an important rule in neuronal orga- fect the cochlea. ese corollary actions of the brain’s output are responsible for
nization: functions that are important for survival are widely and redundantly making the sounds of the heavy timpani in Handel’s Messiah pleasing, whereas
distributed in the brain—a robust safety mechanism. the same sounds encountered unexpectedly would startle us.
Action-induced corollary signaling thus can transiently dampen the sensi-
tivity of the auditory system to predictable sounds, while maintaining its re-
Hearing sponsiveness to unexpected stimuli. ese same mechanisms, when derailed
by disease, are implicated in tinnitus and auditory hallucinations.43 Neuronal
When our ears are plugged, we have trouble not only hearing others but also excitation sent from the motor areas to the auditory areas that does not occur
speaking or singing properly. Auditory feedback is very useful in speech. during self-induced sounds are interpreted by the brain as real.

40. Neurons respond to various constellation of inputs. e most robust response is believed to Body Sensation
re ect the “relevant” input. Many visual neurons have preferred orientation selectivity, motion
direction selectivity, color preference, etc., which we call the receptive eld. e traditional separation of motor and sensory functions is even more
41. Many excellent original studies on eye saccades are compressed in these short paragraphs. prominent in the somatosensory system. is view is explicitly illustrated
Some of these studies are reviewed by Yarbus (1967) and Carpenter (1980) and Martinez- by the classical textbook gures of motor and sensory “homunculi” whose
Conde et al. (2004). Martinez-Conde et al. emphasized the importance of eye movements in
vision but considered them only to a ect retinal processing: “microsaccades primarily mod-
jobs are to perceive and act, respectively, with a one-way, sensory-to-motor
ulate neural activity in early visual areas through retinal motion.” Corollary discharge within
the brain as a mechanism for visual gain is not discussed, even though saccades can a ect the
entire visual system. See also Otero-Millan (2008). Hofmann et al. (2013) discusses the many
42. Paus et al. (1996); Houde and Jordan (1998); Zatorre et al. (2007); Eliades and Wang (2008);
forms of active sensing in bats (echolocation) and invertebrates (e.g., electric communication).
Nelson et al. (2013); Schneider et al. (2014).
Wachowiak (2011) and Morillon et al. (2015) are excellent reviews on the role of motion in
hearing and olfaction, respectively. Visual saccades a ect even hippocampal neurons (Meister 43. Feinberg et  al. (1978); Ford and Mathalon (2004); Langguth et  al. (2005); Nelson et  al.
and Bu alo, 2016). (2013).