See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/6947630

Neuronal ensemble control of prosthetic devices by a human with tetraplegia

Article in Nature · August 2006

DOI: 10.1038/nature04970 · Source: PubMed

CITATIONS READS
3,270 3,096

10 authors, including:

Leigh R Hochberg Mijail Serruya

Massachusetts General Hospital Thomas Jefferson University
241 PUBLICATIONS 20,001 CITATIONS 41 PUBLICATIONS 5,656 CITATIONS

SEE PROFILE SEE PROFILE

Gerhard M Friehs Jon A Mukand

Texas A&M University Brown University
67 PUBLICATIONS 9,160 CITATIONS 51 PUBLICATIONS 3,948 CITATIONS

SEE PROFILE SEE PROFILE

All content following this page was uploaded by Gerhard M Friehs on 23 May 2014.

The user has requested enhancement of the downloaded file.

 Vol 442|13 July 2006|doi:10.1038/nature04970

ARTICLES
Neuronal ensemble control of prosthetic
devices by a human with tetraplegia
Leigh R. Hochberg1,2,4, Mijail D. Serruya2,3, Gerhard M. Friehs5,6, Jon A. Mukand7,8, Maryam Saleh9†,
Abraham H. Caplan9, Almut Branner10, David Chen11, Richard D. Penn12 & John P. Donoghue2,9

Neuromotor prostheses (NMPs) aim to replace or restore lost motor functions in paralysed humans by routeing
movement-related signals from the brain, around damaged parts of the nervous system, to external effectors. To
translate preclinical results from intact animals to a clinically useful NMP, movement signals must persist in cortex after
spinal cord injury and be engaged by movement intent when sensory inputs and limb movement are long absent.
Furthermore, NMPs would require that intention-driven neuronal activity be converted into a control signal that enables
useful tasks. Here we show initial results for a tetraplegic human (MN) using a pilot NMP. Neuronal ensemble activity
recorded through a 96-microelectrode array implanted in primary motor cortex demonstrated that intended hand motion
modulates cortical spiking patterns three years after spinal cord injury. Decoders were created, providing a ‘neural
cursor’ with which MN opened simulated e-mail and operated devices such as a television, even while conversing.
Furthermore, MN used neural control to open and close a prosthetic hand, and perform rudimentary actions with a multi-
jointed robotic arm. These early results suggest that NMPs based upon intracortical neuronal ensemble spiking activity
could provide a valuable new neurotechnology to restore independence for humans with paralysis.

Hundreds of thousands of people suffer from forms of motor and in pilot trials in people with tetraparesis from spinal cord injury,
impairment in which intact movement-related areas of the brain brainstem stroke, muscular dystrophy, or amyotrophic lateral sclero-
cannot generate movements because of damage to the spinal cord, sis. Currently, this system consists of a chronically implanted sensor
nerves, or muscles1. Paralysing disorders profoundly limit indepen- and external signal processors developed from preclinical animal
dence, mobility and communication. Current assistive technologies studies (see Methods)6–8. The participant described in this report, the
rely on devices for which an extant function provides a signal that first in the BrainGate trial, is a 25-yr-old male (MN) who sustained a
substitutes for missing actions. For example, cameras can monitor knife wound in 2001 that transected the spinal cord between cervical
eye movements that can be used to point a computer cursor2. vertebrae C3–C4, resulting in complete tetraplegia (C4 ASIA A)9. The
Although these surrogate devices have been available for some array was implanted in June 2004 into the MI arm area ‘knob’10, as
time, they are typically limited in utility, cumbersome to maintain, identified on pre-operative magnetic resonance imaging (MRI)
and disruptive of natural actions. For instance, gaze towards objects (Fig. 1c). Post-operative recovery was uneventful. The data presented
of interest disrupts eye-based control. By contrast, an NMP is a here are derived from 57 consecutive recording sessions from
type of brain–computer interface (BCI) that can guide movement 14 July 2004 to 12 April 2005 (9 months).
by harnessing the existing neural substrate for that action—that is,
neuronal activity patterns in motor areas. An ideal NMP would Signal quality and variety
provide a safe, unobtrusive and reliable signal from the discon- Action potentials were readily observable on multiple electrodes,
nected motor area that could restore lost function. Neurons in the indicating that MI neural spiking persists three years after SCI, as
primary motor cortex (MI) arm area of monkeys, for example, suggested indirectly by functional MRI data11–14. Recorded signals
provide information about intended arm reaching trajectories3–5, ranged from qualitatively well-isolated single neurons to mixtures of
but this command signal would work for an NMP only if neural a few different waveforms (Fig. 2a). Different waveform shapes were
signals persist and could be engaged by intention in paralysed identified visually, using standard time-amplitude windows, but
humans. there was no further attempt to distinguish between well isolated
In concept, NMPs require a sensor to detect the activity of multiple and intermixed waveforms, both of which we refer to in this report as
neurons, a decoder to translate ensemble firing patterns into motor ‘units’. An average of 26.9 ^ 14.2 units were observed each day
commands, and, typically, a computer gateway to engage effectors. (range 3–57), with mean peak-to-peak spike amplitudes of
BrainGate (Cyberkinetics, Inc.) is an NMP system under development 76.4 ^ 25.0 mV (mean ^ s.d., n ¼ 56 sessions) (see Supplementary
1
Department of Neurology, Massachusetts General Hospital, Brigham and Women’s Hospital, and Spaulding Rehabilitation Hospital, Harvard Medical School, 55 Fruit Street,
Boston, Massachusetts 02114, USA. 2Department of Neuroscience and Brain Science Program, and 3Department of Engineering, Brown University, PO Box 1953, Providence,
Rhode Island 02912, USA. 4Center for Restorative and Regenerative Medicine, Rehabilitation Research and Development Service, Department of Veterans Affairs, Veterans
Health Administration, 830 Chalkstone Avenue, Providence, Rhode Island 02908, USA. 5Department of Clinical Neurosciences (Neurosurgery), Brown University, and
6
Department of Neurosurgery, Rhode Island Hospital, 120 Dudley Street, Suite 103, Providence, Rhode Island 02905, USA. 7Department of Rehabilitation Medicine, Brown
University, 593 Eddy Street, Providence, Rhode Island 02903, USA. 8Sargent Rehabilitation Center, 800 Quaker Lane, Warwick, Rhode Island 02818, USA. 9Cyberkinetics
Neurotechnology Systems, Inc., 100 Foxborough Boulevard–Suite 240, Foxborough, Massachusetts 02035, USA. 10Cyberkinetics Neurotechnology Systems, Inc., 391 Chipeta
Way, Suite G, Salt Lake City, Utah 84108, USA. 11Department of Physical Medicine and Rehabilitation, Rehabilitation Institute of Chicago, 345 E. Superior Street, 1146, Chicago,
Illinois 60611, USA. 12Department of Neurosurgery, University of Chicago Hospitals, 5841 S. Maryland Avenue, MC3026, Chicago, Illinois 60637, USA. †Present address:
Graduate Program in Computational Neuroscience, University of Chicago, Chicago, Illinois 60637, USA.

164
© 2006 Nature Publishing Group
NATURE|Vol 442|13 July 2006 ARTICLES

Information)8. Although details of local field potentials (LFPs) will

be described in a subsequent report, we note that LFPs, which could
be recorded simultaneously with spikes, resembled those observed in
intact monkeys (Fig. 2b)15. A notable decrease in the number of
recorded units was seen approximately 6.5 months after implan-
tation and thereafter. Upon approval of a clinical protocol change at
10 months, which permitted impedance measurements, we observed
a low impedance in 54 of the electrodes, consistent with a physical
short circuit to ground in the array, cable, or connector, but not
consistent with a biological event (for example, gliosis). The precise
reason for this physical change remains under investigation.
Since this report was written, we added a second trial participant
to the study, a 55-yr-old man who has had complete spinal cord
injury at C4 since 1999. Recordings were collected starting in the
seventh month after implant, after making an electrical contacts
repair in the pedestal connector. We recorded an average of
53.2 ^ 6.3 units per session during trial months 7–10, again demon-
strating the presence of neural activity lasting many months. Another
technical issue causing abrupt signal loss at most electrodes, which
may be related to the original repair, occurred at month 11 in
participant 2; the reason for this change is being evaluated.

Figure 2 | Electrical recordings from a sample of four electrodes.

a, Discriminated neural activity at electrodes 33, 34, 22, 95 (n ¼ 80
superimposed action potentials for each unit). On electrode 33, two
Figure 1 | Intracortical sensor and placement, participant 1. a, The neuronal units could be reliably discriminated with peak to peak amplitudes
BrainGate sensor (arrowhead), resting on a US penny, connected by a 13-cm of 206 and 56 mv, respectively. For electrode 34, a single unit is displayed.
ribbon cable to the percutaneous Ti pedestal (arrow), which is secured to the Electrode 22 illustrates a low-amplitude discriminated signal. Electrode 95
skull. Neural signals are recorded while the pedestal is connected to the shows triggered noise. Data are from trial day 90 (90 days after array
remainder of the BrainGate system (seen in d). b, Scanning electron placement). b, Local field potentials during neural cursor control. In the
micrograph of the 100-electrode sensor, 96 of which are available for neural bottom panel, three traces of electrical recording (bandpass: 10–100 Hz)
recording. Individual electrodes are 1-mm long and spaced 400 mm apart, in from one electrode are shown 0.5 s before and 1.9 s after the go cue
a 10 £ 10 grid. c, Pre-operative axial T1-weighted MRI of the brain of instructing MN to move the cursor from the centre position to a target at the
participant 1. The arm/hand ‘knob’ of the right precentral gyrus (red arrow) top of the screen. In the top panel, a Thomson multi-taper time frequency
corresponds to the approximate location of the sensor implant site. A scaled analysis on each trial data segment was performed. This was done by sliding
projection of the 4 £ 4-mm array onto the precentral knob is outlined in red. a 0.3-s window every 0.05 s, using a spectral resolution of 10 Hz. These
d, The first participant in the BrainGate trial (MN). He is sitting in a power spectrograms were averaged across 20 trials to create the resulting
wheelchair, mechanically ventilated through a tracheostomy. The grey box pseudocolour power spectral density (PSD) plot. The diagram is aligned
(arrow) connected to the percutaneous pedestal contains amplifier and such that each point in the PSD plot corresponds to a time window 150 ms
signal conditioning hardware; cabling brings the amplified neural signals to before and after an LFP. In the 20–30-Hz band, a decrease in power is seen
computers sitting beside the participant. He is looking at the monitor, approximately 300 ms after the go cue, followed by an increase in power
directing the neural cursor towards the orange square in this 16-target ‘grid’ from 550–1,200 ms after the go cue, which can also be appreciated in the raw,
task. A technician appears (A.H.C.) behind the participant. single trial data below.
165
© 2006 Nature Publishing Group
ARTICLES NATURE|Vol 442|13 July 2006

Modulation by intent Fig. 1, neurons fired in a relatively time-locked manner upon the
Imagined limb motions modulated neural firing on multiple electro- request to imagine action. These results demonstrate a rich hetero-
des, upon request, beginning at the first experimental session. geneity of firing patterns within a limited sample from a small
Modulation was evaluated during four consecutive sessions when MI region. This diversity is useful in creating a flexible control
MN was asked to imagine a series of movements. This series revealed signal.
a rich variety of firing modulations largely consistent with patterns
observed in monkey MI16. Importantly, this activity was evoked by Linear filter construction
imagined actions in this participant with cervical spinal cord injury. Use of MI neuronal ensemble activity as a control signal by persons
Figure 3a illustrates how certain neurons are selective for one with paralysis requires a novel approach to establishing a transform
imagined action (hands together/apart), whereas others recorded (filter function) between firing patterns and intended action. For
simultaneously are engaged by different imagined actions (elbow or each session, units were used to create a filter (see Methods) to
wrist). This diversity includes neurons that fired with imagined hand provide a two-dimensional output signal displayed as cursor
or distal arm actions (for example, hand open/close, Fig. 3c) and position on a monitor. A simple linear filter algorithm, identical
those that fired during shoulder movements that were actually to that used in intact monkeys, was used to create filters17. Unlike
performed (see also Supplementary Fig. 1). Non-selective neurons, most studies performed using intact monkeys, where hand
active with the onset of any imagined upper extremity action motions are known and kinematics are measured directly, we
(Fig. 3b), were also observed. As shown in Fig. 3c and Supplementary predicted MN’s intended hand movements on the basis of

Figure 3 | Neuronal selectivity for imagined and performed movements. instruction to flex/extend the wrist and to flex/extend the elbow or move the
a, Over an 80-s period, MN was asked to imagine performing a series of left shoulder anteriorly and posteriorly. All movements are imagined except for
limb movements (which are described on the abscissa). Movement shoulder movement, which MN actually performed. b, Go-cue-related
instruction time is indicated by a vertical arrow; the go cue for each activity modulation for a neuron recorded simultaneously with those in a.
alternating movement, presented as text on the video monitor, is indicated Each raster line is centred about the go cue, which requests that the patient
by a small vertical hash mark. Spiking activity of two simultaneously imagine a movement; the seven raster lines represent the epochs
recorded units is displayed. Rasters indicate the time of each spike (thinned surrounding each of the seven different movements in sequence a. The
for visual clarity; every third spike is shown). Normalized, integrated histogram displays the total number of spikes seen in each 500-ms bin. This
firing rates (R) appear beneath each raster, as derived by the equation neuron increased its firing rate during most imagined movement epochs,
R ¼ ½ðR21 þ nÞð1 2 e2b=t Þ
; where R 21 is the previous bin’s integrated firing but demonstrated poor instruction selectivity compared to the neurons
rate value, n ¼ the number of spikes in the current bin, b ¼ bin width, and presented in a. Data are from day 161. c, Hand-instruction-related
t ¼ time constant; bin width ¼ 50-ms window, time constant ¼ 10 ms modulation for three simultaneously recorded neurons. MN was cued to
(adapted from ref. 28); normalization is achieved by dividing by the open and close his hand by text instruction, presented on the screen. Go cues
maximum integrated firing rate from each unit’s spike train over the time are indicated. Each vertical tick represents one action potential (spike). An
period displayed. The top unit (channel 38) increases its firing rate (curved increase in these neurons’ firing rates is directly indicative of the intention to
arrow) with the instruction to move both hands apart/together. The bottom, close the hand. Data are from day 90.
simultaneously recorded unit (channel 16) is activated most clearly after the
166
© 2006 Nature Publishing Group
NATURE|Vol 442|13 July 2006 ARTICLES

instructed actions (instruction-based algorithms have also been task-related information during the intention to move the limb
reported in one study of intact monkeys18). Thus, for filter ordinarily controlled by that MI region.
building, MN was asked to imagine manually tracking a ‘tech-
nician’s cursor’ that was actually being moved by a technician- Quality of neural cursor control
operated mouse through a succession of randomly placed visual Neural cursor position was significantly correlated with technician
targets (see Methods). The filter function was used to decode activity cursor position during the last block of the pursuit filter building task
and drive a ‘neural cursor’. (x coordinate r 2 ¼ 0.56 ^ 0.18 and y coordinate r 2 ¼ 0.45 ^ 0.15,
n ¼ 6 sessions, Fig. 5). These correlations are similar or better than
MI activity during neural cursor control those seen in intact monkeys when linear filters were used to predict
Features of neurons during neural cursor control resembled those real-time hand position from MI neuronal ensembles5,22. The neural
expected from MI. Neurons in MI of intact monkeys characteristi- cursor could be directed towards targets with a form qualitatively
cally begin to modulate their firing before movement onset and similar to that seen for intact monkeys using closed-loop neural
activity is tuned to hand movement direction19–21. To compare this control17,18,23. As in intact monkeys, neural cursor motion had
neural activity with MI of a human with spinal cord injury, MN underlying instabilities and variable oscillatory components com-
performed a step-tracking, ‘centre-out’ task using the neural cursor. pared to hand motions of able-bodied individuals. Continuous
The task requires that the neural cursor be moved from a centre target neural cursor motion with the linear filter made cursor fixation at
to one of four radially displaced targets (screen location: up, down, a single location difficult to achieve.
left, right; see Supplementary Video 1). For each of six sessions, MN Data from the centre-out task were used to evaluate the speed and
performed this task by imagining hand motion (see Methods) as accuracy of cursor control, which are essential design parameters for
soon as the target cue appeared. The task was performed immediately any future practical NMP. As shown in Fig. 6, the participant
after filter building without intervening practice. Timing and direc- correctly acquired 73–95% of targets (control 6.5%; n ¼ 80, paired
tional tuning features of MI neurons during imagined actions were t-test, P , 0.0001, see Methods) when measured in a series of six
consistent with those observed in MI of intact non-human primates. sessions (see also Supplementary Fig. 2). Performance errors
Figure 4 shows that spike-rate modulation occurs soon after the ‘go’ reflected both instabilities in cursor direction control and the ability
cue and that modulation varied by target location, as would to hold at the target location. Mean time to target was 2.51 ^ 0.16 s
be predicted for MI if actual arm motions were performed17. (^s.e.m.) for successfully acquired targets. Although the best 13% of
Furthermore, 66 out of 73 discriminated units (90.4%) significantly MN’s trials were within the range consistently achieved by able-
changed their firing rate in relation to the appearance of the go cue bodied controls using a computer mouse (n ¼ 3, mean 1.06 ^ 0.08 s
(Kolmogorov–Smirnov test, a ¼ 0.05, rate calculated over a sliding (^s.e.m.)), the distribution of times for MN using neural control is
1-s window, overlapping every 0.05 s; 60-s data set for each con- skewed to longer acquisition times (Fig. 6b). Effective use of the
dition, n ¼ 3 sessions). These results indicate that, even years after neural cursor in more complex spatial control tasks was evident when
spinal cord injury and in the absence of kinaesthetic feedback and MN directed the cursor to randomly placed targets while attempting
limb movement, MI neurons can still be actively engaged and encode to avoid obstacles in the cursor’s path (see Supplementary Video 5).

Figure 4 | Directional tuning during centre-out task. Peristimulus time related modulation. Each column shows the firing of one unit in the four
histograms show spike rates for five neurons recorded simultaneously directions, aligned on the cue to move. Note, for example, the time-locked
during the performance of a four-direction centre-out task (day 90) in which increase in firing of unit 6 when MN was cued to move the cursor downward
MN used the neural cursor to acquire a target presented at the right, top, left, (lower right corner) and the lack of change in firing rate for upward
or bottom of the screen. Twenty trials are displayed for each target location. instruction. Changes in firing across the five neurons reveal directional
Increases in activity after the go cue demonstrate movement-intention- tuning.
167
© 2006 Nature Publishing Group
ARTICLES NATURE|Vol 442|13 July 2006

Further information concerning spatial and temporal accuracy was (see Supplementary Fig. 1). NMPs will nearly always operate in the
obtained from a ‘grid task’ (see Supplementary Information). As context of some existing movement capabilities (given the consider-
shown in Supplementary Video 8, participant 2 also performed the able variability of remaining sensory and motor functions in people
centre-out task, highlighting the ability for this second participant with CNS injury); thus, during filter building and use MN was not
with spinal cord injury to modulate his motor cortical activity asked to remain completely still, and he sometimes moved his head
voluntarily for external device control. This participant’s neural or neck. It is thus possible that retained movements influenced cursor
control, however, was generally less accurate and consistent than action, much as such movements are known to influence activity
MN’s; we are investigating the extent to which technical or other correlated with ordinary hand actions24. These still-intact move-
factors may underlie this performance. This and other participants’ ments, however, did not appear to be essential for MN’s cursor
data will be reported in subsequent manuscripts. control, as can be appreciated particularly when he played ‘Neural
Although he is tetraplegic, MN retains shoulder, neck and head Pong’ (Supplementary Video 4). In this video, movement of the head
mobility, and some recorded MI cells fired during shoulder movement or the shoulders sometimes accompanied cursor control, but at other
times MN moved the cursor purposefully while his head remained
stationary, and during other epochs his head moved but the cursor
motion appeared to be unrelated to head movement. We compared
neural cursor position in Supplementary Video 4 to head position
(both assessed by coordinates on a video playback monitor) and
found no consistent relationship (r 2 ¼ 0.063). This finding is incon-
sistent with a unique causal relation between head and cursor
control. In addition, it was our repeated observation that, at least
some of the time, MN performed neural control (and open-loop)
tasks without moving his shoulder. Thus, based upon our instruc-
tions and MN’s self-reporting that he was actively imagining arm
action, we conclude that cursor motion is under the guidance of
imagined/intended actions.

Direct control of prosthetic devices

Continuous computer cursor control could be used to provide many
valuable new outputs for a person with paralysis to carry out

Figure 5 | Reconstruction of neural cursor position during pursuit

tracking. a, Technician (red line) and neural cursor (blue line) paths during
a 5-s epoch during which MN was asked to track the technician cursor with
his neural cursor in real time. MN was able to track the general direction of Figure 6 | Centre-out task performance. a, Target acquisition accuracy
the technician cursor with the neural cursor, changing directions quickly, during the centre-out task. For each of six sessions, MN acquired between
while having some difficulty in overlaying the cursors precisely. Trial day 90. 73–95% of the radially placed targets. Control targets were not present on
b, x, y position control over time during one tracking trial (last 1-min epoch the monitor during task performance, but were marked as acquired if,
of filter building). The top panel displays the x coordinate position of the during post-hoc analysis of the cursor movement, the cursor had traversed
technician cursor (red) and the neural cursor (blue); y positions for the same the location of one of the other three pseudo-randomly selected targets
movement are shown in the bottom panel. c, Neural cursor position during a before the correct target (see Supplementary Video 1). Data from days 72,
target acquisition/obstacle avoidance task. The four panels represent the 77, 83, 84, 86, 90 are shown. b, Time-to-target performance during centre-
four epochs shown in Supplementary Video 5. Green circles indicate targets; out task for MN (blue) and three able-bodied controls (red). Only successful
red squares indicate obstacles. The thick blue line indicates the path taken by target acquisitions in ,7 s are shown for MN. Arrows on the abscissa
the neural cursor and illustrates the ability to avoid most obstacles and represent median times to target for each distribution. Controls’
acquire most targets within a randomly arranged field. Data are from trial performances (n ¼ 3 controls, 80 trials each) are collapsed into 0.2-s bins.
day 90. MN’s performance (398 trials) is collapsed into 0.5-s bins for visual clarity.
168
© 2006 Nature Publishing Group
NATURE|Vol 442|13 July 2006 ARTICLES

activities of daily living. Such a control signal could be used not only criteria for a clinically useful interface for paralysed humans. For
to direct computer software, but also to operate external physical example, electroencephalogram (EEG)-based control systems have
assistive devices. MN tested his ability to perform potentially useful been improved by limiting trial time, re-centring the cursor location
actions in a series of demonstrations. after each trial, and stopping cursor motion as soon as targets were
MN used a simplified computer interface to open simulated e-mail hit26. By contrast, our participant had to maintain cursor control at
and to draw an approximately circular figure using a paint program all times without these interface enhancements. However, we found
(Supplementary Video 2). Using the neural cursor coupled to a that trial success was greater when target dwell time was decreased
simple hardware interface, he adjusted the volume, channel and (grid task). Incorporation of additional feedback (for example,
power to his television. He was also able to play video games such as visual, auditory, somatosensory) may also be useful in enhancing
Neural Pong (see Supplementary Videos 3–5). Control of two robotic performance27.
devices was achieved, allowing MN to manipulate directly the Cursor and external device control may also be improved through
environment. In one example, neural output was coupled to a learning28. MI of intact animals shows learning-related plasticity29–32,
prosthetic hand (Liberating Technologies, Inc.; see Methods) and and such plasticity undoubtedly contributes to the acquisition of
MN was able to open and close the hand under volitional control direct cortical neuronal control of an NMP. Thus, we expect that
(see Supplementary Video 6). Although only a one-dimensional learning will further improve control, as has been reported in
form of proportional control, he achieved this action after a few monkeys18. The circle drawing task (Supplementary Video 2), for
trials while looking at the hand and without requiring feedback example, provides some evidence of improved performance with
from the cursor display. Lastly, MN used a simple multi-jointed brief practice, but a better understanding of learning will require
robotic limb to grasp an object and transport it from one location additional, more systematic investigation.
to another (see Supplementary Video 7 and Supplementary Infor- Other human BCIs are under development33. Foremost for com-
mation). These demonstrations illustrate transfer of control parison here is one by Kennedy and collaborators, who have reported
directly to a physical device without depending on continued results with their neurotrophic (cone) electrode for three humans:
viewing of computer cursor feedback, and suggest that manipu- one with amyotrophic lateral sclerosis34 , one with brainstem
lation of the environment to enable activities such as self-paced stroke35,36, and one with mitochondrial myopathy36 (see Supplemen-
eating, as well as other goal-directed actions, could be achieved in tary Information). In contrast to previous studies, here we present
tetraplegic humans. evidence that cortical ensemble patterns can be decoded and used for
Notably, each of these tasks was achieved rapidly and could be real-time, two-dimensional control of a computer cursor and other
performed while the participant was conversing. Thus, the MI-based external devices by a person with spinal cord injury. Scalp-based
NMP may have the property of allowing external device control with EEG-driven BCIs also have been tested with some success in people
little more disruption than encountered in able-bodied humans with amyotrophic lateral sclerosis, spinal cord injury and other forms
when they are using their arms or hands and simultaneously carrying of paralysis26,27,37–39. Such indirect systems use a substitute for motor
out other motor or cognitive functions. ensembles to achieve control. Although two-dimensional control has
been achieved in paraplegic patients using independent modulation
Discussion of scalp-recorded signals26, this system requires significant learning,
The research shown here from the first participant in an ongoing concentration to the exclusion of other actions, and daily scalp sensor
pilot clinical trial provides initial evidence that a human unable to application, but notably, not surgical placement of the sensor. It
move or sense his limbs can operate a NMP using MI neuronal also has limited scalability to multiple functions because two-
ensemble spiking activity as a control source. In addition, we dimensional tasks appear to engage all controllable signals. In
demonstrate that neural spiking remains in the MI arm area and contrast, the direct NMP can be used during natural activities such
can be modulated by intention years after spinal cord injury. These as speech, and requires minimal learning beyond filter creation. It is
findings provide a number of novel insights concerning cortical also plausible that NMPs could be scaled so that parallel commands
function and the impact of spinal cord injury in humans (see could be derived simultaneously from multiple sensors each in
Supplementary Information). separate cortical regions. If achieved, relatively independent outputs
Although MN used direct neural control to perform reasonably potentially could emanate bilaterally from arm and leg areas to allow
challenging tasks, the level of the control is considerably less than that for the reanimation of paralysed limbs via functional electrical
of an able-bodied person using a manually controlled computer stimulation devices.
mouse. A number of factors might affect control, including: (1) the The relative merits of EEG, transcranial40, electrocorticographic
small set of randomly selected neurons recorded by this single array, (ECoG)41–43 and intracortically based BCIs (NMPs) continue to be
compared to the very large number usually engaged; (2) the influence explored and expanded32,33. Myriad issues including efficacy, safety
of spinal cord injury mechanisms or duration since injury; (3) the (particularly with regards to surgery), reliability, longevity, required
cortical layers recorded; (4) our approach to filter building; (5) the training and support, cosmesis, cost and availability will undoubtedly
nature of the linear filter (as compared, for example, to Kalman filters22, affect BCI success; individuals with impairments in communication
adaptive algorithms or support vector machine approaches25, not yet and/or mobility may prioritize these factors differently when choosing
tested in our work); (6) attention and motivational state during filter a BCI, and it is possible that combinations of techniques will provide
building or control; and (7) the user interface. Changes in the most useful restoration of control. Current BCIs (including the NMP
recorded population across days may also contribute to both varia- reported here) depend upon the insights and assistance of trained
bility and instability of control. Shifting ensembles may result from experts. The need for this assistance must be eliminated through
small motions of the array or through other poorly understood system automation.
mechanisms. Despite these variables, it is important to note that The goal of NMPs and other BCI research is to create safe, reliable
useful filters could be created daily from that neural population, and and unobtrusive neural interfaces to devices that will restore the
that advances in knowledge and technology are likely to improve communication, mobility, and independence of paralysed humans.
recording and decoding. For example, cursor control might be NMPs could provide significant advances over current technologies
enhanced further by adaptive algorithms or more selective choice because they engage natural substrates for control (that is, the MI
of neural signals. Combinations of spiking activity and simul- arm/hand area), do not encumber other actions, and do not require
taneously recorded LFPs would provide additional signals that extensive learning. It will be important to collect results from
might improve performance. additional participants in order to understand the generality of
Efforts to optimize performance may help to identify the design these statements. Improvements in the decoder and user interface
169
© 2006 Nature Publishing Group
ARTICLES NATURE|Vol 442|13 July 2006

are likely to provide control closer to that achieved by able-bodied rate over a 1-s history for each neuron (twenty 50-ms bins), and regressing this
humans. The potential to use implanted sensors to ‘re-route’ neural matrix onto technician-cursor position using a pseudoinverse technique. The
signals may also prove valuable in stroke and other neurological least-squares formulation comprises a closed-form solution:
disorders where pathways are disconnected. The present NMP u ¼ Rzf ¼ RðR T RÞ21 R T k
system incorporates a transcutaneous connection that tethers a
participant to a bulky cart and requires operation by a trained where R is the neural response matrix, f is the linear filter, k is the kinematic value
technician. A wireless, implantable and miniaturized system com- (x,y position), T indicates the transpose of the matrix, and u is the reconstruc-
tion5. Software for displays was custom generated for this purpose and run on
bined with automation will be required for practical use. Emerging
standard PC computers (Orbit Micro, dual Intel Pentium 4 Xeon processors,
and available technologies appear to be sufficient to overcome these 2 GHz).
obstacles, although the challenges of creating a fully implantable External devices. The robot arm was obtained from Lynxmotion, Lynx 5 Series.
system may be formidable. The current data provide initial proof of The electric prosthetic hand was generously provided by Liberating Technologies.
concept that spiking activity from neuronal ensembles can provide a
control signal after spinal cord injury sufficient to perform at least Received 22 March; accepted 6 June 2006.
basic operations for a human with tetraplegia, justifying further 1. Jackson, A. B., Dijkers, M., Devivo, M. J. & Poczatek, R. B. A demographic
engineering efforts towards a human NMP. profile of new traumatic spinal cord injuries: change and stability over 30
years. Arch. Phys. Med. Rehabil. 85, 1740–-1748 (2004).
METHODS 2. Thoumie, P. et al. Clinical and functional evaluation of a gaze controlled system
for the severely handicapped. Spinal Cord 36, 104–-109 (1998).
See Supplementary Information for additional Methods.
3. Humphrey, D. R., Schmidt, E. M. & Thompson, W. D. Predicting measures of
Approval for these studies was granted by the US Food and Drug Adminis- motor performance from multiple cortical spike trains. Science 170, 758–-762
tration (Investigational Device Exemption) and the Rhode Island Hospital, New (1970).
England, Rehabilitation Institute of Chicago, University of Chicago, and 4. Schwartz, A. B., Taylor, D. M. & Tillery, S. I. Extraction algorithms for cortical
Spaulding Rehabilitation Hospital Institutional Review Boards. The participant control of arm prosthetics. Curr. Opin. Neurobiol. 11, 701–-707 (2001).
described in this report has provided permission for photographs, videos and 5. Paninski, L., Fellows, M. R., Hatsopoulos, N. G. & Donoghue, J. P.
portions of his protected health information to be published for scientific and Spatiotemporal tuning of motor cortical neurons for hand position and velocity.
educational purposes. After completion of informed consent and medical and J. Neurophysiol. 91, 515–-532 (2004).
6. Maynard, E. M., Nordhausen, C. T. & Normann, R. A. The Utah intracortical
surgical screening procedures, the 4 £ 4-mm array of electrodes was implanted
electrode array: a recording structure for potential brain-computer interfaces.
into the motor cortex using a pneumatic insertion technique8,44. Details of the Electroencephalogr. Clin. Neurophysiol. 102, 228–-239 (1997).
human surgical procedure are in preparation for publication. 7. Guillory, K. S. & Normann, R. A. A 100-channel system for real time detection
BrainGate system. The sensor is a 10 £ 10 array of silicon microelectrodes that and storage of extracellular spike waveforms. J. Neurosci. Methods 91, 21–-29
protrude 1 mm from a 4 £ 4-mm platform (Fig. 1b). At manufacture, electrodes (1999).
had an impedance of 322 ^ 138 kOhm (mean ^ s.d.). The array was implanted 8. Suner, S., Fellows, M. R., Vargas-Irwin, C., Nakata, K. & Donoghue, J. P.
onto the surface of the MI arm/hand region; electrodes penetrate into the cortex Reliability of signals from chronically implanted, silicon-based electrode array
to attempt to record neurons in intermediate layers. Recorded electrical signals in non-human primate primary motor cortex. IEEE Trans. Neural Syst. Rehabil.
Eng. 13, 524–-541 (2005).
pass externally through a Ti percutaneous connector, which is secured to the
9. Maynard, F. M. Jr et al. International standards for neurological and functional
skull. Cabling attached to the connector during recording sessions routes signals classification of spinal cord injury. American Spinal Injury Association. Spinal
to external amplifiers and then to a series of computers in a cart that process the Cord 35, 266–-274 (1997).
signals and convert them into an output (the neural cursor) that can be viewed 10. Yousry, T. A. et al. Localization of the motor hand area to a knob on the
by the participant on a computer monitor (Fig. 1d). Currently, the system must precentral gyrus. A new landmark. Brain 120, 141–-157 (1997).
be set up and managed by an experienced technician. 11. Lotze, M., Laubis-Herrmann, U., Topka, H., Erb, M. & Grodd, W. Reorganization
Recording sessions. Research sessions were scheduled at least once per week at in the primary motor cortex after spinal cord injury—A functional magnetic
the participant’s residence. Sessions could be cancelled (for example, due to resonance (fMRI) study. Restor. Neurol. Neurosci. 14, 183–-187 (1999).
participant request or schedule conflict) or ended early at the participant’s 12. Shoham, S., Halgren, E., Maynard, E. M. & Normann, R. A. Motor-cortical
activity in tetraplegics. Nature 413, 793 (2001).
request. Sessions would commence with neural recording and spike discrimi-
13. Sabbah, P. et al. Sensorimotor cortical activity in patients with complete spinal
nation, followed by filter building and structured clinical end-point (cursor cord injury: a functional magnetic resonance imaging study. J. Neurotrauma 19,
control) trials. Performance of video games and external device control demon- 53–-60 (2002).
strations followed. The electrodes and neural signals selected immediately before 14. Mikulis, D. J. et al. Adaptation in the motor cortex following cervical spinal cord
filter building remained constant for any given session’s cursor control trials; injury. Neurology 58, 794–-801 (2002).
additional neural signals (detected either with alternative spike discrimination 15. Donoghue, J. P., Sanes, J. N., Hatsopoulos, N. G. & Gaal, G. Neural discharge
methods, or as may have become evident during the cursor control trials) could and local field potential oscillations in primate motor cortex during voluntary
be used during subsequent video game or external device use. movements. J. Neurophysiol. 79, 159–-173 (1998).
16. Sanes, J. N. & Donoghue, J. P. Plasticity and primary motor cortex. Annu. Rev.
Spike discrimination. Units were manually discriminated by a technician using
Neurosci. 23, 393–-415 (2000).
visual features to place time-amplitude windows on waveforms displayed within 17. Serruya, M. D., Hatsopoulos, N. G., Paninski, L., Fellows, M. R. & Donoghue, J. P.
1.6-ms windows triggered when the signal crossed a manually adjusted threshold Instant neural control of a movement signal. Nature 416, 141–-142 (2002).
(Fig. 2) (Cyberkinetics Central Software) while the participant was at rest. We 18. Taylor, D. M., Tillery, S. I. & Schwartz, A. B. Direct cortical control of 3D
applied no objective criteria to determine whether any particular unit (or neuroprosthetic devices. Science 296, 1829–-1832 (2002).
admixture of units) recorded on one day was the same as that recorded on a 19. Evarts, E. V. in Handbook of Physiology (ed. Brooks, V.) 1083–-1120 (Williams
subsequent day. Smaller amplitude signals were likely to have consisted of and Wilkins, Baltimore, 1981).
admixtures of spikes from .1 neuron. These signals were nevertheless used for 20. Georgopoulos, A. P., Kalaska, J. F., Caminiti, R. & Massey, J. T. On the relations
sessions. between the direction of two-dimensional arm movements and cell discharge
in primate motor cortex. J. Neurosci. 2, 1527–-1537 (1982).
Filter building. For each session, single and multiunit data were used to create a
21. Humphrey, D. R. & Tanji, J. in Motor Control: Concepts and Issues
filter to generate a two-dimensional output signal. During filter building, MN (eds Humphrey, D. R. & Freund, H. J.) 413–-443 (John Wiley, London, 1991).
was asked to imagine moving his hand as if he were controlling a computer 22. Wu, W. et al. Modeling and decoding motor cortical activity using a switching
mouse. Specifically, he was asked to imagine tracking a cursor on the computer Kalman filter. IEEE Trans. Biomed. Eng. 51, 933–-942 (2004).
screen; this technician cursor was moved (by the technician) through a succes- 23. Carmena, J. M. et al. Learning to control a brain-machine interface for reaching
sion of randomly positioned targets. During the first 4 min of imagined tracking, and grasping by primates. PLoS Biol. 1, E42 (2003).
only the technician cursor and targets were visible on the screen. An initial filter 24. Baker, J. T., Donoghue, J. P. & Sanes, J. N. Gaze direction modulates finger
was built with the neural activity collected during this epoch, then allowing a movement activation patterns in human cerebral cortex. J. Neurosci. 19,
10044–-10052 (1999).
neural cursor to be placed on the screen. MN continued to track the technician
25. Olson, B. P., Si, J., Hu, J. & He, J. Closed-loop cortical control of direction using
cursor with his neural cursor over a series of four 1-min blocks. The filter was support vector machines. IEEE Trans. Neural Syst. Rehabil. Eng. 13, 72–-80 (2005).
updated at the end of each block. To create the final filter for the research session, 26. Wolpaw, J. R. & McFarland, D. J. Control of a two-dimensional movement
neural data were used from only the four neural cursor blocks. signal by a noninvasive brain-computer interface in humans. Proc. Natl Acad.
Linear filters were constructed from a response matrix containing the firing Sci. USA 101, 17849–-17854 (2004).
170
© 2006 Nature Publishing Group
NATURE|Vol 442|13 July 2006 ARTICLES

27. Birbaumer, N. et al. A spelling device for the paralysed. Nature 398, 297–-298 40. Kennedy, P. et al. Using human extra-cortical local field potentials to control a
(1999). switch. J. Neural Eng. 1, 72–-77 (2004).
28. Fetz, E. E. & Baker, M. A. Operantly conditioned patterns on precentral unit 41. Leuthardt, E. C., Schalk, G., Wolpaw, J. W., Ojemann, J. G. & Moran, D. W. A
activity and correlated responses in adjacent cells and contralateral muscles. brain-computer interface using electrocorticographic signals in humans.
J. Neurophysiol. 36, 179–-204 (1973). J. Neural Eng. 1, 63–-71 (2004).
29. Fetz, E. E. & Finocchio, D. V. Operant conditioning of specific patterns of neural 42. Graimann, B., Huggins, J. E., Levine, S. P. & Pfurtscheller, G. Toward a direct
and muscular activity. Science 174, 431–-435 (1971). brain interface based on human subdural recordings and wavelet-packet
30. Jacobs, K. M. & Donoghue, J. P. Reshaping the cortical motor map by analysis. IEEE Trans. Biomed. Eng. 51, 954–-962 (2004).
unmasking latent intracortical connections. Science 251, 944–-947 (1991). 43. Lal, T. N., et al. in Advances in Neural Information Processing Systems
31. Gandolfo, F., Li, C., Benda, B. J., Schioppa, C. P. & Bizzi, E. Cortical correlates of (eds Saul, L. K., Weiss, Y. & Bottou, L.) (MIT Press, Cambridge, Massachusetts,
learning in monkeys adapting to a new dynamical environment. Proc. Natl Acad. 2005).
Sci. USA 97, 2259–-2263 (2000). 44. Rousche, P. J. & Normann, R. A. A method for pneumatically inserting an array of
32. Schwartz, A. B. Cortical neural prosthetics. Annu. Rev. Neurosci. 27, 487–-507 penetrating electrodes into cortical tissue. Ann. Biomed. Eng. 20, 413–-422 (1992).
(2004).
33. Hochberg, L. R. & Donoghue, J. P. Sensors for brain-computer interfaces. IEEE Supplementary Information is linked to the online version of the paper at
Eng. Med. Biol. (in the press). www.nature.com/nature.
34. Kennedy, P. R. & Bakay, R. A. Restoration of neural output from a paralyzed
patient by a direct brain connection. Neuroreport 9, 1707–-1711 (1998). Acknowledgements The authors thank J. Joseph and D. Morris for assistance;
35. Kennedy, P. R., Bakay, R. A., Moore, M. M., Adams, K. & Goldwaithe, J. Direct L. Mermel for clinical planning advice; V. Zerris and M. Park for surgical
control of a computer from the human central nervous system. IEEE Trans. assistance; G. Polykoff for clinical trial assistance; W. Truccolo for power
Rehabil. Eng. 8, 198–-202 (2000). spectral density analysis development; and the employees of Cyberkinetics for
36. Kennedy, P. R., Kirby, M. T., Moore, M. M., King, B. & Mallory, A. Computer device engineering, manufacturing and clinical trial design and management.
control using human intracortical local field potentials. IEEE Trans. Neural Syst. The authors also thank MN for his participation in this trial, and the nursing staff
Rehabil. Eng. 12, 339–-344 (2004). at his assisted care facility for their assistance. The authors are grateful to
37. Mason, S. G., Bohringer, R., Borisoff, J. F. & Birch, G. E. Real-time control of a M. Serra and Sargent Rehabilitation Center, the study site, for administrative
video game with a direct brain–-computer interface. J. Clin. Neurophysiol. 21, support. The photograph of MN (Fig. 1) is copyright 2005 Rick Friedman. This
404–-408 (2004). work was supported by Cyberkinetics Neurotechnology Systems, Inc.
38. Vaughan, T. M. et al. Brain-computer interface technology: a review of the
Second International Meeting. IEEE Trans. Neural Syst. Rehabil. Eng. 11, 94–-109 Author Information Reprints and permissions information is available at
(2003). npg.nature.com/reprintsandpermissions. The authors declare competing
39. Muller-Putz, G. R., Scherer, R., Pfurtscheller, G. & Rupp, R. EEG-based financial interests: details accompany the paper on www.nature.com/nature.
neuroprosthesis control: a step towards clinical practice. Neurosci. Lett. 382, Correspondence and requests for materials should be addressed to J.P.D.
169–-174 (2005). ([email protected]).

171
© 2006 Nature Publishing Group
View publication stats