International REVIEW OF
Neurobiology Volume 86 SERIES EDITORS RONALD J. BRADLEY Department of Psychiatry, College of Medicine The University of Tennessee Health Science Center Memphis, Tennessee, USA
R. ADRON HARRIS Waggoner Center for Alcohol and Drug Addiction Research The University of Texas at Austin Austin, Texas, USA
PETER JENNER Division of Pharmacology and Therapeutics GKT School of Biomedical Sciences King’s College, London, UK EDITORIAL BOARD ERIC AAMODT PHILIPPE ASCHER DONARD S. DWYER MARTIN GIURFA PAUL GREENGARD NOBU HATTORI DARCY KELLEY BEAU LOTTO MICAELA MORELLI JUDITH PRATT EVAN SNYDER JOHN WADDINGTON
HUDA AKIL MATTHEW J. DURING DAVID FINK MICHAEL F. GLABUS BARRY HALLIWELL JON KAAS LEAH KRUBITZER KEVIN MCNAUGHT JOSE´ A. OBESO CATHY J. PRICE SOLOMON H. SNYDER STEPHEN G. WAXMAN
Academic Press is an imprint of Elsevier 360 Park Avenue South, New York, NY 10010-1700 525 B Street, Suite 1900, San Diego, California 92101-4495, USA 32 Jamestown Road, London NW1 7BY, UK
This book is printed on acid-free paper.
Copyright ß 2009, Elsevier Inc. All Rights Reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording, or any information storage and retrieval system, without permission in writing from the Publisher. The appearance of the code at the bottom of the first page of a chapter in this book indicates the Publisher’s consent that copies of the chapter may be made for personal or internal use of specific clients. This consent is given on the condition, however, that the copier pay the stated per copy fee through the Copyright Clearance Center, Inc. (www.copyright.com), for copying beyond that permitted by Sections 107 or 108 of the U.S. Copyright Law. This consent does not extend to other kinds of copying, such as copying for general distribution, for advertising or promotional purposes, for creating new collective works, or for resale. Copy fees for pre-2007 chapters are as shown on the title pages. If no fee code appears on the title page, the copy fee is the same as for current chapters. 0074-7742/2007 $35.00 Permissions may be sought directly from Elsevier’s Science & Technology Rights Department in Oxford, UK: phone: (þ44) 1865 843830, fax: (þ44) 1865 853333, E-mail:
[email protected]. You may also complete your request on-line via the Elsevier homepage (http://elsevier.com), by selecting ‘‘Support & Contact’’ then ‘‘Copyright and Permission’’ and then ‘‘Obtaining Permissions.’’ For information on all Academic Press publications visit our Web site at www.elsevierdirect.com ISBN-13: 978-0-12-374821-8 PRINTED AND BOUND IN THE UNITED STATES OF AMERICA 08 09 10 11 12 8 7 6 5 4 3 2 1
CONTRIBUTORS
Numbers in parentheses indicate the pages on which the authors’ contributions begin.
Dino Accoto (39), Biomedical Robotics and Biomicrosystems Laboratory, Universita` Campus Biomedico di Roma, Roma, Italy F. Aloise (133), IRCCS Fondazione Santa Lucia, Rome, Italy L. Astolfi (133), Dip. Fisiologia e Farmacologia, Univ. La Sapienza, Rome, Italy; and IRCCS Fondazione Santa Lucia, Rome, Italy Claudio Babiloni (67), Department of Biomedical Sciences, University of Foggia, Foggia, Italy; and Hospital San Raffaele Cassino, Cassino, Italy F. Babiloni (133), Dip. Fisiologia e Farmacologia, Univ. La Sapienza, Rome, Italy; and IRCCS Fondazione Santa Lucia, Rome, Italy A. Bengoetxea (171), Laboratory of Neurophysiology and Biomechanics of Movementa, Universite´ Libre de Bruxelles, CP 168, 50 Av. F. Roosevelt, Brussels, Belgium Antonella Benvenuto (39), Biomedical Robotics and Biomicrosystems Laboratory, Universita` Campus Biomedico di Roma, Roma, Italy Alain Berthoz (159), Laboratoire de Physiologie de la Perception et de l’Action, UMR 7152, Colle`ge de France, CNRS, Paris, France Niels Birbaumer (107), Ospedale San Camillo—IRCCS, Istituto di Ricovero e Cura a Carattere Scientifico, Venezia, Lido, Italy; and Institute of Medical Psychology and Behavioral Neurobiology, University of Tuebingen, Tuebingen, Germany Domenico Campolo (39), Biomedical Robotics and Biomicrosystems Laboratory, Universita` Campus Biomedico di Roma, Roma, Italy Federico Carpi (3), University of Pisa, Interdepartmental Research Centre ‘‘E. Piaggio’’, School of Engineering, 56100 Pisa, Italy A. M. Cebolla (171), Laboratory of Neurophysiology and Biomechanics of Movementa, Universite´ Libre de Bruxelles, CP 168, 50 Av. F. Roosevelt, Brussels, Belgium G. Cheron (171), Laboratory of Electrophysiology, Universite´ de Mons-Hainaut, Mons, Belgium; and Laboratory of Neurophysiology and Biomechanics of Movementa, Universite´ Libre de Bruxelles, CP 168, 50 Av. F. Roosevelt, Brussels, Belgium F. Cincotti (133), IRCCS Fondazione Santa Lucia, Rome, Italy
xi
xii
CONTRIBUTORS
Luca Citi (199), School of Computer Science and Electronic Engineering, University of Essex, Wivenhoe Park, CO4 3SQ Colchester, UK B. Dan (171), Department of Neurology, Hopital Universitaire des Enfants Reine Fabiola, Universite´ Libre de Bruxelles, Belgium; and Laboratory of Neurophysiology and Biomechanics of Movementa, Universite´ Libre de Bruxelles, CP 168, 50 Av. F. Roosevelt, Brussels, Belgium F. De Vico Fallani (133), IRCCS Fondazione Santa Lucia, Rome, Italy Antonio Ferretti (67), Department of Clinical Sciences and Biomedical Imaging, Institute for Advanced Biomedical Technologies, University G. D’Annunzio of Chieti, Chieti, Italy Pierre W. Ferrez (189), Idiap Research Institute, Rue Marconi 19, 1920 Martigny, Switzerland Samson Freyermuth (159), Laboratoire de Physiologie de la Perception et de l’Action, UMR 7152, Colle`ge de France, CNRS, Paris, France Cosimo Del Gratta (67), Department of Clinical Sciences and Biomedical Imaging, Institute for Advanced Biomedical Technologies, University G. D’Annunzio of Chieti, Chieti, Italy Eugenio Guglielmelli (39), Biomedical Robotics and Biomicrosystems Laboratory, Universita` Campus Biomedico di Roma, Roma, Italy Dario Izzo (213), Advanced Concepts Team, European Space Agency, Keplerlaan 1, 2201 AZ Noordwijk, The Netherlands Karim Jerbi (159), INSERM U821, Brain Dynamics and Cognition Laboratory, Lyon 69500, France; and Laboratoire de Physiologie de la Perception et de l’Action, UMR 7152, Colle`ge de France, CNRS, Paris, France Philippe Kahane (159), Department of Neurology and INSERM U704, Grenoble Hospital, Grenoble, France Dean J. Krusienski (147), School of Engineering, University of North Florida, Jacksonville, Florida, USA Jean-Philippe Lachaux (159), INSERM U821, Brain Dynamics and Cognition Laboratory, Lyon 69500, France A. Leroy (171), Laboratory of Neurophysiology and Biomechanics of Movementa, Universite´ Libre de Bruxelles, CP 168, 50 Av. F. Roosevelt, Brussels, Belgium M. Marciani (133), IRCCS Fondazione Santa Lucia, Rome, Italy Martina Marinelli (199), Scuola Superiore Sant’Anna, piazza Martiri della Liberta` 33, 56127 Pisa, Italy D. Mattia (133), IRCCS Fondazione Santa Lucia, Rome, Italy Silvestro Micera (23), Institute for Automation, Swiss Federal Institute of Technology, CH-8092 Zurich, Switzerland; and ARTS and CRIM Labs, Scuola Superiore Sant’Anna, I-56127 Pisa, Italy
CONTRIBUTORS
xiii
Jose´ del R. Milla`n (189), Ecole Polytechnique Fe´de´rale de Lausanne (EPFL), Lausanne, Switzerland; and Idiap Research Institute, Rue Marconi 19, 1920 Martigny, Switzerland Lorella Minotti (159), Department of Neurology and INSERM U704, Grenoble Hospital, Grenoble, France Pedro Montoya (107), Department of Psychology, Universidad Illes Baleares, Palma de Mallorca, Spain Gernot R. Mu¨ller-Putz (119), Institute for Knowledge Discovery, Laboratory of Brain–Computer Interfaces, Graz University of Technology, Graz, Austria Ander Ramos Murguialday (107), Fatronik Foundation, San Sebastian, Spain; and Institute of Medical Psychology and Behavioral Neurobiology, University of Tuebingen, Tuebingen, Germany Xavier Navarro (23), Centro de Investigacio´n Biome´dica en Red sobre Enfermedades Neurodegenerativas (CIBERNED), Spain; and Institute of Neurosciences, Universitat Auto`noma de Barcelona, E-08193 Bellaterra, Spain E. Palmero-Soler (171), Laboratory of Electrophysiology, Universite´ de Mons-Hainaut, Mons, Belgium; and Laboratory of Neurophysiology and Biomechanics of Movementa, Universite´ Libre de Bruxelles, CP 168, 50 Av. F. Roosevelt, Brussels, Belgium M. Petieau (171), Laboratory of Neurophysiology and Biomechanics of Movementa, Universite´ Libre de Bruxelles, CP 168, 50 Av. F. Roosevelt, Brussels, Belgium Gert Pfurtscheller (119), Institute for Knowledge Discovery, Laboratory of Brain–Computer Interfaces, Graz University of Technology, Graz, Austria Giovanni Di Pino (39), Biomedical Robotics and Biomicrosystems Laboratory, Universita` Campus Biomedico di Roma, Roma, Italy Vittorio Pizzella (67), Department of Clinical Sciences and Biomedical Imaging, Institute for Advanced Biomedical Technologies, University G. D’Annunzio of Chieti, Chieti, Italy Gian Luca Romani (67), Department of Clinical Sciences and Biomedical Imaging, Institute for Advanced Biomedical Technologies, University G. D’Annunzio of Chieti, Chieti, Italy Danilo De Rossi (3), University of Pisa, Interdepartmental Research Centre ‘‘E. Piaggio’’, School of Engineering, 56100 Pisa, Italy Luca Rossini (213), Biomedical Robotics and Biomicrosystems Laboratory, Universita` Campus Biomedico di Roma, Via Alvaro del Portillo 21, 00128 Roma, Italy Paolo Maria Rossini (39, 81), Department of Neurology, Universita` Campus Biomedico di Roma, Via Alvaro del Portillo 21, 00128 Roma, Italy John Rothwell (51), Sobell Department, Institute of Neurology, Queen Square, London WC1N 3BG, UK
xiv
CONTRIBUTORS
S. Salinari (133), Dip. Informatica e Sistemistica, Univ. La Sapienza, Rome, Italy; and ARTS and CRIM Labs, Scuola Superiore Sant’Anna, Pisa, Italy Reinhold Scherer (119), Computer Science and Engineering, University of Washington, Seattle, Washington, USA; and Institute for Knowledge Discovery, Laboratory of Brain–Computer Interfaces, Graz University of Technology, Graz, Austria Tobias Seidl (39, 189), Advanced Concepts Team, European Space Agency, Keplerlaan 1, 2201 AZ Noordwijk, The Netherlands Francisco Sepulveda (93), Brain–Computer Interfaces Group, Department of Computing and Electronic Systems, University of Essex, Wivenhoe Park, Colchester, Essex CO4 3SQ, United Kingdom Fabrizio Sergi (39), Biomedical Robotics and Biomicrosystems Laboratory, Universita` Campus Biomedico di Roma, Roma, Italy Leopold Summerer (213), Advanced Concepts Team, European Space Agency, Keplerlaan 1, 2201 AZ Noordwijk, The Netherlands Oliver Tonet (199), CRIM Lab, Scuola Superiore Sant’Anna, Pisa, Italy Cornelia Weber (107), Institute of Medical Psychology and Behavioral Neurobiology, University of Tuebingen, Tuebingen, Germany Jonathan R. Wolpaw (147), Laboratory of Neural Injury and Repair, Wadsworth Center, Albany, New York, USA
FOREWORD
In writing this foreword, I am sitting comfortably on my chair in front of my computer and using my fingers to type character by character, word by word as they are formulated in my head. While this is a very common way of transmitting information indeed (emails, papers, books, SMS), it relies on a rather ‘‘un-natural’’ interface, compared to the ‘‘natural’’ ways of human information exchange: gesture, mimic, and speech. We have increased the eYciency of communicating in terms of speed, distance, and reach by adapting our means of communication to the interfaces imposed by the limitations of machines (typewriter, computer, mobile phone, etc.). We have therefore reduced our own interface to our most powerful tools: our hands and its 10 fingers. This has become possible since for many of these tasks, we have freed our hands from any additional manual work to be done at the same time. With the introduction of increasing computational power, we have been gradually introducing more human-like aspects to the interaction with machines and especially computers: the mouse and a graphical interface instead of text-only command-line inputs and more recently touch and multitouch displays as well as gradually speech recognition. In space, we are confronted with a whole range of additional complications and requirements which demand the use of our main tools: our hands! while continuing to interface with computers. In microgravity, we actively use our feet and hands to keep a certain posture and attitude, to perform experiments and manipulate objects. Given the risks associated with handling errors and the underlying complications of some of these activities combined with the high value of experimental and thus astronaut time in space, most of our activities are highly regulated, written down in detailed manuals and we are training most procedures during weeks and months on ground. Once in space, we therefore usually use a laptop close by, displaying the steps of our procedures while being concentrated to performing them with our hands at the same time and trying to keep posture essentially by the use of our feet. Many times we even have two laptops close by, one with the procedure and one that displays expected intermediate results (pressure values, temperatures, etc.) of the operations we perform. Very simple tasks on the laptop, like scrolling down a page, or increasing an image or following a link with further explanations become therefore already a distraction that take time since they require taking at least one hand oV the experiment or maintenance/repair task, find the trackball on the computer keyboard, move the cursor to the right direction and then click, scroll, etc. before being able to turn back to the actual manual task. xv
xvi
FOREWORD
As an astronaut, I am therefore highly interested in any technology that would allow me to keep concentrated on my actual manual tasks and have my two hands freed to perform them. Just to give you an example from my own experience, I even tried sometimes to use the track-pad with my nose just to keep my hands inside the gloves of a glovebox. We are currently experimenting with a range of such tools like speech recognition, gaze tracking, head-mounted cameras, and overlay displays, etc. All of these techniques have their advantages and disadvantages, and at this stage it is not clear which one would prove the most eYcient. Recent advances in brain science in understanding the functioning of our brain and in brain–machine interfaces (BMIs) have opened another possibility. The prospect of being able—with some prior training—to keep our hands and eyes on the experiment and perform some initially even very simple tasks like scrolling in a document, closing windows, increasing font sizes, etc. by ‘‘thinking’’ is very attractive and I am thus welcoming the initiative of this book, where the scientific community working on BMIs attempts to describe the state-of-the-art in BMIs research for the purpose of human space activities and critically reviews the requirements that space puts on the design of these types of interfaces. FRANK DE WINNE ESA Astronaut
PREFACE WHY THE BMI ‘‘RENAISSANCE’’ COULD AFFECT THE FUTURE OF SPACE EXPLORATION
I. Introduction
The Advanced Concepts Team of the European Space Agency has been investigating the potential application of brain–machine interfaces (BMIs) in cooperation with a number of European universities.1 It was a preliminary eVort during which our group collected data, ideas, and started a vivid discussion which confirmed the initial interest and led to a more in-depth analysis of possible solutions oVered by BMIs to some of the limitations astronauts face when operating in space. Hence, this publication on BMIs evolved naturally, targeting primarily the aerospace community, and aiming to provide the knowledge and the reflections that we hope will help overcoming the healthy skepticism sometimes expressed about the application of BMIs in space. While preparing this book we have come to appreciate the importance such a publication could also have for BMI specialists, as it describes the needs and requirements the space environment puts on interface designs and on their performances, and discusses the resulting potential and limitations of the technology. This book is therefore primarily written by, and targeted to, two communities: the space community, always interested in innovative solutions to renowned problems (especially nowadays, as the renewed Moon and Mars human exploration plans are going to put the lights on some unsolved issues related to the human presence in space beyond low Earth orbit), and the wider scientific community (Neuroscientists, Bioinformatics, Bioengineers, Roboticists, Neurorehab scientists) investigating into BMIs and new generation interfaces in general. This community, even if notoriously overloaded with new publications on the topic, is particularly aware of the importance of 1
Brain-Machine Interfaces, Advanced Concepts Team, European Space Agency, March 2009, http:// www.esa.int/gsp/ACT/bng/op/BMI.htm xvii
xviii
PREFACE
understanding the requirements BMI systems need to meet in order to reach new applications, especially those having a huge inspirational value such as space exploration. This book aspires to lay the first foundation and eventually become a reference point for future routine collaborations between space actors and the wide field of neuroprosthesis and BMIs experts.
II. BMIs Renaissance
To share the enthusiasm of many in BMIs, one has to appreciate the diVerence between the potential advantages of a brain interface and other kinds of interfaces. The worldwide amazement generated by the work of Nicolelis in the year 2001 was caused by the clarity with which this diVerence was demonstrated.2 In a long video sequence, the world witnessed a monkey in the act of controlling a robotic arm and feeding itself by doing so. The monkey was sitting and behaving normally while, at the same time, controlling with its neurons the actions of the robotic arm. The message was so clear and revolutionary in its nature that it inevitably started a new fervor we dare to refer to as ‘‘BMIs renaissance.’’ It was suddenly clear that BMIs are not only potentially able to restore lost abilities, but they are also a realistic option to augment capabilities and eventually adding new ‘‘peripherals’’ to the body. An insight that equally generated hopes and fears in the general public and among scientists and opened a healthy discussion on ethical implications related to the use of this technology.3
III. Natural Interfaces Mean Technological Democracy
The proliferation of more and more sophisticated electronic devices is a characteristic of our time. The direct access to informational contents and to applications’ interfacing and controlling is steadily increasing, while the new phenomenon of digital divide (here intended in its general sense) is also growing at the same rate: the use, and thus its related benefits, of new technologies is not accessed by all potential users. Geographical discrimination is only one of the gaps, which the digital revolution is struggling to bridge. Another discrimination regards knowledge. Any new device often requires the know-how related with the interface of its predecessor, plus a learning process associated to the new features oVered. Hence, who does not manage to keep the pace, for reasons related with 2
Nicolelis, M. A. L., ‘‘Actions from thoughts’’, Nature 2001, 409, 403–407 Clausen, J., ‘‘Man, machine and in between’’, Nature 2009, 457, 1080–1081
3
PREFACE
xix
motivation, age, education level, work experience among the others, is eventually overwhelmed by devices each time more complex and advanced. Henry Ford once said that ‘‘true progress is made only when the advantages of a new technology are within reach of everyone’’: a great challenge that lies ahead our digital era. A solution is the development of better interfaces, since they are responsible for our communication with machines, and thus for the definition of a universal language able to provide all with the possibility of controlling present and future devices. It is possible to argue that the best accessibility will derive from interfaces able to exploit at best the natural communication pathways characteristic of the human being. A ‘‘natural’’ interface learns and adapts itself to the user’s interfacing modalities, and not the opposite, extending the use of new technologies towards population sectors which are typically struggling to interface with technologies (i.e., personal computers). Within this context, BMIs are being seen as a technology aimed at improving the everyday life of a large number of common people by providing such a universal language. Many classical interfaces fail to fulfill such an ambitious goal. They are operated by predefined user’s motor actions, and typically diVerent action kinematics and geometries are associated with diVerent interfaces (i.e., computer keyboards and mouse). As a consequence, their use inevitably requires a re-modulation of connections between user’s brain sensorimotor areas involved with the device utilization (i.e., the neuronal networks involved with the actual formulation of a task), and those dedicated to the motor task for the interface operation (i.e., the hand and fingers control to obtain a correct typing of the related task’s commands). Obviously, operating a classical interface carries a high cognitive load which—when several diVerent interfaces have to be operated within a short period, or when the operator is less re-active in its internal re-modulation—can easily decrease the operator’s mental performance due to mental fatigue and increase the error rate.
IV. Interfaces for Astronauts
Astronauts (a particular highly motivated, trained, and skilled category) are also suVering, when in orbit, from what can be seen as a case of digital divide. On Earth, the high versatility of humans allows for a huge range of elaborated behaviors, such as dancing, playing soccer, doing acrobatics, or playing music instruments, some even at the same time. However, the human motor performances are strictly bounded to the physical conditions that govern our planet. Our perception and planning of movement, for example, are related to the identification of the gravity axis. Inevitably, our sensory–motor system encounters a loss of performance in situations of changed or annihilated gravity. This loss is so important that, from certain perspectives and for the purpose of an academic
xx
PREFACE
comparison, we can assume that astronauts are in a similar situation to people aVected by motor disabilities on Earth: both have, for diVerent reasons, a deficit in the performances required to accomplish their motor tasks. On Earth, disabled people can take advantage of assistive systems, which are technologies designed to fill the gap between user’s residual abilities and required ones. The reduction of physical and mental ability suVered by astronauts is addressed (at least partially) by a number of assistive technologies redesigned for functioning in space. As for any activity which relies on motor coordination, weightlessness conditions aVect astronauts uses of any kind of classical interface. It may be argued that a complex integration of ‘‘behavior monitoring technologies’’ like speech recognition, gesture recognition, facial expression recognition, and gaze tracking provides an appropriate supportive system and can provide valid natural interfaces. A correct use of these technologies has been demonstrated to have the potential to handle the control of complex systems, yet their potentials for severely disabled people and—in an analogue way—for astronauts is limited. Gesture and facial expression recognition cannot be eYcient if the user does not perform within strict tolerances and speech recognition reliability presents great challenges when background noise is present (this can be as high as 64 dBA for the air conditioning to 100 dBA for some vent relief valves in case of space stations). Gaze tracking alone is not precise enough and rich in contents to permit the control of complex systems. In this work, we propose to further explore BMI technologies as alternatives to other, more extensively studied devices that are inherently limited by their functioning principle. Completely independent from the user’s physical abilities, BMIs are instead able to access the user’s intentions at a higher level, were they naturally origin: in the brain. They predict directly the user’s motor intentions, not related with users’ abilities, and are thus accessible to the widest range of users. By monitoring the activity of neurons (individually or in networks) and translating them into actions, BMIs have the potential to provide the most direct control over complex systems and since they are expected to operate, in principle, very similar in space and on Earth could assist astronauts by helping them to perform in space as eYciently as on the Earth.
V. BMIs for Space Applications: An Outlook
Understanding the real potentials of BMIs for space applications requires an eVort in correctly framing BMIs themselves in the first place. When a user controls with his thoughts the operations of a robotic hand, for example, his brain is being part of a hybrid system in which living tissues (neural cells among others) and artificial elements (i.e., the robotic hand) are working together. Systems like this are referred to as Hybrid Bionic Systems (HBSs), and BMIs
PREFACE
xxi
are one of their manifestation. The first section is entirely dedicated to these systems, and focuses on HBSs other than BMIs which could be beneficial for space applications, in an increasing ‘‘scale of invasiveness’’ fashion, from the noninvasive electromyographic and gaze-tracking interfaces, where an intact human body controls by means of natural actions diVerent systems, on which Danilo de Rossi and Federico Carpi give an in-depth review; passing through the invasive monitoring of peripheral nerve activity for bidirectional interfaces, able to read the planned limb motions and to stimulate limb sensations at the same time, actually fusing the user’s body with the controlled system, as two of the pioneers of interfaces with the peripheral nervous system, Xavier Navarro and Silvestro Micera, explain in a comprehensive review; to the end point of hybridization, where small portions of an insect’s brain are included in and connected to a complex artificial system, which can then benefit from the amazing and otherwise un-reproducible level of intelligence which was developed in a millenarian evolution towards autonomous and successful behaviors. Tobias Seidl, a behavioral biologist from the Advanced Concepts Team of the European Space Agency, together with Eugenio Guglielmelli and his Biomedical Robotics and Biomicrosystems group, focused for the first time on such an architecture and present their ideas and outcomes in the last chapter of the section on HBS. As for any other HBS, it is first of all essential to understand the biological components included in the architecture, in order to find the right way to integrate it with the artificial part and to foresee the potential applications and possible shortcomings of the complete system. For this reason the second section is fully dedicated to describe the basic neuroscience knowledge, a task addressed by some of the most renown researchers in the field of neurophysiology. John Rothwell explains what a neuron is, how it works, and how it competes with other neurons in neuronal networks and functional areas to the production of the overall brain activity. Claudio Babiloni, Gianluca Romani, and Vittorio Pizzella explain how this activity is globally organized, how we can monitor and analyze it for clinical or research activities, like for the control of BMIs. Finally, Paolo Rossini describes how, thanks to plastic changes, the brain learns and adapts itself under internal and environmental pressures, and how this adaptations could interact with BMIs control performances in the short and in the long time. The next inevitable step towards understanding BMIs is in the interface architecture itself. As Francisco Sepulveda explains, opening the third section specifically dedicated to the architectures of BMIs, BMIs are composed of many diVerent components, each of which playing crucial roles in the final performance of the system. Few diVerent architectures are then presented directly by their inventors, exploring some of the variety of applications to which those systems can be a solution. Clinical applications, like restoring a communication pathway for patients which otherwise would be completely cut out from any possible interaction with the environment and with other people, or controlling a limb on which
xxii
PREFACE
the natural control ways were pathologically interrupted, can be addressed with BMI systems exploiting self-controlled slow cortical potentials, as Niels Birbaumer demonstrated during the last 30 years. Almost the same time separated Gert Pfurtscheller from demonstrating the possibility of correlating event-related potentials (ERP) to limb movement planning and execution, to the current implementation of automatic and real-time ERP interpreter which is able to drive many diVerent BMI systems, like virtual keyboards, virtual navigation systems, and even Google EarthW software. While groups with long-time experience are synonymous of scientific quality, new research teams have in the past shown higher likelihood to introduce unconventional innovation and novelty. Between the new protagonists of BMI research, Francesco Babiloni and his group at Santa Lucia excelled in applying their experience on EEG signal processing in the optimization of a BMI system to work as a remote controller for a complex domotic environment, in which severely paralyzed patients can control devices autonomously. Noteworthy in this respect is also the work of Dean Krusienski, who has built on his experience matured with the BMI group of Jonathan Wolpaw in the Wadsworth Center, one of the most renowned research centers on noninvasive BMIs: Krusienski and Wolpaw present in this book what, in the opinion of the editors, is one of the most advanced BMI systems for humans currently developed. Finally, invasive BMI systems can be evaluated on human subjects, whenever the subjects are already, and for clinical reasons related to other pathologies (like epilepsy), implanted with intracortical electrodes (i.e., for electrocorticography—ECoG). The first results of ECoG–BMI systems are indeed holding promising results, as detailed in the contribution of Karim Jerbi, Alain Berthoz, and Jean-Philippe Lachaux. In the fourth and last section of the book, we wanted to introduce what we think might be pivot elements for future BMI for space application. Despite the many possible solutions presented in the third section, some of which are already able to address questions related to space applications, exposition to microgravity causes some very specific diVerences in the way the human brain works. As the brain is the main element of BMIs, any changes due to short-time or long-time exposition to ‘‘space environments’’ have to be taken in consideration in the specific design and set up of BMI for space applications. In particular, as Guy Cheron demonstrated in tests performed during several experiments on the International Space Station (ISS), there are adaptive changes of rhythmic EEG oscillations under microgravity exposure which can have profound implications for BMI systems’ design. These eVects and implications are presented in the opening chapter of the fourth section. A first step towards appreciating the eVects of microgravity on BMI operations was already done during a parabolic flight campaign of ESA in December 2007. Thanks to parabolic flights the human body can experience repetitive short exposures to microgravity. Together with Jose´ del R. Millan, the Advanced Concepts Team of the European Space Agency tested
PREFACE
xxiii
the eVects of such exposures on the brain activity related with BMI control. The complete results of this study are reported for the first time in this book. Once the potential eVects of microgravity on BMI have been taken under consideration and some of the initial problems overcome, interface engineers are expected to design brain-driven systems for many types of space applications. However, early adopters would probably quickly be lost in trying to rate the diVerent architectures in terms of performance and reliability, since diVerent BMI research groups tend to apply very diVerent approaches to performance evaluation. One of the outcomes of the preliminary studies on BMIs for space applications we performed in 2006 with several European universities came from three young scientists, Oliver Tonet, Luca Citi, and Martina Marinelli. It is a valuable tool to address the matching between BMI systems and space applications, and its results are presented in this book. The book is closed with a short communication chapter from the editors, which intends to highlight the research directions of BMIs with the perceived highest potential impact on future space applications, and to present an overview of the long-term plans with respect to human space flight, and is concluded by suggesting research and development steps considered necessary to include BMI technology in future architectures for human space flight. The research field of BMI grew with progressive impetus during the last 10 years, giving birth to a wide number of publications on the topic. The present book is not trying to complement these. While not introducing new discoveries or interpretations of elements of BMI, it intends to summarize the state-of-art of the field in order to address, for the first time, the possible relationship between BMIs and space exploration. The question, which, at first glance, could be regarded as a mere science fiction dilemma, is indeed well founded in the eVects that microgravity has on the human motor coordination. LEOPOLD SUMMERER DARIO IZZO LUCA ROSSINI
EMG-BASED AND GAZE-TRACKING-BASED MAN–MACHINE INTERFACES
Federico Carpi and Danilo De Rossi University of Pisa, Interdepartmental Research Centre ‘‘E. Piaggio’’, School of Engineering, 56100 Pisa, Italy
I. Introduction II. EMG-Based Interfaces A. Fundamentals B. Characteristics and Issues C. Examples of Applications III. Gaze-Tracking-Based Interfaces A. Fundamentals B. Characteristics and Issues C. Examples of Applications IV. Final Remark References
A great demand for brain–machine and, more generally, man–machine interfaces is arising nowadays, pushed by several promising scientific and technological results, which are encouraging the concentration of eVorts in this field. The possibility of measuring, processing and decoding brain activity, so as to interpret neural signals, is often looked at as a possibility to bypass lost or damaged neural and/or motor structures. Beyond that, such interfaces currently show a potential for applications in other fields, space science being certainly one of them. At present, the concept of ‘‘reading’’ the brain to detect intended actions and use these to control external devices is being studied with several technical and methodological approaches; among these, interfaces based on electroencephalographic signals play today a prominent role. Within such a context, the aim of this section is to present a brief survey on two types of noninvasive man–machine interfaces based on a diVerent approach. In particular, they rely on the extraction of control signals from the user with techniques that adopt electromyography and gaze tracking. Working principles, implementations, typical features, and applications of these two types of interfaces are reported.
INTERNATIONAL REVIEW OF NEUROBIOLOGY, VOL. 86 DOI: 10.1016/S0074-7742(09)86001-7
3
Copyright 2009, Elsevier Inc. All rights reserved. 0074-7742/09 $35.00
4
CARPI AND DE ROSSI
I. Introduction
Integrating human and robotic machines into one system has the potential to oVer multiple opportunities for creating assistive technologies that can be used in space, biomedical, and industrial applications. In this context, the development and use of noninvasive man–machine interfaces is progressively gaining a considerable importance. The most direct and, for certain aspects, fascinating (although challenging as well) implementation of a man–machine interface relies on so-called brainmachine interfaces (BMIs) or brain–computer interfaces (BCI), or neuroprostheses (Wolpaw et al., 2002). Such systems are typically conceived as technological interfaces between a machine (usually a computer) and a human brain. They should allow the user to perform certain tasks without any or with minimal motor action. This implies that neural impulses generated by the user should be detected, elaborated and used by the machine, approximately in real time, to perform specific tasks. Accordingly, BMI might represent fundamental systems functional to rehabilitation, communication or assistance. For instance, they are intended to implement brain controls of devices deputed either to perform external actions (such as active prostheses and orthoses, computer virtual keyboards, home electronic equipment, or aid systems) or to trigger functional electrical stimulations (FES) of muscles and nerves (bypassing degenerated or interrupted biological electrical routes) (Navarro et al., 2005; Wolpaw et al., 2002). Beyond biomedical applications, the availability of reliable, eYcient and noninvasive BMIs may provide advantages for diVerent disciplines, space science being one of them. As an example, extravehicular activities may be performed by robotic systems tele-operated by astronauts by means of noninvasive BMIs. Such interfaces may be used also to perform multitask operations. So far, the concept of ‘‘reading’’ the brain to detect intended actions and to use extrapolated signals to perform tasks has been developed in several ways, by adopting diVerent technical and methodological approaches and achieving diVerent results. Interfaces based on electroencephalography (EEG) are certainly one of the most promising solutions to this problem (Wolpaw et al., 2002), as extensively described in the rest of this book. Nevertheless, in addition to EEG-based interfaces, diVerent kinds of man– machine interfaces are currently drawing considerable attention as well. The main purpose of this section is to provide a brief description of two types of such diVerent systems: those based on electromyography (EMG) recordings and those based on gaze-tracking techniques. These might be regarded as complementary rather than alternative approaches in comparison with EEG-based interfaces.
EMG-BASED AND GAZE-TRACKING-BASED INTERFACES
5
The following sections describe these technologies, by providing for each of them brief insights in the fundamental aspects, typical features and most significant applications. II. EMG-Based Interfaces
A. FUNDAMENTALS Electromyography consists in a recording of bioelectric signals generated by neuromuscular activity. As such, EMG signals are an electrical display of neuromuscular activation associated with a contraction of a skeletal muscle, regulated by the central and peripheral nervous system (De Luca, 1988; Henneberg, 2000). The bioelectric genesis of EMG potentials can be synthetically described as follows. Motoneurons lead nervous pulses from the spinal cord to the neuromuscular junctions, where action potentials trigger muscle contraction, as the final result of a series of electrochemomechanical events (their description goes beyond the scope of this book) (Guyton and Hall, 2005). The electrical signal generated by the activation of the muscular fibers belonging to a certain motor unit is called motor unit action potential (MUAP) and represents an elementary basic component of an EMG signal. A MUAP drives a force twitch of the muscle fibers. To sustain a prolonged contraction, a motor unit should be activated repeatedly; accordingly, muscle fibers belonging to the considered motor unit should be driven by a sequence of MUAP; such a sequence is called motor unit action potential train (MUAPT). As a result, an EMG signal results from a space and time varying superposition of multiple MUAPTs (De Luca, 1988), as schematically represented in Fig. 1. EMG measurements can be performed noninvasively by means of surface electrodes (also known as skin or cutaneous electrodes); in this case, the technique is usually referred to as surface electromyography (SEMG) (De Luca, 1997). Figure 2 reports an example of arrangement of skin electrodes for EMG recordings from the arm (e.g., biceps muscle); as for any other type of bioelectric signal, a single-channel measurement always requires a primary, a secondary, and a ground electrode. EMG signals can have amplitudes up to the order of 1–10 mV, depending on the measurement site and adopted electrodes (De Luca, 1997; Webster, 1997); maximum values of the peak-to-peak amplitude close to 5 mV can be commonly obtained with the appropriate use of surface electrodes. Typically, the most significant frequency components of EMG biopotentials are included approximately between 10 Hz and some hundreds of Hz, as indicative orders of magnitudes; accordingly, usual recommendations suggest a recording bandwidth covering the range 25 – 500 Hz (De Luca, 1997). An example of EMG spectrum is reported in Fig. 3.
6
CARPI AND DE ROSSI
EMG signal
EMG electrode
Muscle MUAPT signals
Motoneuron FIG. 1. Schematic drawing of the bioelectric genesis of EMG signals.
Primary electrode Secondary electrode Ground electrode
EMG recording unit
Vin+ GND Vin−
FIG. 2. Schematic drawing of an EMG recording by means skin electrodes.
EMG-BASED AND GAZE-TRACKING-BASED INTERFACES
7
Amplitude [a.u.]
100 80 60 40 20 0
0
100
200
300
400
500
Frequency [Hz] FIG. 3. Frequency spectrum of an EMG signal recorded with skin electrodes from an anterior tibial muscle during a voluntary isometric activation, about 50% of the maximum value.
Since the half of the 1800 century, electromyographic signals collected from the skin surface have been used as an easy and noninvasive access to the electrophysiological processes that drive muscular contraction. As an application of the knowledge arisen in the following decades, since the early 1960s the use of EMG signals for controlling prosthetic devices has progressively increased. To such an extent, EMG biopotentials can be captured from diVerent body portions, depending on the application (Navarro et al., 2005); for instance, the forearm is typically adopted in the case of prosthetic hands (Castellini and van der Smagt, 2009; Zecca et al., 2002). The simplest strategy of use of an EMG signal for control purposes consists in extracting its amplitude or rate of change. More information can be obtained by using two channels, corresponding to two primary electrodes placed on two antagonist muscles (e.g., biceps and triceps brachii or flexor and extensor of the forearm). Figure 4 reports some examples of signals that can be recorded in such a manner. To extract useful information, the recorded raw signals should undergo an adequate preprocessing, aimed at emphasizing specific features. The estimation of the so-called ‘‘normalized muscle activation level’’ (NAL) is one of the most relevant of those; it is achieved with a signal preprocessing procedure including the following blocks (see, for instance, Rosen et al., 2001): (1) a high-pass filtering (to reject low-frequency motion artifacts); (2) a full signal rectification (to extrapolate the signal absolute value); (3) a low-pass filter (to reduce noise, while limiting the bandwidth to the most useful components); (4) a signal normalization (typically with respect to the mean value during maximal voluntary isometric contraction).
8
CARPI AND DE ROSSI
Elbow flexion
Elbow extension Biceps
Biceps
Triceps
Triceps Forearm pronation
Forearm supination
Biceps
Biceps
Triceps
Triceps
FIG. 4. Examples of EMG signal couples detectable from agonist–antagonist muscle couples for diVerent types of movements.
B 500
Recorded EMG [mV]
3
1000
1500
2000
2500 3
2
2
1
1
0
0
−1
−1
−2
−2
−3 500
1000
1500
Time [n⬚ of samples]
2000
−3 2500
500
Pre-processed EMG [mV]
A
2.0
1000
1500
2000
2500 2.0
1.5
1.5
1.0
1.0
0.5
0.5
0.0
0.0 500
1000
1500
2000
2500
Time [n⬚ of samples]
FIG. 5. Example of an EMG signal recorded from the forearm, before (A) and after (B) a preprocessing.
As an example, Fig. 5 presents an EMG signal, both as-detected and after an elementary preprocessing; in this example, the final part of the preprocessing phase included a leveling of the signal above a certain threshold. Aimed at driving an external device, the controller should then further process the EMG signal, typically according to the following successive actions: (1) action onset detection (i.e., identification of the time instant when the muscle goes from the relaxed to the contracted state); (2) feature extraction; (3) pattern classification. Several algorithms can be used for detecting the movement onset (Micera et al., 1998, 2001) and for extracting features and classifying related patterns (Crawford et al., 2005; Zecca et al., 2002). Real-time pattern discrimination and classification is certainly one of the most delicate issues for EMG signals (as for any other type of bioelectric information); with respect to this, neural network-based algorithms
EMG-BASED AND GAZE-TRACKING-BASED INTERFACES
9
(Bishop, 1995) are today commonly adopted (Castellini and van der Smagt, 2009; Soares et al., 2003; Zecca et al., 2002), in addition to Bayesian classifiers (Zecca et al., 2002) and fuzzy logic systems (Ajiboye and Weir, 2005; Chan et al., 2000; Zecca et al., 2002).
B. CHARACTERISTICS AND ISSUES One of the most significant characteristic of SEMG is represented by its capability of evaluating ongoing muscular activity noninvasively and in a comfortable manner for the user. This feature makes the EMG technology easily and readily applicable for controlling robotic devices (Navarro et al., 2005). Such a use is also favored by the relative ease of detection of EMG signals, due to their quite high amplitudes (as reported above). Moreover, their band-pass spectrum, typically excluding frequencies lower than 25 Hz (De Luca, 1997), provides an advantage for a rejection of motion artifacts; in fact, these contain very low frequencies that can be eVectively filtered out, while keeping most of the useful information. Nevertheless, although surface EMG is a useful measure of muscle activation and assessment, the information that can be extracted from this type of signals is aVected by some intrinsic limitations, briefly reported below. An important issue is that EMGs capture potentials generated mostly by superficial muscles and that are sensitive to the electrical activity of a greater muscular area (large number of motor units), with consequent poor selectivity. With respect to this, the phenomenon known as crosstalk plays a relevant role: it refers to a signal that is recorded over a certain muscle but is actually generated by a nearby muscle, following a conduction through the intervening volume to the recording electrodes (De Luca and Merletti, 1988). In the past, crosstalk signals have been regarded as low-frequency phenomena (due to both their far origin and the low-pass spatial filtering they undergo), so that high-pass filtering has been adopted as a remedy (De Luca, 1997). Nevertheless, more recent investigations have suggested the impossibility of generalizing such an approach, by describing the limitations that these and other types of eVects provide while attempting to use surface EMG to infer several parameters of muscular activations, such as the level of the activation itself, the type of motor unit recruited, the upper limit of motor unit recruitment, the average discharge rate and the degree of synchronization between motor units (Farina et al., 2004). As a second issue, given the complexity of the task and the variability of the EMG signals, EMG controlled systems usually require custom calibrations for each user and specific training for the pattern recognition algorithms. One of the major diYculties with EMG-based interfaces is represented by the needed of considerable mental eVorts during the training phase (Soares et al., 2003).
10
CARPI AND DE ROSSI
An equally important problem is the stochastic nature of EMG signals, resulting in parameter estimation errors that, in turn, cause classification and/ or control diYculties. Moreover, some control errors are generally introduced by the inability of the subject to reliably generate and reproduce the target contraction signals. Besides, current techniques make it typically very diYcult to control more than two degrees of freedom (DoF ) (Castellini and van der Smagt, 2009; Costanza et al., 2004; Zecca et al., 2002). Such a problem is further complicated by the time-variant characteristics of the EMG signal, due to changes in the electrical impedance of the skin, electrode locations, muscular fatigue, sweat, and so on. Such variations can cause significant problems, because of the usually adopted calibration process. In fact, most current EMG controlled devices are tuned only in the oZine phase; the user learns to reproduce one or two diVerent signals and the device is tuned to these signals. With such controllers it is not possible to successfully control more than 1 active DoF, because the diVerences between tuned signals and actual ones tend to increase gradually with time. For practical usage, the number of EMG channels is typically limited to 2, although the implementation of pattern recognition approaches can potentially lead to a much higher number of control commands (Fukuda et al., 2003).
C. EXAMPLES OF APPLICATIONS 1. EMG Controlled Cybernetic Hands EMG signals are today largely used as control inputs for myoelectrically based powered prostheses (Bitzer and van der Smagt, 2006; Castellini and van der Smagt, 2009; Zecca et al., 2002). In such systems, the human operator provides command signals in a feed-forward open-loop mode, utilizing only visual feedback as the primary source of information. So-called myoprocessors for controlling such prosthetic devices are today commercially available (see, for instance, examples mentioned in Castellini and van der Smagt, 2009). These systems are frequently based on one dimension only of the EMG signal (the variance or mean absolute value). Users are trained to produce a constant level of activation of muscles and the prostheses are tuned accordingly. A steady-state EMG signal, however, has very little temporal structure because of the active modification of recruitment and firing patterns needed to sustain a contraction (De Luca, 1988, 1997; Henneberg, 2000). The parameters that could be extracted to quantify its amplitude (e.g., variance or mean absolute value) or its frequency characteristics (e.g., Fourier spectrum or median frequency) are often not suYcient to distinguish between more than two classes of movement. As a result, despite the interesting results being achieved, it is typically
EMG-BASED AND GAZE-TRACKING-BASED INTERFACES
11
very diYcult to control reliably more than two DoF (Castellini and van der Smagt, 2009; Costanza et al., 2004; Zecca et al., 2002). Among the diVerent types of EMG controlled devices and systems studied at present, prosthetic hands (myoelectric hands) and, more generally, cybernetic hands are object of considerable study (Bitzer and van der Smagt, 2006; Castellini and van der Smagt, 2009; Ferguson and Dunlop, 2002; Zecca et al., 2002). We believe that such systems might also have a relevant impact for the space field. As an example, future space robots might benefit from articulated multifinger hands with bioelectric control. However, replicating the performance of the human hand is beyond current technical capabilities. In fact, the human hand is extremely complex: it has 22 DoF, controlled by about 38 muscles in the hand itself (almost twice the number of DoF ). Currently available commercial hand prostheses have a limited number of DoF (1 or 2 for finger movements and thumb opposition), thus typically showing low grasping functionality (Castellini and van der Smagt, 2009). They are controlled either in proportional or on/oV mode, by using a couple of primary EMG electrodes placed on two antagonist muscles. The use of a larger number of electrodes to control more DoF introduces several issues related to the movements coding and to the number of electrodes required that complicates the structure and the use of the socket. 2. EMG Controlled Exoskeletons Powered exoskeletons represent a type of wearable active orthotic systems that, in the opinion of the authors, could find significant applications in the space field. They are typically designed as an external structural mechanism that can be worn by the subject, so that one or more human joints correspond to those of the structure. A powered exoskeleton can be used as a system belonging to one of the following categories of: (1) power amplifier; (2) master device of a master/slave teleoperator system; (3) haptic device (Rosen et al., 2001; Tsuji et al., 2003). They can be synthetically described as follows. Within the use of an exoskeleton as a human power amplifier, the human provides control signals for the exoskeleton, while the actuators of the latter provide most of the power to perform the task. The application of an exoskeleton as a master device (in a master/slave system) enables the operator (master) to control a remote robotic arm (slave). Finally, the adoption of an exoskeleton as a haptic device is aimed at simulating human interactions with virtual objects (virtual reality); in such a case, a computer simulation acts as a virtual slave component of a master/slave system (Rosen et al., 2001). EMG signals can be adopted as useful command inputs for an exoskeleton (Di Cicco et al., 2004; Lucas et al., 2004; Mulas et al., 2005; Rosen et al., 2001). The EMG signal along with joint kinematics of the exoskeleton is fed into a myoprocessor that implements a muscle model, used to predict muscular variables relative to each involved joint (Rosen et al., 2001).
12
CARPI AND DE ROSSI
Nevertheless, as opposed to controlling a myoelectrically powered prosthesis, the operation of a myoelectrically powered exoskeleton is more challenging. In fact, such systems imply the existence of a kinematic link between the human and the exoskeleton; as a result, the human neural control system and exoskeleton control system coexist and have to cooperate by sharing the same kinematic and dynamic constraints. Moreover, when an exoskeleton is used, the muscular force cannot be estimated from the EMG signals; in fact, the angle of each human joint coupled with an exoskeleton joint constantly changes during operation, causing a modification of the length and end points velocities of the muscles attached to that joint. As a result, muscle models implemented by the myoprocessor have to take into account the muscle’s length and velocity, in addition to the EMG signal (which defines the muscle activation level), in order to predict the force that will be developed by the muscle (Rosen et al., 2001).
III. Gaze-Tracking-Based Interfaces
A. FUNDAMENTALS Gaze reflects our attention, intention, and desire. Gaze information plays an important role in identifying a person’s focus of attention; the information can provide useful communication cues to a multimodal interface. For example, it can be used to identify where a person is looking and what he is paying attention to. Thus, detection of gaze direction makes possible to extract such information that is valuable in human–computer interaction. Accordingly, computers integrated with gaze-tracking functions can provide an intuitive and eVective interactive system (Brunner et al., 2007; Ding et al., 2005; Hutchinson et al., 1989; Kaufman et al., 1993; Morimoto and Mimica, 2005). A person’s gaze direction is determined by two factors: the orientation of the head and the orientation of the eyes. While the first determines the overall direction of the gaze, the orientation of the eyes provides the exact gaze direction and is limited by the head orientation. The clear vision of an object is possible only when its image falls on the centre zone of an ocular portion, called fovea. Figure 6 shows the structure of the eye. To explore a scene, it is necessary that the eyes complete the movements that concur to carry and maintain the image stable on the fovea. The ocular movements of a subject, therefore, can tell us where he is watching and what and how long he is observing. Saccades are the faster ocular movements that the oculomotor apparatus can complete and they have the task to move the visual axis during the exploration of the scene. Fixation consists of a pause between two successive saccades and represents the time interval during which the visual information is
13
EMG-BASED AND GAZE-TRACKING-BASED INTERFACES
Cornea Iris Conjunctiva y ar
dy bo
Aqueous
humor
Ciliary muscle
li
Ci
Rectus tendon
Lens Zonal fibers Vitreous humor
Optic axis
Ora serrata
Visual axis
Papilla optic disk blind spot
Sclera
Retina
Choroid Lamina cribrosa Sheath
Fovea
Optic nerve
Macula lutea Center
Right
FIG. 6. Structure of the eye (from Malmivuo and Plonsey, 1995).
acquired. Therefore, visual exploration is made of a succession of saccades and fixations. Gaze determines the user’s current line of sight or point of fixation. The fixation point is defined as the intersection of the line of sight with the surface of the object being viewed. Therefore, gaze tracking may be used to interpret user’s intentions for interactions. Since the beginning of the 1990s, gaze-tracking technologies have progressively become more accurate, less cumbersome and today are available as commercial products. Several techniques for tracking the direction of eye-gaze are
14
CARPI AND DE ROSSI
FIG. 7. Examples of gaze-tracking systems: (A) head mounted device for corneal reflection; (B) EOG electrodes placed around the eyes; and (C) scleral contact lens equipped with an induction coil, to be arranged in contact with the eye (adapted from Matsumoto, 2003).
now available (Duchowski, 2003; Morimoto and Mimica, 2005). The most relevant for the purposes of this book are briefly described below. 1. Corneal Reflection This technique consists of a detection of the eye rotation by measuring the reflection of a light beam that is shone onto the cornea of the eye (Morimoto and Mimica, 2005; Reulen et al., 1988; Yoo et al., 2002). Typically, infrared (IR) light is used, in order to distract the user as little as possible and to avoid interference from other light sources (e.g., lamps). Two types of systems for corneal reflection can be used, depending on the arrangement of the IR video camera: (1) solidary with the subject’s head (Fig. 7A), being typically mounted on a helmet; (2) solidary with the scene to be explored (e.g., a computer screen) (Duchowski, 2003; Morimoto and Mimica, 2005). The higher suitability of one type over the other should be evaluated for each specific application. Corneal reflection techniques typically exhibit an accuracy of 1–2 (Morimoto and Mimica, 2005). 2. Electrooculography Electrooculography (EOG) is today a technique routinely used as a readily applicable diagnostic tool for studying the human oculomotor system (Malmivuo and Plonsey, 1995). It allows a detection of eye rotations, by measuring electric biopotentials from the skin surrounding the eyes, with simple surface electrodes (Fig. 7B) (Kaufman et al., 1993; Malmivuo and Plonsey, 1995; Morimoto and Mimica, 2005). This technique relies on the following bioelectric principle. The cornea of the eye is electrically positive relative to the back of the eye, with a resting potential of the order of 1 mV. This bioelectric source behaves as a corneoretinal dipole, whose orientation is varied by the eyeball movements. Accordingly, ocular rotations can be detected externally with skin electrodes that measure a potential diVerence, whose sign and amplitude depends on the
15
EMG-BASED AND GAZE-TRACKING-BASED INTERFACES
Corneoretinal dipole
Eyeball
−
+ −
EOG [ mV]
+
+ −
200
ϑ
ϑ = 30⬚
100 0
ϑ = 0⬚
ϑ = 0⬚ 1
−100
2 Time [s]
FIG. 8. Schematic drawing of the bioelectric genesis of EOG signals (adapted from Malmivuo and Plonsey, 1995).
rotation: the electrode in the direction of movement becomes positive with respect to the other electrode, as represented in Fig. 8 (Malmivuo and Plonsey, 1995). Two couples of electrodes arranged around the eye (external/internal couple and superior/inferior couple) can be used to detect both horizontal and vertical ocular movements, respectively. Signal amplitudes are typically of the order of 10 mV/degree and the achievable accuracy is 2 (Malmivuo and Plonsey, 1995; Morimoto and Mimica, 2005). Maximum detectable rotations are of approximately 70 , although the linearity of the response progressively worsens beyond 30 (Malmivuo and Plonsey, 1995). Two subdivisions of the EOG can be distinguished: saccadic response and nystagmography (Malmivuo and Plonsey, 1995). A saccadic response consists in a quick rotation of the eye from one fixation point to another, with an angular speed up to 700/s. These fast movements are aimed at rapidly moving the sight to a new visual object in a way that minimizes the transfer time. As an example, in order to follow a target that moves with stepwise jumps, typically the eyes undergo accelerations by reaching the maximum velocity about midway to the target. EOG allows detections of several parameters of saccadic movements, including the maximum angular speed (typically 400/s), the amplitude (typically 20 ), and the duration (typically 80 ms). As an important feature, it is worth stressing that the trajectory and the speed of saccades cannot be altered voluntarily, while they are influenced by fatigue, along with diseases, of course, but also drugs and alcohol (Malmivuo and Plonsey, 1995). Nystagmography is here mentioned for the unique sake of completeness of the discussion, since at present it does not find any use within the field of man–machine interfaces for controlling external devices. In fact, nystagmography has a purely clinical relevance and consist in an evaluation of the response of the visual control system to both vestibular and visual stimuli (Malmivuo and Plonsey, 1995).
16
CARPI AND DE ROSSI
Such a response is known as a nystagmoid movement (or nystagmus), which can be of two diVerent types (according to their origin): vestibular nystagmus and optokinetic nystagmus. The analysis of vestibular nystagmi allows investigations of the vestibular system; in fact, when the vestibular system is damaged, it sends erroneous signals to the oculomotor system and the subject can be aVected by dizziness. As a diVerence, an optokinetic nystagmus originates when multiple targets are in rapid motion with respect to the subject. To keep the image focused on the fovea, a saccadic reflex successively restores the eye to new target positions (Malmivuo and Plonsey, 1995). EOG is today one of the most used gaze-tracking techniques for noninvasive human computer interfaces (Brunner et al., 2007; Ding et al., 2005; Morimoto and Mimica, 2005). 3. Search coil in Scleral Contact Lens This technique consists of a detection of the eye rotation by exploiting electromagnetic induction in a search coil embedded into a flexible contact lens (Fig. 7C). In particular, the user’s gaze is detected by measuring the voltage induced in the search coil by an electromagnetic field generated externally; in fact, the direction and angular displacement of the eye change the polarity and amplitude of the induced voltage (Collewijn et al., 1975; Robinson, 1963). To detect eye movements in two dimension, that is, up/down and left/right rotations, a couple of external electromagnetic sources, arranged along orthogonal directions, can be used; in this case, two electromagnetic fields with diVerent frequency are generated, so as to distinguish (by frequency filtering) each induced voltage within the same search coil. Although very intrusive, search coil-based systems typically have a very high accuracy, about 0.08 (Morimoto and Mimica, 2005). Latest developments of this technology include the useful development of wireless devices (Roberts et al., 2008), so as to avoid the limitations typically introduced by the presence of the wire (Bergamin et al., 2004).
B. CHARACTERISTICS AND ISSUES 1. Corneal Reflection This technique for gaze tracking is highly accurate in suitable conditions (Matsumoto, 2003). Recent improvements have made the setup more accessible and adaptable for desktops and large projection screens. However, head movements are still a limitation for external systems, so that head mounted devices are certainly the best performing (Ciger et al., 2004). In the case of use of a measurement system not solidary with the head, during measurements the head should not move or its movements should be carefully
EMG-BASED AND GAZE-TRACKING-BASED INTERFACES
17
measured and compensated. Alternatively, head mounted devices are uncomfortable, cumbersome and prevent a natural behavior of the user. Another issue worth being reported of this gaze-tracking technique is the need of personalized accurate calibrations (Ciger et al., 2004; Matsumoto, 2003; Morimoto and Mimica, 2005). 2. Electrooculography The above reported description of EOG measurements points out two main features that characterize this technology: it allows recordings with minimal interference with the activities of the subject and it provides minimal discomfort for the subject. Furthermore, EOG recordings can be made in total darkness and/ or with closed eyes. In the opinion of the authors, such advantage might be significant for applications within a manned space module, for instance, since the recoding systems can easily be worn by astronauts and it is not optically aVected by the actual lighting conditions of the environment. Although both horizontal and vertical ocular movements can be recorded, no movements of torsion on the anteroposterior axis can be detected. In fact, these do not modify the dipole and they do not determine therefore potential diVerences them on the derivation electrodes (Patmore and Knapp, 1998). The most important issues relate to the fact that the value of the corneoretinal potential is not constant, since it can vary diurnally and can be aVected by light and muscular fatigue (Malmivuo and Plonsey, 1995). Consequently, there is a need for frequent calibration and recalibration. Additional diYculties may arise, owing to muscle artifacts and the nonlinearity of the technique (Malmivuo and Plonsey, 1995). 3. Scleral Contact Lens/Search Coil This technique is highly accurate, due to the close contact between the eye and the lens equipped with the measurement coil (Collewijn et al., 1975; Morimoto and Mimica, 2005). Despite that, this technique allows a limited measurement range (almost 5 ). Moreover, it is the most intrusive: its discomfort for the user requires the operator to adopt a particular care. Accordingly, in the opinion of the authors it is not realistically applicable for the applications of interest in this book, related to environments like those of spacecrafts.
C. EXAMPLES OF APPLICATIONS Gaze-tracking systems have several application domains, which include psychiatry, cognitive science, behavioral analysis and ophthalmology (Ciger et al., 2004; Conati and Merten, 2007; Herbelin et al., 2007; Malmivuo and Plonsey, 1995). Additionally, applications in the human–computer interaction field are certainly among the most studied at present (Duchowski, 2003; Morimoto and
18
CARPI AND DE ROSSI
Mimica, 2005). A couple of significant examples for the purposes of this book are reported below. 1. Task Selection Systems Computer interfaces represent one of the most studied fields of application of gaze tracking. Many interactive systems based on such a technique have been proposed (Hutchinson et al., 1989; Jacob, 1993). A typical application consists in task selection controlled by eye movements. For example, without using any vocal or manual commands, a user can control a pointing device on a screen and select icons or menus, by looking at them for a while (Norris and Wilson, 1997, Patmore and Knapp, 1998; Zhu and Ji, 2004). The potential benefits for incorporating eye movements into interactions between humans and computers are numerous. For example, reading a user’s gaze may help a computer not only to interpret a user’s request, but also to ascertain some cognitive states of the user (e.g., confusion or fatigue) (Ciger et al., 2004; Conati and Merten, 2007; Herbelin et al., 2007). Corneal reflection and EOG are of course the two technologies of primary choice for such applications. Nevertheless, at present the use of visual-evoked potentials (not discussed in this section) captured from electroencephalography is being investigated as well (Patmore and Knapp, 1998). 2. Attention Tracking Systems Real-time gaze monitoring can also be used to develop so-called attentive interfaces, intended as systems capable of tracking the user’s attention (Vertegaal, 2002; Zhai, 2003). For example, in this type of interfaces gaze can be employed to determine fixation points on a screen, so as to infer what information the user is interested in (Bojko, 2006). Appropriate actions could then be taken; for instance, they might enable the introduction of display changes, according to the spatial or temporal characteristics of eye movements. 3. Vigilance Monitoring Systems Another typical field of application of gaze-tracking systems consists in monitoring the vigilance of specific categories of subjects; for instance, a significant example concerns those at risk of falling suddenly asleep while working, with possibly fatal consequences (e.g., drivers and pilots). The main component of such types of systems consists of a computer vision setup, based on a remotely located device that monitors pupil movements ( Ji and Yang, 2002). Corneal reflection is the most used gaze-tracking techniques in this field, although it can also be advantageously integrated with additional information collected by parallel systems, detecting additional variables such as head position or eye lead movements ( Ji and Yang, 2002).
EMG-BASED AND GAZE-TRACKING-BASED INTERFACES
19
IV. Final Remark
This section described some examples of noninvasive man–machine interfaces, by emphasizing their fundamental features, performances, and applications. In the opinion of the authors, it could be worth studying in depth, for each of these interfaces, its possible use not only as the unique means of bioelectric control of a device, but also in combination with a BMI. In fact a BMI could require and may benefit from an auxiliary system to be used for specific tasks. For instance, this might be the case of astronauts engaged in multitask operations, requiring several activities to be accomplished in parallel. Some tasks may be performed by using signals detected by a BMI, while others, at the same time, may be accomplished by exploiting alternative means of communication enabled by diVerent types of noninvasive interfaces. The challenge is to properly combine these technologies, by making the overall system robust and intelligent. Fulfillment of such issues may open completely new approaches to manage space operations.
References
Ajiboye, A. B., and Weir, R. F. (2005). A heuristic fuzzy logic approach to EMG pattern recognition for multifunctional prosthesis control. IEEE Trans. Neural Syst. Rehabil. Eng. 13(3), 280–291. Bergamin, O., Ramat, S., Straumann, D., and Zee, D. S. (2004). Influence of orientation of exiting wire of search coil annulus on torsion after saccades. Invest. Ophthalmol. Vis. Sci. 45, 131–137. Bishop, C. M. (1995). ‘‘Neural Networks for Pattern Recognition.’’ Oxford University Press, Oxford. Bitzer, S., and van der Smagt, P. (2006). Learning EMG control of a robotic hand: Towards active prostheses. In Proceedings of ICRA, International Conference on Robotics and Automation, Orlando, Florida, USA, May 2006, pp. 2819–2823. Bojko, A. (2006). Using eye tracking to compare Web page designs: A case study. J. Usability Stud. 1(3), 112–120. Brunner, S., Hanke, S., Wassertheuer, S., and Hochgatterer, A. (2007). EOG pattern recognition trial for a human computer interface. In C. Stephanidis, Ed., Universal Access in HCI, Part II, HCII 2007, LNCS 4555, pp. 769–776. Springer, Berlin. Castellini, C., and van der Smagt, P. (2009). Surface EMG in advanced hand prosthetics. Biol. Cybern. 100(1), 35–47. Chan, F. H. Y., Yong-Sheng, Y ., Lam, F. K., Yuan-Ting, Z ., and Parker, P. A. (2000). Fuzzy EMG classification for prosthesis control. IEEE Trans. Rehabil. Eng. 8(3), 305–311. Ciger, J., Herbelin, B., and Thalmann, D. (2004). Evaluation of gaze tracking technology for social interaction in virtual environments. In Proceedings of the Second International Workshop on Modelling and Motion Capture Techniques for Virtual Environments (CAPTECH). Collewijn, H., Van der Mark, F., and Jansen, T. C. (1975). Precise recording of human eye movements. Vis. Res. 15, 447–450. Conati, C., and Merten, C. (2007). Eye-tracking for user modeling in exploratory learning environments: An empirical evaluation. Knowl. Based Syst. 20(6), 557–574.
20
CARPI AND DE ROSSI
Costanza, E., Perdomo, A., Inverso, S. A., and Allen, R. (2004). EMG as a subtle input interface for mobile computing. In Mobile HCI 2004: 6th International Symposium, Glasgow, UK, pp. 426–430. Crawford, B., Miller, K., Shenoy, P., and Rao, R. (2005). Real-time classification of electromyographic signals for robotic control. Technical Report No. 2005-03-05, Department of Computer Science, University of Washington, March 2005. De Luca, C. J. (1988). Electromyography. In ‘‘Encyclopedia of Medical Devices and Instrumentation’’ ( J. G. Webster, Ed.), Wiley, New York. De Luca, C. J. (1997). The use of surface electromyography in biomechanics. J. Appl. Biomechan. 13, 135–163. De Luca, C. J., and Merletti, R. (1988). Surface myoelectric signal cross-talk among muscles of the leg. Electroencephalogr. Clin. Neurophysiol. 69, 568–575. Di Cicco, M., Lucas, L., and Matsuokd, Y. (2004). Comparison of control strategies for an EMG controlled orthotic exoskeleton for the hand. In Proceedings of the 2004 IEEE International Conference on Robotics and Automation, New Orleans, LA, April 2004, pp. 1622–1627. Ding, Q., Tong, K., and Li, G. (2005). Development of an EOG electro-based human-computer interface. In Proceedings of the 2005 IEEE Engineering in Medicine and Biology 27th Annual Conference, Shanghai, China, September 1–4. Duchowski, A. T. (2003). ‘‘Eye Tracking Methodology: Theory and Practice.’’ Springer-Verlag, Berlin. Farina, D., Merletti, R., and Enoka, R. M. (2004). The extraction of neural strategies from the surface EMG. J. Appl. Physiol. 96, 1486–1495. Ferguson, S., and Dunlop, G. R. (2002). Grasp recognition from myoelectric signals. In Proceedings of the Australasian Conference on Robotics and Automation, Auckland, New Zealand. Fukuda, O., Tsuji, T., Kaneko, M., and Otsuka, A. (2003). A human-assisting manipulator teleoperated by EMG signals and arm motions. IEEE Trans. Robot. Autom. 19(2), 210–221. Guyton, A. C., and Hall, J. E. (2005). ‘‘Textbook of Medical Physiology,’’ 10th Edition W. B. Saunders Coompany, Philadelphia. Henneberg, K. (2000). Principles of electromyography. In ‘‘The Biomedical Engineering Handbook’’ ( J. D. Bronzino, Ed.), Second Edition. CRC Press LLC, Boca Raton. Herbelin, B., Grillon, H., De Heras Ciechomski, P., and Thalmann, D. (2007). Coding gaze tracking data with chromatic gradients for VR Exposure Therapy. In Proceedings of the 17th IEEE International Conference on Artificial Reality and Telexistence, pp. 7–14. Hutchinson, T., White, K. Jr., Reichert, K., and Frey, L. (1989). Human–computer interaction using eye-gaze input. IEEE Trans. Syst. Man Cybern. 19, 1527–1533. Jacob, R. (1993). What you look at is what you get. IEEE Comput. 26(7), 65–66. Ji, Q., and Yang, X. (2002). Real-time eye, gaze, and face pose tracking for monitoring driver vigilance. Real Time Imaging Arch. 8(5), 357–377. Kaufman, A., Bandopadhay, A., and Shaviv, B. (1993). An eye tracking computer user interface. In ‘‘Proceedings of the Research Frontier in Virtual Reality Workshop,’’, pp. 78–84. IEEE Computer Society Press, Los Alamitos, CA. Lucas, L., Di Cicco, M., and Matsuoka, Y. (2004). An EMG-controlled hand exoskeleton for natural pinching. J. Robot. Mechatr. 16, 482–488. Malmivuo, J., and Plonsey, R. (1995). ‘‘Bioelectromagnetism – Principles and Applications of Bioelectric and Biomagnetic Fields.’’ Oxford University Press, Oxford. Matsumoto, Y. (2003). ‘‘Real-Time Computer Vision for Human Interfaces.’’ EURON Summer School, Hamburg, Germany. Available online at http://tams-www.informatik.uni-hamburg.de/ lehre/2003ss/euron_summer_school/Matsumoto.09.07.pdf Website visited on October 15, 2008. Micera, S., Sabatini, A. M., and Dario, P. (1998). An algorithm for detecting the onset of muscle contraction by EMG signal processing. Med. Eng. Phys. 20(3), 211–215.
EMG-BASED AND GAZE-TRACKING-BASED INTERFACES
21
Micera, S., Vannozzi, G., Sabatini, A. M., and Dario, P. (2001). Improving detection of muscle activation intervals. IEEE Eng. Med. Biol. Mag. 20(6), 38–46. Morimoto, C. H., and Mimica, M. R. M. (2005). Eye gaze tracking techniques for interactive applications. Comput. Vis. Image Underst. 98(1), 4–24. Mulas, M., Folgheraiter, M., and Gini, G. (2005). An EMG-controlled exoskeleton for hand rehabilitation. In Proceedings of the 9th IEEE International Conference on Rehabilitation Robotics, June 28–July 1, 2005, Chicago, IL, USA, pp. 371–374. Navarro, X., Krueger, T. B., Lago, N., Micera, S., Stieglitz, T., and Dario, P. (2005). A critical review of interfaces with the peripheral nervous system for the control of neuroprostheses and hybrid bionic systems. J. Peripher. Nerv. Syst. 10(3), 229–258. Norris, G., and Wilson, E. (1997). The eye mouse, an eye communication device. In Proceedings of the IEEE 1997 Bioengineering Conference, 21–22 May 1997, pp. 66–67. Patmore, D. W., and Knapp, R. B. (1998). Towards an EOG-based eye tracker for computer control. In Proceedings of the Third International ACM Conference on Assistive Technologies, Marina del Rey, California, pp. 197–203. ACM Press, New York, NY. Reulen, J., Marcus, J. T., Koops, D., de Vries, F., Tiesinga, G., Boshuizen, K., and Bos, J. (1988). Precise recording of eye movement: The iris technique, part 1. Med. Biol. Eng. Comput. 26(1), 20–26. Roberts, D., Shelhamer, M., and Wong, A. (2008). A new ‘‘wireless’’ search-coil system. In Proceedings of the Eye Tracking Research & Application Symposium, ETRA 2008, Savannah, Georgia, USA, March 26–28, pp. 197–204. Robinson, D. A. (1963). A method of measuring eye movements using a scleral search coil in a magnetic field. IEEE Trans. Biomed. Eng. 10, 137–145. Rosen, J., Brand, M., Fuchs, M. B., and Arcan, M. (2001). A myosignal-based powered exoskeleton system. IEEE Trans. Syst. Man Cybern. A 31(3), 210–222. Soares, A., Andrade, A., Lamounier, E., and Carrijo, R. (2003). The development of a virtual myoelectric prosthesis controlled by an EMG pattern recognition system based on neural networks. J. Intell. Inf. Syst. 21(2), 127–141. Tsuji, O. F. T., Kaneko, M., and Otsuka, A. (2003). A human-assisting manipulator teleoperated by EMG signals and arm motions. IEEE Trans. Robot. Autom. 19(2), 210–222. Vertegaal, R. (2002). Designing attentive interfaces. In Proceedings of the Eye Tracking Research & Applications Symposium, New Orleans, LA, pp. 23–30. Webster, J. G. (1997). ‘‘Medical instrumentation: Application and Design,’’ Third Edition Wiley, New York. Wolpaw, J. R., Birbaumer, N., McFarland, D. J., Pfurtscheller, G., and Vaughan, T. M. (2002). Brain– computer interfaces for communication and control. Clin. Neurophysiol. 113(6), 767–791. Yoo, D., Kim, J., Lee, B., and Chung, M. (2002). Non contact eye gaze tracking system by mapping of corneal reflections. In Proceedings of the International Conference on Automatic Face and Gesture Recognition, Washington, DC, pp. 94–99. Zecca, M., Micera, S., Carrozza, M. C., and Dario, P. (2002). Control of multifunctional prosthetic hands by processing the electromyographic signal. Crit. Rev. Biomed. Eng. 30(4–6), 459–485. Zhai, S. (2003). What’s in the eyes for attentive input. Commun. ACM 46(3), 34–39. Zhu, Z., and Ji, Q. (2004). Eye and gaze tracking for interactive graphic display. Mach. Vis. Appl. 15(3), 139–148.
BIDIRECTIONAL INTERFACES WITH THE PERIPHERAL NERVOUS SYSTEM
Silvestro Micera*,y and Xavier Navarroz,} *ARTS and CRIM Labs, Scuola Superiore Sant’Anna, I-56127 Pisa, Italy Institute for Automation, Swiss Federal Institute of Technology, CH-8092 Zurich, Switzerland z Institute of Neurosciences, Universitat Auto`noma de Barcelona, E-08193 Bellaterra, Spain } Centro de Investigacio´n Biome´dica en Red sobre Enfermedades Neurodegenerativas (CIBERNED), Spain
y
I. Introduction II. Organization and Function of the PNS III. Nerve Electrodes: Types and Applications A. Extraneural Electrodes B. Intraneural Electrodes C. Regenerative Electrodes IV. Stimulation and Recording Neural Signals A. Stimulation of the PNS B. Recording and Processing Neural Signals V. Biomedical Applications A. Neuroprostheses for CNS Injured Patients B. Cybernetic Prostheses References
Considerable scientific and technological eVorts have been devoted to develop neuroprostheses and hybrid bionic systems that link the human nervous system with electronic or robotic prostheses, with the main aim of restoring motor and sensory functions in disabled patients. Such developments have also the potential to be applied to normal human beings to improve their physical capabilities for bidirectional control and feedback of machines. A number of neuroprostheses use interfaces with peripheral nerves or muscles for neuromuscular stimulation and signal recording. This chapter provides a general overview of the peripheral neural interfaces available and their use from research to clinical application in controlling artificial and robotic prostheses and in developing neuroprostheses. Extraneural electrodes, such as cuV and epineurial electrodes, provide simultaneous interface with many axons in the nerve, whereas intrafascicular, penetrating, and regenerative electrodes may selectively contact small groups of axons within a nerve fascicle. Biological and technical issues are reviewed relative to the problems of electrode design and tissue
INTERNATIONAL REVIEW OF NEUROBIOLOGY, VOL. 86 DOI: 10.1016/S0074-7742(09)86002-9
23
Copyright 2009, Elsevier Inc. All rights reserved. 0074-7742/09 $35.00
24
MICERA AND NAVARRO
injury. The last sections review diVerent strategies for the use of peripheral neural interfaces in biomedical applications.
I. Introduction
The possibility of interfacing and controlling artificial prostheses and machines with biological signals has a long history of multidisciplinary research. Many scientific and technological eVorts have been devoted to develop bionic systems that link, via neural interfaces, the human nervous system with electronic and robotic prostheses, with the main aim of restoring motor and sensory functions in patients with spinal cord injuries, brain injuries, neurodegenerative diseases, or limb amputations (Chapin and Moxon, 2000). A number of such neuroprostheses include interfacing the peripheral nervous system (PNS) by means of electrodes that allow neuromuscular stimulation and neural signal recording. Several architectures have been developed and tested: (1) functional electrical stimulation (FES) systems to artificially replace central motor control and directly stimulate the intact peripheral nerves or muscles of patients with central nervous system (CNS) injuries; (2) artificial prostheses aimed at substituting lost parts of the body; (3) exoskeletons intended to augment or restore reduced human capabilities; (4) teleoperated robots to carry out tasks in environments where it is not possible the direct intervention of human beings. The combination of the artificial system with the human–machine interface (HMI) is often called ‘‘hybrid bionic system’’ (HBS). It is characterized by three main attributes (Fig. 1, Micera et al., 2006): (i) level of Hybridness: the system to be controlled can be a prosthesis, an exoskeleton, a personal computer or a scalable robotic alias; (ii) level of Augmentation: the motor and/or sensory channels of the user that can be involved in the development of the HBS; (iii) level of Connection: multimodal devices or (invasive or noninvasive) interfaces with diVerent parts of the nervous system can be used. Sophisticated prostheses and robotic devices ask for HMIs able to take full advantage of their potentials. Therefore, a fast, intuitive, reliable, and bidirectional flow of information between the nervous system of the user and the robotic device needs to be established. In recent years, various types of HBSs have been developed for such purpose (Dhillon et al., 2004; Hochberg et al., 2006; Navarro et al., 2005; Rutten, 2002; Velliste et al., 2008; Warwick et al., 2003). Among the possible choices, interfaces with the PNS are interesting because they represent a trade-oV between a potentially good ability to restore a natural link with the nervous system and a reduced invasiveness. For example, in the case of a cybernetic prosthesis, the interface should be able to stimulate diVerent aVerent
25
PERIPHERAL NERVE INTERFACES
Hybridness Prosthesis
Exoskeleton
Robotic alias
Connectivity Indirect
Direct to PNS
Direct to CNS
Sensory or motor Sensory and motor Sensory-motor and cognitive
Augmentation FIG. 1. Examples of diVerent systems with diVerent levels of Hybridness, Augmentation, and Connection.
nerves to deliver sensory feedback information originating from sensors in the prosthesis, and to record signals from eVerent nerves or from muscles to derive motor commands to the prosthesis. Similarly, kinematic and kinetic information for the closed-loop control of a neuroprosthesis could be detected from signals originating from natural sensors intercepted by the neural interface. Starting from these needs, several neural interfaces have been developed with diVerent characteristics (Fig. 2). The aim of this chapter is to describe the potentials and limits of the use of PNS neural interfaces to develop advanced HBSs.
II. Organization and Function of the PNS
The PNS is constituted by neurons whose cell bodies are located in the spinal cord or within spinal ganglia, their central connections, and their axons, which extend through peripheral nerves to reach target organs. Peripheral nerves contain several types of nerve fibers (Table I). AVerent sensory fibers terminate
26
MICERA AND NAVARRO
Regenerative
PNS electrodes
Intraneural
Invasivity
d
Intrafascicular
c
b Extraneural
a Selectivity FIG. 2. The diVerent types of electrodes applied to interface peripheral nerves classified regarding invasiveness and selectivity. Micrographs show examples of hybrid silicone–polyimide cuV electrode (FraunhoVer IBMT) polyimide tfLIFE (FraunhoVer IBMT), silicon-based Utah MEA (Cyberkinetics), and polyimide sieve electrode (FraunhoVer IBMT).
TABLE I GENERAL CLASSIFICATION OF THE PERIPHERAL NERVE FIBERS
Fiber type Myelinated A A A A B Unmyelinated sC dC
Function
Diameter (m)
Conduction velocity (m/s)
Alpha-motor eVerents, Proprioceptive aVerents Tactile, proprioceptive aVerents Gamma-motor eVerents Pain, cold aVerents Preganglionic autonomic eVerents
12–22 6–12 3–5 2–5 1–5
60–120 40–70 30–45 10–30 3–15
Postganglionic autonomic eVerents Thermal, pain, mechanical aVerents
0.3–1.3 0.3–1.3
0.7–2.3 0.5–2.0
at the periphery either as free endings or in specialized sensory receptors in the skin, the muscle, and deep tissues. Sensory fibers convey various classes of sensory inputs, mainly mechanical, thermal, and noxious stimuli. EVerent motor fibers
PERIPHERAL NERVE INTERFACES
27
originate from motoneurons in the spinal cord and end in neuromuscular junctions in skeletal muscles. The majority can be divided into alpha-motor fibers that innervate skeletal extrafusal muscle fibers, and gamma-motor fibers that innervate the spindle muscle fibers. EVerent autonomic nerve fibers in somatic peripheral nerves are unmyelinated postganglionic sympathetic fibers that innervate smooth muscle and glandular targets. Most of the somatic peripheral nerves are mixed, providing motor, sensory, and autonomic innervation to the corresponding projection territory. Nerve fibers, both aVerent and eVerent, are grouped in fascicles that eventually give origin to branches that innervate distinct targets, muscular, cutaneous, or visceral. Peripheral nerves are organized somatotopically and functionally at the fascicular level. The fascicular architecture changes throughout the length of the nerve, with an increasing number of fascicles of smaller size in distal with respect to proximal segments. Fascicles innervating a given target remain well localized within the nerve for some long distances (Brushart, 1991), thus facilitating the selective interface of diVerent fascicles within a common nerve (Branner et al., 2001, Veraart et al., 1993). Peripheral nerves are composed of three supportive sheaths: epineurium, perineurium, and endoneurium (Peters et al., 1991) (Fig. 3). The epineurium is the outermost layer, composed of loose connective tissue and carries the blood vessels supplying the nerve. The perineurium that surrounds each fascicle in the nerve is composed of inner layers of flat perineurial cells and an outer layer of collagen fibers. The endoneurium is composed of collagen and reticular fibers and an extracellular matrix occupying the space between nerve fibers within the fascicle. The endoneurial collagen fibrils form the walls of the endoneurial tubules, in which axons are accompanied by Schwann cells, which either myelinate or just surround them. The actions of the body are controlled by neural signals conducted by eVerent nerve fibers to activate diVerent muscles. Each spinal motoneuron makes synaptic contacts with a number of muscle fibers, constituting a motor unit. Graded contraction of each muscle is produced by increasing the number of motor units activated, and by increasing the frequency of impulses to each motor unit. Recruitment of motor units follows a size-dependent order, with slow fatigueresistant motor units activated first and large fast fatigue motor units activated only at high levels of tension (Henneman et al., 1974). On the other hand, the information transduced by the natural receptors is conducted to the CNS by the aVerent nerve fibers. Each somatic sensory neuron is specified to a sensory modality, touch, proprioception, temperature, or pain, depending upon the specialized terminal receptor. Each sensory neurons subsides a receptive field in the peripheral tissue, of variable size according to the body segment. Signals are transmitted by the corresponding axons in series of action potentials, with intensity of the signal mainly coded by impulse frequency.
28
MICERA AND NAVARRO
A
B
C
FIG. 3. Structure of the peripheral nerve. (A) Transverse section of the rat sciatic nerve showing subfascicles of nerve fibers. The endoneurial compartment is encircled by the perineurium and an outer loose epineurium. (B) Semithin cross section showing at high magnification myelinated fibers of diVerent sizes. (C) Electron microscopy section showing a small myelinated fiber surrounded by collagen fibrils and flat fibroblastic cells. Bars ¼ 100 m in (A); 20 m in (B); 2 m in (C).
III. Nerve Electrodes: Types and Applications
Most peripheral nerve interfaces use an electrical coupling method to detect the bioelectrical activity of the nerve fibers and/or to induce their excitation. Thus, most nerve electrodes are implanted around or within a peripheral nerve or
PERIPHERAL NERVE INTERFACES
29
spinal root to reduce tissue resistance and stimulus intensity. Nerve electrodes can be classified into three main classes: extraneural, intraneural, and regenerative. The selectivity of stimulation or recording individual nerve fibers increases with the invasiveness of the electrode implantation (Fig. 2).
A. EXTRANEURAL ELECTRODES CuV electrodes are composed of an insulating tubular sheath that encircles the nerve and contains electrode contacts exposed at the inner surface that are connected to lead wires (Fig. 2a). CuV electrodes placed around the nerves allow for more precise positioning and reduced stimulus intensity compared to surface and epimysial electrodes (Loeb and Peck, 1996). In comparison with more invasive, penetrating and regenerative electrodes, cuV electrodes of spiral type are less prone to damage the nerve and easier to implant (Naples et al., 1990). However, because they are placed around the nerve cuV electrodes have a reduced selectivity; they can record only mixed sensorimotor activity of the enclosed fibers whenever a mixed nerve is utilized. The stimulation of as well as the activity recorded from a nerve with cuV electrodes is dominated by the activation of large myelinated fibers and of those located at superficial locations. Multisite cuV electrodes (Tarler and Mortimer, 2004; Veraart et al., 1993; Walter et al., 1997), innovative cuV structures (Tyler and Durand, 1997), and advanced processing algorithms (Cavallaro et al., 2003; Tesfayesus and Durand, 2006) have increased cuVs selectivity. The flat interface nerve electrode (FINE) is an extraneural electrode designed to reshape peripheral nerves into a favorable geometry for selective stimulation (Leventhal and Durand, 2003; Tyler and Durand, 2002). By flattening the nerve, fascicles become more accessible and central fibers are moved closer to the electrode contacts in comparison with cylindrical cuVs. The surface area of the nerve is also enlarged, increasing the interface surface and allowing more contacts placed around the nerve. Studies in laboratory animals with FINEs implanted over months showed that electrodes applying small forces did not cause detectable changes in nerve physiology and histology. However, high reshaping forces can induce nerve damage (Tyler and Durand, 2003).
B. INTRANEURAL ELECTRODES Longitudinal intrafascicular electrodes (LIFEs), constructed from thin insulated conducting wires (such as Pt–Ir or metalized polymers), are inserted longitudinally into the nerve to lay in-between and parallel to the nerve fibers (Lawrence et al., 2004; Yoshida and Horch, 1993; Yoshida and Stein, 1999). Gathering signals
30
MICERA AND NAVARRO
from only a small number of axons, they allow more selective interfacing than extraneural electrodes that wrap the whole nerve. LIFE electrodes are less invasive than intraneural multielectrode arrays (MEAs) that are inserted transversely into the nerve leading to a higher risk of damage. Recently, thin-film LIFEs (tfLIFE) have been fabricated on micropatterned polyimide substrate, allowing several contacts in one device and selective multiunit nerve recording and stimulation (Navarro et al., 2007). MEAs are composed of tens of needles inserted transversally into the nervous system and developed on materials such as silicon, glass, or polyimide (Kipke et al., 2003; Nordhausen et al., 1996; Rutten et al., 1999). They have been mainly used as CNS microelectrodes to record or stimulate the brain cortex (Hochberg et al., 2006; Velliste et al., 2008). MEAs have been also tested in peripheral nerves in experimental works with animal models and also in a human volunteer (Warwick et al., 2003). Recently, a modified version (named Utah Slanted Electrode Array, USEA) has been used within the PNS, allowing selective recording of single unit responses and low-current highly selective stimulation of motor fibers (Branner et al., 2001, 2004). MEAs present the advantage of a high number of electrical contacts. However, they have also some limitations, such as the rigid structure of the electrodes and the tethering forces by lead wires that may damage the nerves during movements, and the lack of stability of recorded signals over time during chronic studies (Branner et al., 2004; Warwick et al., 2003).
C. REGENERATIVE ELECTRODES Regeneration-type (or sieve) electrodes are designed to interface a high number of nerve fibers by using an array of holes with electrodes around them (Fig. 2d), implanted between the severed stumps of a nerve. Regenerating axons grow through the holes, making it possible to record action potentials from and to stimulate individual axons or small groups. The most logical and challenging application of regenerative electrodes is the implantation in severed nerves of an amputee limb for a bidirectional interface with a prosthesis. They may allow interfacing with the injured axons that originally innervated the lost limb. However, regenerative electrodes can only be applied to transected peripheral nerves and need time for regenerating axons to grow through the interface, thus precluding acute experiments. Although promising results on the use of regenerative electrodes have been achieved in experimental models (Bradley et al., 1997; Kovacs et al., 1994; Navarro et al., 1998), some challenges remain limiting their clinical usability (Lago et al., 2005). Axonal regeneration through polyimide sieve electrodes occurs in most of the animals implanted, with higher quality and quantity than reported for the earlier used silicon-based sieve electrodes, but still in some animals there appeared signs of compressive axonopathy at the
PERIPHERAL NERVE INTERFACES
31
sieve electrode level on the long term (Lago et al., 2005). Nevertheless, stimulation of small numbers of regenerated fibers is feasible using regenerative electrodes, and by matching regenerated axons with original receptive fields an adequate sensory feedback might be delivered (Lago et al., 2007).
IV. Stimulation and Recording Neural Signals
PNS electrodes can be used for stimulation of nerve fibers as well as for recording of neural impulses, constituting a bidirectional interface with the nervous system.
A. STIMULATION OF THE PNS From a functional point of view, cuV electrodes can be used to stimulate the enclosed nerve leading to activation of eVerent motor or autonomic nerve fibers. Simple configurations are bipolar and tripolar, which reduce current leaks out of the cuV. Nerve stimulation produces larger movements and more reproducible than intramuscular or intraspinal stimulation (Aoyagi et al., 2004). In addition, the stimulus current required to activate nerve axons by stimulation with epineurial or cuV electrodes is much lower than that required for intramuscular stimulation. Multichannel cuV electrodes enable selective stimulation of separate axonal fascicles within the nerve, each one supplying innervation to a diVerent muscle (Navarro et al., 2001; Tarler and Mortimer, 2004; Veraart et al., 1993). The reduced size and thickness of polymer cuVs opens the possibility of implanting several small cuVs around diVerent branches of nerves, thus achieving selective functional stimulation of a higher number of muscles (Stieglitz et al., 2000). Short pulse widths and subthreshold transverse currents from a steering anode in the cuV can significantly improve the selectivity of stimulation by restricting the region of excitation of the nerve trunk. Large myelinated fibers are activated before small myelinated and unmyelinated fibers when applying electrical stimulation to the nerve. This is advantageous for stimulation of aVerent fibers to provide sensory feedback, since tactile or position sensations can be elicited without concomitantly evoking pain. On the contrary, large motor fibers innervating fast fatigue motor units are activated earlier than thinner motor fibers controlling slow fatigue-resistant motor units, resulting into an inverse recruitment that causes fast muscle fatigue. Strategies for achieving a more physiological recruitment order include the application of anodal blocking and of quasi-trapezoidal pulses (Fang and Mortimer, 1991).
32
MICERA AND NAVARRO
FINE electrodes have been also used to stimulate the PNS in acute experiments showing that it is possible to selectively activate individual fascicles of the cat sciatic nerve, as well as groups of fibers within the fascicles, and revealed the strong dependency of selectivity on the relative locations of the fascicle and the electrode contacts (Leventhal and Durand, 2003; Tyler and Durand, 2002). Intrafascicular metal and polymer-based electrodes have been shown to be valuable for functional neuromuscular stimulation with high selective characteristics (Yoshida et al., 2000). LIFEs were able to produce equivalent levels of activation using stimulation levels that were an order of magnitude smaller than with hook electrodes. LIFEs can be used to activate nerve fibers in diVerent fascicles independently of each other, and also to activate subsets of axonal populations within a single fascicle (Yoshida and Horch, 1993). Microtechnologies allowing to place multiple contacts in one electrode device for stimulation or recording of individual nerve fibers can considerably increase the number of channels and resolution whereas minimizing the invasiveness and the number of electrodes to implant in a nerve. The nerve stimulation capabilities of multisite USEA (Branner et al., 2001) and of tfLIFE (Navarro et al., 2007) have been assayed in animal nerves.
B. RECORDING AND PROCESSING NEURAL SIGNALS The possibility of extracting potential actions from neural signal recordings is an important issue for the development of HBSs. The dynamic time-variant properties of the musculoskeletal system makes desirable to develop closed-loop FES systems. Feedback information can be gathered by using external (Carpaneto et al., 2003) or implantable (Cavallaro et al., 2005; Johnson et al., 1999) artificial sensors, or by processing electroneurographic (ENG) signals recorded by means of electrodes in the PNS, and used to correct deviations caused by unexpected changes and nonlinearities. The processing of neural signals is related to the type of electrodes used. Because of the insulating properties of perineurial and epineurial layers, electrodes placed inside a peripheral nerve allow enhanced selectivity with respect to extraneural electrodes and increase the signal-to-noise ratio (SNR) of recordings. With extraneural electrodes (e.g., cuV signals), the contribution of single axons cannot be extracted because of the low SNR and the overlapping between the frequency range of the signals (few hundred Hz to a few kHz) and the noise. In most cases, the use of recorded neural activity is limited to the onset detection for the closed-loop control of FES systems (Haugland et al., 1994; Inmann and Haugland, 2004) and for the control of hand prostheses (Stein et al., 1980). Nevertheless, pattern recognition techniques allow identifying complex motor commands from the compound recorded signal, and kinematic information has
PERIPHERAL NERVE INTERFACES
33
been extracted from ENG signals recorded from aVerent activities (Cavallaro et al., 2003; Micera et al., 2001). With intraneural electrodes (e.g., LIFE and MEA), it is possible to record spikes from diVerent nerve fibers. Such signals have been used (Aoyagi et al., 2003; McNaughton and Horch, 1994; Mirfakhraei and Horch, 1997) to diVerentiate single units in multiunit peripheral nerve recordings using diVerent features and classification schemes. Moreover, as individual spikes can be resolved in aVerent activity from natural sensors, the interspike interval can provide meaningful feedback to FES systems (Yoshida and Horch, 1996). The use of discrete and continue wavelet transform was recently applied to ENG signals recorded using LIFEs demonstrating that LIFEs were able to record spikes and to sort diVerent classes of spikes in a robust way for several weeks, and that it was possible to use the extracted spikes to identify diVerent neural stimuli (Citi et al., 2008).
V. Biomedical Applications
Although most interfaces were originally developed for FES systems, they can also be the key component of neurocontrolled prostheses and robotic machines (Table II).
A. NEUROPROSTHESES FOR CNS INJURED PATIENTS Nowadays, the most frequent use of PNS interfaces resides in FES. FES systems have been developed in order to artificially replace the central motor control and directly stimulate the intact peripheral nerves or muscles of spinal cord, or brain injured patients, attempting to generate movements or functions that mimic normal actions. There are clinical applications in a variety of systems designed to control micturition and defecation by stimulating the sacral roots (Brindley, 1994; RijkhoV, 2004), for phrenic nerve pacing in ventilatory assistance (Chervin and Guilleminault, 1997; Creasey et al., 1996), for treating neuropathic pain by stimulating somatic nerves (Stanton-Hicks and Salamon, 1997), for activation of lower extremity movements (Burridge et al., 2007; Triolo et al., 1996; Waters et al., 1985), and for control of hand movements (Kilgore, 1997; Peckham and Keith, 1992) by stimulating paralyzed muscles or nerves. DiVerent studies have demonstrated that patients are able to use such systems for activities of daily living and enhancing their quality of life and independency. The introduction of closed-loop control by recorded sensory neural activity related to the stimulated actions is attempting to improve the usability and functionality of FES systems (Inmann and Haugland, 2004; Sinkjaer et al., 2003).
34
MICERA AND NAVARRO
TABLE II MAIN TYPES OF ELECTRODES USED FOR INTERFACING THE PNS IN BIOMEDICAL APPLICATIONS Electrode Type
Mode
No.
Contact site
Application
Epineurial Epineurial Epineurial Epineurial Helical Book CuV CuV CuV FINE LIFE MEA Regenerative
stim stim stim stim stim stim stim rec stim rec/stim rec/stim rec/stim rec/stim
4 2 2 16–25 1 3 1 1 4 13 1–4 20–100 10–30
Phrenic nerve Peroneal nerve Somatic nerves Retina/ganglion Cells Vagus nerve Sacral spinal roots Peroneal nerve Sural nerve Optic nerve Peripheral nerve Peripheral nerve Nerve, brain cortex Injured nerve
Breathing Foot drop Pain relieve Blindness Epilepsy, sleep apnea Bladder control FES FES control Blindness Artificial limb control Motion control
Mode: rec, recording; stim, stimulation. FINE, flat interface nerve electrode; LIFE, longitudinal intrafascicular electrode; MEA, multielectrode array.
B. CYBERNETIC PROSTHESES The restoration of sensorimotor functions to those who lost limbs due to disease, trauma, or amputation is also an active field of research. Commercial prosthetic limb devices are unable to provide enough functionality and to deliver appropriate sensory feedback to the user so as to functionally replace the lost limb. In recent years, Kuiken and colleagues have developed a novel interface called targeted reinnervation (Kuiken et al., 2007; Miller et al., 2008). The amputated nerves that originally provided innervation to the missing limb are surgically transferred to innervate other arm and chest muscles that remain after the amputation. Once reinnervated, these muscles produce electromyographic (EMG) signals that now correspond to the original arm motor orders and can be used to control several degrees of freedom of the prosthesis. Concurrently, sensory nerves forced to reinnervate the skin overlying the target muscles may provide a pathway for sensory information of the amputated arm. This approach, however, is still dependent on surface electrodes and limited to a few EMG signals. Future perspectives rely on directly interfacing the amputated nerves by multipolar electrodes that may re-create the bidirectional link between the user’s nervous system and the prosthesis in a more physiologically based manner.
PERIPHERAL NERVE INTERFACES
35
In the recent past LIFE electrodes have been used to this aim in selected chronic amputees (Dhillon et al., 2004, 2005). LIFEs were implanted into median or ulnar nerves proximal to the stump. The results indicated that the motor signals recorded using LIFEs could be used to control prosthetic systems. During these trials the possibility of stimulating aVerent nerves to provide sensory feedback to the subjects was also investigated. Subjects were able to feel tactile or proprioceptive sensations localized to individual phantom digits elicited through diVerent electrodes (Dhillon et al., 2005). In some cases plastic stimulus-induced reorganization on somatosensory cortex by the aVerent stimulation made the localization of elicited sensations better defined with time. Interesting results were also achieved with a MEA implanted in the median nerve of an able-bodied subject (Warwick et al., 2003). An eVective bidirectional link between sensorized dexterous hand prosthesis and the nervous system could be achieved. In particular, the ENG-based control was more natural than the standard EMG-based approach. The sensory feedback allowed the subject to minimize the required grasping force after training. However, after the 96-day trial only three channels (over 100) were still working, due to mechanical fatigue of the connection wires.
References
Aoyagi, Y., Stein, R. B., Branner, A., Pearson, K. G., and Normann, R. A. (2003). Capabilities of a penetrating microelectrode array for recording single units in dorsal root ganglia of the cat. J. Neurosci. Methods 128, 9–20. Aoyagi, Y., Mushahwar, V. K., Stein, R. B., and Prochazka, A. (2004). Movements elicited by electrical stimulation of muscles, nerves, intermediate spinal cord, and spinal roots in anesthetized and decerebrate cats. IEEE Trans. Neural Syst. Rehabil. Eng. 12, 1–11. Bradley, R. M., Cao, X., Akin, T., and Najafi, K. (1997). Long term chronic recordings from peripheral sensory fibers using a sieve electrode array. J. Neurosci. Methods 73, 177–186. Branner, A., Stein, R. B., and Normann, R. A. (2001). Selective stimulation of cat sciatic nerve using an array of varying-length microelectrodes. J. Neurophysiol. 85, 1585–1594. Branner, A., Stein, R. B., Ferna´ndez, E., Aoyagi, Y., and Normann, R. A. (2004). Long-term stimulation and recording with a penetrating microelectrode array in cat sciatic nerve. IEEE Trans. Biomed. Eng. 51, 146–157. Brindley, G. S. (1994). The first 500 patients with sacral anterior root stimulator implants: General description. Paraplegia 32, 795–805. Brushart, T. M. (1991). Central course of digital axons within the median nerve of Macaca mulatta. J. Comp. Neurol. 311, 197–209. Burridge, J. H., Haugland, M., Larsen, B., Pickering, R. M., Svaneborg, N., Iversen, H. K., Christensen, P. B., Haase, J., Brennum, J., and Sinkjaer, T. (2007). Phase II trial to evaluate the ActiGait implanted drop-foot stimulator in established hemiplegia. J. Rehabil. Med. 39, 212–218. Carpaneto, J., Micera, S., Zaccone, F., Vecchi, F., and Dario, P. (2003). A sensorized thumb for force closed-loop control of hand neuroprostheses. IEEE Trans. Neural Syst. Rehabil. Eng. 11, 346–353.
36
MICERA AND NAVARRO
Cavallaro, E., Micera, S., Dario, P., Jensen, W., and Sinkjaer, T. (2003). On the intersubject generalization ability in extracting kinematic information from aVerent nervous signals. IEEE Trans. Biomed. Eng. 50, 1063–1073. Cavallaro, E., Cappiello, G., Micera, S., Carrozza, M. C., Rantanen, P., and Dario, P. (2005). On the development of a biomechatronic system to record tendon sliding movements. IEEE Trans. Biomed. Eng. 52, 1110–1119. Chapin, J. K., and Moxon, K. A. (2000). ‘‘Neural Prosthesis for Restoration of Sensory and Motor Function.’’ CRC Press, Boca Raton. Chervin, R. D., and Guilleminault, C. (1997). Diaphragm pacing for respiratory insuYciency. J. Clin. Neurophysiol. 14, 369–377. Citi, L., Carpaneto, J., Yoshida, K., HoVmann, K. P., Koch, K. P., Dario, P., and Micera, S. (2008). On the use of wavelet denoising and spike sorting techniques to process electroneurographic signals recorded using intraneural electrodes. J. Neurosci. Methods 172, 294–302. Creasey, G., Elefteriades, J., DiMarco, A., Talonen, P., Bijak, M., Girsch, W., and Kantor, C. (1996). Electrical stimulation to restore respiration. J. Rehabil. Res. Dev. 33, 123–132. Dhillon, G. S., Lawrence, S. M., Hutchinson, D. T., and Horch, K. (2004). Residual function in peripheral nerve stumps of amputees: Implications for neural control of artificial limbs. J. Hand Surg. 29A, 605–615. Dhillon, G. S., Kruger, T., Sandhu, J., and Horch, K. (2005). EVects of short term training on sensory and motor function in severed nerves of long term human amputees. J. Neurophysiol. 93, 2625–2633. Fang, Z. P., and Mortimer, J. T. (1991). A method to eVect physiological recruitment order in electrically activated muscle. IEEE Trans. Biomed. Eng. 38, 175–179. Haugland, M., HoVer, J., and Sinkjaer, T. (1994). Skin contact force information in sensory nerve signals recorded by implanted cuV electrodes. IEEE Trans. Rehabil. Eng. 2, 18–28. Henneman, E., Clamann, H. P., Gillies, J. D., and Skinner, R. D. (1974). Rank order of motoneurons within a pool: Law of combination. J. Neurophysiol. 37, 1338–1349. Hochberg, L., Serruya, S., Friehs, G., Mukand, J., Saleh, M., Caplan, A., Branner, A., Chen, D., Penn, R., and Donoghue, J. (2006). Neuronal ensemble control of prosthetic devices by a human with tetraplegia. Nature 442, 164–171. Inmann, A., and Haugland, M. (2004). Implementation of natural sensory feedback in a portable control system for a hand grasp neuroprosthesis. Med. Eng. Phys. 26, 449–458. Johnson, M. W., Peckham, P. H., Bhadra, N., Kilgore, K. L., Gazdik, M. M., Keith, M. W., and Strojnik, P. (1999). Implantable transducer for two-degree of freedom joint angle sensing. IEEE Trans. Rehabil. Eng. 7, 349–359. Kilgore, K. L. (1997). An implanted upper-extremity neuroprosthesis. J. Bone Joint Surg. 79A, 533–541. Kipke, D. R., Vetter, R. J., Williams, J. C., and Hetke, J. F. (2003). Silicon-substrate intracortical microelectrode arrays for long-term recording of neuronal spike activity in cerebral cortex. IEEE Trans. Rehabil. Eng. 11, 151–155. Kovacs, G. T. A., Storment, C. W., Halks-Miller, M., Belczynski, C. R., Della Santina, C. C., Lewis, E. R., and Maluf, N. I. (1994). Silicon-substrate microelectrode array for parallel recording of neural activity in peripheral and cranial nerves. IEEE Trans. Biomed. Eng. 41, 567–577. Kuiken, T. A., Marasco, P. D., Lock, B. A., Harden, R. N., and Dewald, J. P. (2007). Redirection of cutaneous sensation from the hand to the chest skin of human amputees with targeted reinnervation. Proc. Natl. Acad. Sci. USA 104, 20061–20066. Lago, N., Ceballos, D., Rodrı´guez, F. J., Stieglitz, T., and Navarro, X. (2005). Long term assessment of axonal regeneration through polyimide regenerative electrodes to interface the peripheral nerve. Biomaterials 26, 2021–2031. Lago, N., Udina, E., Ramachandran, A., and Navarro, X. (2007). Neurobiological assessment of regenerative electrodes for bidirectional interfacing injured peripheral nerves. IEEE Trans. Biomed. Eng. 54, 1129–1137.
PERIPHERAL NERVE INTERFACES
37
Lawrence, S. M., Dhillon, G. S., Jensen, W., Yoshida, K., and Horch, K. W. (2004). Acute peripheral nerve recording characteristics of polymer-based longitudinal intrafascicular electrodes. IEEE Trans. Neural Syst. Rehabil. Eng. 12, 345–348. Leventhal, D. K., and Durand, D. M. (2003). Subfascicle stimulation selectivity with the flat interface nerve electrode. Ann. Biomed. Eng. 31, 643–652. Loeb, G. E., and Peck, R. A. (1996). CuV electrodes for chronic stimulation and recording of peripheral nerve. J. Neurosci. Methods 64, 95–103. McNaughton, T. G., and Horch, K. W. (1994). Action potential classification with dual channel intrafascicular electrodes. IEEE Trans. Biomed. Eng. 41, 609–616. Micera, S., Jensen, W., Sepulveda, F., Riso, R. R., and Sinkjaer, T. (2001). Neuro-fuzzy extraction of angular information from muscle aVerents for ankle control during standing in paraplegic subjects: An animal model. IEEE Trans. Biomed. Eng. 48, 787–794. Micera, S., Carrozza, M. C., Beccai, L., Vecchi, F., and Dario, P. (2006). Hybrid bionic systems for the replacement of hand function. Proc. IEEE 94, 1752–1762. Miller, L. A., Stubblefield, K. A., Lipschutz, R. D., Lock, B. A., and Kuiken, T. A. (2008). Improved myoelectric prosthesis control using targeted reinnervation surgery: A case series. IEEE Trans. Neural Syst. Rehabil. Eng. 16, 46–50. Mirfakhraei, K., and Horch, K. (1997). Recognition of temporally changing action potentials in multiunit neural recordings. IEEE Trans. Biomed. Eng. 44, 123–131. Naples, G. G., Mortimer, J. T., and Yuen, T. G. H. (1990). Overview of peripheral nerve electrode design and implantation. In ‘‘Neural Prostheses: Fundamental Studies’’ (W. F. Agnew and D. B. McCreery, Eds.), pp. 107–144. Prentice-Hall, New Jersey. Navarro, X., Calvet, S., Rodrı´guez, F. J., Stieglitz, T., Blau, C., Butı´, M., Valderrama, E., and Meyer, J. U. (1998). Stimulation and recording from regenerated peripheral nerves through polyimide sieve electrodes. J. Peripher. Nerv. Syst. 2, 91–101. Navarro, X., Valderrama, E., Stieglitz, T., and Schuttler, M. (2001). Selective fascicular stimulation of the rat sciatic nerve with mutipolar polyimide cuV electrodes. Restor. Neurol. Neurosci. 18, 9–21. Navarro, X., Krueger, T., Lago, N., Micera, S., Stieglitz, T., and Dario, P. (2005). A critical review of interfaces with the peripheral nervous system for the control of neuroprostheses and hybrid bionic systems. J. Peripher. Nerv. Syst. 10, 229–258. Navarro, X., Lago, N., Vivo´, M., Yoshida, K., Koch, K. P., Poppendieck, W., and Micera, S. (2007). Neurobiological evaluation of thin-film longitudinal intrafascicular electrodes as a peripheral nerve interface. In Proc. IEEE 10th Int. Conf. Rehabil. Robotics, pp. 643–649. Nordhausen, C. T., Maynard, E. M., and Normann, R. A. (1996). Single unit recording capabilities of a 100 microelectrode array. Brain Res. 726, 129–140. Peckham, H. P., and Keith, M. W. (1992). Motor prostheses for restoration of upper extremity function. In ‘‘Neural Prostheses. Replacing Motor Function After Disease or Disability’’ (R. B. Stein, P. H. Peckham, and D. B. Popovic, Eds.), pp. 162–187. Oxford University Press, New York. Peters, A., Palay, S. L., and Webster, H. F. (1991). ‘‘The Fine Structure of the Nervous System: Neurons and their Supporting Cells’’, 3rd Ed. Oxford University Press, New York. RijkhoV, N. J. M. (2004). Neuroprostheses to treat neurogenic bladder dysfunction: Current status and future perspectives. Child Nerv. Syst. 20, 75–86. Rutten, W. L. C. (2002). Selective electrical interfaces with the nervous system. Annu. Rev. Biomed. Eng. 4, 407–452. Rutten, W. L. C., Smit, J. P. A., Frieswijk, T. A., Bielen, J. A., Brouwer, A. L. H., Buitenweg, J. R., and Heida, C. (1999). Neuro-electronic interfacing with multielectrode arrays. IEEE Eng. Med. Biol. 18, 47–55. Sinkjaer, T., Haugland, M., Inmann, A., Hansen, M., and Nielsen, K. D. (2003). Biopotentials as command and feedback signals in functional electrical stimulation systems. Med. Eng. Phys. 12, 29–40.
38
MICERA AND NAVARRO
Stanton-Hicks, M., and Salamon, J. (1997). Stimulation of the central and peripheral nervous system for the control of pain. J. Clin. Neurophysiol. 14, 46–62. Stein, R. B., Charles, D., HoVer, J. A., Arsenault, J., Davis, L. A., Moorman, S., and Moss, B. (1980). New approaches for the control of powered prostheses particularly by high-level amputees. Bull. Prosthet. Res. 10–33, 51–62. Stieglitz, T., Beutel, H., Schuettler, M., and Meyer, J. U. (2000). Micromachined, polyimide-based devices for flexible neural interfaces. Biomed. Microdev. 2, 283–294. Tarler, M., and Mortimer, J. (2004). Selective and independent activation of four motor fascicles using a four contact nerve-cuV electrode. IEEE Trans. Neural Syst. Rehab. Eng. 12, 251–257. Tesfayesus, W., and Durand, D. M. (2006). Blind source separation of neural recordings and control signals. Conf. Proc. IEEE Eng. Med. Biol. Soc. 1, 731–734. Triolo, R. J., Bieri, C., Uhlir, J., Kobetic, R., Scheiner, A., and Marsolais, E. B. (1996). Implanted FNS systems for assisted standing and transfers for individuals with cervical spinal cord injuries. Arch. Phys. Med. Rehabil. 77, 1119–1128. Tyler, D. J., and Durand, D. M. (1997). A slowly penetrating interfascicular nerve electrode for selective activation of peripheral nerves. IEEE Trans. Rehabil. Eng. 5, 51–61. Tyler, D. J., and Durand, D. M. (2002). Functionally selective peripheral nerve stimulation with a flat interface nerve electrode. IEEE Trans. Neural Syst. Rehabil. Eng. 10, 294–303. Tyler, D. J., and Durand, D. M. (2003). Chronic response of the rat sciatic nerve to the flat interface nerve electrode. Ann. Biomed. Eng. 31, 633–642. Velliste, M., Perel, S., Spalding, M. C., Whitford, A. S., and Schwartz, A. B. (2008). Cortical control of a prosthetic arm for self-feeding. Nature 453, 1098–1101. Veraart, C., Grill, W. M., and Mortimer, J. T. (1993). Selective control of muscle activation with a multipolar nerve cuV electrode. IEEE Trans. Biomed. Eng. 40, 640–653. Walter, J. S., GriYth, P., Sweeney, J., Scarpine, V., Bidnar, M., McLane, J., and Robinson, C. (1997). Multielectrode nerve cuV stimulation of the median nerve produces selective movements in a raccoon animal model. J. Spinal Cord. Med. 20, 233–243. Warwick, K., Gasson, M., Hutt, B., Goodhew, I., Kyberd, P., Andrews, B., Teddy, P., and Shad, A. (2003). The application of implant technology for cybernetic systems. Arch. Neurol. 60, 1369–1373. Waters, R. L., McNeal, D. R., Faloon, W., and CliVord, B. (1985). Functional electrical stimulation of the peroneal nerve for hemiplegia: Long-term clinical follow-up. J. Bone Joint Surg. 67, 792–793. Yoshida, K., and Horch, K. (1993). Selective stimulation of peripheral nerve fibers using dual intrafascicular electrodes. IEEE Trans. Biomed. Eng. 40, 492–494. Yoshida, K., and Horch, K. (1996). Closed-loop control of ankle position using muscle aVerent feedback with functional neuromuscular stimulation. IEEE Trans. Biomed. Eng. 43, 167–176. Yoshida, K., and Stein, R. B. (1999). Characterization of signals and noise rejection with bipolar longitudinal intrafascicular electrodes. IEEE Trans. Biomed. Eng. 46, 226–234. Yoshida, K., Jovanovic, K., and Stein, R. B. (2000). Intrafascicular electrode for stimulations and recording from mudpuppy spinal roots. J. Neurosci. Methods 96, 47–55.
INTERFACING INSECT BRAIN FOR SPACE APPLICATIONS
Giovanni Di Pino,* Tobias Seidl,y Antonella Benvenuto,* Fabrizio Sergi,* Domenico Campolo,* Dino Accoto,* Paolo Maria Rossini,z and Eugenio Guglielmelli* *Biomedical Robotics and Biomicrosystems Laboratory, Universita` Campus Biomedico di Roma, Roma, Italy y Advanced Concepts Team, European Space Agency, Keplerlaan 1, 2201 AZ Noordwijk, The Netherlands z Department of Neurology, Universita` Campus Biomedico di Roma, Via Alvaro del Portillo 21, 00128 Roma, Italy
I. Introduction II. Interfaces A. Levels of Interfacing B. Natural Interfaces C. In Vivo Neuronal Bidirectional Interfaces in Insects III. Sensory and Motor Mapping IV. Proposing a Model of Hybrid Control Architecture V. Conclusions and Outlook References
Insects exhibit remarkable navigation capabilities that current control architectures are still far from successfully mimic and reproduce. In this chapter, we present the results of a study on conceptualizing insect/machine hybrid controllers for improving autonomy of exploratory vehicles. First, the diVerent principally possible levels of interfacing between insect and machine are examined followed by a review of current approaches towards hybridity and enabling technologies. Based on the insights of this activity, we propose a double hybrid control architecture which hinges around the concept of ‘‘insect-in-a-cockpit.’’ It integrates both biological/artificial (insect/robot) modules and deliberative/reactive behavior. The basic assumption is that ‘‘low-level’’ tasks are managed by the robot, while the ‘‘insect intelligence’’ is exploited whenever high-level problem solving and decision making is required. Both neural and natural interfacing have been considered to achieve robustness and redundancy of exchanged information.
INTERNATIONAL REVIEW OF NEUROBIOLOGY, VOL. 86 DOI: 10.1016/S0074-7742(09)86003-0
39
Copyright 2009, Elsevier Inc. All rights reserved. 0074-7742/09 $35.00
40
DI PINO et al.
I. Introduction
The present book impressively highlights how astronaut’s capabilities can be enhanced by interfacing brains with machines, but we are well aware that human presence drastically limits space travel beyond earth orbiting. Robotic missions can reach further but require either remote access or a high level of autonomy. Despite the great scientific success of nonhuman missions they still lack important capabilities such as autonomous navigation on ground, problem solving and decision making in conflict situations, learning and adapting to perturbations of the original program. One possible approach is the integration of animal brains into unmanned spacecraft to achieve an intermediate type of mission combining features of the commonly known and strictly separated manned and unmanned mission types. In the framework of an Ariadna-study ESA’s Advanced Concepts Team, the Biomedical Robotics and Biomicrosystems Laboratory and the Neuroscience Department-Area of Neurology of the Campus Bio-Medico University in Rome jointly evaluated the major technologies required and conceptualized a control architecture integrating insect brain tissue in an engineered control architecture (Benvenuto et al., 2008). Among several animal species, insects have been chosen because they developed navigation mechanisms which are optimized in terms of simplicity and robustness ( Dacke and Srinivasan, 2007; Wehner, 2007), that are invaluable features of robotic systems. Since insect neuronal systems diVer considerably from those of humans the approaches and technologies to be used diVer accordingly. Given the inherent technological challenges, some simplifying assumptions are required; in particular we assume that it is feasible to keep alive and functional the animal brain tissue (or the whole insect) for a period of time appropriate for space missions. Moreover, we focus on the use of predeveloped living tissue and do not consider in vitro development of biological neuronal networks. Eventually, we will not take care of control issues which can be solved with a smart (e.g., biomimetic) design. In this chapter, at first the levels of interfacing between living tissues, environment and robot are described and the state-of-the-art technologies for both natural and neural interfacing are critically reviewed. Then elementary behaviors associated to exploration/navigation tasks and their triggers are schematized to better address the sensory and motor mapping issues. Finally, we present a double hybrid control architecture which includes both biological/artificial (insect/robot) modules and deliberative/reactive behaviors.
INTERFACING INSECT BRAIN
41
II. Interfaces
A. LEVELS OF INTERFACING The interface between living tissues and the engineered counterpart of the hybrid controller could be established at several levels involving the whole body of the animal or only part of it. Various solutions diVer also for the invasiveness and for the amount of information exchanged. First of all there is the choice between non-invasive cockpit like interfaces, where the intact animal’s sensors and eVectors are used to transmit information by providing quasi-natural stimuli and invasive interfaces with a direct electric contact between controller and living tissue. The invasive technique then allows choosing between neural interfaces, that is, information is decoded from neurons, or non-neural approaches (i.e., from muscles). Finally, neural interfaces roughly divide into three subgroups: electrodes stimulate and monitor (i) cultured neurons or (ii) pregrown brain tissue but being separated from the organism, and (iii) brain tissue in the organism. Experiments with self-organized neuronal cultures have returned promising results (Novellino et al., 2007) such as sensing and reacting to various stimuli. However, tasks as defined above might not be in the scope of such an arrangement. In consequence, we investigated on more complex neural networks as they can be extracted from living and developed organisms. It is worth noting that the brain stem of a lamprey is already able to control a small mobile device via bidirectional information exchange ( Reger et al., 2000). However, it would be desirable to interface an entire brain and have it maintained by a functional organism. This work has been focused on insect interfacing, that could be achieved either by insertion of microelectrodes in the ganglia or implanting electrodes into the musculature. While the first solution principally lacks mechanical stability, muscular interfacing requires higher electric current (Mavoori et al., 2004) and leads to a loss in information bandwidth. Finally, it is feasible also to invasively interface sensory organs of the organism similar to, for example, cochlear implants in humans.
B. NATURAL INTERFACES Natural interfaces exploit the existing sensorial and eVectorial means of an organism. In insect studies, natural interfacing is part of neurobiological and neuroethological experimental setups focusing on insect flight, visual flight
42
DI PINO et al.
stabilization or navigation. In such setups, the insect is situated in an arena— tethered or freely moving—and provided with visual stimuli by, for example, a circular matrix of light-emitting diodes (compare Fig. 1A; for exemplary description see Dacke and Srinivasan, 2007; Reiser and Dickinson, 2008; Srinivasan et al., 1996). For instance, in freely moving animals, magnetic coils attached to the thorax allow to monitor the insect’s motor response (Schilstra and Van Hateren, 1999) (Fig. 1C), while in tethered insects MEMS-based force sensors register the insect’s attempts and translates them into a visual reaction (Sun et al., 2005) (Fig. 1B). Experimental setups usually focus on the research of certain eVects rather than on achieving bidirectional control and hence the degree of integration of several technologies is kept rather low. There are, however, successful attempts where an insect autonomously controls a robot. The roachbot—developed by Hertz and coworkers (http://www.conceptlab.com/roachbot/)—uses proximity sensors to acquire data about the surrounding. LED panels display the results of these measurements to the tethered cockroach and a trackball monitors the animal’s reactions to the stimuli. The system allows the cockroach to interact with obstacles but is not capable of exploiting high-level autonomous behaviors.
C. IN VIVO NEURONAL BIDIRECTIONAL INTERFACES IN INSECTS Neuronal interfaces can exploit the comparative simplicity of the insect neuronal system, since most of the relevant neurons can be directly approached in the ventral ganglia or the connectives. Moreover, the nervous system presents a straight one-to-one correspondence between nerve stimulation and muscle
FIG. 1. Natural interfaces: (A) Programmable arena (adapted from Reiser and Dickinson, 2008); (B) MEMS-based force sensor (adapted from Sun et al., 2005); (C) sensor coils (adapted from Schilstra and Van Hateren, 1999).
INTERFACING INSECT BRAIN
43
activation and requires only little voltage to excite an axon, due to the sparse insulation. Indeed, direct stimulation of a nerve works with a tenth of the electric current compared to excitation via the musculature ( Mavoori et al., 2004). The technological development in the recent past has led to miniaturized and versatile implantable electrodes that greatly extend the application range. Modern techniques involve flexible polymer carrying multielectrode arrays. They can be wrapped a nerve or an insect appendage enabling multiunit recording and stimulation, without aVecting the kinematics of the animal locomotion (Spence et al., 2007). Other techniques were reported by Ye and coworkers (1995) using cuV-shaped electrodes for chronic recordings in tethered cockroach (Periplaneta americana). The electrode was placed close to the thoracic ganglion and remained for the remarkably long period of 2 months. With the same species Takeuchi and colleagues recently demonstrated a radiotelemetry system that allows recording neural activity in freely walking animal in the range of 16 m ( Takeuchi and Shimoyama, 2004). The system uses a shape memory alloy (SMA) electrode that, when actuated by electric heating, clips around the nerve cord along the thorax (Fig. 2A). Since it was developed only as a recording system, it misses the ability of stimulating and hence to operate as bidirectional interface.
A
Telemeter
Polyimide ribbon cable
B
Polyimide film
SMA microelectrode
C
D
FIG. 2. (A) Telemetric system implanted in freely moving Periplaneta americana (adapted from Takeuchi and Shimoyama, 2004); (B) Neuron-chip implanted in Manduca sexta (adapted from Diorio and Mavoori, 2003); (C) Cyborg beetle microsystem (adapted from Sato et al., 2008); (D) moth–robot (adapted from http://neuromorph.ece.arizona.edu/).
44
DI PINO et al.
Fully integrated bidirectional systems should ideally be able to stimulate and to record from the nervous system of a flying insect, while including an on-board memory and being suYciently small and light to be carried by the animal. Mavoori and coworkers (Diorio and Mavoori, 2003) presented such a microsystem carried by a freely moving Manduca sexta (Fig. 2B). An even higher degree of integration was achieved by Sato and coworkers (2008). The flight of the insect was controlled via an ensemble of muscular stimulators, an embedded microcontroller with batteries, microfluidic tubes and LED visual stimulator, together with a silicon neural probe. The four neural stimulators are implanted in the flight control area of the brain and close to the flight musculature, while the base of the optic flow device is mounted positioning the LED array in front of the head (Fig. 2C). Another remarkable example is the moth–robot (http://neuromorph.ece.arizona.edu/). Its composite bidirectional interfaces consist of (i) a natural interface through a continuous optic flow provided by a revolving wall painted with vertical stripes and (ii) a neural interface for measurements of electrical activity of visual motion neurons. The moth is immobilized inside a plastic tube mounted on a wheeled robot, which turns left or right, according to neural signals from the insect (Fig. 2D).
III. Sensory and Motor Mapping
The operating environment in which an exploratory spacecraft would be situated diVers considerably from the natural habitat of any potential model insect and therefore inputs need to be translated into signals that are readable for the insect. This translation does not only take place at the physical level, that is, presenting stimuli within the signal range of the sensors, but also requires that the neuronal system is able to decipher the meaning of an input and elicit appropriate reaction toward it. Complex behaviors as navigation or exploration may be abstracted as a product of behavioral elements following attraction and repulsion. Depending on their individual characteristics, the behavior elicited has either static or dynamic nature. A typical example of a static attractor is the pursuit of food or the nest, while a predator would elicit fleeting behavior, a dynamic repeller (Table I).
TABLE I ELEMENTARY BEHAVIORS AND THEIR TRIGGERS IN EXPLORATION/NAVIGATION TASKS
Static Dynamic
Attractors
Repellers
Pursuit (food, nest) Hunting (prey)
Avoidance (obstacle) Fleeting (predator)
45
INTERFACING INSECT BRAIN
IV. Proposing a Model of Hybrid Control Architecture
Interfacing insect brains strongly depends on the task the final architecture will be confronted with. In our pilot study, we identified an exploratory scenario where the animal/insect shall explore an unknown environment and autonomously navigate back to, for example, a base station that provides extended functionality such as energy, communication or automated sample analysis. Within such a scenario, high-level tasks like navigation, exploration and maintaining the energy budget would be handed over to the insect, while obstacle avoidance and locomotory issues would be dealt with by the engineered controller. Based on these considerations, a robotic platform including the hybrid control architecture was conceptualized as shown in Fig. 3. The proposed architecture (Benvenuto et al., 2009) comprises three main modules (i) the hybrid controller (HC), (ii) the adaptor (ADP), and (iii) the underlying mechatronic system (UMS). Moreover, two subsets have been defined in sensor modules: low-level sensors (LLS) and embiotic sensors (ES). The LLS are included in the UMS and they can be
Hybrid controller (HC) Sensory mapping
-Visual cues (programmable LEDs) -Olfactory cues
Neural stimulation unit
Neural interface
Environment
Embiotic sensors
Neural registration unit
Sensor modules
Natural stimulation unit
Natural interface
Cockpit Natural registration unit
Motor mapping
-Force sensors -Magnetic coils
Adaptor
Low level sensors Proprioceptors
Low level controller
Underlying mechatronic system (UMS) Actuators
Robot mechanical system
FIG. 3. Scheme of a robotic platform including the hybrid control architecture (Benvenuto et al., 2009).
46
DI PINO et al.
used for executing low-level behaviors; typically they include proximity sensors, inertia modules, wheel\leg encoders and wheel\leg slide sensors for selfstabilization. ‘‘embiotic’’ is a neologism that we introduced to indicate an additional subset of conventional sensors (e.g., vision systems, temperature sensors, polarization sensors, etc.), which complement LLS and depend on the selected insect. The hybrid controller is composed of ES, the cockpit, the interfacing module, and mapping modules. The cockpit surrounds the tethered insect which receives stimuli both via natural and neural interfaces. The neural interface module includes stimulation and registration units as well as sensory and motor mapping modules forming core elements of the proposed architecture. The ADP is designed to process the insect’s motor commands and to give input to the low-level controller. Actually, this is the only module which directly exchanges data between the hybrid controller and the mechatronic system. By changing the ADP module, the proposed hybrid controller could in principle be used with diVerent mechatronic systems, which can be typically tailored to the specific application scenario. The UMS is composed of the LLS, the proprioception sensors, and the low-level controller. The proprioception sensors monitor the internal state of the robot, in particular, for energy budget and mechanical failures. The main task of the low-level controller is to properly weight inputs from the sensor module, the proprioceptors and the adaptor, thus allowing for undisturbed mobility of the robot. V. Conclusions and Outlook
Our studies showed that the integration of predeveloped insect intelligence in robotic platforms could create an intermediate type of mission bridging between purely robotic and human controlled missions capable of performing complex behaviors exploiting the neuronal capabilities of the animal. However, success of automated mission vehicles strongly correlates with the capability of the control architecture to successfully integrate a whole range of decision parameters. The architecture we present here delegates high-level decision making and planning to the insect, while low-level tasks are executed by the robotic platform. The current state of the art of both neural and natural interfaces allows implementing in the short/medium term a ‘‘cockpit’’ interface with tethered insects; while additional challenges have to be considered if an ‘‘arena’’ interface with freely moving insects (i.e., highly ecological environment) will be developed, since dimensions and weight of neural bidirectional interfaces integrating batteries and telemetry systems should be reduced to allow implantation in small insects. Moreover, experiments to assess insect capabilities by reproducing space operating conditions, where the insect brain could also beforehand be trained, should be performed. Eventually, a novel performance/benchmarking metrics should be defined to assess the obtained results and to
INTERFACING INSECT BRAIN
47
compare them with the state-of-the-art autonomous agent performances. Even if much work is required for development and validation of the proposed control architecture, it represents a new challenging approach merging biomimetics, neurophysiology, ethology, and microengineering. References
Benvenuto, A., Di Pino, G., Sergi, F., Campolo, D., Accoto, D., Assenza, G., Rossini, P. M., and Guglielmelli, E. (2008). Machine/animal hybrid controllers for space applications. European Space Agency, the Advanced Concepts Team, Ariadna Final Report (07/6301). Benvenuto, A., Sergi, F., Di Pino, G., Seidl, T., Campolo, D., Accoto, D., Guglielmelli, E. (2009). Beyond biomimetics: Towards insect/machine hybrid controllers for space applications. Advanced Robotics, in press. Dacke, M., and Srinivasan, M. V. (2007). Honeybee navigation: Distance estimation in the third dimension. J. Exp. Biol. 210, 845–853. Diorio, C., and Mavoori, J. (2003). Computer electronics meet animal brains. Computer 36, 69–75. Mavoori, J., Millard, B., Longnion, J., Daniel, T., and Diorio, C. (2004). A miniature implantable computer for functional electrical stimulation and recording of neuromuscular activity. In IEEE International Workshop on Biomedical Circuits & Systems, pp. S1/7/INV–S1/13–16. Novellino, A., D’Angelo, P., Cozzi, L., Chiappalone, M., Sanguinetti, V., and Martinoia, S. (2007). Connecting neurons to a mobile robot: An in vitro bidirectional neural interface. Comput. Intell. Neurosci. 2007, 13. Reger, B. D., Fleming, K. M., Sanguineti, V., Alford, S., and Mussa-Ivaldi, F. A. (2000). Connecting brains to robots: An artificial body for studying the computational properties of neural tissues. Artif. Life 6, 307–324. Reiser, M. B., and Dickinson, M. H. (2008). A modular display system for insect behavioral neuroscience. J. Neurosci. Methods 167, 127–139. Sato, H., Berry, C., Casey, B., Lavella, G., Yao, Y., VandenBrooks, J., and Maharbiz, M. (2008). A cyborg beetle: Insect flight control through an implantable, tetherless microsystem. In MEMS 2008, pp. 164–167. Tucson, AZ, USA. Schilstra, C., and Van Hateren, J. H. (1999). Blowfly flight and optic flow, I. Thorax kinematics and flight dynamics. J. Exp. Biol. 202, 1481–1490. Spence, A. J., Neeves, K. B., Murphy, D., Sponberg, S., Land, B. R., Hoy, R. R., and Isaacson, M. S. (2007). Flexible multielectrodes can resolve multiple muscles in an insect appendage. J. Neurosci. Methods 159, 116–124. Srinivasan, M., Zhang, S., Lehrer, M., and Collett, T. (1996). Honeybee navigation en route to the goal: Visual flight control and odometry. J. Exp. Biol. 199, 237–244. Sun, Y., Fry, S. N., Potasek, D. P., Bell, D. J., and Nelson, B. J. (2005). Characterizing fruit fly behavior using a microforce sensor with a new comb-drive configuration. J. Microelectromech. Syst. 14(1), 4–11. Takeuchi, S., and Shimoyama, I. (2004). A radio-telemetry system with a shape memory alloy microelectrode for neural recording of freely moving insects. IEEE Trans. Biomed. Eng. 51, 133–137. Wehner, R. (2007). The desert ant’s navigational toolkit: Procedural rather than positional knowledge. In Proceedings of 63rd Annual Meeting of the Institute of Navigation, April 23–25, 2007, Cambridge, Massachusetts. Ye, S., Dowd, J. P., and Comer, C. M. (1995). A motion tracking system for simultaneous recording of rapid locomotion and neural activity from an insect. J. Neurosci. Methods 60, 199–210.
MEET THE BRAIN: NEUROPHYSIOLOGY
John Rothwell Sobell Department, Institute of Neurology, Queen Square, London WC1N 3BG, UK
I. II. III. IV.
Introduction How Do Neurons Transmit Information? Synapses The Motor Areas of the Cerebral Cortex A. Historical Background B. Present Day Anatomical and Electrophysiological Definitions of the Motor Areas of Cortex C. Output of Cortical Motor Areas D. Inputs to Cortical Motor Areas E. Activity of Motor Cortical Neurons F. Effect of Lesions of Motor Cortical Areas V. Plasticity of Primary Motor Cortex VI. Conclusions References
The central nervous system is composed of two main types of cells: the most numerous are glia, which have a supportive, protective and regulatory role, and neurons, which are the primary computing element. Neurons transmit information as a pulsed electrical code which is conducted down a specialized process (axon) that connects with other neurons. Each neuron can connect with many others, and each neuron can receive input from many others. At the sites of connection (synapses), information is transmitted across a small gap; small molecules (neurotransmitters) are released from the end of the axon, and these diVuse to receptor molecules on the receiving neuron. The latter then convert the chemical code back into an electrical signal that can be transmitted along the next axon. An important feature of synapses is that they are modifiable according to the prior history of activity in the system. This gives them an important role in learning, memory, and in adaptation to damage. Networks of neurons perform particular tasks. Those controlling movement are located in a number of adjacent areas of cerebral cortex. Some of these have axons that project to the spinal cord where they contact motoneurons that control particular sets of muscles; some have axons that interconnect the areas of the cortex; some have axons that project to subcortical groups of neurons in the basal
INTERNATIONAL REVIEW OF NEUROBIOLOGY, VOL. 86 DOI: 10.1016/S0074-7742(09)86004-2
51
Copyright 2009, Elsevier Inc. All rights reserved. 0074-7742/09 $35.00
52
JOHN ROTHWELL
ganglia and cerebellum. The presence of modifiable synapses in this complex network means that it is capable of learning new tasks and can react to injury by rearranging its connections to optimize function in undamaged parts.
I. Introduction
The central nervous system (CNS) contains two main types of cell: glia and neurons. Even though we always tend to think of the neurons as being the most important part of the CNS, they are in fact vastly outnumbered by the glia. In fact there are about 10–50 times more glia than neurons. The glia, so-called because early anatomists thought they were the ‘‘glue’’ that held the neurons together, come in a variety of forms and perform a large number of diVerent and important functions. The most numerous are the astrocytes, which are roughly star shaped with long processes, some of which contact neurons, where they may have a role in transferring nutrients, whereas others may line the outside of blood capillaries where they form the blood–brain barrier, determining what chemicals are allowed from the blood into the brain. Astrocytes also seem to have an important role in regulating the concentration of potassium ions around the neurons, an important factor in determining the voltage across the neuronal membrane. They are also involved in mopping up neurotransmitters after release from synapses and even manufacturing the raw chemicals that are transformed into neurotransmitters by neurons. Two other forms of glia have a role in insulating the neurons and thus helping them conduct their electrical impulses. These are the oligodendrocytes and the Schwann cells. Oligodendrocytes are found in the CNS where their processes envelope 10 or more diVerent neurons; Schwann cells are found around neurons in the periphery (i.e., in nerves outside the brain and spinal cord). Each Schwann cell contacts only one neuron while each neuron is contacted by many Schwann cells, each of which covers the length between two nodes of Ranvier (see below). Neurons are the active signaling units of the nervous system. Within the CNS they receive input from other neurons, process it, and then send it on to other neurons in a one way flow of information. Neurons in the periphery have other connections: sensory neurons contact (or have specialized terminations themselves) sensory receptors that transform the five senses (touch, vision, hearing, smell, taste) into electrical signals that eventually enter the CNS. Motor neurons form the only output of the CNS: at their ends they contact muscles that move the body. Essentially the CNS receives sensory inputs and uses this information to drive movement (Fig. 1). Each neuron has four main regions. The cell body contains the nucleus and most of the other specialized structures necessary to manufacture proteins. It has
53
NEUROPHYSIOLOGY
Dendrite Sensory neuron Receptor (free nerve ending)
Cell body Axon
Myelin sheath Motor end plate
Axon Motor neuron www.infovisual.info
Dendrite Cell body
FIG. 1. Typical sensory and motor neuron illustrating the main features and connectivity of each. The arrows indicate the direction of information flow.
two sorts of processes, dendrites, of which there may be very many, that branch out over a very large volume in comparison with the volume of the cell body itself, and a single axon that forms the output of the neuron and which may travel considerable distances from the cell body before contacting other neurons at specialized terminations called synapses. The dendrites receive input from the synapses of other neurons and together with the cell body, they integrate input from multiple sources. The axon then transmits the processed information to the next neurons. Each neuron may typically have hundreds of inputs and will synapse with hundreds of other neurons.
II. How Do Neurons Transmit Information?
All neurons have an electrical potential across the membrane that makes the inside about 70 mV negative with respect to the outside. The potential is caused by diVerences in the concentration of charged ions, particularly Naþ and Kþ, in the inside versus the outside. Amino acids and proteins tend to have a negative charge and are located both inside and outside the neuron, but a special pump (the Naþ/Kþ pump) in the cell membrane pumps Naþ out of the cell and pumps Kþ into the cell. The result is that the concentration of Naþ outside is about 10 times that inside whereas the concentration of Kþ is about 20 times higher
54
JOHN ROTHWELL
inside than outside. The membrane itself is fairly impermeable to Naþ (and to the large amino acids and proteins), but is leaky for Kþ. This means that Kþ leaks out of the neuron down its concentration gradient, leaving behind an excess of negatively charged large proteins and amino acids that are responsible for the net negative charge inside the cell. The pumping of ions occurs continuously to maintain the diVerential concentrations of ions; if it stops, the potential disappears. Axons transmit information in an all or none fashion as a series of identical electrical impulses known as action potentials. Thus an axon can only code information in terms of the timing of the pulses that it transmits. The action potential begins at a specialized region of the cell body known as the initial segment, at the start of the axon. It is initiated as the result of processing the inputs collected by the dendrites. This causes the membrane potential at the axon hillock to become less negative than in the resting state (see below). If the potential across the membrane reaches a value that is less negative than a certain value (i.e., more positive than, say, 50 mV ) there is a sudden change in the permeability of the cell membrane at that point and it becomes highly permeable to Naþ (much more permeable than it was to Kþ). Naþ ions then enter the neuron down their concentration gradient and cause the inside at that point to become positive with respect to the outside, eVectively reversing the potential across the membrane. This state lasts a few milliseconds, after which the permeability to Naþ drops back to normal, and the membrane reverts to its original state with exit of Kþ ions. The key to the whole process of changing the membrane permeability to Naþ lies with a specialized protein in the membrane called the voltage sensitive Naþ channel. At a resting potential of 70 mV, the channel is closed; when depolarized beyond a certain value (known as the threshold potential), the Naþ channel opens transiently and then closes. The job of the axon is to transmit this transient change in potential to other neurons that are connected at the ends of the axon. It achieves this via two processes. One is passive spread of the potential to adjacent regions of the neuronal membrane (electrotonic conduction). The neuronal membrane can be regarded as an electrical capacitance in which the depolarized part begins to change the potential on the adjacent membrane as current flows through the resistance of the extra- and intracellular fluids. This change takes time and does not spread very far because the original depolarization is transient. However, the length of the axon is populated by voltage sensitive Naþ channels that open when the potential in their vicinity exceeds the threshold. When they do this, they reinforce the spread of depolarization along the length of the axon. Within the CNS, impulses are transmitted along axons at speeds of up to about 10 m/s. The larger the diameter of the axon, the faster the conduction. This is because a larger axon has a smaller surface:volume ratio than a small axon, and therefore a proportionally smaller membrane capacitance to charge up. It also has a smaller
NEUROPHYSIOLOGY
55
internal electrical resistance which also speeds up the rate at which adjacent sections of membrane are depolarized.
III. Synapses
The synapse is the point at which information is transmitted between two neurons. It consists of a small gap across which information is transferred by diVusion of chemicals (neurotransmitters) that are released from the specialized termination of the axon. They are detected by molecules inserted into the dendrite on the opposite side of the gap. In the presynaptic axon, information transmitted as electrical impulses has to be transformed into a chemical code, while in the postsynaptic dendrite the process must be reversed and the chemical code must be transformed back into an electrical code. Synapses are highly complex parts of the neuron and are designed to be modifiable according to the past history of activity in the pre- and postsynaptic neurons. The details of transmission at the synapse are as follows. At the region of the synapse, the presynaptic axon contains voltage sensitive Ca2þ channels that open when the wave of depolarization produced by the action potential reaches the synapse. Ca2þ exists at a much higher concentration outside than inside the neuron, so that on arrival, the action potential causes an influx of Ca2þ ions. Within the presynaptic terminal, a neurotransmitter is stored in small pockets called vesicles surrounded by a membrane. The sudden Ca2þ influx via a series of intermediate steps causes these vesicles to fuse with the neuronal membrane and release their contents into the synaptic space. In the postsynaptic membrane of the dendrite receptor molecules can bind to the neurotransmitter, and in doing so open ion channels that cause changes in the postsynaptic membrane potential. Exactly which ion channels are open and for how long depends on the neurotransmitter and receptor which it couples to. Typically, postsynaptic membrane polarizations last about 10–20 ms, although some may continue for several hundred milliseconds. In contrast, the depolarization in an action potential lasts about 5 ms (Fig. 2). As an example, glutamate is the most common excitatory neurotransmitter in the brain. When it binds to one of its receptors, the AMPA receptor, an ion channel opens in the postsynaptic dendritic membrane that is permeable to Naþ and Kþ. The net result is that it depolarizes the dendrite. This depolarization spreads electrotonically to the initial segment region of the axon where it can contribute to the initiation of an action potential. GABA is the most common inhibitory neurotransmitter. When bound by its receptor, it increases permeability to Cl ions, which then enter the neuron down a concentration gradient and make the inside of the cell more negative than at rest.
56
JOHN ROTHWELL
Ca++ V-sens
Pre-synaptic Glu
Na+
Mg++ NMDA
K+
AMPA EPSP
Post-synaptic dendritic spine
FIG. 2. Presynaptic release of neurotransmitter from vesicles that fuse with the cell membrane when calcium ions enter flowing depolarization of the membrane. Transmitter molecules (in this case glutamate) diVuse across the synapse and combine with AMPA receptors in the postsynaptic membrane. These then allow sodium and potassium ions to enter the membrane, depolarizing the potential. Another receptor that can bind glutamate is also drawn, the NMDA receptor. This is usually blocked by a magnesium ion at normal membrane potentials, but can be expelled during depolarization and then bind glutamate.
An important feature of the synaptic connections between most neurons is that the input from any one synapse has only a very small eVect on the potential at the trigger zone in the axon hillock; an action potential requires the summation of inputs from many synapses, all active at the same time. In simple terms the neuron
NEUROPHYSIOLOGY
57
adds up the synaptic inputs that occur and if these at any time reach the threshold, an impulse is generated that travels onward to the next set of synapses. There are a large number of refinements to this simple model of a working neuron. The first, as mentioned above is that the eVectiveness of a synapse is not fixed, but can change according to the pattern of activity that has occurred in the recent past. These changes occur over a range of time scales from milliseconds to hours and days, with multiple molecular pathways involved in each stage. In addition, the eVectiveness of inputs to a neuron can be regulated by ‘‘neuromodulatory’’ transmitters such as dopamine, acetylcholine, or serotonin. When these substances are released from synapses, they do not typically make neurons discharge, but instead regulate the response of the neuron to other inputs.
IV. The Motor Areas of the Cerebral Cortex
A. HISTORICAL BACKGROUND Hughlings Jackson was one of the first physicians to speculate that the cortex around the central sulcus contained an organized representation of body movements. He observed that motor epilepsies often began with small twitches in the hand or the corner of the mouth and then spread to involve adjacent muscles and finally the whole body. In a small number of cases that came to pathology, he saw that there was limited damage to part of the cerebral cortex around the central sulcus. He suggested that this indicated that there was a discrete representation of movements of diVerent body parts in this area, and that ‘‘irritation’’ could produce movements of the corresponding part of the contralateral body. He further noted that some parts were likely to have a larger or more excitable eVect than others, explaining the propensity for twitches to begin in the hands or face. His ideas were confirmed later by Fritsch and Hitzig and David Ferrier in the 1870s, who showed that electrical stimulation of the central area in dogs and monkeys could produce movements of the opposite side of the body. Movements of diVerent parts of the body were produced by diVerent locations of the stimulating electrode, with the lowest threshold eVects being observed in the distal limbs. Bartholow carried the first stimulation of the human motor cortex out only a few years later in a patient whose cortex was exposed by a large ulcer on her scalp. These experiments defined the motor cortex as the area from which movements could be elicited at lowest intensity. Within this area there was a map of the body in which movements of the legs were represented medially, with the trunk, arms and face progressively more lateral. As predicted by Hughlings Jackson, movements of the lower face and hands were much more readily evoked, and from a wider area of cortex, than movements of other parts of the body.
58
JOHN ROTHWELL
This arrangement was later popularized in the now-familiar motor ‘‘homunculus’’ drawn by Penfield and coworkers.
B. PRESENT DAY ANATOMICAL AND ELECTROPHYSIOLOGICAL DEFINITIONS OF THE MOTOR AREAS OF CORTEX Neurophysiologists now recognize that there are several representations of the body in what are now termed the ‘‘motor areas’’ of the cerebral cortex. Anatomically, these occupy Brodmann’s areas 4 and 6 on the lateral and medial surfaces of the hemispheres anterior to the central sulcus, together with areas 23 and 24 in the cingulate gyrus. In monkey experiments, electrical stimulation over these areas has revealed a total of seven diVerent representations of the contralateral body. The primary motor cortex has been studied in most detail. This occupies area 4 of Brodmann, which is mostly buried in the anterior bank of the central sulcus. It is distinguished from adjacent sensory areas by its lack of a pronounced cortical layer IV (agranular cortex). It diVers from area 6 by the presence of large pyramidal neurons in layer V (Betz cells). Electrical stimulation of the primary motor cortex has a lower threshold than any other motor area, and produces twitch-like movements of a small number of muscles in the contralateral body. These may be, for example, a flick of the fingers or a twitch of biceps or the corner of the mouth depending on the point of stimulation. The location of the primary motor cortex is frequently mapped out during neurosurgical operations in man. Patients are often awake during such operations, and they point out that the movements feel involuntary, as if imposed by an external force. The implication is that awareness of the eVort of a voluntary movement must arise in other areas of cortex. Patients also note that during stimulation they feel unable to move that part of the body. Presumably activation of the cortex by electrical current prevents patients from using that area in voluntary movements. About one-third of the primary motor cortex is devoted to control of the hand. MRI images show that in most subjects, this region is marked by folding of the central sulcus into an ‘‘omega’’ shape when viewed from the surface. It is a rare example of an anatomical marker for a specific cortical function (Fig. 3). In the monkey two further representations of the body are found anterior to area 4 in the lateral part of area 6 around the arcuate sulcus. These are known as the dorsal and ventral premotor areas. Stimulation here has a higher threshold than for area 4 and provokes more complex movements, often involving more than one part of the body simultaneously. This area has not been extensively studied in humans. One problem is to define the limits of human premotor areas. The precentral sulcus is thought to be the human analogue of the arcuate sulcus, yet human area 6 extends further anterior to this point than it does in the monkey.
59
NEUROPHYSIOLOGY
Cingulate
l
SPC
cu
at
e
ntr al
Ar
Ce
Princ ipal
ta
rie
pa
6
8
4
ra nt
I
6 PCS
CS
44
CMA
SMA PMd
PMv
MCx
Dorsal compartment of area 6 on superior and middle frontal gyrus (PMC d) Ventral compartment of area 6 on precentral gyrus (PMC v) PCS Precentral sulcus CS Central sulcus
FIG. 3. Diagrammatic summary of some of main motor areas of the macaque cortex (left) and homologous areas on the lateral surface of the hemisphere in human (right). The macaque brain on the left shows both the lateral and medial walls of the hemisphere, with the upper panel labeled to show the names of the principal sulci, and the lower panel labeled to indicate the approximate areas of the primary motor cortex (MCx), dorsal and ventral premotor cortex (PMd, PMd), supplementary motor area (SMA), and cingulate motor areas (CMA). The dotted lines indicate the bottom of the central and cingulated sulci. The human brain on the right shows only the lateral surface and indicates the approximate extent of the premotor and primary motor cortices.
Another problem is that there have been no detailed mapping studies to compare with the monkey work. In fact, the original somatotopic maps of human cortex mark this region as part of the trunk representation. The medial portion of area 6 anterior to the leg representation in the primary motor cortex comprises the supplementary area (SMA). This is organized with the legs posterior, adjacent to the primary leg area, and the arms and face anterior. The eVects of stimulation are relatively well described in humans. The threshold is higher than for the primary motor cortex, and the movements more complex, often involving for example combined turning of the head and extension of the arm. Bilateral movements and vocalization can also be produced. In the last 10 years three more motor representations have been described around the cingulate gyrus, approximately ventral to the SMA. These lie in parts of area 6, 23, and 24 of Brodmann, and called the dorsal, ventral, and rostral cingulate motor areas (CMAd, CMAv, CMAr). The main evidence for these representations came initially from anatomical studies that showed they had direct projections to the spinal cord. Electrical stimulation studies are rare,
60
JOHN ROTHWELL
although it has been confirmed in humans that at least one similar representation of the body lies in the cingulate gyrus at this level. Stimulation produced tonic extension of the arm or the leg.
C. OUTPUT OF CORTICAL MOTOR AREAS All the cortical motor areas have a direct output to the spinal cord. This is composed of axons from pyramidal neurons in cortical layer V, which run in the corticospinal (or pyramidal) tract and innervate all levels of the contralateral spinal cord. In humans there are about half a million fibers in the corticospinal tract on each side. Two percent of these are large diameter and conduct impulses rapidly at speeds of up to 80 m/s. They are probably the axons of the large Betz cells of primary motor cortex. The majority of corticospinal fibers are smaller diameter and slower conducting (10–30 m/s). The fibers run in the lateral and anterior columns with the majority traveling in the former. Most of the projection is contralateral, although a small percentage (estimated at about 10%), particularly fibers in the anterior columns, run ipsilaterally. Terminations are mostly onto interneurons in the gray matter of the intermediate zone. However, especially in primates and man, there are extensive monosynaptic projections directly to spinal motoneurons in lamina IX. This projection is particularly prominent to distal muscles of the forearm and hand and is thought to be one factor that contributes to the increased dexterity of humans compared with other species. The pattern of corticospinal projections to antagonist and synergist muscles has been examined in detail for muscles of the hand and arm. In most cases, corticospinal connections excite close synergists and often inhibit the antagonists. However, there are also smaller numbers of neurons that excite agonist and antagonist muscles simultaneously. It is thought that this could be of use when it is necessary to stabilize a joint, such as is needed, for example at the wrist when individual finger movements are made. Although the corticospinal tract is large, it is important to remember that motor cortical areas also communicate with spinal motoneurons via projections to nuclei in the brainstem. Some of these are collaterals of corticospinal fibers, while some project to the brainstem only. These brainstem nuclei have descending fibers that form the reticulospinal tracts and innervate all segments of the cord, sometimes bilaterally. The density of these cortico-reticulospinal projections is higher from premotor, SMA and cingulate motor areas than it is from the primary motor cortex. The relative roles of corticospinal and noncorticospinal pathways are illustrated by experiments in which the corticospinal fibers are surgically cut. This has been performed several times in monkey by lesioning the pyramids in
NEUROPHYSIOLOGY
61
the medulla; in man corticospinal fibers have been cut for relief of involuntary movements by lesioning the middle third of the cerebral peduncle. In both cases, the loss of a million or so fibers is accompanied by little evidence of gross movement deficit. There is no spasticity and little weakness. The main deficit is in control of manipulative movements of the hands: the fingers are no longer used independently and precision grip is lost. The implication is that although the corticospinal system is important for fine distal movement, the noncorticospinal projections are of equal or even greater importance for other types of movement. When projections from cortex to the brainstem nuclei are lesioned, as happens in a capsular stroke, these noncorticospinal projections lose their input from the cerebral cortex. The resulting movement deficit is much greater than seen after pure pyramidal lesions. Motor areas of the cortex also send projections to pontine nuclei that innervate the cerebellum and to the ascending sensory systems in the gracile and cuneate nuclei (in both cases, often as collateral of corticospinal fibers). The cerebellar projection provides a copy of the motor command that could potentially be used to update movement more quickly than relying on sensory feedback. The projection to sensory nuclei is important in controlling the flow of sensory information during movement. D. INPUTS TO CORTICAL MOTOR AREAS The motor areas are also distinguished by diVerences in the inputs they receive from other parts of the brain. The nature of these inputs is presumably an important factor in determining the contribution of each area to specific types of movement (see section below). The primary motor cortex receives input mainly from sensory cortex, and from areas of thalamus that receive input from the cerebellum, and to a lesser extent, the basal ganglia. There are also extensive connections both to and from the other motor areas. The premotor areas receive a major input from regions of the posterior parietal cortex that are involved in the combined processing of visual and somatosensory input, as well as input from cerebellum via the thalamus. The SMA receives a large input from parietal cortex and from thalamic nuclei that receive input from basal ganglia. Cingulate motor areas are thought to have extensive connections with regions in the frontal lobes.
E. ACTIVITY OF MOTOR CORTICAL NEURONS The pattern of input is reflected in the way neurons in each area contribute to diVerent parts of the preparation and execution of movement. Neurophysiological recordings of cell discharge in behaving animals show that neurons in primary motor cortex change their firing rate just before and during a movement.
62
JOHN ROTHWELL
Neurons with monosynaptic connections to motoneurons tend to behave much like the muscles that they drive, being active just before and during contraction. The relation of cortical interneurons or pyramidal neurons without direct connections to spinal motoneurons tends to be more complex. Their activity may reflect stages in cortical processing or the input to spinal interneurons rather than motoneurons. In general, activity in these neurons occurs around the time of movement and tends to be related to parameters such as the force of muscle contraction or the direction of the movement. The activity of neurons in SMA and premotor areas is more likely to be related to preparation, rather than execution of movement. The neurons here may discharge well before a movement as if they were specifying what movement is to be made next rather than specifying the nature of the movement currently underway. There appear to be few diVerences in the contribution of SMA and premotor cortex to simple tasks, such as pushing a button when a light comes on. However, activity in these two areas diVers substantially in more complex tasks, such as those requiring a sequence of movements. Premotor neurons are more likely to change their activity when visual cues are being used to guide the sequence of movements while SMA activity is more intense when the sequence is made from memory, without visual cues. An example would be to point at a series of lights as they are illuminated in random order, or to point to the same positions from memory. The former is said to be an ‘‘externally guided’’ movement and involves activity in premotor cortex; the latter is an ‘‘internally generated’’ movement and involves activity in the SMA. The main input that drives externally guided movements comes from parietal cortex. It can be imagined that processed sensory input from this area can help the premotor regions to select appropriate movements from a set of stored commands. Indeed, there are neurons in this area that have been labeled ‘‘mirror’’ neurons, because they discharge both when a monkey makes a particular movement and also when it observes another animal make the same movement. Such neuronal types could obviously help us to learn movements by observing them.
F. EFFECT OF LESIONS OF MOTOR CORTICAL AREAS Small lesions of the primary motor cortex produce transient weakness that often resolves. This is presumably because neighboring areas of cortex can compensate for the lost function of the damaged area (see Section V ). Larger lesions, particularly if they involve premotor areas, result in permanent weakness and spasticity. The latter is thought to be a consequence of removal of input to reticular centers of the brainstem that influence muscle tone. Pure lesions of the premotor and SMA areas are rare in humans but have been studied extensively in monkey. Lesions of premotor areas aVect the ability of the monkey to
NEUROPHYSIOLOGY
63
retrieve the correct movement on the basis of external cues, whereas SMA lesions aVect the ability of monkeys to perform a series of movements from memory (‘‘internally guided’’ movements). SMA lesions also aVect coordination in actions involving object manipulation in both hands. In humans, rare patients with limited lesions of presumed premotor areas also have diYculty in associating visual cues with particular movements. They cannot learn, for example, to produce diVerent hand postures when they view particular visual symbols. Lesions of the SMA in humans are most commonly associated with diYculty in initiating or maintaining speech, a feature that is not evident in the monkey data. In addition patients with larger midline lesions may exhibit the ‘‘alien hand’’ sign, a condition in which the hand contralateral to the lesion behaves independently of the subject’s will, and may reach out to grasp objects within reach. This can be interpreted as a possible uncoupling of the externally guided premotor system from the internally guided SMA system. Finally, the movements of patients with SMA lesions, even after apparently good recovery, are often slower than normal. Interestingly, bradykinesia is a principal symptom of Parkinson’s disease, and since the SMA is an important output target of the basal ganglia, slowness of movement after a lesion may be due to lack of basal ganglia input to the motor system.
V. Plasticity of Primary Motor Cortex
It has been known for many years that electrical stimulation at the same site in primary motor cortex does not always produce exactly the same movement on every occasion over a period of several hours. However, this capacity of the cortex to change its organization, even in the adult, has only been investigated in detail in the last 10–20 years. For example, in rats, there is a large representation of the whiskers in the primary motor cortex. In the somatotopic representation of the rats’ body, this area is bordered by a region representing the periocular muscles and a region representing the arm muscles. If the VII nerve is lesioned, the rats can no longer use the whiskers. In such circumstances, what happens to that area of motor cortex that was devoted to control of the whiskers? Does it become a ‘‘silent’’ area, from which no movements of any sort can be obtained, or does reorganization occur so that it can contribute to movement of other body parts? In fact the latter seems to be true: stimulation of sites that previously had evoked whisker movement now give rises to movements of the periocular or arm muscles. These changes occur so rapidly that they cannot involve growth of new connections or synapses. Instead it is thought that they rely on changes in horizontal connections between cortical areas. Thus, after VII nerve section, the excitability of connections from the whisker area to other regions becomes more excitable.
64
JOHN ROTHWELL
Electrical stimulation of the whisker area activates these connections and evokes movements of the arm or periocular muscles indirectly via adjacent cortex. The excitability of the horizontal connections that are involved in this remodeling of cortical representation is controlled by GABAergic inhibition. This inhibitory system in turn is regulated by sensory information about the state of the peripheral motor apparatus. Similar remodeling occurs if the damage is in the motor cortex rather than the periphery. After small lesions, healthy surrounding areas can evoke movements previously elicited by stimulation of the damaged part. It is particularly interesting to note that such reorganization is enhanced if animals undergo training of the aVected part of the body. The implication is that physiotherapy may have an important influence in outcome from cortical damage. The situation with larger lesions is unexplored. However, horizontal connections in the cortex are short (2 mm or so) and often do not cross boundaries between cortical areas. Thus recovery from large lesions may require a diVerent from of reorganization. Finally, there is good evidence for similar patterns of reorganization in human primary motor cortex. The recent technique of transcranial magnetic stimulation, in which a magnetic field is used to induce stimulating electrical currents in the brain, allows coarse mapping of the motor cortex to be carried out in intact conscious subjects. Mapping the cortex of amputees, for example, shows that stimulation of the area previously controlling movements of the lost limb can evoke movement in immediately adjacent parts of the body. As in the rat experiments, it seems as if the pattern of representation of the body on the cortex can change after injury. Indeed, recent experiments suggest that there may even be changes in the pattern of representation in motor cortex when subjects learn new tasks. It appears as if the motor cortex map is maintained by a dynamic balance in the excitability of short corticocortical connections.
VI. Conclusions
In conclusion, control of movement depends on the coordinated action of a large number of anatomically separate but interconnected areas of the brain and spinal cord that operate in parallel to determine the final movement outcome. From the viewpoint of a brain–machine interface, this design poses a problem since there is no one point where the final command is represented apart from at its final convergence at the spinal motoneuron. In the case of voluntary movements of the hand and arm, however, a close second point of convergence is the primary motor cortex which is the source of much of the command for grasping and reaching movements. Recording activity here may give a good representation of the final movement that is intended by the brain. However, other types of
NEUROPHYSIOLOGY
65
movement, perhaps involving postural control of legs and trunk are more likely to depend on integration of many inputs from other subcortical areas of the motor system that may be less accessible to noninvasive recording devices.
References
Porter, R., and Lemon, R. N. (1993). ‘‘Corticospinal Function and Voluntary Movement.’’ Oxford University Press, Oxford. Rizzolatti, G., and Luppino, G. (2001). The cortical motor system. Neuron 31, 889–901. Rothwell, J. C. (2001). First studies of the organisation of the human motor cortex. In ‘‘Classics in Movement Science’’ (M. L. Latash and V. M. Zatisiorsky, Eds.), pp. 273–288. Human Kinetics, Champaign, IL. Sanes, J. N., and Donoghue, J. P. (2001). Plasticity and primary motor cortex. Ann. Rev. Neurosci. 23, 393–415.
FUNDAMENTALS OF ELECTROENCEFALOGRAPHY, MAGNETOENCEFALOGRAPHY, AND FUNCTIONAL MAGNETIC RESONANCE IMAGING
Claudio Babiloni,*,z Vittorio Pizzella,y Cosimo Del Gratta,y Antonio Ferretti,y and Gian Luca Romaniy y
*Department of Biomedical Sciences, University of Foggia, Foggia, Italy Department of Clinical Sciences and Biomedical Imaging, Institute for Advanced Biomedical Technologies, University G. D’Annunzio of Chieti, Chieti, Italy z Hospital San Raffaele Cassino, Cassino, Italy
I. II. III. IV. V. VI. VII.
Introduction to Electroencephalography and Magnetoencephalography Physiological Generation of EEG/MEG Signals EEG and MEG Techniques Allow the Study of Brain Rhythms Functional Magnetic Resonance Imaging Physiological Generation of Blood Oxygen Level-Dependent Signal Typical f MRI Experimental Designs BOLD-f MRI Techniques in Clinical Environment References
This review introduces readers to fundamentals of electroencephalography (EEG), magnetoencephalography (MEG), and functional magnetic resonance imaging (fMRI). EEG and MEG signals are mainly produced by postsynaptic ionic currents of synchronically active pyramidal cortical neurons. These signals reflect the integrative information processing of neurons representing the output of cortical neural modules. EEG and MEG signals have a high temporal resolution (1 s) and quite high
INTERNATIONAL REVIEW OF NEUROBIOLOGY, VOL. 86 DOI: 10.1016/S0074-7742(09)86005-4
67
Copyright 2009, Elsevier Inc. All rights reserved. 0074-7742/09 $35.00
68
BABILONI et al.
spatial resolution (40 Hz, gamma rhythms). Due to these eVects, the amplitude of EEG potentials at a given scalp site cannot reflect the intensity of neural currents in the underlying cerebral generators, posing the problematic issue of the estimation of brain EEG sources. To solve the problem of low spatial resolution and high-frequency filtering of EEG activity, brain potentials can be recorded by thin electrodes inserted into the human head when some special clinical conditions are present. For example, intracranial EEG recordings are used to localize the brain regions generating seizures in epilepsy patients who do not respond to pharmacologic therapy and need neurosurgery. For obvious ethical reasons, the event of intracranial EEG is quite rare, so the above important limitations of EEG techniques have motivated the development of sophisticated mathematical approaches to take into account the eVects of electrical reference and head volume conduction. Furthermore, techniques have been
FUNDAMENTALS OF EEG, MEG, AND FMRI
69
developed to record the magnetic component of brain electromagnetic activity, the so-called magnetoencephalography (MEG). Specifically, MEG technique detects the magnetic field generated by brain electrical activity. MEG activity has been first observed in the late 1960s, but only later it became feasible thanks to the development of superconducting quantum interference devices (SQUIDs). These superconducting devices are extremely sensitive magnetic flux to voltage converters, with an ultimate sensitivity limited only by quantum limit. As a matter of fact, the magnetic field generated by brain electric activity (neuromagnetic field) is extremely weak, often below 100 f T (100 f T ¼ 1013 T ), while the detectors today used in MEG devices (low-Tc SQUIDs operate at 4.2 K, the liquid helium temperature) feature a sensitivity of few f T/sqrt(Hz). Moreover, the neuromagnetic field is much weaker than the ordinary magnetic fields generated by the earth (104 T) or by the 50/60 Hz current flowing inside power lines (about 108 T). The use of a heavy magnetical shielded room is thus required to attenuate these magnetic fields in the sensor area. These technical requirements make the MEG device a relatively expensive one, with an overall present price of more than one million Euro compared to few tens of thousands of Euro needed for EEG equipment. Similarly to EEG, the MEG technique is absolutely noninvasive.
II. Physiological Generation of EEG/MEG Signals
EEG and MEG signals are very large-scale measure of brain source activity, apparently recording synaptic activity synchronized over macroscopic (centimeter), regional, and even whole brain spatial scales (Nunez et al., 2001). Synchrony among neural populations in compact regions of the brain produces localized dipole current sources. Synchrony among neural populations distributed across the cortex can give rise to regional or global networks consisting of many dipole sources. EEG and MEG signals both derive from electric activity of neurons of cerebral cortex. The human cerebral cortex is formed by small macrocolumns (diameter of
