Enhancing Performance for Action and perception - Multisensory Integration, Neuroplasticity and Neuroprosthetics, Part II, Volume 192 (Progress in Brain Research)

SERIES EDITORS

STEPHEN G. WAXMAN Bridget Marie Flaherty Professor of Neurology Neurobiology, and Pharmacology; Director, Center for Neuroscience & Regeneration/Neurorehabilitation Research Yale University School of Medicine New Haven, Connecticut USA

DONALD G. STEIN Asa G. Candler Professor Department of Emergency Medicine Emory University Atlanta, Georgia USA

DICK F. SWAAB Professor of Neurobiology Medical Faculty, University of Amsterdam; Leader Research team Neuropsychiatric Disorders Netherlands Institute for Neuroscience Amsterdam The Netherlands

HOWARD L. FIELDS Professor of Neurology Endowed Chair in Pharmacology of Addiction Director, Wheeler Center for the Neurobiology of Addiction University of California San Francisco, California USA

Elsevier Radarweg 29, PO Box 211, 1000 AE Amsterdam, The Netherlands Linacre House, Jordan Hill, Oxford OX2 8DP, UK 360 Park Avenue South, New York, NY 10010-1710 First edition 2011 Copyright Ó 2011 Elsevier B.V. All rights reserved No part of this publication may be reproduced, stored in a retrieval system or transmitted in any form or by any means electronic, mechanical, photocopying, recording or otherwise without the prior written permission of the publisher Permissions may be sought directly from Elsevier’s Science & Technology Rights Department in Oxford, UK: phone (þ44) (0) 1865 843830; fax (þ44) (0) 1865 853333; email: [email protected]. Alternatively you can submit your request online by visiting the Elsevier web site at http://elsevier.com/locate/permissions, and selecting Obtaining permission to use Elsevier material Notice No responsibility is assumed by the publisher for any injury and/or damage to persons or property as a matter of products liability, negligence or otherwise, or from any use or operation of any methods, products, instructions or ideas contained in the material herein. Because of rapid advances in the medical sciences, in particular, independent verification of diagnoses and drug dosages should be made Library of Congress Cataloging-in-Publication Data A catalog record for this book is available from the Library of Congress British Library Cataloguing in Publication Data A catalogue record for this book is available from the British Library ISBN: 978-0-444-53355-5 ISSN: 0079-6123

For information on all Elsevier publications visit our website at elsevierdirect.com

Printed and bound in Great Britain 11 12 13 14

10 9 8

7

6 5

4

3

2 1

List of Contributors D. Barthélemy, School of Rehabilitation, Faculty of Medicine, Université de Montréal, Montréal, Québec, Canada N. Birbaumer, University of Tübingen, Tübingen, Germany L. Bouyer, CIHR Multidisciplinary Team in Locomotor Rehabilitation; Centre for Interdisciplinary Research in Rehabilitation and Social Integration, IRDPQ, and Department of Rehabilitation, Faculty of Medicine, Université Laval, Québec, Canada S.M. Chase, Department of Neurobiology and the Center for the Neural Basis of Cognition, University of Pittsburgh, Pittsburgh, Pennsylvania, USA M.M. Churchland, Department of Electrical Engineering, and Neurosciences Program, Stanford University, Stanford, California, USA N. Dancause, Département de Physiologie, Université de Montréal, Montréal, Québec, Canada B. Darbandi, Department of Biomedical Engineering, The Johns Hopkins University, Baltimore, Maryland, USA C. Duclos, Pathokinesiology Laboratory, Centre for Interdisciplinary Research in Rehabilitation, Institut de réadaptation Gingras-Lindsay-de-Montréal; School of Rehabilitation, Université de Montréal, Montréal, and CIHR Multidisciplinary Team in Locomotor Rehabilitation, Québec, Canada A. Fedorov, Institute of Medical Psychology, Medical Faculty, Otto-von-Guericke-University Magdeburg, Magdeburg, Germany M. Fettiplace, Department of Biomedical Engineering, The Johns Hopkins University, Baltimore, Maryland, USA J.A. Fishel, Department of Biomedical Engineering, University of Southern California, Los Angeles, California, USA C. Gall, Institute of Medical Psychology, Medical Faculty, Otto-von-Guericke-University Magdeburg, Magdeburg, Germany M. Goyette, Institut Philippe-Pinel de Montréal, Université de Montréal, Montréal, Québec, Canada M.J. Grey, School of Sport and Exercise Sciences, University of Birmingham, Edgbaston, Birmingham, United Kingdom P. Henrich-Noack, Institute of Medical Psychology, Medical Faculty, Otto-von-Guericke-University Magdeburg, Magdeburg, Germany N. Hogan, Department of Mechanical Engineering, and Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, Massachusetts, USA S.S. Hsiao, Department of Neuroscience and the Krieger Mind/Brain Institute, The Johns Hopkins University, Baltimore, Maryland, USA L.A. Jones, Department of Mechanical Engineering, Massachusetts Institute of Technology, Cambridge, Massachusetts, USA C. Joyal, Université du Québec à Trois-Rivières, Institut Philippe-Pinel de Montréal, Montréal, Québec, Canada v

vi

M.T. Kaufman, Neurosciences Program, Stanford University, Stanford, California, USA J. Kowalczewski, Centre for Neuroscience, School of Molecular and Systems Medicine, University of Alberta, Edmonton, Alberta, Canada H.I. Krebs, Department of Mechanical Engineering, Massachusetts Institute of Technology, Cambridge, Massachusetts, and Department of Neurology, University of Maryland School of Medicine, Baltimore, USA R. Kupers, Institute of Neuroscience and Pharmacology, Panum Institute, University of Copenhagen, Copenhagen, Denmark G.E. Loeb, Department of Biomedical Engineering, University of Southern California, Los Angeles, California, USA L.B. Merabet, Department of Ophthalmology, Massachusetts Eye and Ear Infirmary, Harvard Medical School, and The Boston Retinal Implant Project, Center for Innovative Visual Rehabilitation, Boston VA Medical Center, Boston, Massachusetts, USA L.E. Miller, Department of Physiology; Department of Physical Medicine and Rehabilitation, Feinberg School of Medicine, Northwestern University, Chicago, and Department of Biomedical Engineering, Northwestern University, Evanston, Illinois, USA S. Nadeau, Pathokinesiology Laboratory, Centre for Interdisciplinary Research in Rehabilitation, Institut de réadaptation Gingras-Lindsay-de-Montréal; School of Rehabilitation, Université de Montréal, Montréal, and CIHR Multidisciplinary Team in Locomotor Rehabilitation, Québec, Canada J.B. Nielsen, Department of Exercise and Sport Sciences, and Department of Neuroscience and Pharmacology, Panum Institute University of Copenhagen, Copenhagen N, Denmark R.J. Nudo, Landon Center on Aging, Kansas University Medical Center, Kansas City, Kansas, USA A. Prochazka, Centre for Neuroscience, School of Molecular and Systems Medicine, University of Alberta, Edmonton, Alberta, Canada M. Ptito, Chaire de recherche Harland Sanders en Sciences de la vision, École d'Optométrie, Université de Montréal, Montréal, Québec, Canada J.M. Rebesco, Department of Physiology, Feinberg School of Medicine, Northwestern University, Chicago, Illinois, USA P. Renaud, Université du Québec en Outaouais, Institut Philippe-Pinel de Montréal, Montréal, Québec, Canada C.L. Richards, CIHR Multidisciplinary Team in Locomotor Rehabilitation; Centre for Interdisciplinary Research in Rehabilitation and Social Integration, IRDPQ, and Department of Rehabilitation, Faculty of Medicine, Université Laval, Québec, Canada B.A. Sabel, Institute of Medical Psychology, Medical Faculty, Otto-von-Guericke-University Magdeburg, Magdeburg, Germany M. Sahani, Department of Electrical Engineering, Stanford University, Stanford, California, USA, and Gatsby Computational Neuroscience Unit, University College London, London, United Kingdom S. Schaal, Department of Biomedical Engineering, and Department of Computer Science, University of Southern California, Los Angeles, California, USA A.B. Schwartz, Department of Neurobiology and the Center for the Neural Basis of Cognition, University of Pittsburgh, Pittsburgh, Pennsylvania, USA K.V. Shenoy, Department of Electrical Engineering; Neurosciences Program; Department of Bioengineering, and Department of Neurobiology, Stanford University, Stanford, California, USA

vii

S. Stoleru, INSERM 669, Paris, France G.A. Tsianos, Department of Biomedical Engineering, University of Southern California, Los Angeles, California, USA N. Weiskopf, University College London, London, United Kingdom N. Wettels, Department of Biomedical Engineering, University of Southern California, Los Angeles, California, USA

Preface Basic neuroscience research over the past several decades has defined many of the central neuronal mechanisms underlying the functioning of the various sensory and motor systems. These systems work together in a cooperative fashion: perception is required to plan action, while movements often serve to acquire desired sensory inputs (e.g., eye movements to capture a visual target of interest). Performance levels reach their apogee in examples ranging from high-performance athletes to concert-level pianists, with both requiring highly refined sensorimotor abilities. There is now considerable interest in how the brain adapts its functioning in health and disease. Extensive progress has been made in understanding how use and disuse influence motor and sensory performance, and the underlying mechanisms of neuronal plasticity responsible for adaptive changes. Increasingly, basic scientists working in these fields are being challenged to translate this scientific knowledge into applications that provide new and innovative methods to restore lost function in humans following injury or disease (e.g., amputation, myopathies, neuropathies, spinal cord lesions, or stroke). In recent years, scientists have risen to this challenge, collaborating with engineers and clinicians to help develop novel technologies. These have progressed to the point where devices such as cochlear implants are now commonly used in clinical practice, while applications such as neuroprosthetic devices controlled by the brain are rapidly becoming a realistic possibility for restoring lost motor function. This two-volume set of books is the result of a symposium, inspired by these new initiatives, that was held at the Université de Montréal on May 10–11, 2010 (see http://www.grsnc.umontreal.ca/32s/). It was organized by the Groupe de Recherche sur le Système Nerveux Central (GRSNC) as one of a series of annual international symposia held on a different topic each year. The symposium included presentations by world-renowned experts working on the neuronal mechanisms that play critical roles in learning new motor and sensory skills in both health and disease. The objective was to provide an overview of the various neural mechanisms that contribute to learning new motor and sensory skills as well as adapting to changed circumstances including the use of devices and implants to substitute for lost sensory or motor abilities (neural prosthetics). The symposium emphasized the importance of basic science research as the foundation for innovative technological developments that can help restore function and improve the quality of life for disabled individuals. It equally emphasized how such new technologies can contribute to our basic scientific understanding of the neural mechanisms of sensorimotor control and adaptation. Many of the participants of that meeting have contributed chapters to this book, including symposium speakers and poster presenters. In addition, we invited a number of other well-known experts who could not participate in the conference itself to submit chapters. This two-volume collection of over 30 chapters can only cover a fraction of the topics and extensive range of work that pertains to adapting our motor and sensory systems to changed conditions and to the development of technologies that substitute for lost abilities. However, it does address a range of motor functions and sensory modalities; considers adaptive changes at both behavioral and neurophysiological levels; and presents perspectives on basic research, clinical approaches, and technological innovations. The result is a collection that includes chapters broadly separated into five key themes: (1) mechanisms to enhance motor performance, (2) mechanisms to enhance sensory perception, 3) multisensory interactions to enhance action and perception, (4) assistive technologies to enhance sensorimotor performance, and (5) neurorehabilitation. Volume I focuses on the basic mechanisms underlying ix

x

performance changes (themes 1–3). The current volume (Volume II) complements the first volume by focussing on the translation of this scientific knowledge into technological applications and clinical strategies that can help restore lost function and improve quality of life following injury or disease (themes 4 and 5). The conference and this book would not have been possible without the generous support of the GRSNC, the Fonds de la Recherche en Santé de Québec (FRSQ), the Canadian Institutes of Health Research (CIHR), the Institute of Neuroscience, Mental Health and Addiction (CIHR), the Institute of Musculoskeletal Health and Arthritis (CIHR), and the Faculty of Arts and Sciences and Faculty of Medicine of the Université de Montréal. We gratefully acknowledge these sponsors as well as our contributing authors who dedicated their time and effort to present their perspectives on the neural mechanisms and technological advances that enhance our performance for action and perception. Andrea M. Green C. Elaine Chapman John F. Kalaska Franco Lepore

A. M. Green, C. E. Chapman, J. F. Kalaska and F. Lepore (Eds.) Progress in Brain Research, Vol. 192 ISSN: 0079-6123 Copyright Ó 2011 Elsevier B.V. All rights reserved.

CHAPTER 1

Building the bionic eye: an emerging reality and opportunity Lotfi B. Merabet{,{,* {

{

Department of Ophthalmology, Massachusetts Eye and Ear Infirmary, Harvard Medical School, Boston, Massachusetts, USA The Boston Retinal Implant Project, Center for Innovative Visual Rehabilitation, Boston VA Medical Center, Boston, Massachusetts, USA

Abstract: Once the topic of folklore and science fiction, the notion of restoring vision to the blind is now approaching a tractable reality. Technological advances have inspired numerous multidisciplinary groups worldwide to develop visual neuroprosthetic devices that could potentially provide useful vision and improve the quality of life of profoundly blind individuals. While a variety of approaches and designs are being pursued, they all share a common principle of creating visual percepts through the stimulation of visual neural elements using appropriate patterns of electrical stimulation. Human clinical trials are now well underway and initial results have been met with a balance of excitement and cautious optimism. As remaining technical and surgical challenges continue to be solved and clinical trials move forward, we now enter a phase of development that requires careful consideration of a new set of issues. Establishing appropriate patient selection criteria, methods of evaluating long-term performance and effectiveness, and strategies to rehabilitate implanted patients will all need to be considered in order to achieve optimal outcomes and establish these devices as viable therapeutic options. Keywords: retinal implant; neuroprosthesis; blind; retina; rehabilitation.

Introduction: history, an unmet demand, and state of the art

We can rebuild him. . .we have the technology. —from the television series “The Six Million Dollar Man”

Our fascination with building a bionic human mirrors the technological advances that ubiquitously characterize the modern era. Today, this idea has become less the subject of science fiction and more the pursuit of intense scientific research. Advances

*Corresponding author. Tel.: þ (617) 573-4130; Fax: þ (617) 573-4178 E-mail: [email protected] DOI: 10.1016/B978-0-444-53355-5.00001-4

3

4

within the realms of microfabrication, microelectronics, material science, wireless technology, and high-speed computer processing power have allowed for the development of neuroprosthetic devices designed to assist individuals living with sensory loss and/or motor impairment. The basic premise underlying all neuroprosthetic approaches is that targeted and controlled delivery of electrical stimulation to nerves or muscles can potentially restore (to a certain degree) the physiological function of a damaged organ or limb (Marbach et al., 1982). The success of cochlear implants, developed over 30 years ago, serves as a well-known example. This neuroprosthetic device has helped thousands of profoundly deaf individuals regain hearing and develop speech communication (Jones et al., 2008; Loeb, 1990). Similarly, sophisticated artificial limbs have led to improved walking mobility and even grasping skills for amputees (Allin et al., 2010; Craelius, 2002; Laferrier and Gailey, 2010). The continued development of brain–machine interfaces (BMIs) is also providing exciting hope for paralyzed patients. By recording neuronal signals from the brain that code for movement, these signals can be converted and used to control external devices such as a robotic limb prosthesis (Donoghue, 2002; Hochberg et al., 2006; Nicolelis, 2003). Rapid progress in all of these arenas continues and in many ways serves as inspiration for the development of a visual neuroprosthesis for the blind. Today, several device designs and approaches are being developed and human clinical trials are well underway (for extensive reviews see: Chader et al., 2009; Dagnelie, 2006; Dowling, 2005; Humayun, 2007; Javaheri et al., 2006; Merabet et al., 2005). According to the World Health Organization, there are 314 million visually impaired individuals worldwide (2009 WHO fact sheet; http://www. who.int/mediacentre/factsheets/fs282/en/). While an astonishingly large number, it is worthy to note that only a minority of individuals ( 45 million) are actually considered profoundly blind (defined as best-corrected visual acuity worse than 20/400 Snellen acuity) and have some degree of residual visual function. Further, a tragic reality

exists. The greatest portion of these individuals live in developing countries and the majority of the leading causes of blindness are actually avoidable and/or treatable (e.g., surgery for cataracts or antibiotic treatment for trachoma). Thus, the restoration of functional vision through a visual prosthesis will likely target only a restricted segment of the blind population. Moreover, it is important to realize that not all individuals and all forms of visual impairment could potentially benefit from a visual neuroprosthesis. As presently conceived, visual prosthetic devices have been designed for individuals with profound vision loss and who have had normal visual development (as opposed to congenital causes of blindness). Further, as these devices are designed to interface with viable neuronal tissue, the site of damage and nature of pathology will largely dictate whether a prosthetic device can be feasibly implemented. For example, in conditions where the overall functional and structural integrity of the retina is compromised (e.g., trauma, glaucoma, or retinal complications related to diabetes), a retinal-based visual prosthesis is unlikely to be effective in restoring visual function (see discussion on various visual prosthesis approaches). These limitations notwithstanding, it is also important to highlight advances being made in other areas of biomedical research such as gene therapy and cell transplantation. These molecular-based approaches may in time provide new treatments and help halt the progression of vision loss particularly with respect to hereditary causes of blindness (Acland et al., 2001; MacLaren et al., 2006). At the same time, blind individuals will continue to benefit from the use of sensory-substitution devices (also discussed in this edition). These devices are specially designed to leverage sensory information obtained from the intact senses (e.g., hearing and touch) to substitute for the vision. This allows a blind user to interact with his or her surrounding environment (Bach-y-Rita, 2004; Bach-y-Rita and Kercel, 2003). Thus, the future rehabilitation of individuals with visual impairment will likely continue to encompass multidisciplinary approaches

5

and include molecular-based therapies designed to halt the progression of vision loss, the use of sensory-substitution devices, and potentially restore a certain level of functional vision through the use of visual neuroprosthetic devices. Here, we will highlight advances and discuss future perspectives relating to visual neuroprosthesis development. Summary of visual neuroprosthetic approaches: from retina to cortex Generally speaking, the operating premise underlying a visual neuroprosthesis is to artificially replace the function of damaged neuronal elements

that make up the visual pathway (Fig. 1). Typically, patterned microelectrical stimulation is delivered through an array of tiny microelectrodes to elicit the perception of organized patterns of light (however, see also the development of submillimeter, geometrically constrained microfluidic channels to deliver targeted and controlled release of neurotransmitters; Peterman et al., 2004). The electrical stimulation of these surviving visual neuronal elements evokes the subjective sensation of discrete points of light (referred to as “phosphenes”; Gothe et al., 2002; Marg and Rudiak, 1994). In principle, by delivering appropriate multisite patterns of electrical stimulation (i.e., characterizing the shape of the intended visual target and reflecting the neural structure's retinotopic organization),

(c) Cortex Mioroelectrode array

(a) Retina EPIRETINAL

Ganglion Bipolar

Microelectrode array

(b) Optic nerve

Photoreceptor

SUBRETINAL

Spiral cuff electode

Fig. 1. Summary diagram illustrating various neuroprosthesis approaches to restore vision. Theoretically, any point along the visual pathway can be electrically stimulated to generate the perception of phosphenes and thus represents a potential site to implant a visual prosthesis. At the level of the retina, an implanted device generates electrical current to stimulate cells of the inner retina (i.e., ganglion and bipolar cells). Two approaches are possible: (i) epiretinal; in which the device is attached to the inner surface of the retina, and (ii) subretinal; in which the device is placed within the underside of the retina. The optic nerve can be stimulated by implanting a cuff electrode around the nerve. In the cortical approach, electrodes are placed either intracortically or on the cortical surface in order to stimulate the visual cortex directly and thus bypassing afferent visual structures.

6

geometrical visual percepts can be generated. This allows for the perception of visual images (much akin to viewing a stadium electronic scoreboard or the images generated by an ink jet printer). The pattern of electrical stimulation delivered is determined by analyzing an image captured by a digital camera or in response to the images captured by the optics of the eye itself. With regards to visual perception, this “scoreboard” approach certainly represents a great oversimplification. It is clear that many attributes characterize a visual scene such as color, motion, and form. However, as currently conceived, visual prostheses are designed to address only one of the most basic components of vision, that is, spatial detail. Among the biggest challenges of prosthetic vision is the puzzle of the neural code for perception. The complexity of the neural code suggests that prosthetic devices should rely on intact neural circuitry whenever possible in order to take advantage of any intact sensory processing available (Dagnelie and Schuchard, 2007). Thus, reducing the complexity of neural coding necessary could potentially be achieved by implanting the prosthetic device at the earliest point along the visual pathway that retains functional integrity. Following to this premise, the retina would represent the earliest site of potential neuronal interface. Retinitis pigmentosa (RP) and age-related macular degeneration (AMD) are two retinal disorders that contribute greatly to the incidence of inherited blindness and blindness in the elderly, respectively (Bunker et al., 1984; Klein et al., 1997). Profound vision loss results largely due to the progressive degeneration of the lightcapturing component of the outer-segment of the retina, that is, the photoreceptor cells. However, the remaining retinal elements within the inner retinal layers (e.g., the bipolar and ganglion cells that converge to form the optic nerve) appear to survive in large numbers. Further, these elements remain responsive to electrical simulation even in highly advanced stages of the disease (Humayun et al., 1996). In

essence, a retinal-based visual prosthesis would replace the function of the degenerated photoreceptor cells by stimulating the surviving retinal neuronal machinery. A set of pivotal human experiments demonstrated that electrical stimulation of the retina of RP patients (Humayun et al., 1996; Rizzo et al., 2003b) as well as one patient with AMD (Humayun et al., 1999) led to the generation of phosphenes despite the fact that patients were profoundly blind for many years. Experiments lasted minutes to hours while patients remained awake in order to describe their visual experiences. Following electrical stimulation, patients reported visual patterned perceptions that were initially relatively crude. However, the gross geometric structure of the phosphene patterns could be altered in a controlled fashion by varying the position and number of the stimulating electrodes and the strength or duration of the delivered current (Humayun et al., 1996; Rizzo et al., 2003a,b). This demonstration of proof of principle has led many groups worldwide to pursue development of a variety of retinal-based designs and approaches. Currently, the retinal-based approach is arguably receiving the most attention as evidenced by size and number of on-going human clinical trials. Two retinal-based approaches are being pursued that are largely differentiated by their location of implantation with respect to the retina. In the subretinal approach, the implant is placed in the region of degenerated photoreceptors by creating a pocket between the sensory retina and retinal pigment epithelium (RPE) layer. In the epiretinal approach, the implant device is attached to the inner surface of the retina, close to the ganglion cell side (Fig. 1a). The subretinal visual neuroprosthesis design is currently being pursued by the Boston Retinal Implant Project (a large joint collaborative effort that includes the Massachusetts Eye and Ear Infirmary and Harvard Medical School, the Massachusetts Institute of Technology, the Boston Veterans Affairs Healthcare System, and other partnering institutions; see Fig. 2; Shire

7

Fig. 2. Artist conception of the Boston Retinal Implant device. (a) Specially designed glasses house a miniature camera used for the capture of images. The image is analyzed by an image processing unit and appropriate stimulus pulse information and power are sent via a transmission coil. A secondary receiving coil (sutured around the iris of the eye) captures the wireless information transmitted. (b) The transmitted information is relayed through a series of electronic components (hermetically sealed in a titanium case) and then ultimately to the stimulating electrode array that is inserted into subretinal space through a scleral flap created behind the eye. (c) Cross-sectional view of the eye and implant device. Note that only the electrode array penetrates the sclera and that the bulk of the implant components lie outside the eye.

et al., 2009). By virtue of being placed in juxtaposition to the nearest layer of surviving neurons (i.e., bipolar cells), the subretinal approach affords greater inherent mechanical stability. This is due to the fact that the ultrathin electrode array is effectively “sandwiched” between the inner-segment of retina and the RPE layer. Further, this approach has the theoretical advantage of not only being closer to surviving neuronal elements (thus potentially requiring lower amounts of electrical stimulating current) but also exploiting retinal signal preprocessing inherent to the bipolar cell layer. The placement of a subretinal device does require elaborate and complex surgical methods. For the Boston Retinal Implant device, this includes inserting an ultrathin flexible microelectrode array through an incision made on the outside scleral wall of the ocular globe. This surgical approach allows the device to reside within the subretinal space created (referred to as the “ab

externo approach” as opposed to “ab interno”; where ones passes through the vitreous humor of the eye and inserts the device through an incision made directly in the retina; see Javaheri et al., 2006). Another feature of this configuration is that it leaves the bulk of the electronic hardware outside of the eye thus avoiding complications related to heat generation and corrosion and facilitates the exchange of electronic components as needed. For its operation, a miniature camera mounted on a pair of eyeglasses is used for image capture. These images are then analyzed by an externally worn portable microprocessor used to convert the image data into an electronic signal. The appropriate signal pulses (delivering data and power) are transferred to the implant wirelessly via radio frequency (RF) coils. The resulting signal is transmitted to the subretinal microelectrode array driving the surviving retinal neural elements (i.e., bipolar and ganglion cells) with

8

appropriate patterned electrical stimulation. It is here that the signal processing begins and is further integrated as it passes down the optic nerve on to the visual cortex for final perception of the visual image. All electronic parts are hermetically sealed in a titanium case connected to an external flex circuit and the microelectronic array (Kelly et al., 2009). To date, the group has succeeded in developing a wireless retinal prosthesis prototype as the first step toward a human subretinal prosthesis implant. Initial studies in animal models have been successful in implanting active versions of the device and refining surgical techniques and mechanical design (Kelly et al., 2009). Human clinical trials are now being planned. Variations of the subretinal implant design have also been pursued by several large consortia efforts. The Artificial Silicone Retina (ASR) developed by Optobionics Corporation contains  5000 microphotodiodes, each containing its own stimulating electrode (Chow et al., 2004). When implanted under the retina, photocurrents generated by absorbed light stimulate adjacent retinal neurons in a multisite fashion. In a phase 1 trial of safety and efficacy carried out in six patients with profound vision loss from RP (followed from 6 to 18 months after implantation), patients reported an improvement in visual function after implantation. These reports were evidenced by an increase in visual field size and the ability to name more letters using a standardized visual acuity chart (Chow et al., 2004). While the relatively simple design of this device was intuitively appealing (note that no camera and subsequent image processing is required with this device), the apparent improvement in vision was not attributed to true prosthetic vision per se, but rather to a potential neurotrophic (or “cell rescue”) effect related to microelectric currents generated by the device (Pardue et al., 2005a,b). With this limitation in mind, a multilayered subretinal chip device incorporating signal amplification is now being pursued by a German consortium (Retina Implant AG). This device has

recently been implanted in profoundly blind RP patients and recent results have been encouraging. Early human clinical trial data suggests that stable visual percepts can be obtained and implanted patients profoundly blind with RP have been able to identify objects and letters (Besch et al., 2008; Zrenner, 2002). As a contrasting design approach, the epiretinal strategy entails placing an electrode array along the inner surface of the retina to stimulate the underlying ganglion cells. This procedure employs more typical vitreoretinal surgery techniques so as to affix the microelectrode array on to the retinal surface (e.g., using a retinal tack). The Artificial Retina Project has been actively pursued by a collaborative effort between the Doheny Eye Institute (University of Southern California) and Second Sight Medical Products. Like the Boston Retinal Implant design, this device incorporates a digital camera mounted on a pair of eyeglass capturing an image that in turn is converted into an electrical signal that is delivered to the retina (Humayun et al., 2003). Initial testing with a 16 electrode device (Argus I) in human volunteers with advanced RP has been successful. A large-scale multicentered phase II FDA-sponsored clinical trial is currently underway to evaluate a second generation implant (Argus II; 60 electrodes) in the largest cohort of visual prosthesis recipients to date. Results suggest that patients chronically implanted with this device can detect phosphenes at individual electrodes, discriminate crude shapes upon multiple electrode stimulation, and recognize simple stimuli presented via a headmounted camera (Humayun et al., 2009; Weiland et al., 2004). Very recently, the group reported that implanted subjects showed a significant improvement in accuracy in a spatial visual-motor target localization task comparing performance in patients implanted with their second generation device. Subjects were instructed to locate and touch a high contrast square target presented on a monitor. Nearly all subjects (26/27) showed a significant improvement in accuracy (Ahuja

9

et al., 2011). This is consistent with the observation that implanted subjects were able to develop appropriate head-scanning techniques and good “camera-hand” coordination in using their visual prosthetic device (Ahuja et al., 2011). Other groups are also pursuing the epiretinal approach including a variety of German-based consortiums. While still in earlier stages of development, early results have also been encouraging (Eckmiller et al., 2005; Gerding, 2007; Gerding et al., 2007). Other notable downstream approaches have been developed. A Belgian consortium has developed a prosthesis designed to stimulate the optic nerve using a four-electrode cuff electrode design and driven by stimuli captured by an external camera (Fig. 1b). Two patients have been chronically implanted to date. Reports from one blind volunteer demonstrated that electrical stimulation evoked the perception of localized, and often colored, phosphenes throughout the visual field (Veraart et al., 2003). After 4 months of psychophysical testing, the patient could recognize and distinguish orientations of lines, some shapes and even certain letters (Brelen et al., 2005; Veraart et al., 2003). Finally, there have also been attempts to deliver electrical stimulation to the visual cortex itself (Fig. 1c). Historically, this represents the oldest approach in developing a visual neuroprosthesis. By stimulating the visual cortex directly (thus bypassing earlier visual structures), this strategy has the appealing feature of potentially helping all forms of blindness regardless of ocular pathology. Early seminal work in a profoundly blind volunteer demonstrated that electrical stimulation delivered to the cortex (using surface electrodes) evoked the perception of discrete phosphenes (Brindley and Lewin, 1968). While the phosphene perceptions were rather crude, their spatial location approximately corresponded to the known cortical retinotopic representation of visual space. Later efforts incorporated a digital video camera mounted onto a pair of glasses interfaced with a

cortical stimulating array via a cable attached in the patient's skull (Dobelle and Mladejovsky, 1974). Several blind volunteers have been implanted and reportedly, one patient could distinguish the outline of a person and identify the orientation of certain letters using this device (Dobelle et al., 1974). While certainly a pioneering effort, the cortical approach still faces several technical challenges. These include determining the appropriate encoding strategies that are necessary to generate patterns of stimulation, safety concerns due to the inherent invasiveness of surgical implantation and the risk of focal seizures induced by direct cortical stimulation. However, new electrode designs (such as the 100-electrode array developed at the University of Utah; Normann et al., 1999) and advances in wireless technology have stimulated renewed interest and several groups are further pursuing this approach (Fernandez et al., 2005; Normann et al., 2009; Tehovnik et al., 2009; Troyk et al., 2003, 2005). Current technical challenges As with all neuroprosthesis efforts, the development and realization of a visual neuroprosthetic device will require continued and extensive collaborative effort among basic scientists, engineers, and clinicians. Despite great technical progress, certain technical challenges are immediately apparent and must be solved before a visual neuroprosthesis can be considered a viable clinical therapy (for further discussion see Chader et al., 2009; Cohen, 2007a,b; Dagnelie and Schuchard, 2007; Winter et al., 2007). For example, electrode geometry poses inherent limitations that must be carefully considered. This is particularly true with regards to how closely electrodes can be placed next to each other thus impacting the theoretical resolution the visual prosthesis can provide. Further, electrode geometry is intimately related to the amount of current that can be delivered safely

10

to the target neuronal tissue (i.e., charge/density limits). As these neuroprosthetic devices are meant to be implanted and used for very longperiods of time, the effect of prolonged and focal electrical stimulation delivered to delicate (and even potentially further degenerating) neuronal tissue remains unknown. In this direction, new electrode designs, materials, and coatings are being actively pursued in order to improve and expand safety profiles. One intriguing possibility is the development of pillar electrode arrays. Implanting this electrode array design has shown that penetrating pillars are able to attract neuronal elements (e.g., ganglion cells of the retina). The closer electrode–neuron interface may allow for lower currents to be used and thus safer injection of current for prolonged electrical stimulation (Butterwick et al., 2009). There also exists the issue of how a captured image is coregistered with the natural movement of the eye. Inappropriate compensatory eye movements may lead to perceptual mismatch, causing the patient wearing the implant to mislocalize objects in the external world. This potential confound is particularly true of implant designs that incorporate the use of an external mounted camera. To solve this issue, sophisticated eye-tracking mechanisms have been proposed and designed to generate appropriate shifts in the image (e.g., Palanker et al., 2005). However, these solutions await further development. Interestingly, recent work with visual simulations suggests that following training, implanted patients may learn to carry out appropriate compensatory head and camera movements to generate more stable percepts (Chen et al., 2009; Srivastava et al., 2009). Identifying appropriate candidates for implantation and determining the optimal placement of the implant are also of crucial importance (Merabet et al., 2007). Diagnostic techniques typically found in the clinical setting (such as the electroretinogram, visual evoked potentials, and visual field perimetry) are certainly intuitive choices to help characterize the profoundness of

an individual's visual impairment. Establishing predictive testing methods that allow for correlations between objective measures of visual function and eventual implant success would be highly desirable (Bach et al., 2010; Dagnelie, 2008). Along these lines, work has been done to develop extensive methodologies aimed at determining “best candidates” for long-term implantation of a microelectronic retinal implant (specifically defined as those requiring lowest current levels delivered to the retina to elicit visual perceptions; Yanai et al., 2003). This includes a series of preoperative visual, psychophysical, and electrophysiological tests (including dark-adapted bright flash and flicker electroretinograms and electrical-evoked responses; Yanai et al., 2003). Novel applications of other imaging techniques such as functional magnetic resonance imaging (fMRI), diffusion tensor imaging (DTI), and transcranial magnetic stimulation (TMS) may also prove helpful for the direct evaluation of overall visual cortical function and excitability (Fernandez et al., 2002; Merabet et al., 2007). With regard to retinal implants, the use of high-resolution optical coherence tomography (OCT) provides detailed analysis and characterization of retinal laminar anatomy (Matsuo and Morimoto, 2007; Fig. 3a). This is particularly relevant in considering more recent detailed anatomical findings indicating that there is extensive retinal reorganization of cellular components and interconnections in patients with longstanding retinal pathologies such as RP (Marc et al., 2003). It would follow that a degenerating retina may respond very differently to electrical stimulation over time (Dagnelie, 2006). Implantation of a retinal prosthesis during stages of complete photoreceptor loss, but with minimally altered inner retinal structure, may prove beneficial in increasing the likelihood that a visual prosthesis will function. Thus, continued assessment of retinal structural and functional viability could assist in not only selecting appropriate candidates, but also identifying the optimal location and timing of implantation as a function of disease progression (Fig. 3b).

11

Fig. 3. Use of advanced imaging methodology to ascertain implant positioning and retinal integrity. (a) Using optical coherence tomography (OCT), the cross-sectional position of the implanted microelectrode array (arrow) can be viewed in subretinal space. (b) Three-dimensional combined OCT with retinal microperimetry allows for simultaneous assessment of structure and function at specific points of retina. The resulting topographic map (values indicate luminance levels detected in decibels; dB) could potentially be used for postoperative evaluation as well as preoperative assessment of candidate implant locations. Image generated using an OPKO/OTI combined optical coherence tomography and scanning laser ophthalmoscope with microperimetry feature (Opko Health, Inc. Miami, FL).

Moving forward: new challenges and remaining questions Experimental evidence from numerous groups has demonstrated, at least in principle, that patterned electrical stimulation can evoke patterned light perceptions. However, as human clinical trials expand and patients continue to interact

with these visual neuroprosthetic devices on a more long-term basis, we now move away from the goal of simply demonstrating proof of principle toward establishing the fact that a visual neuroprosthesis can indeed improve the quality of life of an implanted patient. In that direction, we also need to define what are tractable milestones of success. Overall success can certainly be interpreted differently particularly when taken from the perspective of the device user. Moving forward, the implementation and potential benefit of a visual prosthesis needs to consider outcome measures and performance assessments that translate directly into improvements in the quality of life of blind individuals (e.g., accurate recognition and grasping of objects or skillful navigation in an unfamiliar environment and carrying out activities of daily living; Dagnelie, 2008). There is a clear need for new standardized testing and assessment of device efficacy that can be quantified in a manner that is scientifically testable and verifiable. In addition, the selection criteria for potential candidates must be clear. Not only would this allow for easier comparison of results across design efforts but also establish and evaluate patients’ visual demands and needs within the context of what a visual prosthesis can ultimately deliver. A review of human testing reveals that implanted recipients experience difficulties in fully understanding the visual information provided by these visual prosthetic devices. In the initial studies, the reported patterns of visual percepts often did not correspond to what was predicted based on the patterns of electrical stimulation delivered (e.g., Rizzo et al., 2003a,b). This key observation suggests that our intuitive sense as to how to generate patterned visual percepts (i.e., the “scoreboard approach”) may not prove to be an adequate strategy (Fernandez et al., 2005). This might be related to the fact that stimulation is carried out on neuronal tissue that is severely degenerated and, therefore, physiologically compromised. Certainly, the quality of visual percepts is likely to improve as remaining

12

technical challenges continue to be solved. However, there may arrive a point when engineering and surgical issues will no longer represent the greatest impediment to future progress. Rather, the greatest barrier will likely lie in our ignorance of how to introduce visual information that is meaningful to the visually deprived brain. It is a misconception that simple perceptions generated from patterned light are sufficient to generate meaningful vision. Further, increasing the resolution of images (e.g., by increasing the number of stimulating electrodes) with the goal of generating more complex perceptions would initially be perceptually meaningless rather than helpful. Several studies have highlighted that following the loss of vision, the brain undergoes profound neuroplastic transformation and that the occipital visual cortex is a major site of these changes (Bavelier and Neville, 2002; Fine et al., 2003; Merabet et al., 2005; Pascual-Leone et al., 1999). The extent and magnitude of these neuroplastic changes is likely to be influenced by such factors as the cause, onset, and duration of blindness. The plasticity of the visual system may allow for a considerable degree of adaptation. However, understanding the precise constraints of these neuroplastic processes will be crucial and have clear implications for rehabilitative training and progress in device development (Fernandez et al., 2005). A better understanding of how the brain adapts to the loss of sight and how the remaining senses process information in the visually deprived brain are necessary to appropriately modulate restored visual input and, ultimately, to allow meaningful vision with a neuroprosthetic device. One possibility might be to envision a patientcontrolled system that coordinates and registers the visual perceptions generated by a visual prosthesis with the identification of objects perceived through other senses (such as touch and audition). Patients could learn to integrate these concordant sources of sensory stimuli into meaningful percepts and, ultimately, gain the

ability to identify objects in the visual world (Merabet et al., 2005). Related studies in the development of sensory-substitution devices will likely contribute greatly to our knowledge in this arena. Further, these issues of training an implanted patient to “see again” are directly related to the realm of visual rehabilitation. The adaptive strategies and structured training necessary to interpret newly acquired visual percepts that ultimately translate to useful functional vision should not be left to chance (Dagnelie and Schuchard, 2007). This should be carried out within the context of a patient's current rehabilitation program (such as sensory-substitution devices as well as mobility aids such as a guidedog) to ensure an appropriate functional overlap. Clearly, any functional advantage gained through the use of a visual prosthetic device should meet, if not exceed, current rehabilitative options. Ultimately, the implementation of a visual prosthesis should not interfere with an individual's on-going rehabilitative program. Conclusion The loss of sight can have a devastatingly negative impact on the quality of life of an individual. The goal of restoring functional vision to blind, while certainly valiant, still faces formidable challenges before it will ever become a tractable reality. However, there are grounds to be cautiously optimistic and there is every reason to believe we are on the path to achieve this goal. It is also important to realize that the rehabilitation of the blind is a very complex problem, requiring extraordinarily diverse, lengthy, and intimate collaborations among basic scientists, engineers, clinicians, educators, and rehabilitative experts. As technical challenges continue to be solved, there also remains the issue of understanding how the brain adapts to the loss of vision itself. Success in restoring functional vision depends on our understanding of how blindness affects the

13

brain and what it means to “see” again. The neural changes that result from loss of vision need to be addressed if the restoration of visual input is to lead to functional vision. These issues of neuroplasticity also lead to questions regarding the feasibility of the visual prosthesis approach and its potential to benefit blind individuals. Therefore, it is essential that future research explores the mechanisms that underlie brain plasticity following the loss of vision. Such insight could help to develop and refine strategies for merging visual sensations that are generated by the prosthesis. Uncovering these adaptive strategies may ultimately assist in the rehabilitation process itself. References Acland, G. M., Aguirre, G. D., Ray, J., Zhang, Q., Aleman, T. S., Cideciyan, A. V., et al. (2001). Gene therapy restores vision in a canine model of childhood blindness. Nature Genetics, 28, 92–95. Ahuja, A. K., Dorn, J. D., Caspi, A., McMahon, M. J., Dagnelie, G., Dacruz, L., Stanga, P., Humayun, M. S., & Greenberg, R. J. (2011). Blind subjects implanted with the Argus II retinal prosthesis are able to improve performance in a spatial-motor task. British Journal of Ophthalmology, 95 (4), 539–543. Allin, S., Eckel, E., Markham, H., & Brewer, B. R. (2010). Recent trends in the development and evaluation of assistive robotic manipulation devices. Physical Medicine and Rehabilitation Clinics of North America, 21, 59–77. Bach, M., Wilke, M., Wilhelm, B., Zrenner, E., & Wilke, R. (2010). Basic quantitative assessment of visual performance in patients with very low vision. Investigative Ophthalmology and Visual Science, 51, 1255–1260. Bach-y-Rita, P. (2004). Tactile sensory substitution studies. Annals of the New York Academy of Sciences, 1013, 83–91. Bach-y-Rita, P., & Kercel, W. S. (2003). Sensory substitution and the human-machine interface. Trends in Cognitive Sciences, 7, 541–546. Bavelier, D., & Neville, H. J. (2002). Cross-modal plasticity: Where and how? Nature Reviews Neuroscience, 3, 443–452. Besch, D., Sachs, H., Szurman, P., Gulicher, D., Wilke, R., Reinert, S., et al. (2008). Extraocular surgery for implantation of an active subretinal visual prosthesis with external connections: Feasibility and outcome in seven patients. The British Journal of Ophthalmology, 92, 1361–1368. Brelen, M. E., Duret, F., Gerard, B., Delbeke, J., & Veraart, C. (2005). Creating a meaningful visual perception

in blind volunteers by optic nerve stimulation. Journal of Neural Engineering, 2, S22–S28. Brindley, G. S., & Lewin, W. S. (1968). The sensations produced by electrical stimulation of the visual cortex. Journal of Physiology, 196, 479–493. Bunker, C. H., Berson, E. L., Bromley, W. C., Hayes, R. P., & Roderick, T. H. (1984). Prevalence of retinitis pigmentosa in Maine. American Journal of Ophthalmology, 97, 357–365. Butterwick, A., Huie, P., Jones, B. W., Marc, R. E., Marmor, M., & Palanker, D. (2009). Effect of shape and coating of a subretinal prosthesis on its integration with the retina. Experimental Eye Research, 88, 22–29. Chader, G. J., Weiland, J., & Humayun, M. S. (2009). Artificial vision: Needs, functioning, and testing of a retinal electronic prosthesis. Progress in Brain Research, 175, 317–332. Chen, S. C., Suaning, G. J., Morley, J. W., & Lovell, N. H. (2009). Simulating prosthetic vision: II Measuring functional capacity. Vision Research, 49, 2329–2343. Chow, A. Y., Chow, V. Y., Packo, K. H., Pollack, J. S., Peyman, G. A., & Schuchard, R. (2004). The artificial silicon retina microchip for the treatment of vision loss from retinitis pigmentosa. Archives of Ophthalmology, 122, 460–469. Cohen, E. D. (2007a). Prosthetic interfaces with the visual system: Biological issues. Journal of Neural Engineering, 4, R14–R31. Cohen, E. D. (2007b). Safety and effectiveness considerations for clinical studies of visual prosthetic devices. Journal of Neural Engineering, 4, S124–S129. Craelius, W. (2002). The bionic man: Restoring mobility. Science, 295, 1018–1021. Dagnelie, G. (2006). Visual prosthetics 2006: Assessment and expectations. Expert Review of Medical Devices, 3, 315–325. Dagnelie, G. (2008). Psychophysical evaluation for visual prosthesis. Annual Review of Biomedical Engineering, 10, 339–368. Dagnelie, G., & Schuchard, R. A. (2007). The state of visual prosthetics—Hype or promise? Journal of Rehabilitation Research and Development, 44, xi–xiv. Dobelle, W. H., & Mladejovsky, M. G. (1974). Phosphenes produced by electrical stimulation of human occipital cortex, and their application to the development of a prosthesis for the blind. Journal of Physiology, 243, 553–576. Dobelle, W. H., Mladejovsky, M. G., & Girvin, J. P. (1974). Artificial vision for the blind: Electrical stimulation of visual cortex offers hope for a functional prosthesis. Science, 183, 440–444. Donoghue, J. P. (2002). Connecting cortex to machines: Recent advances in brain interfaces. Nature Neuroscience, 5(Suppl.), 1085–1088. Dowling, J. (2005). Artificial human vision. Expert Review of Medical Devices, 2, 73–85.

14 Eckmiller, R., Neumann, D., & Baruth, O. (2005). Tunable retina encoders for retina implants: Why and how. Journal of Neural Engineering, 2, S91–S104. Fernandez, E., Alfaro, A., Tormos, J. M., Climent, R., Martinez, M., Vilanova, H., et al. (2002). Mapping of the human visual cortex using image-guided transcranial magnetic stimulation. Brain Research. Brain Research Protocols, 10, 115–124. Fernandez, E., Pelayo, F., Romero, S., Bongard, M., Marin, C., Alfaro, A., et al. (2005). Development of a cortical visual neuroprosthesis for the blind: The relevance of neuroplasticity. Journal of Neural Engineering, 2, R1–R12. Fine, I., Wade, A. R., Brewer, A. A., May, M. G., Goodman, D. F., Boynton, G. M., et al. (2003). Long-term deprivation affects visual perception and cortex. Nature Neuroscience, 6, 915–916. Gerding, H. (2007). A new approach towards a minimal invasive retina implant. Journal of Neural Engineering, 4, S30–S37. Gerding, H., Benner, F. P., & Taneri, S. (2007). Experimental implantation of epiretinal retina implants (EPI-RET) with an IOL-type receiver unit. Journal of Neural Engineering, 4, S38–S49. Gothe, J., Brandt, S. A., Irlbacher, K., Roricht, S., Sabel, B. A., & Meyer, B. U. (2002). Changes in visual cortex excitability in blind subjects as demonstrated by transcranial magnetic stimulation. Brain, 125, 479–490. Hochberg, L. R., Serruya, M. D., Friehs, G. M., Mukand, J. A., Saleh, M., Caplan, A. H., et al. (2006). Neuronal ensemble control of prosthetic devices by a human with tetraplegia. Nature, 442, 164–171. Humayun, M. S. (2007). Artificial sight: Basic research, biomedical engineering, and clinical advances. New York: Springer. Humayun, M. S., de Juan, E., Jr, Dagnelie, G., Greenberg, R. J., Propst, R. H., & Phillips, D. H. (1996). Visual perception elicited by electrical stimulation of retina in blind humans. Archives of Ophthalmology, 114, 40–46. Humayun, M. S., de Juan, E., Jr, Weiland, J. D., Dagnelie, G., Katona, S., Greenberg, R., et al. (1999). Pattern electrical stimulation of the human retina. Vision Research, 39, 2569–2576. Humayun, M. S., Dorn, J. D., Ahuja, A. K., Caspi, A., Filley, E., Dagnelie, G., et al. (2009). Preliminary 6 month results from the Argus II epiretinal prosthesis feasibility study. Conference Proceedings of IEEE Engineering in Medicine and Biology Society, 2009, 4566–4568. Humayun, M. S., Weiland, J. D., Fujii, G. Y., Greenberg, R., Williamson, R., Little, J., et al. (2003). Visual perception in a blind subject with a chronic microelectronic retinal prosthesis. Vision Research, 43, 2573–2581.

Javaheri, M., Hahn, D. S., Lakhanpal, R. R., Weiland, J. D., & Humayun, M. S. (2006). Retinal prostheses for the blind. Annals of the Academy of Medicine, Singapore, 35, 137–144. Jones, S., Harris, D., Estill, A., & Mikulec, A. A. (2008). Implantable hearing devices. Missouri Medicine, 105, 235–239. Kelly, S. K., Shire, D. B., Chen, J., Doyle, P., Gingerich, M. D., Drohan, W. A., et al. (2009). Realization of a 15-channel, hermetically-encased wireless subretinal prosthesis for the blind. Conference Proceedings of IEEE Engineering in Medicine and Biology Society, 2009, 200–203. Klein, R., Klein, B. E., Jensen, S. C., & Meuer, S. M. (1997). The five-year incidence and progression of age-related maculopathy: The Beaver Dam Eye Study. Ophthalmology, 104, 7–21. Laferrier, J. Z., & Gailey, R. (2010). Advances in lower-limb prosthetic technology. Physical Medicine and Rehabilitation Clinics of North America, 21, 87–110. Loeb, G. E. (1990). Cochlear prosthetics. Annual Review of Neuroscience, 13, 357–371. MacLaren, R. E., Pearson, R. A., MacNeil, A., Douglas, R. H., Salt, T. E., Akimoto, M., et al. (2006). Retinal repair by transplantation of photoreceptor precursors. Nature, 444, 203–207. Marbach, W. D., Zabarsky, M., Hoban, P., & Nelson, C. (1982). Building the bionic man. Newsweek, 100, 78–79. Marc, R. E., Jones, B. W., Watt, C. B., & Strettoi, E. (2003). Neural remodeling in retinal degeneration. Progress in Retinal and Eye Research, 22, 607–655. Marg, E., & Rudiak, D. (1994). Phosphenes induced by magnetic stimulation over the occipital brain: Description and probable site of stimulation. Optometry and Vision Science, 71, 301–311. Matsuo, T., & Morimoto, N. (2007). Visual acuity and perimacular retinal layers detected by optical coherence tomography in patients with retinitis pigmentosa. The British Journal of Ophthalmology, 91, 888–890. Merabet, L. B., Rizzo, J. F., Amedi, A., Somers, D. C., & Pascual-Leone, A. (2005). What blindness can tell us about seeing again: Merging neuroplasticity and neuroprostheses. Nature Reviews Neuroscience, 6, 71–77. Merabet, L. B., Rizzo, J. F., 3rdPascual-Leone, A., & Fernandez, E. (2007). 'Who is the ideal candidate?': Decisions and issues relating to visual neuroprosthesis development, patient testing and neuroplasticity. Journal of Neural Engineering, 4, S130–S135. Nicolelis, M. A. (2003). Brain-machine interfaces to restore motor function and probe neural circuits. Nature Reviews Neuroscience, 4, 417–422. Normann, R. A., Greger, B., House, P., Romero, S. F., Pelayo, F., & Fernandez, E. (2009). Toward the development of a cortically based visual neuroprosthesis. Journal of Neural Engineering, 6, 035001.

15 Normann, R. A., Maynard, E. M., Rousche, P. J., & Warren, D. J. (1999). A neural interface for a cortical vision prosthesis. Vision Research, 39, 2577–2587. Palanker, D., Vankov, A., Huie, P., & Baccus, S. (2005). Design of a high-resolution optoelectronic retinal prosthesis. Journal of Neural Engineering, 2, S105–S120. Pardue, M. T., Phillips, M. J., Yin, H., Fernandes, A., Cheng, Y., Chow, A. Y., et al. (2005). Possible sources of neuroprotection following subretinal silicon chip implantation in RCS rats. Journal of Neural Engineering, 2, S39–S47. Pardue, M. T., Phillips, M. J., Yin, H., Sippy, B. D., WebbWood, S., Chow, A. Y., et al. (2005). Neuroprotective effect of subretinal implants in the RCS rat. Investigative Ophthalmology and Visual Science, 46, 674–682. Pascual-Leone, A., Hamilton, R., Tormos, J. M., Keenan, J. P., & Catala, M. D. (1999). Neuroplasticity in the adjustment to blindness. In J. Grafman & Y. Christen (Eds.), Neuronal plasticity: Building a bridge from the laboratory to the clinic. Berlin Heidelberg, New York: Springer-Verlag. Peterman, M. C., Noolandi, J., Blumenkranz, M. S., & Fishman, H. A. (2004). Localized chemical release from an artificial synapse chip. Proceedings of the National Academy of Sciences of the United States of America, 101, 9951–9954. Rizzo, J. F., 3rd, Wyatt, J., Loewenstein, J., Kelly, S., & Shire, D. (2003a). Methods and perceptual thresholds for short-term electrical stimulation of human retina with microelectrode arrays. Investigative Ophthalmology and Visual Science, 44, 5355–5361. Rizzo, J. F., 3rd, Wyatt, J., Loewenstein, J., Kelly, S., & Shire, D. (2003b). Perceptual efficacy of electrical stimulation of human retina with a microelectrode array during short-term surgical trials. Investigative Ophthalmology and Visual Science, 44, 5362–5369. Shire, D. B., Kelly, S. K., Chen, J., Doyle, P., Gingerich, M. D., Cogan, S. F., et al. (2009). Development and implantation of a minimally invasive wireless subretinal neurostimulator. IEEE Transactions on Biomedical Engineering, 56, 2502–2511.

Srivastava, N. R., Troyk, P. R., & Dagnelie, G. (2009). Detection, eye-hand coordination and virtual mobility performance in simulated vision for a cortical visual prosthesis device. Journal of Neural Engineering, 6, 035008. Tehovnik, E. J., Slocum, W. M., Smirnakis, S. M., & Tolias, A. S. (2009). Microstimulation of visual cortex to restore vision. Progress in Brain Research, 175, 347–375. Troyk, P., Bak, M., Berg, J., Bradley, D., Cogan, S., Erickson, R., et al. (2003). A model for intracortical visual prosthesis research. Artificial Organs, 27, 1005–1015. Troyk, P. R., Bradley, D., Bak, M., Cogan, S., Erickson, R., Hu, Z., et al. (2005). Intracortical visual prosthesis research—Approach and progress. Conference Proceedings of IEEE Engineering in Medicine and Biology Society, 7, 7376–7379. Veraart, C., Wanet-Defalque, M. C., Gerard, B., Vanlierde, A., & Delbeke, J. (2003). Pattern recognition with the optic nerve visual prosthesis. Artificial Organs, 27, 996–1004. Weiland, J. D., Yanai, D., Mahadevappa, M., Williamson, R., Mech, B. V., Fujii, G. Y., et al. (2004). Visual task performance in blind humans with retinal prosthetic implants. Conference Proceedings of IEEE Engineering in Medicine and Biology Society, 6, 4172–4173. Winter, J. O., Cogan, S. F., & Rizzo, J. F. 3rd. (2007). Retinal prostheses: Current challenges and future outlook. Journal of Biomaterials Science, Polymer Edition, 18, 1031–1055. Yanai, D., Lakhanpal, R. R., Weiland, J. D., Mahadevappa, M., Van Boemel, G., Fujii, G. Y., et al. (2003). The value of preoperative tests in the selection of blind patients for a permanent microelectronic implant. Transactions of the American Ophthalmological Society, 101, 223–228, discussion 228–230. Zrenner, E. (2002). The subretinal implant: Can microphotodiode arrays replace degenerated retinal photoreceptors to restore vision? Ophthalmologica, 216 (Suppl. 1), 8–20, discussion 52–53.

A. M. Green, C. E. Chapman, J. F. Kalaska and F. Lepore (Eds.) Progress in Brain Research, Vol. 192 ISSN: 0079-6123 Copyright Ó 2011 Elsevier B.V. All rights reserved.

CHAPTER 2

Insights from darkness: what the study of blindness has taught us about brain structure and function Ron Kupers{,* and Maurice Ptito{ {

Institute of Neuroscience and Pharmacology, Panum Institute, University of Copenhagen, Copenhagen, Denmark { Chaire de recherche Harland Sanders en Sciences de la vision, École d'Optométrie, Université de Montréal, Montréal, Québec, Canada

Abstract: Vision plays a central role in how we represent and interact with the world around us. Roughly, one-third of the cortical surface in primates is involved in visual processes. The loss of vision, either at birth or later in life, must therefore have profound consequences on brain organization and on the way the world is perceived and acted upon. In this chapter, we formulate a number of critical questions. Do blind individuals indeed develop supra-normal capacities for the remaining senses in order to compensate for their loss of vision? Do brains from sighted and blind individuals differ, and how? How does the brain of someone who has never had any visual perception form an image of the external world? We discuss findings from animal research as well from recent psychophysical and functional brain imaging studies in sighted and blind individuals that shed some new light on the answers to these questions. Keywords: cross-modal plasticity; visual cortex; sensory substitution; supramodal cortical organization; qualia; brain rewiring. incapacitating events that can befall a person. The importance that vision plays in everyday life is We see with our brains not eyes. —Paul Bach-y-Rita already reflected at the level of the cortical organization. Indeed, roughly one-third of the cortical surface in primates is involved in visual Introduction functions. This raises the question of what happens to this cortex when vision is lacking from Since we are living in a world in which vision birth or when vision is lost at a later stage in plays a very important role, the loss of vision, development. For a long time, it was believed either from birth or later in life, is one of the most that the visually deprived cortex would remain devoid of any particular functional role. However, a wealth of studies in animals *Corresponding author. (reviewed in Ptito and Desgent, 2006), followed Tel.: þ45-3545-6890; Fax: þ45-3545-8949 by studies in humans (reviewed in Merabet and E-mail: [email protected] DOI: 10.1016/B978-0-444-53355-5.00002-6

17

18

Pascual-Leone, 2010; Pietrini et al., 2009), have shown in an unequivocal way that the visually deprived cortex not only reorganizes structurally but also becomes functionally involved in a multitude of nonvisual tasks. Whereas the first studies focused on its acquired role in nonvisual forms of sensory processing, in particular, tactile and auditory functions, more recent studies revealed a broader picture showing that the visually deprived occipital cortex is also involved in various cognitive processes (Amedi et al., 2003; Bonino et al., 2008; Cattaneo et al., 2008; Kupers et al., 2007, 2010; Raz et al., 2005; Stevens et al., 2007). This raises a number of interesting questions. First, how does nonvisual information reach the visual cortex? Is this accomplished through the formation of new anatomical connections or by a strengthening or unmasking of existing pathways in the sighted person's brain? Thanks to modern MRI-based brain imaging techniques, we are starting to understand the reorganization of the connectivity in the blind person's brain. At the functional level, does the fact that blind subjects have extra cortical territory available to process nonvisual information make them more proficient in nonvisual tasks? A final and crucial question is what can we offer to blind people to (partially) restore their lack of visual input? Throughout history, many attempts have been undertaken to develop substitutes for vision. The best known example is undoubtedly Braille reading, which replaces the visual input of letters by embossed arrays of dots which are sensed by the tactile system and translated into meaningful words. Although Braille reading meant an important leap forward in the quality of life of blind persons, its limitations are apparent. In the past decades, many efforts have therefore been undertaken to develop devices that convey “visual” information from objects that are placed outside the immediate egocentric space of the blind individual. The legacy of Professor Paul Bach-y-Rita needs to be acknowledged here. He was one of the pioneers in sensory substitution and in the

field of neuroscience that later became known as cross-modal plasticity. His pioneering work on sensory substitution systems (Bach-y-Rita, 1967; Bach-y-Rita et al., 1969), although met at the beginning with much skepticism, has paved the way for a generation of new sensory substitution devices of which the tongue display unit (TDU) and the vOICe system are the best known examples (reviewed in Bubic et al., 2010).

Visual deprivation models in animals The cerebral cortex has a remarkable capacity for plasticity resulting in anatomical reorganization and behavioral recovery, both in animals (Kaas, 2002) and humans (Pascual-Leone et al., 2005). Bilateral enucleation in hamsters (Izraeli et al., 2002), congenital blindness in mice (Chabot et al., 2007, 2008), and naturally very low vision as in the blind mole rat (Bronchti et al., 2002; Doron and Wollberg, 1994) yield the formation of new ectopic projections from the inferior colliculus to the lateral geniculate nucleus, the primary visual relay in the thalamus. These new aberrant projections are probably responsible for the auditory evoked activity in the visual cortex, as measured in electrophysiological recordings. Rebillard et al. (1977) were the first to report that the primary auditory cortex is driven by visual stimuli in congenitally deaf cats. Conversely, studies on the microphthalmic mole rat (Spalax ehrenbergi) showed that auditory stimulation can drive cells in the primary visual cortex (Bronchti et al., 2002). Cells in the primary visual cortex of visually deprived cats, rats, or mice can also be triggered by somatosensory or auditory inputs, suggesting cross-modal reorganization (Toldi et al., 1994). The same has been shown in nonhuman primates. For instance, neurons in visual cortical areas respond to somatic inputs following early visual deprivation in monkeys (Hyvarinen et al., 1991). This is in sharp contrast with results in normal-seeing animals, in which area 19 neurons respond exclusively to visual

19

inputs. Peripheral inputs play a pivotal role in the organization of the neocortex, as cortical territories usually involved in visual processing are invaded by the auditory and somatosensory systems. It seems therefore that the visual cortex is capable of rewiring in order to accommodate these nonvisual inputs. However, in the case of early (perinatal) brain damage, abnormal neuronal connectivity patterns can be produced and an alternative approach to study cross-modal plasticity resides in the tampering with “blueprints” during prenatal development. Relevant to this approach are the numerous studies on “rewiring” in hamsters (Ptito and Desgent, 2006) and ferrets (reviewed in Lyckman and Sur, 2002). The “rewired” hamster brain Early brain damage results in abnormal neuronal connectivity patterns. By destroying central retinal targets, it must therefore be possible to induce the formation of new and permanent retinofugal projections into nonvisual thalamic sites such as the auditory nucleus (Frost and Metin, 1985; Ptito et al., 2001; Fig. 1a). These surgically induced retinal projections are retinotopically organized and make functional synapses (Metin and Frost, 1989). Neurons in the somatosensory cortex of animals with ectopic retinal projections have visual response properties similar to those of neurons in the primary visual cortex of normal animals (Metin and Frost, 1989). Ferrets with retinofugal projections to the auditory thalamus but no visual cortex appear to perceive light stimuli as visual (Von Melchner, et al., 2000). The question concerning the parallelism between a different brain organization (produced by lesions) and behavioral recovery is still debated although recent experiments both in rewired ferrets and hamsters seem to indicate a large degree of recovery in visual functions (reviewed in Ptito et al., 2001). For example, responses to visual stimuli have been observed in the auditory cortex of hamsters with robust

and permanent projections to the auditory thalamic nucleus (medial geniculate nucleus) but which are lacking a visual cortex. Single neurons in the auditory cortex of these animals respond to visual stimuli and some of them respond equally well to visual as to auditory stimuli (Ptito et al., 2001). Moreover, cells responding to visual stimuli show orientation selectivity, and motion and direction sensitivity. These receptive field properties compare favorably well with those obtained from cells in the visual cortex of normal hamsters. At the behavioral level, rewired hamsters can learn visual discrimination tasks as well as normal animals and a lesion of the auditory cortex abolishes this function (Fig. 1b; Frost et al., 2000). In fact, rewired hamsters with auditory cortex lesions exhibit cortical blindness similar to nonrewired hamsters with visual cortex lesions. This cross-modal processing of sensory information in the cortex is not fully understood. Recent work carried out in our laboratory has led to the suggestion that the observed changes may be due to modifications in GABAergic interneurons that express calcium-binding proteins (CaBPs) like parvalbumin (PV) and calbindin (CB; Desgent et al., 2010). In deaf and cross-modal rewired ferrets, for example, qualitative changes were observed in the morphology and proportion of interneurons containing PV and CB (Pallas, 2001, 2002). Since the laminar distribution of these proteins is significantly different in the primary visual and auditory cortices of normal hamsters (Desgent et al., 2005), the induction of aberrant connectivity to these cortices should also be evident at the neurochemical level. Indeed, hamsters enucleated at birth show significant changes in the distribution of CaBPs within their visual cortex. Compared to intact hamsters, the density of PV-immunoreactive neurons is higher in layer IV and lower in layer V, whereas the density of CB-immunoreactive cells is significantly lower in layer V of V1 in enucleated animals. These results suggest that the affected primary visual cortex may adopt chemical features of the auditory cortex through cross-modal rewiring.

20 (a)

MG

O T

LGv

Trials to criterion

500

NON REWIRED Intact

VC lesion

REWIRED 900

AC lesion

Trials to criterion

(b)

400 300 200

Pre-AC lesion

Post-AC lesion

600

300

100

* 0

Light/Dark

Gratings

* Squares

*

*

0 Light/Dark

Gratings

Squares

Fig. 1. Anatomical rewiring and behavioral outcome following loss of visual input at birth. (a) Aberrant ectopic projections from the retina to the medial geniculate nucleus (MG) following neonatal lesions of the superior colliculus in hamsters (after Ptito et al., 2001). (b) Performance of rewired hamsters (right panel) in a visual pattern discrimination task compared to controls (left panel). Note that the performance of rewired hamsters following the additional lesion of the auditory cortex is similar to that of controls with a lesion of the primary visual cortex (illustrated by the asterisk; after Frost et al., 2000) Abbreviations: AC = auditory cortex; VC = visual cortex.

Anatomy of the blind human brain How does absence of vision from birth affect the macrostructural organization of the human brain and through which pathways can nonvisual information be funneled to the occipital cortex in the visually deprived brain? In recent years, MRI-based brain imaging techniques such as voxel-based morphometry (VBM) and diffusion tensor imaging (DTI) and diffusion tensor tractography (DTT) have been successfully applied for the in vivo investigation of alterations in gray matter (GM) and white matter (WM) in the blind human brain. The results of these studies seem to concur that there is significant

GM atrophy of all structures of the visual pathways, including the lateral geniculate and posterior pulvinar nuclei, the striate and extrastriate visual areas, and the inferior temporal gyrus and lateral orbital cortex, regions that are part of the ventral visual stream which is involved in object recognition (Noppeney et al., 2005; Pan et al., 2007; Ptito et al., 2008b; Shimony et al., 2006; Fig. 2a). These changes can be massive, with volume reductions ranging from 25% in the primary visual cortex up to 20% in extrastriate visual areas (Ptito et al., 2008b). Volume reductions also occur in nonvisual areas such as the hippocampus (Chebat et al., 2007; Fortin et al., 2008). Besides the volume

21

Fig. 2. The congenitally blind brain. (a) Axial brain slices showing regional reductions in gray matter (red) and white matter (blue) in congenitally blind compared to matched sighted control subjects. Note that all components of the visual system in the blind are reduced in volume (after Ptito et al., 2008b). (b) Differences in cortical thickness between congenitally blind and sighted control subjects. Despite a reduction in volume of the occipital cortex, blind subjects show an increase in thickness of the cuneus (unpublished data from our lab). (c) Glucose metabolism at rest in a normal sighted control (upper row) and a congenitally blind subject (lower row). Increases in glucose metabolism in the occipital cortex in the blind are shown on sagittal sections (unpublished data from our lab).

reductions in GM, congenitally blind subjects show increases in cortical thickness in the cuneus (Fig. 2b) which are likely due to a reduction in cortical pruning in early maturation stages of the cortex as a consequence of the loss of visual input. Changes in WM include atrophy of the optic tracts and optic chiasm, the optic radiations, the splenium of the corpus callosum (Noppeney et al., 2005; Pan et al., 2007; Ptito et al., 2008a,b; Shimony et al., 2006), and the inferior longitudinal fasciculus (ILF; Ptito et al., 2008b), a pathway connecting the occipital cortex with the temporal lobe. Lesions of the ILF may induce visual agnosia, prosopagnosia, and disturbances in visual recent memory. No studies found direct evidence for the establishment of new pathways, although volume increases in the occipitofrontal fasciculus, the superior longitudinal fasciculus,

and the genu of the corpus callosum have been reported (Ptito et al., 2008a,b). There is also indirect evidence for an increased functional connectivity between parietal and visual areas in the blind (Kupers et al., 2006; Ptito et al., 2005; Wittenberg et al., 2004). Taken together, since no de novo tracts have been demonstrated in congenitally blind subjects, the data suggest that cross-modal functionality of the visual cortex in early blindness is primarily mediated by preserved or strengthened corticocortical connections. Finally, there are also important metabolic changes in the congenitally blind person's brain. Using PET-FDG, we showed that glucose metabolism at rest is increased by around 15% in striate and extrastriate cortex of blind subjects. Figure 2c shows FDG uptake at rest in a congenitally blind and a blindfolded control subject.

22

Cross-modal plasticity: substituting vision with touch There are two major classes of sensory substitution devices for blindness, based upon either tactile or auditory input. These are referred to as respectively tactile-to-vision sensory substitution (TVSS) and auditory-to-vision sensory substitution (AVSS) devices. The best known examples of the latter category are the PVSA (prosthesis substituting vision with audition) system developed by Veraart and colleagues (Capelle et al., 1998) and the vOICe system (Meijer, 1992). Both systems translate visual images taken by a video camera into auditory soundscapes. Bright pixels sound loud and pixels in the upper field of view get a higher pitch. Subjects need training to be able to interpret the resulting soundscapes correctly. A discussion of the latter category is beyond the scope of this chapter and we refer the interested reader to a recent review (Ward and Meijer, 2010). Already in the 1960s, Bach-y-Rita (1967) developed the first TVSS device. The system consisted of a video camera, a computer, and 400 small pneumatic stimulators which were mounted in the back of a dental chair. The blind subject, seated with his back against the stimulators, used a video camera to scan an object that was placed in front of him/her and the visual input was translated into vibrotactile stimulation that was delivered to his back. With training, blind subjects could recognize increasingly more complex shapes, detect movement, and certain visual features such as shades and depth. This system was later replaced by the TDU, named after the fact that the tongue became the substrate for stimulation (Bach-y-Rita and Kercel, 2003). This has lead to a system that is much smaller and became portable, allowing it to be used also outside the laboratory in real-life situations. We have used the TDU in a series of behavioral and brain imaging studies. In a first study, we trained a group of congenitally blind and blindfolded sighted control subjects to use the

TDU in an orientation discrimination task (Ptito et al., 2005). We scanned subjects before and after a 1-week training period. Both groups learned the task equally well, although the blind tended to be faster than the sighted participants. As expected, the brain imaging results before training did not show activations in visual cortical areas in either group. In sharp contrast, after training, blind but not blindfolded sighted control subjects activated large parts of their visual cortex (Fig. 3a). Interestingly, the activated clusters in the occipital and occipitoparietal areas showed a strong resemblance with the areas reported to be activated when sighted subjects do a visual orientation task. These results are in line and extend earlier results showing occipital cortical activation in blind subjects during Braille reading (Burton et al., 2002, 2004; Cohen et al., 1999).

A dorsal and a ventral visual stream: also in the absence of vision? The visual system is classically subdivided into a dorsal “where” and a ventral “what” pathway (Ungerleider and Mishkin, 1982). After having shown that the visual cortex in the blind can be recruited by an orientation task, we next addressed the question whether the basic architecture of a dorsal and ventral pathway is preserved in subjects lacking vision from birth. To that end, we did a series of experiments with the TDU, in which we used tasks tailored to activate either the dorsal or the ventral visual pathway. In the first study, congenitally blind and sighted participants used the TDU to detect the motion direction of a random dot pattern (Matteau et al., 2010). Stimuli were moving in a coherent manner (left, right), randomly, or remained static. The fMRI data showed that following training, blind subjects activated large parts of the dorsal extrastriate visual pathway. Both groups activated the motion-sensitive hMTþ complex, although at different anatomical locations (Fig. 3b). The observation that the

23

Fig. 3. Cross-modal activation of the occipital cortex in congenitally blind subjects. (a) Brain activation pattern showing that trained blind (upper row) but not sighted controls (lower row) activate their visual cortex in an orientation discrimination task with the TDU. The values below refer to the z-coordinate of the slices as defined in Montreal Neurological Institute (MNI) space (after Ptito et al., 2005). (b) fMRI data showing activations of area hMTþ in blind (upper row) and sighted control (lower row) subjects for the contrasts “coherent motion versus rest,” “incoherent motion versus rest,” and “coherent þ incoherent motion versus rest.” The numbers next to the slices refer to the positioning of the slice in the z-direction in MNI space (after Matteau et al., 2010). (c) Blind subjects showed significantly stronger BOLD responses in the occipital cortex compared to sighted controls in an odor detection task (after Kupers et al., 2011). (d). Cortical flatmap representation of fMRI data showing activation of the occipital cortex and right parahippocampus in blind subjects performing a spatial navigation task with the TDU (upper row). When sighted subjects performed the same navigation task visually (lower row), they activated a highly similar network (after Kupers et al., 2010).

hMTþ complex can be activated by tactile motion in congenitally blind subjects demonstrates that its recruitment is not mediated by visual-based mental imagery and that visual experience is not

necessary for the development of this cortical system. The fact that area hMTþ was activated at different anatomical locations in sighted and congenitally blind individuals, however, suggests that

24

lack of vision leads to functional rearrangements of this supramodal area. Indeed, results from an earlier brain imaging study showed that visual experience plays a critical role in the functional segregation of hMTþ into a more anterior part that is involved in the representation of both optic and tactile motion and a more posterior part that is uniquely involved in the representation of optic flow (Ricciardi et al., 2007). In the case that hMTþ develops in the absence of visual experience, the entire structure becomes involved in tactile motion representation. This suggests that competitive interactions between visual and tactile inputs in normal development lead to a functional specialization in hMTþ that does not develop without visual input. In a subsequent study, we trained blind and blindfolded sighted subjects to use the TDU in a shape recognition task. Participants were presented four different shapes (a triangle, rectangle, square, and the letter E) and they had to indicate which of the four shapes had been presented. In line with our hypothesis, the fMRI data showed that during nonhaptic shape recognition, blind subjects activated large portions of the ventral visual stream, including the cuneus, inferotemporal (IT) cortex, lateral occipital tactile vision area (LOtv), and fusiform gyrus (Matteau et al., 2008). Control subjects activated area LOtv and precuneus but not cuneus, IT, and fusiform gyrus. These results indicate that congenitally blind subjects recruit key regions in the ventral visual pathway during nonhaptic tactile shape discrimination. The activation of LOtv by nonhaptic tactile shape processing in blind and sighted subjects adds further support to the notion that this area subserves an abstract or meta-modal representation of shape (Amedi et al., 2001, 2002; Pietrini et al., 2004). The above results lead us to the following conclusions. First, the segregation of the efferent projections of the primary visual cortex into a dorsal and ventral visual stream is preserved in individuals blind from birth. Second, cortical “visual” association areas are capable of processing and interpreting

information carried by nonvisual sensory modalities. This is not merely the consequence of brain reorganization following congenital blindness, as this ability also exists in sighted subjects. Third, the differences in the extent and magnitude of the activated areas in blind and sighted subjects are likely due to the effects of rearrangements that follow the lack of sight. The supramodal nature of this functional cortical organization explains how individuals who never had any visual experience are able to acquire normal knowledge about objects and their position in space and form mental representations of and interact effectively with the external world.

There is more than touch to activate the occipital cortex in the blind The studies we have been discussing so far mainly concern the somatosensory system. However, there is ample evidence that other sensory inputs also activate the visual cortex. For instance, studies in the auditory domain have demonstrated that congenitally blind subjects have superior auditory capacities (Lessard et al., 1998; Röder et al., 1999) and that this is related to activation of their visual cortex by auditory stimuli (Gougoux et al., 2005). Not much is known about the other remaining senses. We recently started investigating olfactory processing in the blind. The few published studies in this field have reported highly contradictory results, some finding no performance differences between congenitally blind and sighted subjects, whereas others concluded that the blind have a better developed sense of smell. We studied odor detection threshold, odor discrimination, and odor identification in congenitally blind subjects and a group of matched sighted subjects. We also assessed self-reported odor awareness—that is, consciousness of olfactory sensations—by means of the Odor Awareness Scale (OAS). The OAS questionnaire measures to which degree participants notice, pay attention to, or attach importance to smells

25

(Smeets et al., 2008). Our results showed that blind subjects have a lower odor detection threshold compared to sighted subjects (Kupers et al., 2011). However, we found no differences in odor discrimination or identification. Interestingly, blind subjects scored higher on the OAS, indicating an increased awareness for smells. Among the OAS items that were rated significantly higher by blind subjects, most were related to fragrances or to the smell of people. This suggests that in the absence of vision, more attention is directed toward other people's smell, which can provide information about a person's identity. Next, we conducted an event-related fMRI study in which congenitally blind and sighted controls performed an odor detection task (BeaulieuLefebvre et al., 2010). Blind participants showed significantly stronger blood oxygenation leveldependent (BOLD) responses in primary (right amygdala) and higher order (right orbitofrontal and bilateral hippocampus) olfactory cortex and occipital cortex during odor detection (Fig. 3c). These data provide the first demonstration that the visual cortex of the blind can also be recruited by odorants, thus adding new evidence to its multimodal function. The increased BOLD responses in higher order olfactory cortex and visual cortex may provide a neurobiological substrate for the increased odor awareness in blind subjects. There is strong evidence that congenitally blind subjects also recruit their visual cortex in a variety of cognitive tasks such as lexical and phonological processing, episodic memory, and visuospatial imagery (Amedi et al., 2003; Cattaneo et al., 2008; Raz et al., 2005; Röder et al., 2002; Stevens et al., 2007). We investigated the possible role of the occipital cortex in the congenitally blind in repetition priming which is a nonconscious (implicit) form of learning (Kupers et al., 2007). Repetition priming involves a change in the ability to identify an object or generate a word as a consequence of a specific prior encounter with it. At the behavioral level, it manifests itself by an increase in accuracy or speed of task

performance following earlier encounter(s) with the task or stimulus (Schacter et al., 2007). We asked a group of congenitally blind subjects to read a list of Braille words in a language (Finnish) unknown to them. Participants read the list three times in a row, as fast and as accurately as possible. The improvement in performance between the first and the third reading provides an index of the magnitude of the repetition priming effect. Next, participants read a new list of words immediately following a 15-min period of repetitive transcranial magnetic stimulation (rTMS) over the mid-occipital cortex or over a control region. The data showed that the repetition priming effect was largely abolished when rTMS was applied over the occipital cortex but not when applied over a control area. Participants also made significantly more reading errors following mid-occipital rTMS. These data further highlight the role of the occipital cortex in the blind in higher cognitive functions.

A journey through the dark: navigation in the absence of vision Vision is undoubtedly an important facilitator of navigation. The access to visual information explains why sighted individuals can easily select a navigational path through a hallway scattered with obstacles. Avoiding obstacles and creating a cognitive map of the environment is obviously more difficult in the absence of vision and remains one of the greatest navigational challenges faced by blind individuals. Notwithstanding, congenitally blind subjects are able to generate spatial representations, probably through tactile, auditory, and olfactory cues, as well as motion-related cues arising from the vestibular and proprioceptive systems, and they preserve the ability to recognize a traveled route and to represent spatial information mentally (Passini et al., 1990; Thinus-Blanc and Gaunet, 1997).

26

One area for which the TDU could be particularly useful in the daily life of the blind is therefore spatial navigation. We tested the potential of the TDU in a series of behavioral and brain imaging studies. In a first study, we trained a group of blind and blindfolded sighted control subjects to use the TDU in a life-size obstacle course (Chebat et al., 2011). The obstacle course was composed of two hallways in which obstacles of different sizes and shapes were placed. Although both groups learned to detect and avoid the obstacles, blind subjects performed significantly better than the sighted. These data underscore the potential of the TDU as a navigational aid in people lacking vision from birth. This brings us to the question concerning the cortical network that is recruited for navigation in the blind. The neural correlates of navigation in congenital blindness have remained largely elusive, mainly because of the difficulty in testing navigational skills of blind subjects in a functional brain imaging study. We circumvented this difficulty by using the TDU (Kupers et al., 2010). During 4 consecutive days, congenitally blind and blindfolded sighted participants were trained in a route navigation and a route recognition task. In the route navigation task, they learned to navigate through two virtual routes that were presented via the TDU, by using the arrow keys of a keyboard. In the route recognition task, the computer program guided the participants automatically through the routes and they then had to indicate which route had been presented. Both groups learned the navigation tasks with the blind slightly outperforming the sighted controls. Following behavioral training, subjects repeated the route recognition task inside the MRI scanner. The fMRI data revealed that during route recognition, blind subjects showed increased BOLD responses in large parts of the visual cortex, the right parahippocampus, posterior parietal cortex, precuneus, and dorsolateral prefrontal cortex (Fig. 3d). These data are in sharp contrast with those of the blindfolded sighted controls who did not show task-dependent BOLD signal increases

in the parahippocampus or in any region of the visual cortex. In a second fMRI experiment, we demonstrated that the areas activated by the blind participants are the same as those activated by sighted subjects when they did the same navigational task under full vision. These data suggest cross-modal plasticity in spatial coding. They also suggest that visual experience is not necessary for the development of a spatial navigation network in the brain, as visual association cortical areas are capable of processing and interpreting spatial information carried by nonvisual sensory modalities.

Subjective experience associated with activation of the visual cortex In the preceding sections, we provided evidence that the occipital cortex in the congenitally blind is recruited by a wide variety of different sensory stimuli and cognitive tasks. It is generally accepted that cortical activity in a certain area produces a subjective sensation within the same domain. Thus, electrical stimulation experiments by Penfield showed that when stimulating the somatosensory cortex, tactile sensations referred to a particular body area are induced. Moreover, the body is somatotopically mapped: inputs from neighboring body parts are encoded in adjacent parts within the primary somatosensory cortex (Penfield and Boldrey, 1937). Transcranial magnetic stimulation (TMS) is a technique that allows stimulation of the cortex in a noninvasive manner (Cowey and Walsh, 2001). A large number of studies have shown that TMS applied over the occipital cortex in normal sighted subjects produces transient perceptions of light, called phosphenes (Kammer et al., 2005). In view of the above finding of cross-modal responses in the occipital cortex of the blind, the question is now which type of sensations will be induced by TMS of the occipital cortex in these subjects. In a first study, we used the TDU to examine the subjective character of experience associated with

27

the activation of occipital cortex before and after the establishment of cross-modal plasticity (Kupers et al., 2006). More specifically, we wanted to test the possibility that stimulation of the occipital cortex can induce subjective sensations associated with the new (tactile) input. Before training, TMS of the occipital cortex elicited phosphenes in the control subjects but not in the congenitally blind. In sharp contrast, following a 1-week training with the TDU, occipital TMS evoked “tactile sensations” on the tongue in the blind subjects (Fig. 4a). These were described as short-lasting tingling sensations, varying in intensity, extent, and topography depending on the precise occipital stimulation site. We found a positive correlation between the amount of occipital cortex activated in a

PET study with the TDU and the number of occipital sites from which TMS-induced tactile sensations could be induced. None of the trained sighted participants reported TMS-induced tactile sensations on the tongue. If tactile sensations referred to the tongue can be induced by stimulating the occipital cortex in blind subjects trained with the TDU, TMS should also be able to induce tactile sensations referred to the fingertips in proficient blind Braille readers. We addressed the question of remapping of the fingers onto the visual cortex in a group of blind Braille readers and Braille-naive normal sighted controls (Ptito et al., 2008a). Again, TMS of the occipital cortex in control subjects evoked only phosphenes. As predicted, blind subjects reported tactile sensations in the fingers,

Fig. 4. Tactile sensations evoked by TMS of the occipital cortex in congenitally blind subjects. (a) Somatotopically organized tactile sensations in the tongue induced by TMS over the occipital cortex in four blind subjects trained with the TDU. The figure shows the areas of the tongue where tactile sensations were felt after TMS of visual cortex. The numbers on the scales refer to the distance (in centimeters) from the inion (after Kupers et al., 2006). (b) Representation of occipital cortex sites that induced tactile sensations in the fingers in two proficient Braille readers. Shown in colors are the areas of the fingers where tactile sensations were felt after TMS stimulation of the occipital cortex. Color scale, red indicates the highest number of cortical sites that induced paresthesiae in a particular finger and purple the lowest number (after Ptito et al., 2008a).

28

varying in intensity, extent, and topography depending on the stimulated occipital area (Fig. 4b). We found again important interindividual differences with respect to the number of sites from which tactile sensations could be induced and in the topography of the referred sensations. The subjects reporting the highest amount of finger paresthesiae were the ones with the best Braille reading performance. The results of these experiments constitute the first direct demonstration that the subjective experience of activity in the visual cortex after sensory remapping is tactile, not visual. They provide new insights into the ongoing scientific debate on cortical dominance or deference (Hurley and Noë, 2003; James, 1890). What is the experience of a subject in whom areas of cortex receive input from sensory sources not normally projecting to those areas? Our studies suggest that the qualitative character of the subject's experience is not determined by the area of cortex that is active (cortical dominance), but by the source of input to it (cortical deference). Our results are also in line with recent evidence that sensory cortical areas receive input from multiple sensory modalities early in development (Falchier et al., 2002; Rockland and Ojima, 2003; Wallace et al., 2004).

Cortical reorganization or unmasking? Two competing hypotheses have been put forward to explain cross-modal plasticity in congenital blindness. According to the cortical reorganization hypothesis, cross-modal brain responses are mediated by the formation of new pathways in the sensory-deprived brain (Bronchti et al., 2002; Chabot et al., 2007, 2008; Desgent et al., 2010). According to the unmasking hypothesis, loss of a sensory input induces unmasking and strengthening of already existing neuronal connections. Although our results with the TDU are compatible with both hypotheses, the rapid onset of cross-modal responses excludes the

possibility of mediation by the establishment of new connections and therefore favors the unmasking hypothesis. One possibility is that training unmasks and strengthens preexisting connections between the parietal and the occipital cortices. There is indeed electrophysiological (Fishman and Michael, 1973) and anatomical (Falchier et al., 2002; Rockland and Ojima, 2003) evidence that primary visual cortex in normal mammals receives input not only from the visual thalamus but also from somatosensory and auditory modalities. These nonvisual inputs conveying tactile and auditory inputs to the occipital cortex may modulate the processing of visual information (Macaluso et al., 2000), while not giving rise to subjective nonvisual sensations under normal circumstances due to masking by the dominant visual input. It is interesting to mention the results of Zangaladze et al. (1999) showing that disrupting the function of the visual cortex by TMS impairs tactile discrimination of grating orientation in normal-seeing subjects. This confirms that although the visual cortex receives tactile input, this normally does not lead to subjective tactile sensations. Thus, in our trained control subjects, TMS over occipital cortex produced only phosphenes, without tactile sensations. However, under certain circumstances, nonvisual processing in the occipital cortex can be strengthened or unmasked. In line with the dynamic sensorimotor hypothesis, training with the TDU device results in new highly specific learned dynamic interaction patterns between sensory stimulation and active movement (O'Regan and Noe, 2001), thereby further strengthening and unmasking existing connections between the parietal and occipital cortices.

Conclusion The study of the blind person's brain has offered new insights regarding the plastic rearrangements that take place when visual input is lacking. It has also lead to a better understanding of the

29

functional organization of the sighted person's brain itself. In this respect, the availability of novel noninvasive brain mapping methodologies has provided a framework for our understanding of the neural mechanisms that enable awareness of the surrounding world. New findings from our own studies as well as from others seem to concur that the blind person's brain should not be considered as a “disabled” brain but rather as a truly “differently able” brain.

Acknowledgments Supported by the Lundbeck Foundation, the Danish Medical Research Council, and the Harland Sanders Chair in Visual Sciences, Canada. References Amedi, A., Jacobson, G., Hendler, T., Malach, R., & Zohary, E. (2002). Convergence of visual and tactile shape processing in the human lateral occipital complex. Cerebral Cortex, 12, 1202–1212. Amedi, A., Malach, R., Hendler, T., Peled, S., & Zohary, E. (2001). Visuo-haptic object-related activation in the ventral visual pathway. Nature Neuroscience, 4, 324–330. Amedi, A., Raz, N., Pianka, P., Malach, R., & Zohary, E. (2003). Early 'visual' cortex activation correlates with superior verbal memory performance in the blind. Nature Neuroscience, 6, 758–766. Bach-y-Rita, P. (1967). Sensory plasticity. Applications to a vision substitution system. Acta Neurologica Scandinavica, 43, 417–426. Bach-y-Rita, P., Collins, C. C., Saunders, F. A., White, B., & Scadden, L. (1969). Vision substitution by tactile image projection. Nature, 221, 963–964. Bach-y-Rita, P., & Kercel, S. (2003). Sensory substitution and the human-machine interface. Trends in Cognitive Sciences, 7, 541–546. Bonino, D., Ricciardi, E., Sani, L., Gentili, C., Vanello, N., Guazzelli, M., et al. (2008). Tactile spatial working memory activates the dorsal extrastriate cortical pathway in congenitally blind individuals. Archives Italiennes de Biologie, 146, 133–146. Bronchti, G., Heil, P., Sadka, R., Hess, A., Scheich, H., & Wollberg, Z. (2002). Auditory activation of “visual” cortical

areas in the blind mole rat (Spalax ehrenbergi). The European Journal of Neuroscience, 16, 311–329. Burton, H., Sinclair, R. J., & McLaren, D. G. (2004). Cortical activity to vibrotactile stimulation: An fMRI study in blind and sighted individuals. Human Brain Mapping, 23, 210–228. Bubic, A., Striem-Amit, E. & Amedi, A. (2010). Large-Scale Brain Plasticity Following Blindness and the Use of Sensory Substitution Devices. In: M.J. Naumer, J. Kaiser (eds.), Multisensory Object Perception in the Primate Brain, 351. Springer, pp. 351–380. Burton, H., Snyder, A. Z., Conturo, T. E., Akbudak, E., Ollinger, J. M., & Raichle, M. E. (2002). Adaptive changes in early and late blind: A fMRI study of Braille reading. Journal of Neurophysiology, 87, 589–607. Capelle, C., Trullemans, C., Arno, P., & Veraart, C. (1998). A real time experimental prototype for enhancement of vision rehabilitation using auditory substitution. IEEE Transactions on Biomedical Engineering, 45, 1279–1293. Cattaneo, Z., Vecchi, T., Cornoldi, C., Mammarella, I., Bonino, D., Ricciardi, E., et al. (2008). Imagery and spatial processes in blindness and visual impairment. Neuroscience and Biobehavioral Reviews, 32, 1346–1360. Chabot, N., Charbonneau, V., Laramée, M. E., Tremblay, R., Boire, D., & Bronchti, G. (2008). Subcortical auditory input to the primary visual cortex in anophthalmic mice. Neuroscience Letters, 433, 129–134. Chabot, N., Robert, S., Tremblay, R., Miceli, D., Boire, D., & Bronchti, G. (2007). Audition differently activates the visual system in neonatally enucleated mice compared with anophthalmic mutants. The European Journal of Neuroscience, 26, 2334–2348. Chebat, D. R., Rainville, C., Kupers, R., & Ptito, M. (2007). Alterations in right posterior hippocampus in early blind individuals. Neuroreport, 18, 329–333. Chebat, D. R., Schneider, F. C., Kupers, R., & Ptito, M. (2011). Navigation with a sensory substitution device in congenitally blind individuals. Neuroreport, 22, 342–347. Cohen, L. G., et al. (1999). Period of susceptibility for crossmodal plasticity in the blind. Annals of Neurology, 45, 451–460. Cowey, A., & Walsh, V. (2001). Tickling the brain: Studying visual sensation, perception and cognition by transcranial magnetic stimulation. Progress in Brain Research, 134, 411–425. Desgent, S., Boire, D., & Ptito, M. (2005). Distribution of calcium binding proteins in visual and auditory cortices of hamsters. Experimental Brain Research, 163, 159–172. Desgent, S., Boire, D., & Ptito, M. (2010). Altered expression of parvalbumin and calbindin in interneurons within the primary visual cortex of neonatal enucleated hamsters. Neuroscience, 171(4), 1326–1340.

30 Doron, N., & Wollberg, Z. (1994). Cross-modal neuroplasticity in the blind mole rat Spalax ehrenbergi: A WGA-HRP tracing study. Neuroreport, 5, 2697–2701. Falchier, A., Clavagnier, S., Barone, P., & Kennedy, H. (2002). Anatomical evidence of multimodal integration in primate striate cortex. The Journal of Neuroscience, 22, 5749–5759. Fishman, M. C., & Michael, P. (1973). Integration of auditory information in the cat's visual cortex. Vision Research, 13, 1415–1419. Fortin, M., Voss, P., Lord, C., Lassonde, M., Pruessner, J., Saint-Amour, D., Rainville, C., & Lepore, F. (2008). Wayfinding in the blind: larger hippocampal volume and supranormal spatial navigation. Brain, 131, 2995–3005. Frost, D. O., Boire, D., Gingras, G., & Ptito, M. (2000). Surgically created neural pathways mediate visual pattern discrimination. Proceedings of National Academy of Sciences of the United States of America, 97, 11068–11073. Frost, D. O., & Metin, C. (1985). Induction of functional retinal projections to the somatosensory system. Nature, 317, 162–164. Gougoux, F., Zatorre, R. J., Lassonde, M., Voss, P., & Lepore, F. (2005). A functional neuroimaging study of sound localization: Visual cortex activity predicts performance in early-blind individuals. PLoS Biology, 3(2), e27. Hurley, S. L., & Noë, A. (2003). Neural plasticity and consciousness. Biology and Philosophy, 18, 131–168. Hyvarinen, J., Hyvärinen, L., & Linnankoski, I. (1991). Modification of parietal association cortex and functional blindness after binocular deprivation in young monkeys. Experimental Brain Research, 42, 1–8. Izraeli, R., Koay, G., Lamish, M., Heicklen-Klein, A. J., Heffner, H. E., Heffner, R. S., et al. (2002). Cross-modal neuroplasticity in neonatally enucleated hamsters: Structure, electrophysiology and behaviour. The European Journal of Neuroscience, 15, 693–712. James, W. (1890). Principles of psychology. New York: Dover. Kaas, J. H. (2002). Sensory loss and cortical reorganization in mature primates. Progress in Brain Research, 138, 167–176. Kammer, T., Puls, K., Erb, M., & Grodd, W. (2005). Transcranial magnetic stimulation in the visual system. II. Characterization of induced phosphenes and scotomas. Experimental Brain Research, 160, 129–140. Kupers, R., Beaulieu-Lefebvre, M., Schneider, F., Paulson, O., Siebner, H., & Ptito, M. (2011). Neural correlates of olfactory processing in congenital blindness. Neuropsychologia, 49, 2037–2044. Kupers, R., Chebat, D. R., Madsen, K. H., Paulson, O. B., & Ptito, M. (2010). Neural correlates of virtual route recognition in congenital blindness. Proceedings of the National Academy of Sciences of the United States of America, 107, 12716–12721.

Kupers, R., Fumal, A., de Noordhout, A. M., Gjedde, A., Schoenen, J., & Ptito, M. (2006). Transcranial magnetic stimulation of the visual cortex induces somatotopically organized qualia in blind subjects. Proceedings of National Academy of Sciences of the United States of America, 103, 13256–13260. Kupers, R., Pappens, M., Ptito, M., & Fumal, A. (2007). rTMS of the occipital cortex abolishes Braille reading and repetition priming in blind subjects. Neurology, 68, 691–693. Lessard, N., Pare, M., Lepore, F., & Lassonde, M. (1998). Early blind human subjects localize sound sources better than sighted subjects. Nature, 395, 278–280. Lyckman, A. W., & Sur, M. (2002). Role of afferent activity in the development of cortical specification. Results and Problems in Cell Differentiation, 39, 139–156. Macaluso, E., Frith, C. D., & Driver, J. (2000). Modulation of human visual cortex by crossmodal spatial attention. Science, 289, 1206–1208. Matteau, I., Kupers, R., & Ptito, M. (2008). Tactile shape recognition through the tongue in the congenitally blind. FENS Abstract, 4, 83, 153.8. Matteau, I., Kupers, R., Ricciardi, E., Pietrini, P., & Ptito, M. (2010). Beyond visual, aural and haptic movement perception: hMTþ is activated by electrotactile motion stimulation of the tongue in sighted and in congenitally blind individuals. Brain Research Bulletin, 82, 264–270. Meijer, P. B. (1992). An experimental system for auditory image representations. IEEE Transactions on Biomedical Engineering, 39, 112–121. Merabet, L. B., & Pascual-Leone, A. (2010). Neural reorganization following sensory loss: The opportunity of change. Nature Reviews. Neuroscience, 11, 44–52. Metin, C., & Frost, D. O. (1989). Visual responses of neurons in somatosensory cortex of hamsters with experimentally induced retinal projections to somatosensory thalamus. Proceedings of the National Academy of Sciences of the United States of America, 86, 357–361. Noppeney, U., Friston, K. J., Ashburner, J., Frackowiak, R., & Price, C. J. (2005). Early visual deprivation induces structural plasticity in gray and white matter. Current Biology, 15, 488–490. O'Regan, J. K., & Noe, A. (2001). A sensorimotor account of vision and visual consciousness. The Behavioral and Brain Sciences, 24, 939–973. Pallas, S. L. (2001). Intrinsic and extrinsic factors that shape neocortical specification. Trends in Neurosciences, 24, 417–423. Pallas, S. L. (2002). Cross-Modal plasticity as a tool for understanding the ontogeny and phylogeny of cerebral cortex. In A. Schuz & R. Miller (Eds.), Cortical areas: Unity and diversity (pp. 245–271). London and New York: Taylor and Francis, Inc.

31 Pan, W. J., Wu, G., Li, C. X., Lin, F., Sun, J., & Lei, H. (2007). Progressive atrophy in the optic pathway and visual cortex of early blind Chinese adults: a voxel-based morphometry magnetic resonance imaging study. Neuroimage, 37, 212–220. Pascual-Leone, A., Amedi, A., Fregni, F., & Merabet, L. B. (2005). The plastic human brain cortex. Annual Review of Neuroscience, 28, 377–401. Passini, R., Proulx, G., & Rainville, C. (1990). The spatio-cognitive abilities of the visually impaired population. Environment and Behavior, 22, 91–118. Penfield, W., & Boldrey, E. (1937). Somatic motor and sensory representation in the cerebral cortex of man as studied by electrical stimulation. Brain, 60, 389–443. Pietrini, P., Furey, M. L., Ricciardi, E., Gobbini, M. I., Wu, W. H., Cohen, L., et al. (2004). Beyond sensory images: Object-based representation in the human ventral pathway. Proceedings of the National Academy of Sciences of the United States of America, 101, 5658–5663. Pietrini, P., Ptito, M., & Kupers, R. (2009). Blindness and consciousness: New lights from the dark. In G. Tononi & S. Laureys (Eds.), The neurology of consciousness (pp. 360–374). Amsterdam: Elsevier. Ptito, M., & Desgent, S. (2006). Sensory input-based adaptation and brain architecture. In P. Baltes, P. ReuterLorenz & F. Rösler (Eds.), Lifespan development and the brain (pp. 111–123). Cambridge: Cambridge University Press. Ptito, M., Fumal, A., de Noordhout, A. M., Schoenen, J., Gjedde, A., & Kupers, R. (2008a). TMS of the occipital cortex induces tactile sensations in the fingers of blind Braille readers. Experimental Brain Research, 184, 193–200. Ptito, M., Giguère, J. F., Boire, D., Frost, D. O., & Casanova, C. (2001). When the auditory cortex turns visual. Progress in Brain Research, 134, 447–458. Ptito, M., Moesgaard, S., Gjedde, A., & Kupers, R. (2005). Cross-modal plasticity revealed by electrotactile stimulation of the tongue in the congenitally blind. Brain, 128, 606–614. Ptito, M., Schneider, F., Paulson, O. B., & Kupers, R. (2008b). Alterations of the visual pathways in congenital blindness. Experimental Brain Research, 87, 41–49. Raz, N., Amedi, A., & Zohary, E. (2005). V1 activation in congenitally blind humans is associated with episodic retrieval. Cerebral Cortex, 15, 1459–1468. Rebillard, G., Carlier, E., Rebillard, M., & Pujol, R. (1977). Enhancement of visual responses on the primary auditory cortex of the cat after an early destruction of cochlear receptors. Brain Research, 129, 162–164. Ricciardi, E., Vanello, N., Sani, L., Gentili, C., Scilingo, E. P., Landini, L., et al. (2007). The effect of visual experience on the development of functional architecture in hMTþ. Cerebral Cortex, 17, 2933–2939.

Rockland, K. S., & Ojima, H. (2003). Multisensory convergence in calcarine visual areas in macaque monkey. International Journal of Psychophysiology, 50, 19–26. Röder, B., Stick, O., Bien, S., Neville, H., & Rösler, F. (2002). Speech processing activates visual cortex in congenitally blind humans. The European Journal of Neuroscience, 16, 930–936. Röder, B., Teder-Sälejärvi, W., Sterr, A., Rösler, F., Hillyard, S. A., & Neville, H. (1999). Improved auditory spatial tuning in blind humans. Nature, 400, 162–166. Schacter, D. L., Wig, G. S., & Stevens, W. D. (2007). Reductions in cortical activity during priming. Current Opinion in Neurobiology, 17, 171–176. Shimony, J. S., Burton, H., Epstein, A. A., McLaren, D. G., Sun, S. W., & Snyder, A. Z. (2006). Diffusion tensor imaging reveals white matter reorganization in early blind humans. Cerebral Cortex, 16, 1653–1661. Smeets, M. A. M., Schifferstein, H. N. J., Boelema, S. R., & Lensvelt-Mulders, G. (2008). The Odor Awareness Scale (OAS): A new scale for measuring positive and negative odour awareness. Chemical Senses, 33, 725–734. Stevens, A. A., Snodgrass, M., Schwartz, D., & Weaver, K. (2007). Preparatory activity in occipital cortex in early blind humans predicts auditory perceptual performance. The Journal of Neuroscience, 27, 10734–10741. Thinus-Blanc, C., & Gaunet, F. (1997). Representation of space in blind persons: Vision as a spatial sense? Psychological Bulletin, 121, 20–42. Toldi, J., Rojik, I., & Fehér, O. (1994). Neonatal monocular enucleation-induced cross-modal effects observed in the cortex of adult rat. Neuroscience, 62, 105–114. Ungerleider, L. G., & Mishkin, M. (1982). Two cortical visual systems. In D. J. Ingle, M. A. Goodale & R. J. W. Mansfield (Eds.), Analysis of visual behavior (pp. 549–586). Cambridge, MA: MIT Press. Von Melchner, L., Pallas, S. L., & Sur, M. (2000). Visual behaviour mediated by retinal projections directed to the auditory pathway. Nature, 404, 871–876. Wallace, M. T., Ramachandran, R., & Stein, B. E. (2004). A revised view of sensory cortical parcellation. Proceedings of the National Academy of Sciences of the United States of America, 101, 2167–2172. Ward, J., & Meijer, P. (2010). Visual experiences in the blind induced by an auditory sensory substitution device. Consciousness and Cognition, 19, 492–500. Wittenberg, G. F., Werhahn, K. J., Wassermann, E. M., Herscovitch, P., & Cohen, L. G. (2004). Functional connectivity between somatosensory and visual cortex in early blind humans. The European Journal of Neuroscience, 20, 1923–1927. Zangaladze, A., Epstein, C. M., Grafton, S. T., & Sathian, K. (1999). Involvement of visual cortex in tactile discrimination of orientation. Nature, 401, 587–590.

A. M. Green, C. E. Chapman, J. F. Kalaska and F. Lepore (Eds.) Progress in Brain Research, Vol. 192 ISSN: 0079-6123 Copyright Ó 2011 Elsevier B.V. All rights reserved.

CHAPTER 3

A dynamical systems view of motor preparation: Implications for neural prosthetic system design Krishna V. Shenoy{,{,},k,*, Matthew T. Kaufman{, Maneesh Sahani{,} and Mark M. Churchland{,{ {

}

Department of Electrical Engineering, Stanford University, Stanford, California, USA { Neurosciences Program, Stanford University, Stanford, California, USA } Department of Bioengineering, Stanford University, Stanford, California, USA Gatsby Computational Neuroscience Unit, University College London, London, United Kingdom k Department of Neurobiology, Stanford University, Stanford, California, USA

Abstract: Neural prosthetic systems aim to help disabled patients suffering from a range of neurological injuries and disease by using neural activity from the brain to directly control assistive devices. This approach in effect bypasses the dysfunctional neural circuitry, such as an injured spinal cord. To do so, neural prostheses depend critically on a scientific understanding of the neural activity that drives them. We review here several recent studies aimed at understanding the neural processes in premotor cortex that precede arm movements and lead to the initiation of movement. These studies were motivated by hypotheses and predictions conceived of within a dynamical systems perspective. This perspective concentrates on describing the neural state using as few degrees of freedom as possible and on inferring the rules that govern the motion of that neural state. Although quite general, this perspective has led to a number of specific predictions that have been addressed experimentally. It is hoped that the resulting picture of the dynamical role of preparatory and movement-related neural activity will be particularly helpful to the development of neural prostheses, which can themselves be viewed as dynamical systems under the control of the larger dynamical system to which they are attached. Keywords: premotor cortex; motor cortex; motor preparation; state space; dynamical systems; singletrial analysis; neural prostheses; brain machine interface; brain computer interface.

Introduction *Corresponding author. Tel.: þ1-650-723-4789; Fax: þ1-650-723-4659. E-mail: [email protected] DOI: 10.1016/B978-0-444-53355-5.00003-8

It is difficult to appreciate just how central movement is to everyday life until this ability is lost due 33

34

to neurological injury or disease. Moving is how we interact and communicate with the world. We move our legs and feet to walk, we move our arms and hands to manipulate the objects that surround us, and we move our tongues and vocal cords to speak. Movement is not only central to these critical aspects of life, but also to self-image and psychological well-being. In fact, the fundamental reason that tetrapelgics wish most for the restored use of their arms is to regain some degree of independence (Anderson, 2004). Fortunately, it appears that a confluence of knowledge and technology from the fields of (1) systems motor neuroscience, (2) neuroengineering, and (3) electrical engineering and computer science may soon provide a new class of electronic medical systems (termed neural prosthetic systems, brain machine interfaces, or brain computer interfaces) aimed at increasing the quality of life for severely disabled patients. First, basic neuroscience research across the past several decades has elucidated many of the fundamental principles underlying movement generation and control. A substantial body of knowledge regarding the cortical control of arm movements, particularly in rhesus macaques, now exists (e.g., Evarts, 1964; Georgopoulos et al., 1982, 1986; Schwartz, 1994; Tanji and Evarts, 1976). This literature is reviewed elsewhere (e.g., Kalaska, 2009; Kalaska et al., 1997; Scott, 2004; Wise, 1985). As discussed below, this understanding has been sufficient to help guide the design of first generation prosthetic systems. Yet continued focus on underlying neural mechanisms (in both monkeys and humans), how neural populations behave across timescales, and how neural populations participate in the ongoing control of movement, is essential for creating second generation prostheses capable of higher performance and a greater range of capabilities (e.g., Cunningham et al., 2010; Green and Kalaska, 2010; Truccolo et al., 2008, 2010). Second, basic neuroengineering research has provided proof-of-concept demonstrations of neural prosthetic systems which translate the electrical

activity (action potentials and local field potentials, LFPs) from populations of intracortically recorded neurons into control signals for guiding computer cursors, prosthetic arms, or stimulating the paralyzed musculature. More specifically, a series of designs and demonstrations across the past decade have produced compelling laboratory evidence that intracortical neural signals from rodents (e.g., Chapin et al., 1999), monkeys (e.g., Carmena et al., 2003; Chase et al., 2009; Fetz, 1969; Fraser et al., 2009; Ganguly and Carmena, 2009; Gilja et al., 2010b,c; Heliot et al., 2009; Humphrey et al., 1970; Isaacs et al., 2000; Jackson et al., 2006; Jarosiewicz et al., 2008; Moritz et al., 2008; Mulliken et al., 2008; Musallam et al., 2004; Nuyujukian et al., 2010; Santhanam et al., 2006; Serruya et al., 2002; Shenoy et al., 2003; Taylor et al., 2002; Velliste et al., 2008; Wessberg et al., 2000; Wu et al., 2004), and humans (e.g., Hochberg et al., 2006; Kim et al., 2008) can control prosthetic devices that may provide meaningful quality of life improvement to paralyzed patients. This literature is reviewed elsewhere (e.g., Andersen et al., 2010; Millan and Carmena, 2010; Donoghue, 2008; Donoghue et al., 2007; Fetz, 2007; Hatsopoulos and Donoghue, 2009; Linderman et al., 2008; Nicolelis and Lebedev, 2009; Ryu and Shenoy, 2009; Scherberger, 2009; Schwartz, 2007). electronics, Finally, the semiconductor optoelectronic telecommunications, micro-electromechanical systems (MEMS), and information technology revolutions over the past four decades have produced extraordinary and relevant technologies. These include low-power and high computational-density circuits and systems, lowpower wireless telemetric systems, advanced light sources and imaging modalities, and small sensor systems that are capable of running sophisticated signal processing algorithms. These technologies have progressed extremely quickly, as described by Moore's Law, and have been leveraged and adapted to create new neurotechnologies for basic neuroscience and neuroengineering applications such as neural prosthetic systems. It is now possible to record from hundreds of neurons simultaneously

35

with bio-MEMS electrode arrays (e.g., Chestek et al., 2009a, 2011; Jackson and Fetz, 2007; Mavoori et al., 2005; Santhanam et al., 2007), filter and “spike sort” all channels in real time (e.g., O'Driscoll et al., 2006; Santhanam et al., 2004, 2006), “decode” the intended arm movement with advanced algorithms (e.g., Achtman et al., 2007; Cunningham et al., 2008; Kemere et al., 2004, 2008; Santhanam et al., 2009; Wu et al., 2006; Yu et al., 2007, 2010), wirelessly telemeter the resulting prosthetic arm control signals with just a few tens of milliwatts of power (e.g., Chestek et al., 2009b; Gilja et al., 2010a; Harrison et al., 2007, 2009), and soon, this will likely all be possible in fully implantable systems (e.g., Borton et al., 2009; Harrison, 2008; Nurmikko et al., 2010). While these laboratory proof-of-concept systems and initial FDA phase-I clinical trials are encouraging (e.g., Hochberg, 2008; Hochberg and Taylor, 2007), several barriers remain. If these barriers are unaddressed, they could substantially limit the prospect of intracortically based neural prosthetic systems having a broad and important clinical impact. We recently reviewed what we consider to be three of the most important neuroengineering, bioengineering, electrical engineering, and computer science challenges and opportunities for intracortically based neural prostheses (Gilja et al., 2011). We review here what we consider to be one of the most central and important basic systems-level motor neuroscience questions. The knowledge gained while investigating this question should directly advance our ability to design high-performance neural prostheses. The central question we have been asking is: what are the neural processes that precede movement and lead to the initiation of movement? Neural prostheses will benefit from a deeper and more comprehensive understanding of the neural activity upon which they are based (Green and Kalaska, 2010). This includes activity during both movement preparation and movement generation. We need to understand both because prostheses use both

(e.g., Yu et al., 2010), and because the two are presumably causally linked and likely impossible to understand fully if studied in isolation (discussed further in the final section, and Fig. 12). Prostheses should thus benefit from having a firm scientific understanding of how preparatory activity relates to upcoming arm movements, and how this preparatory activity evolves on a millisecond timescale. These are the questions and topics discussed in this review.

Preparing to move the arm Why should one prepare and then move, as opposed to starting the movement as soon as possible? In some cases, it is critical to move right away, such as when withdrawing a hand from a flame. Animals have evolved low-latency circuits to help in these cases and these circuits underlie a wide range of reflexive movements. However, animals have also evolved circuits to enable voluntary movements which are intentional and purposeful. Voluntary movements require the ability to change, refine, and suppress possible actions before they are actually executed. A simple example is how we swat a fly. One approach would be to see a fly and start moving right away. Unless the nervous system can execute perfectly, this is unlikely to be a good strategy, and if the initial movement is not successful, the fly is likely to depart before a correction can be made. It would thus be beneficial to take slightly more time to initiate the movement, assuming that, in doing so, greater accuracy can be gained. Presumably, we use this slight addition of time to create and refine movement plans until the moment is right and then we initiate the movement. It is this form of deliberate, goal-driven movement that we seek to better understand, both neurally and behaviorally, both out of scientific curiosity and because it could lead to superior prosthetic designs.

36 (a)

(b)

Horizontal target position

cm

12

Horizontal hand position Central spot

0 Hand velocity 100 cm/s

There is indeed evidence that voluntary movements are prepared before they are initiated (e.g., Day et al., 1989; Ghez et al., 1997; Keele, 1968; Kutas and Donchin, 1974; Riehle and Requin, 1993; Rosenbaum, 1980; Wise, 1985). An important line of evidence comes from “instructed-delay tasks” where a temporal delay separates an instruction stimulus from a subsequent “go” cue. Figure 1 illustrates the experimental arrangement and task timing, along with example hand position and electromyographic (EMG) measurements. This task is widely employed and is the behavioral task used in the recent studies reviewed here. At the behavioral level, reaction times (RTs), defined as the time from the go cue until movement onset, are shorter after an instructed-delay period. Figure 2 illustrates how RT decreases and then plateaus as a function of delay period. This RT reduction with delay, largely occurring during the first 200 ms, suggests that some timeconsuming preparatory process is given a head start by the delay (e.g., Crammond and Kalaska, 2000; Riehle and Requin, 1989; Rosenbaum, 1980). It is straightforward to interpret the importance of this head start on preparation in the context of the fly swatting example offered above. There the goal was not to move instantaneously as soon as the fly landed or was seen. Instead, the goal was to move swiftly and accurately, at a particular speed and along a particular path that perhaps approaches from behind, and to be able to start that movement as quickly as possible when it is decided that the time is right. Thus, a good strategy is to prepare the desired movement as soon as possible, so that one is ready to move as soon as possible when called upon to do so. The ability to prepare a movement ahead of time is presumably related to the preparatory activity that is widespread in cortex and subcortical structures. Neurons in a number of cortical areas including dorsal premotor cortex (PMd)

EMG

0

T

200 ms

G

M

Fig. 1. Illustration of the instructed-delay task, hand measurements, and EMG recordings. (a) Monkeys sit in a primate chair approximately 25 cm from a fronto-parallel display. Movements begin and end with the hand touching the display. The hand is a few millimeters from the screen while in flight. The white trace shows the reach trajectory for one trial. (b) Time line of the task and behavior for the same trial. T, target onset; G, go cue; and M, movement onset. Horizontal hand (black) and target (red) position is plotted (top). The target jittered on first appearing and ceased at the go cue. Bottom: Gray trace plots hand velocity (computed in the direction of the target), superimposed on the voltage recorded from the medial deltoid (arbitrary vertical scale). Traces end at the time of the reward. Data are from monkey A in a session focused on EMG recordings. Figure adapted from Churchland et al. (2006b).

37

Fig. 2. Mean RT (in milliseconds) is plotted versus delay period duration. For monkeys A and B, this was for the catch trials with short delays. Although the delay period was selected from a continuum, in practice, delay periods were integer multiples of 16 ms because of video presentation, and this binning is used in the plots. Lines show exponential fits. For monkey G, we did not use catch trials (the minimum delay for most experiments was already quite short, at 200 ms). The plotted data are therefore from one experiment using three discrete delay durations (30, 130, and 230 ms; black symbols) and another (performed the previous day) using a continuous range (200–700 ms; white symbols). For the latter, data have been binned (ranges shown in parentheses). From Churchland et al. (2006c).

and primary motor cortex (M1) show changes in activity during the delay period (e.g., Crammond and Kalaska, 2000; Godschalk et al., 1985; Kalaska et al., 1997; Kurata, 1989; Messier and Kalaska, 2000; Riehle and Requin, 1989; Snyder et al., 1997; Tanji and Evarts, 1976; Weinrich et al., 1984). Figure 3 shows four example PMd neurons. While it is typical for the average action potential emission (firing) rate during the delay period to change following target onset, the temporal structure is widely varying across cells: some increase their firing rate, some decrease, some arrive at an approximate plateau level, while others undulate. This variety of neural responses stands in stark contrast to the simple monotonic decline in behavioral RT as shown in Fig. 2. The central question is, therefore, how does neural activity in the first 200–300 ms of the delay period relate

to the decrease in RT? Asked in the context of the fly swatting example, what does this neural activity need to accomplish during the delay so that we are maximally poised to generate the planned movement and, after initiating the movement, successfully hit the fly?

Optimal subspace hypothesis We have been investigating this question using a dynamical systems perspective (e.g., Briggman et al., 2005; Churchland et al., 2007; Stopfer et al., 2003). What this means in essence is that we wish to understand (1) how the activity of a neural population evolves and achieves the needed preparatory state, (2) how this preparatory state impacts the subsequent arm movement, and (3) what the underlying dynamics (rules) of

Spikes/s

38

Cell B46

Cell A2

Spikes/s

Cell B16

Cell B29

Target

Fig. 3. Examples of typical delay-period responses in PMd. (a) Mean  SE firing rates for four example neurons. Three of these showed increases in firing rate after target onset, whereas one showed a decrease. Data are from experiments using a continuous range of delay periods (500–900 for monkey B and 400–800 for monkey A). For each time point, mean firing rate was computed from only those trials with a delay period at least that long. Labels give the monkey initial and cell number. Details (direction, distance, instructed speed, and trials/condition) were as follows: cell B29, 45 , 85 mm, fast, 23 trials; cell B16, 135 , 60 mm, fast, 20 trials; cell B46, 335 , 85 mm, fast, 41 trials; cell A2, 185 , 120 mm, slow, 42 trials. From Churchland et al. (2006c).

the neural circuitry are (Churchland et al., 2007; Yu et al., 2006, 2009). We start with as simple an assumption as possible: the arm movement made (M(t)) depends upon preparatory activity (P) at the time movement activity begins to be generated (t0). In other words, M(t) depends on P(t0). It is important to note that there are likely sources of variability that impact M(t) but are not accounted for in P(t0), such as downstream variability in the state of the spinal cord or muscles. Thus, to be strictly true, a noise source should be included, or P(t0) would need to be the initial state of the entire animal. However, for the moment, we avoid this issue and simply concentrate on the hypothesis that the movement you make is in large part a function of the plan that was present just before movement began. Also, note that the above conception does not rule out a strong (or even dominant) role for feedback. Such feedback could be part of the causal mechanism by which the plan produces the movement. The central implication of our assumption that M(t) depends on P(t0) is that motor preparation

may be the act of optimizing preparatory activity (i.e., bringing P to the state needed at t0) so that the generated movement has the desired properties. In the case of monkeys performing reaching movements, the desired movement can be defined as a reach that is accurate enough to result in reward. Consider the space of all possible preparatory states (all possible Ps). For a given reach, there is presumably some small subregion of space containing those values of P that are adequate to produce a successful reach. Although the response of each neuron (i.e., tuning) may not be easily parameterized, there is nonetheless a smooth relationship between firing rate and movement. Therefore, the small subregion of space is conceived of as being contiguous. Figure 4 illustrates this idea. We conceive of all possible preparatory states as forming a space, with the firing rate of each neuron contributing an axis. Each possible state—each vector of possible firing rates—is then a point in this space. For a given reach (e.g., rightwards), there will be some subset of states (gray region in Fig. 4, referred to as the optimal subspace) that will

39 Neuron 3

Left reach Right reach Trial 1 Trial 2

Neuron 2

Firing rate, neuron 1 Fig. 4. Illustration of the optimal subspace hypothesis. The configuration of firing rates is represented in a state space, with the firing rate of each neuron contributing an axis, only three of which are drawn. For each possible movement, we hypothesize that there exists a subspace of states that are optimal in the sense that they will produce the desired result when the movement is triggered. Different movements will have different optimal subspaces (shaded areas). The goal of motor preparation would be to optimize the configuration of firing rates so that it lies within the optimal subspace for the desired movement. For different trials (arrows), this process may take place at different rates, along different paths, and from different starting points. From Churchland et al. (2006c).

result in a successful reach that garners a reward. Under this optimal subspace hypothesis, the central goal of motor preparation is to bring the neural state within this subspace before the movement is triggered. This may occur in different ways on different trials (trial 1 and trial 2 in Fig. 4). This framework, though rather general, has provided us with a number of specific and testable predictions, which we review below. Before doing so, it is worth considering that an almost-trivial prediction of the optimal subspace hypothesis is that different movements require different initial states. If preparatory activity has a strong role in determining movement, then making different movements will require different patterns of preparatory activity. The overall

neural state, and thus the state of individual neurons, should therefore vary with different movements. This is of course consistent with the observation that preparatory activity is tuned for reach parameters such as direction and distance (e.g., Messier and Kalaska, 2000). In fact, under the optimal subspace hypothesis, neural activity should appear tuned for essentially every controllable aspect of the upcoming reach (a prediction we will return to shortly). As a brief aside on the topic of tuning, we note that one could conceive of each axis in Fig. 4 as capturing not the activity of a single neuron, but rather the activity of a population of neurons that are all tuned for the same thing. Thus, the three axes might capture, respectively, the average activity of neurons tuned for direction, distance, and speed. If so, the preparatory state could be thought of as an explicit representation of direction, distance, and speed. However, it has been argued that few individual neurons appear tuned for reach parameters in the straightforward and invariant way that one might hope (e.g., Churchland et al., 2006b; Churchland and Shenoy, 2007b; Cisek, 2006; Fetz, 1992; Scott, 2004, 2008; Todorov, 2000). The optimal subspace hypothesis is largely agnostic to this debate. So long as there is a systematic relationship between preparatory activity and movement, the optimal subspace conception remains viable. Put another way, the space illustrated in Fig. 4 could have axes that capture well-defined parameters, but it need not, and there are reasons to suspect that it does not. A related and critical point is that the space in which neural activity evolves is certainly larger than the three dimensions illustrated in Fig. 4. Movements vary from one another in more than three different ways. Similarly, neural activity varies across movements in more than three different ways (Churchland and Shenoy, 2007b). Thus, care should be taken when gleaning intuition from illustrations such as that in Fig. 4, to keep in mind that what is illustrated is a projection of a larger and richer space (Churchland et al., 2007; Yu et al., 2009).

40 (a)

(b)

Perhaps slightly faster? Instructed fast

Perhaps slightly slower?

100 spikes/s

We now review a number of specific and testable predictions of the optimal subspace hypothesis.

Prediction 1: Reach-speed modulation T

Instructed slow

(c)

Delay

G

M

Instructed fast

20

Instructed slow

Δ spikes/s

20 –20

Neuron B24 –20

p < 10–5 –20 Δ cm/s 20 p < 10–6 –20 Δ cm/s 20

(d)

15 Δ spikes/s

Figure 5a illustrates in state space our first prediction under the optimal subspace hypothesis: preparatory activity should covary with other meaningful aspects of movement, including peak reach speed. Confirming this would be consistent with our assumption above, whereas failing to find this would be consistent with preparatory activity having a higher-level, perhaps more sensory role reflecting the target location but not the more detailed aspects of movement. To test this prediction, we trained monkeys to reach to targets in a variant of the instructed-delay task. Reaches must be made somewhat faster ( 1.5 m/s peak speed) when the target was red and somewhat slower ( 1.0 m/s peak speed) when the target was green (Churchland et al., 2006b). All other movement metrics such as reach path remained similar. Delay-period activity was substantially different ahead of fast and slow arm movements to the same target location. Figure 5b shows the average response of an example neuron, ahead of reaches to a particular target, where the delay-period activity was greater ahead of fast reaches (red) than ahead of slow reaches (green). Figure 5c and d show two more example neurons where this difference in preparatory activity ahead of fast (red) and slow (green) reaches is emphasized by collapsing across all reach target locations. Some neurons had higher average firing rates ahead of fast movements (Fig. 5c), while other neurons had higher average rates ahead of slow movements (Fig. 5d). In sum, prediction 1 as illustrated in Fig. 5a appears to be correct.

Neuron A06 –15

15

p < 0.01 –20 Δ cm/s

20

p < 0.005

–15

–20 Δ cm/s 20

Fig. 5. Predicted and measured relationships between neural activity and reach velocity. (a) The prediction that preparatory activity should covary with instructed reach speed (prediction 1) is visualized in the state-space framework. Two optimal subspaces are illustrated: one shaded red for the fast instruction and another shaded green for the slow instruction. The prediction that preparatory activity should correlate, on a trial-by-trial basis, with peak reach speed (prediction 2) can also be visualized in this state space. For example, an instructed-fast trial with a slowerthan-typical actual reach speed should have a preparatory state toward one end of the “fast” optimal subspace, nearer to the “slow” optimal subspace. (b) Examination of the first prediction: the mean firing rate is plotted as a function of time for one neuron, one target location, and both instructed speeds. For this neuron, the mean firing rate was highest when preparing a fast reach. Other neurons showed the opposite pattern. “T,” “G,” and “M” indicate target onset,

41

Prediction 2: Reach-speed (trial-by-trial) correlation Figure 5a also illustrates in state space our second prediction under the optimal subspace hypothesis: preparatory activity should correlate, on a trialby-trial basis, with the peak reach speed. Our assumption that M(t) depends on P(t0) predicts that even a slightly different P(t0) value should lead to a different M(t). If P(t0) reflects the result of a difficult optimization, then variability in P(t0) is likely. Therefore, it should be possible to observe a trial-by-trial correlation between P(t0) and movement metrics M(t). To test this prediction, we again employed the reach-speed variant of the instructed-delay task. We found trial-by-trial correlations between the firing rate of individual neurons before the go cue and peak reach speed (Churchland et al., 2006a). Consider the instructed-fast condition (red) in Fig. 5c. The horizontal spread of points (one point per trial) reflects the trial-to-trial variance in peak reach speed. The vertical spread of points largely reflects the trial-to-trial variance in

the go cue, and the median time of movement onset. (c) Trialby-trial correlation between preparatory firing rate and peak reach speed. Data are from one neuron (B24). Each dot plots the mean delay-period firing rate versus peak reach speed for one trial. Trials have been pooled across target locations (for this neuron all locations involved a preference for the fast instruction). To allow pooling, firing rates and peak speeds are expressed relative to the mean for the relevant condition. The offset between the left subpanel (instructed-slow reaches) and the right subpanel (instructedfast reaches) indicates the degree to which firing rates were on average higher for the instructed-fast condition. This defines a “predicted slope” (gray line) with which one can compare the slopes computed from the trial-by-trial variability (black lines). (d) Similar plot but for a second neuron (A06) for which the average firing rate was higher for instructed-slow reaches. In agreement, the trial-by-trial correlations show negative slopes. Figure adapted from Churchland et al. (2006a).

estimated firing rate, which is the inevitable result of it being difficult to assess the firing rate of a single neuron on a single trial from a handful of stochastically occurring spikes. Nevertheless a statistically significant correlation was found for most neurons and for both instructed speeds. Importantly, the state-space illustration (Fig. 5a) further predicts that within the instructed-fast condition, for example, a trial with a slightly slower peak reach speed should have a preparatory state slightly closer to those found in the instructed-slow condition. In other words, if we assume that movement parameters are mapped smoothly from firing rate, the slope of the within-condition correlation (black lines) should agree with the slope of the across-condition mean line fit (gray line) both when the instructed-fast condition had a higher average firing rate (Fig. 5c) and when it had a lower average firing rate (Fig. 5d). We found this to be the case in the majority of neurons (Churchland et al., 2006a). In sum, prediction 2 as illustrated in Fig. 5a appears to be correct.

Prediction 3: Across-trial firing-rate variance (Fano factor) reduces through time Figure 6a illustrates in state space our third prediction under the optimal subspace hypothesis: preparatory activity should become, through time, quite accurate and therefore quite similar across trials. Before the target appears, “baseline” neural activity can be somewhat different from trial to trial, leading to some amount of across-trial firing-rate variance (black circles in panel labeled “before target onset” in Fig. 6a, top). After target onset, and for the coming 200–300 ms, preparatory activity on each trial is nominally being optimized and brought to reside within the optimal subspace. The optimal subspace is presumably rather restricted by virtue of the behavioral task constraints and thus should have less across-trial firing-rate variance (black

(b)

(a) Neuron 3

Before target onset

Fano factor

42

Hand speed Neuron 2

Target

Firing rate, Neuron 1

30 ms 130 ms 230 ms

Mean RT (ms)

Neuron 3

(d)

(c) Fano factor

~200 ms after target onset

30 ms

130 ms 230 ms

Neuron 2

Target Firing rate, Neuron 1

Fano factor around time of go cue

Fig. 6. State-space view of prediction 3, and Fano factor relationship with RT in the instructed-delay task. (a) State-space illustration, as in Fig. 4, showing the state of several trials (black circles) from before target onset until converging within the optimal subspace (gray shaded region) approximately 200 ms later. (b) Fano factor reduces from the time of target onset, approximately holds at a plateau level throughout delay periods longer than 200 ms, and then reduces further following the go cue. (c) By employing shorter delay periods, where the go cue comes at 30, 130, or 230 ms after target onset (colored arrows), it is possible to ask if the resulting RTs are longer when the Fano factor is higher (prediction 4-I). (d) Mean  SEM measured RT plotted against mean  measured Fano factor for the three delay durations (monkey G). A clear correlation is observed, with longer delay durations leading to both lower Fano factors and lower RTs as predicted. As a technical aside, when measuring across-trial variability our later publications (e.g., Churchland et al. 2007, Churchland et al. 2010c) employed the Fano factor while our original publication (Churchland et al. 2006c) employed the closely related 'normalized variance'. The above plots are reproduced from that original manuscript, and it should thus be kept in mind that the vertical axis is not technically the Fano factor, because spiking was assessed in a Gaussian window rather than a square counting window. That said, results are very similar regardless of the exact window shape (e.g., Churchland et al. 2007, Fig. 4).

circles within the optimal subspace shaded gray, in the panel labeled “200 ms after target onset’ in Fig. 6a, bottom). To test this prediction, we analyzed data from an instructed-delay task using the Fano factor: the across-trial spike-count variance divided by the mean (Churchland et al., 2006c, 2007, 2010c). Normalization, and an additional set of controls, is necessary to ensure that the measured changes in variance are not simply due to the well-known scaling of spike-count variance with spike-count mean (as happens, e.g., for a Poisson

process; Churchland et al., 2007, 2010c; Rickert et al., 2009). As shown in Figure 6b, we found that the Fano factor declines over the course of approximately 200 ms and then approximately plateaus (Churchland et al., 2006c). This is somewhat remarkable, as it so closely resembles the decline and plateau seen in the behavioral curves (RT versus delay, Fig. 2). In sum, prediction 3 as illustrated in Fig. 6a appears to be correct. Moreover, it appears that the across-trial firing-rate variance (as measured by the Fano factor) parallels the reduction in

Fano factor Spikes per s

Mean rate 39

11

Mean matched 1 1.5

7 2

V1

Fano factor

V4

1

1 23 10 1.4

28

7 1.4

MT

1

1

26

17

8

12 1.2

LIP

MT

PRR

1.4

1

1

13 4 1.15

14 12 PMd

1.4

0.95

1.2

31

21

5 1.9

MT

1.3

6 1.7

OFC

MT

1.2 200 ms

200 ms

Fig. 7. Changes in firing-rate variability for 10 datasets (one per panel). Insets indicate stimulus type. Data are aligned on stimulus onset (arrow). For the two bottom panels (MT area/direction and MT speed), the dot pattern appeared at time zero (first arrow) and began moving at the second arrow. The mean rate (gray) and the Fano factor (black with flanking SEM) were computed using a 50-ms sliding window. For OFC, where response amplitudes were small, a 100-ms window was used to gain statistical power. Analysis included all conditions, including nonpreferred. The Fano factor was computed after mean matching (Churchland et al., 2010c). The resulting stabilized means are shown in black. The mean number of trials per condition was 100 (V1), 24 (V4), 15 (MT plaids), 88 (MT dots), 35 (LIP), 10 (PRR), 31 (PMd), 106 (OFC), 125 (MT direction and area), and 14 (MT speed). Figure from Churchland et al. (2010c).

44

Predictions 4-I and 4-II: Lower Fano factor and lower RTs Figure 6c illustrates the first part of our fourth prediction (prediction 4-I) under the optimal subspace hypothesis: the lower the across-trial firingrate variance at the time of the go cue, the lower too should be the RT. Having seen the similarity between how the Fano factor descends and holds as a function of delay duration (Fig. 6b), and how RT descends and holds as a function of RT (Fig. 2), it is natural to predict that there should exist a positive correlation between Fano factor and RT. For example, one expects that short delays should lead to high Fano factors and high RTs, while long delays should lead to low Fano factors and low RTs. To test this prediction, we analyzed short delayduration trials from the instructed-delay task.

(a)

Long RT¢s

Short RT¢s

(b) Go

Fano factor

RT: both drop over the course of approximately 200 ms and then hold at that level. This possibility is explored below, as predictions 4-I and 4-II. As a brief aside, it could be the case that this reduction in across-trial firing-rate variance is principally a motor phenomenon and is specific to the preparation of arm movements. However, we found that this same general structure of a reduction in across-trial firing-rate variance following a stimulus onset is present across much if not all of cerebral cortex (Churchland et al., 2010c). Figure 7 shows a substantial reduction in Fano factor following stimulus onset in numerous areas, across all four cortical lobes, and in a variety of behaviors. This reduction seems to be a general property of the nervous system responding to an input, much as the mean (across-trial) firing-rate changing is a general property of cortical neurons. This reduction in across-trial firing-rate variance in each area may be correlated with the relevant functions performed therein (e.g., sensation, cognition, behavior), again just as the mean firing rate is well known to correlate with the function of each area.

Long RT¢s

Short RT¢s Target Fig. 8. State-space view of prediction 4-II, and relationship of the Fano factor to natural RT variability. (a) State-space view of prediction 4-II. The shaded area represents the optimal subspace for the movement being prepared, as in Fig. 4. Each dot corresponds to one trial and represents the configuration of firing rates around the time of the go cue. For some trials, that configuration may lie within the optimal subspace (green dots), leading to a short RT. For other trials, the configuration may lie outside (red dots), leading to a longer RT. (b) Red and green traces show the Fano factor, around the time of the go cue, for trials with RTs longer and shorter than the median. Data were pooled across the recordings from 7 days (monkey G), including all trials with delay periods > 200 ms. Figure adapted from Churchland et al. (2006c).

Figure 2 shows representative RT data from monkey G when 30, 130, and 230 ms delay durations were used. Figure 6c shows the Fano factor at the three critical times: 30, 130, and 230 ms after target onset. Figure 6d shows RT data plotted against Fano factor data, from the same trials in Monkey G, and a clear relationship is seen. The lower the across-trial firing-rate variance at the time of the go cue (as measured by the Fano factor), the lower the RT (Churchland et al., 2006c).

45

Figure 8a illustrates in state space the second part of our fourth prediction (prediction 4-II) under the optimal subspace hypothesis: the lower the across-trial firing-rate variance at the time of the go cue, the lower too should be the RT, even in long delay-duration trials where sufficient time has elapsed for “complete” motor preparation to result. For long delay durations (e.g., > 200–300 ms), the Fano factor has nominally plateaued, as has the RT, at a low level. But, as depicted in Fig. 8a, there could still remain some variability. On trials that “wander outside” the optimal subspace (red circles), some additional time (i.e., increased RT) should be required to complete preparatory optimization following a go cue. In contrast, on trials where the preparatory state is within the optimal subspace (green dots) and therefore motor preparation is complete and ready for execution, movement can begin with a minimum of latency following the go cue (i.e., low RT). To test this prediction, we started with all trials with 200 ms or longer delay durations, across 7 days of experiments using a 96-channel electrode array. This helped assure sufficient data. Second, we sorted trials according to whether the RT was shorter than or longer than the median RT. Third, we calculated the across-trial firing-rate variance (Fano factor) for the half of trials with shorter than median RT, and the same for the half of trials with longer than median RT. We did so for times ranging from 200 ms before the go cue until 200 ms after the go cue in order to assess the robustness of the result. Figure 8b plots the Fano factor curve for shorter than median trials (green curve) and longer than median trials (red curve). These curves are statistically significantly different (not shown), and as predicted, the lower across-trial firing-rate variance trials (green curve) are associated with lower RTs (Churchland et al., 2006c). In sum, prediction 4-II as illustrated in Fig. 8a appears to be correct. When combined with the experiments and results associated with prediction 4-I, it appears clear that there is a close relationship between the across-trial firing-rate

variability at the time of the go cue and the resulting RT. Recently, similar results have been found in area V4 ahead of saccadic eye movements, suggesting that this relationship is not limited to the arm movement system alone (Steinmetz and Moore, 2010).

Prediction 5: Perturbing neural activity increases RT The inset of Fig. 9 illustrates in state space the fifth prediction under the optimal subspace hypothesis: perturbing the preparatory state out of the optimal subspace should result in an increased RT. But it should not reduce movement accuracy. The first four predictions of the optimal subspace hypothesis were correlative, and their affirmation provides important evidence supporting the optimal subspace hypothesis. The inset of Fig. 9 illustrates a causal prediction, wherein a preparatory state within the optimal subspace is deliberately perturbed (curly line with displaced preparatory state, black circle) and it is predicted that the RT should increase. This follows from the reasoning that if the goal of motor preparation is to help make accurate movements, then the brain must somehow be able to monitor preparatory activity and determine when it is accurate enough to initiate movement. If preparatory activity were optimized and within the optimal subspace, but were then perturbed away from the optimal subspace, the brain should wait for the plan to reoptimize to the optimal subspace (i.e., recover) before initiating movement (red dashed arrow labeled “reoptimization”). Importantly, after taking time to reoptimize preparatory activity, the resulting movements should be as accurate as on nonstimulated trials. To test this prediction, we delivered subthreshold electrical microstimulation to PMd on a subset of trials and did so at various times relative to the go cue (Churchland and Shenoy, 2007a). Figure 9 plots experimental results from all (30)

46 Neuron 3 100

Hand speed (cm/s)

Microstimulation

Reoptimization Neuron 2 Firing rate, Neuron 1 0 Go

300

Reaction time (m/s)

Fig. 9. State-space view of prediction 5 and influence of subthreshold microstimulation in PMd on arm movement RT. Inset: Statespace view of the predicted effect of subthreshold microstimulation on the preparatory state (curly arrow displacing the state, shown as a black circle), and the time consuming process of reoptimizing the preparatory state so that it is returned to within the optimal subspace (red dashed line). Black curve: nonstimulated trials. Red curve: microstimulation occurring just after the go cue, when preparatory activity is most needed. Green curve: stimulation occurring well before the go cue, when time exists following stimulation offset for preparatory activity to recover before the go cue appears. Red and green bars indicate when stimulation was delivered. Data are from 30 experiments in PMd, and curves plot mean  SE.

stimulation sites in PMd in one monkey. RTs are increased when microstimulation is delivered around the time of the go cue. This is seen as a rightward shift in the red hand-speed curve (stimulation around time of go cue, as indicated by the red bar) relative to the black hand-speed curve (no stimulation). This is consistent with time having been consumed (increased RT) to reoptimize preparatory activity. Importantly, aside from delaying the onset of movement, all other movement metrics were extremely similar to the nonstimulated trials (consistent with prediction 5). Note that the red and green averaged curves in Fig. 9 have lower peak hand speed due to staggered RTs, but individual trials do achieve the same, higher peak hand speed (see Churchland and Shenoy, 2007a for details). As it is critical to establish effect specificity when conducting causal perturbation experiments, we performed several additional control experiments (Churchland and Shenoy, 2007a). Four are briefly

summarized here. First, we found that stimulating well before the go cue (Fig. 9, green bar) had little impact on the RT. This can be seen in Fig. 9 by noting that the green curve largely overlaps with the black curve. This result is consistent with there being sufficient time for reoptimization to occur before the go cue appears. This is an important temporal control and indicates that the effect of subthreshold microstimulation exerts its influence just when the preparatory state is most needed (consistent with prediction 5). Second, stimulating on zero delay-duration trials where there was presumably no optimized preparatory activity present to perturb did not alter RT. This is an important control as it confirms the necessity of there first existing a preparatory state near the optimal subspace (consistent with prediction 5). Third, stimulating in M1 where there is relatively less preparatory activity resulted in little RT increase. The importance of this control is twofold. (i) It confirms that perturbing motor preparation is easier in an

47

area where preparatory activity is prevalent (PMd) than in an area where it is not (M1). (ii) It confirms area specificity. M1 is just a few millimeters from PMd, but the effect is dramatically reduced and thereby helps assure that subthreshold microstimulation is not just a generalized distraction. Both findings are consistent with prediction 5. Fourth and finally, the effect of microstimulation was specific to arm movements and produced little increase in saccadic eye movement RT. This is an important control for the possibility that microstimulation is altering attention, which should impact both effectors equivalently. In sum, prediction 5 as illustrated in Fig. 9 is borne out. This contributes causal supportive evidence for the optimal subspace hypothesis, which complements predictions 1–4 as well as 6 and 7 (below).

Prediction 6: Single-trial neural trajectories The conceptual sketch in Fig. 4 illustrates in state space the sixth prediction under the optimal subspace hypothesis: it should be possible to construct single-trial state-space neural trajectories and use them to directly confirm that across-trial firing-rate variability decreases through time. To test this prediction, we must begin by measuring many neurons simultaneously. This is essential as we seek an accurate estimate of the preparatory state on each individual trial and on a fine time scale. Both require data from many neurons, instead of the more traditional technique of trial averaging, in order to mathematically reduce the deleterious effects of spiking noise (Churchland et al., 2007; Yu et al., 2009). These measurements can be made with electrode arrays, which have been developed substantially as part of neural prosthesis research. The analyses can be performed using modern dimensionality reduction and visualization methods such as Gaussian Process Factor Analysis (GPFA; Yu et al., 2009). Dimensionality reduction is needed for two reasons. First, reducing the dimensionality

of the data from its original  100 D space (e.g., 100 neurons measured simultaneously constitutes a 100 D space) down to 10–15 D appears to be possible without significant loss of information and has the benefit of effectively denoising the data (Yu et al., 2009). This can be thought of as essentially performing a weighted average to combine the responses of neurons that share some important aspect of their response. Second, while reducing the dimensionality further (below 10–15 D) does result in a loss of information, it can be quite useful for visualization purposes. This is because the two or three dimensions used in drawings can be the two or three dimensions that capture the greatest variance in the data, and the resulting plots are still sufficient to spur on hypotheses and predictions as described above. Figure 10a shows multiple single-trial neural trajectories in a 2D state space created with GPFA (Churchland et al., 2010c; Yu et al., 2009). This is the first time that true single-trial neural trajectories (gray lines in Fig. 10a), as opposed to the cartoon depictions in Fig. 4, are plotted in this review. It is reassuring to see in Fig. 10a that the scatter in across-trial preparatory states at each point in time (black dots) reduces as the trial progresses. As the trial progresses from before target onset (100 ms pretarget) to just after target onset when the preparatory state is evolving toward the optimal subspace (100 ms post-target), and on to when the neural state on each trial is presumably within the optimal subspace (200 ms post-target), the variance (scatter) of the preparatory state reduces. This is consistent with the results presented above, inferred with Fano factor analyses. Figure 10b again shows multiple single-trial neural trajectories in a 2D state space created with GPFA but now goes on to show data until the time of movement onset (Churchland et al., 2010c). This reveals for the first time that neural trajectories (gray lines) follow a largely stereotyped path through state space, after the initial convergence following target onset. They start in

48 (a)

100 ms pretarget (b)

100 ms posttarget

200 ms posttarget

(c)

Go cue

Pretarget Movement onset Fig. 10. Single-trial neural trajectories computed using GPFA. (a) Projections of PMd activity into a two-dimensional state space. Each black point represents the location of neural activity on one trial. Gray traces show trajectories from 200 ms before target onset until the indicated time. The stimulus was a reach target (135, 60 mm distant), with no reach allowed until a subsequent go cue. Fifteen (of 47) randomly selected trials are shown. (b) Trajectories were plotted until movement onset. Blue dots indicate 100 ms before stimulus (reach target) onset. No reach was allowed until after the go cue (green dots), 400–900 ms later. Activity between the blue and green dots thus relates to movement planning. Movement onset (black dots) was approximately 300 ms after the go cue. For display, 18 randomly selected trials are plotted, plus one hand-selected trial (red, trialID 211). Covariance ellipses were computed across all 47 trials. This is a two-dimensional projection of a 10-dimensional latent space. In the full space, the black ellipse is far from the edge of the blue ellipse. This projection was chosen to accurately preserve the relative sizes (on a per-dimension basis) of the true 10-dimensional volumes of the ellipsoids. Data are from the G20040123 dataset. (c) Data are presented as in (b), with the same target location, but for data from another day's dataset (G20040122; red trial, trialID 793). From Churchland et al. (2010c).

the baseline pretarget state (blue circles), progress, and slow-down (if an extended delay period) in the optimal subspace until the time of the go cue (green), and then arch around and arrive at a small region where the arm movement is first detected (black). Highlighted in red is one outlier trial which had a substantially longer RT than typical. With single-trial visualization of even (entirely) internal neural processing, it is now possible to ask, for the first time, what the reason might be. On this trial, the preparatory process appears to have completed normally. The green

circle is within the (presumed) optimal subspace and surrounded by other trials that had normal RTs. While we cannot conclude what the cause was from this data alone, we appear to be able to rule out incomplete motor preparation. Figure 10c plots data from a different data set. Again, one trial with a particularly long RT is highlighted in red. This trial's neural trajectory undergoes an entire loop between the go cue and movement onset. In sum, prediction 6 as illustrated in Fig. 4 is borne out. It is possible to construct single-trial

49

neural trajectories and use these trajectories to directly see key features that can only be inferred less directly with single-neuron recordings. It also appears to now be possible to begin to investigate the reasons for outlier and other types of unique trials. Intriguingly, it should now also be possible to design experiments aimed at creating inherently single-trial phenomenon such as single-trial decision making, which could shed considerable insight on internal cognitive processing and neural dynamics (Kalmar et al., 2010; Rivera-Alvidrez et al., 2008). Further, as tasks become more complex (e.g., Churchland et al., 2008) and naturalistic (e.g., Chestek et al., 2009b; Gilja et al., 2010a; Jackson et al., 2006; Santhanam et al., 2007), both behavior and the preceding neural processes will likely become less stereotyped and may therefore often require single-trial analyses.

happens to be farther along the “loop” and moving in the standard direction around the loop (see arrows), then RT may be further lowered. Neuron 3

(a)

Initial state of a dynamical system that generates movement

Go cue

Neuron 2 Firing rate, neuron 1 Neuron 3

(b)

Movement onset Longer RT Shorter RT

Prediction 7: Farther and faster along loop reduces RT Trial 1

We have posited that the preparatory state has a large impact on the subsequent movement. We have also seen several predictions that stem from the optimal subspace hypothesis along with evidence supporting these predictions. It does seem to be the case that the preparatory state at the time of the go cue has a substantial influence on the subsequent movement. But why should this be? One possibility is illustrated in Fig. 11a. It could be that the preparatory state at the time of the go cue (green circle) acts as the initial state of a subsequent dynamical system that serves to generate muscle activity and create movement (green and blue arrows; Churchland et al., 2010a). As such, it is important that the preparatory state be within the optimal subspace in order to help create the desired movement. Thus some region or regions of the brain appear to monitor and wait for this to be true before “pulling the trigger” to initiate movement. After the movement trigger has been pulled, if the preparatory state

Trial 2 Neuron 2

Firing rate, neuron 1 Fig. 11. Low-dimensional state space (as in Fig. 4) illustrating prediction 7. (a) A single-trial neural trajectory is illustrated with activity from the time of target onset (red circle) shown in red, from the time the go cue is presented (green circle) shown in green, and from the time of movement onset (blue circle) shown in blue. One or more regions of the brain appear to monitor the preparatory state and initiate movement (“pull the movement trigger”) only if it is within the optimal subspace, so as to assure that the desired movement results. (b) Illustration of prediction 7. After the movement trigger is pulled, a preparatory state that happens to be farther along the loop in the standard direction should have a shorter RT. Trial 1 should have a shorter RT than trial 2. Not shown is the related prediction that a preparatory state that is moving faster in the standard direction of loop travel should also have a shorter RT.

50

Figure 11b illustrates in state space the seventh prediction under the optimal subspace hypothesis: the farther the preparatory state is along the loop when the movement trigger is pulled, and the faster it is moving along the loop in the standard direction (not shown in Fig. 11b), the shorter the RT should be. Figure 11b depicts the “loop” structure seen in Fig. 10b, where most individual trials follow a stereotyped path in state space. The two single-trial neural trajectories in Fig. 11b exit the optimal subspace in a particular direction, as was the case for all single-trial neural trajectories (gray lines) in Fig. 10b and c. Prediction 7 states that a trial like trial 1 in Fig. 11b should have a shorter RT than trial 2, because (i) the preparatory state at the time of the go cue is within the optimal subspace for both trials (and thus the movement trigger is presumably pulled at the same time) and (ii) the preparatory state for trial 1 (green circle) is nearer the exitedge of the optimal subspace and is thus farther along the stereotyped path that it will need to take to generate movement. If the preparatory state also happened to be moving along the stereotyped path in the standard direction, as opposed to not moving or moving in the opposite direction, then the RT ought to be shorter still. To test this hypothesis, we correlated, on a trial-by-trial basis, how far along the loop the preparatory state was (at the time of the go cue) with RT. As predicted, we found a statistically significant negative correlation, and primarily in just the exit-edge direction (Afshar et al., 2011). Also as predicted, we found a statistically significant correlation between the direction of movement of the preparatory state at the time of the go cue and RT: preparatory states that were moving in the direction of (subsequent) loop travel had lower RTs than comparably positioned preparatory states moving in the opposite direction (Afshar et al., 2011). This appears to suggest that a trial with preparatory activity at the time of the go cue (green circle) that is (i) within the optimal subspace and (ii) farther along, and moving in, the standard “loop”

direction is in some sense “doubly advantaged” because it is both (i) a well-optimized preparatory state (i.e., within the optimal subspace so the movement trigger can be pulled straight away) and (ii) fortuitously positioned and already headed along the path it will need to take to generate movement. In sum, prediction 7 as illustrated in Fig. 11b is borne out.

Future directions The above predictions were derived from a dynamical systems perspective, and to some degree their confirmation argues for that perspective. Yet the most central questions remain largely unaddressed. What is the nature of the relevant dynamics (e.g., Yu et al., 2006)? Do they relate to the dynamics of movement-generating circuits in simpler organisms (e.g., Grillner, 2006; Kristan and Calabrese, 1976)? How and why do dynamics change as a function of overall state (e.g., resting vs. planning vs. moving)? What is the nature of the circuitry, both local and feedback, that produces those dynamics? Answering such questions will likely depend on progress in three domains: (1) the ability to better perturb and probe dynamics, (2) the ability to resolve dynamical structure in neural data, and (3) the ability to relate the recorded “neural trajectories” to externally measurable parameters such as muscle activity and hand movement. We consider these in turn. First, when reverse-engineering any system, the ability to perturb and observe is critical. As described above, we used intracortical electrical microstimulation to ask how a perturbation of neural activity influenced RT. Pharmacological manipulations are also possible and would offer cell-type specific manipulation of the system. Recent advances in optogenetic stimulation of neurons in rhesus monkeys may also provide important new insights due to the ability to excite and inhibit neurons in a cell-type specific manner (unlike electrical microstimulation), do so on a millisecond timescale (unlike pharmacological

51

manipulations), and not interfere with simultaneous electrical recordings (unlike electrical microstimulation; Diester et al., 2011; Han et al., 2009). This could enable the direct visualization of neural trajectories throughout trials where the neural state is optically perturbed at various times. Recall the Fig. 9 inset, where the curly black line and dashed red line could only illustrate our speculation about how the neural trajectory might evolve during and directly following stimulation, because electrical microstimulation interferes with electrical array recordings. Also, while permanently altering the underlying neural circuitry transgenically is not currently possible in rhesus monkeys, it is possible to reversibly lesion brain regions, cell-types, and neuronal projections pharmacologically and optogenetically. Understanding how the subsequent alteration of neural trajectories relates to altered behavior would deepen our understanding of motor preparation and generation, and the role of specific cells and connectivity (Kaufman et al., 2009, 2010a; Lerchner et al., 2011). Second, any real understanding of dynamics will hinge upon the ability to go beyond merely plotting state-space trajectories. One wishes to take those seemingly complex neural trajectories, which evolve in many dimensions, and infer meaningful and parsimonious underlying dynamics (Yu et al., 2006). Indeed, if this cannot be done—if the proposed dynamics are not simpler than the data they seek to explain—then the dynamical systems perspective may have little to offer. Fortunately, it appears that simple dynamics may well be able to explain a considerable amount of the structure of the data (Churchland et al., 2010b, 2011; Cunningham et al., 2011; Macke et al., 2011; Petreska et al., 2011), but further progress in this realm will depend on the continued development of analysis methods that can capture how one neural state leads to the next. Finally, while the observed state-space trajectories often appear rather abstract, they must exist for a concrete purpose: producing movement. That is, there must be some direct and causal relationship between the neural

trajectory and some externally measurable quantity such as muscle activation or arm kinematics. Historically, the relationship between movement-period activity and the parameters of movement has been contentious (e.g., Kalaska, 2009). The dynamical systems perspective will not on its own resolve this debate, but there are a number of contributions it can make. The dimensionality-reduction techniques that produce the statespace trajectories force the experimenter to focus on those patterns that are most strongly present in the data (e.g., Rivera-Alvidrez et al., 2009, 2010a,b). Also, the relatively high dimensionality of the state space makes it clear that not all aspects of neural activity can or should be related directly to external factors: some dimensions may be important to the overall dynamics but may not exert any direct influence on the periphery (Kaufman et al., 2010b, 2011). Progress in the above domains should also increase the breadth of questions addressable under the dynamical systems perspective. Already it has been possible to ask whether neural variability decreases during learning (MandelblatCerf et al., 2009). More generally, we wish to know what the “state-space” correlate of motor learning might look like. For example, does the location of the optimal subspace change following learning? Or does learning change the dynamics that determine the trajectory away from that planned state? Of course, one suspects that both such mechanisms might be at play, perhaps depending on the type of learning (e.g., the former strategy might be more rapid but less flexible). The further development of critical tools, as described above, could open the door to many experiments of this type.

Importance to neural prosthetic system design Neural prosthetic system design depends critically on a fundamental and comprehensive scientific understanding of how populations of neurons prepare and generate natural movements (Green and Kalaska, 2010). How neural populations

52 Touch, fixate

Target onset

Go cue

Movement onset

1 m/s

Hand speed

Neuron 1 Neuron 2

Neuron 97

200 ms

Trial G20040508.118

Fig. 12. Instructed-delay reach task and neural activity from a single trial, highlighting the need to understand all phases of volitional arm movement (holding, preparing, and moving) on a trial-by-trial basis as this is the fundamental information source and (millisecond) timescale on which neural prosthetic systems depend. Top: Schematic illustration and timeline of the instructed-delay reach task introduced in Fig. 1 and discussed throughout this review. Middle: measured hand speed. Photo: 100 electrode array used to record from many tens of neurons simultaneously. Bottom: action potential (spike) raster from 97 simultaneously recorded single and multi-units, with the time of an action potential indicated by a black tick mark. Red vertical lines indicate the time of target onset, go cue presentation, and movement onset.

evolve on a fast timescale is of particular interest, as neural prostheses must operate rapidly to ensure accurate and stable control. Figure 12 highlights perhaps the most fundamental problem that basic neuroscience (reviewed above) and neural prosthetic systems have in common: how to understand noisy electrical activity from a population of neurons on a millisecond timescale and on a single-trial basis. Figure 12 shows the now

familiar instructed-delay reach task and, for the first time in this review, reasonably raw and unprocessed electrode-array neural data. The most striking feature is how noisy spiking data really are. Staring at this figure for a few moments reveals the subtle difference in response pattern between the preparatory period following target onset and the baseline period preceding it. More obvious is the difference between the movement period and

53

preparatory period. While the eye is often poor at discerning signal from noise, particularly without the visual benefit of trial averaging, this is a useful exercise as it helps one appreciate the challenge before us as neuroscientists and neuroengineers. We seek to understand how these neural responses arise, how they support behavior, and how we can use these volitional signals as an information source for controlling neural prostheses. While several important design insights for prostheses have already resulted from a deeper scientific understanding, some of which are discussed briefly below, there is little doubt that the most important leaps forward in prostheses will result from future scientific discovery. This has historically been the case, with science deeply informing engineering. In addition to how the basic scientific understanding of brain organization and movement control has already informed neural prosthetic system design, as described in the Introduction, there are two additional points to briefly note here. First, communication prostheses rely on preparatory activity, as can motor prostheses (e.g., Musallam et al., 2004; Santhanam et al., 2006; Shenoy et al., 2003; Yu et al., 2007, 2010). These systems can make use of new discoveries such as preparatory activity in PMd reflecting the speed of the upcoming movement. Second, at the heart of the dynamical systems perspective and the associated quest for single-trial neural trajectories is time. How long does it take for the neural trajectory to actually traverse from baseline to the optimal subspace where, once there, the neural activity can be fruitfully decoded and used to guide a prosthesis? This is precisely the “transit time” we needed to know as part of our recent prosthesis research, so that we could skip this transition period to avoid inadvertently decoding neural activity that is still in flux. This time (Tskip; Santhanam et al., 2006) is approximately 200 ms as measured with single-trial neural trajectories in scientific experiments (as described above) and agrees with measurements from neural prosthetic experiments. Similarly, for prosthetics designs, it is important to know how long neural activity should be integrated (Tint), so as to best

estimate the parameters of interest, and this is related to how stable preparatory neural activity is while in and around the optimal subspace. Single-trial neural trajectories can, and have, revealed important features which will continue to inform the design of neural prosthetic systems.

Summary The ability to move voluntarily is central to the human experience. By pursuing a deeper scientific understanding of the neural control of natural movement, it should be possible to advance the design of neural prostheses, with the goal of helping patients who have lost their ability to move. A potentially underappreciated part of controlling movement is preparing movement. Preparation is, after all, how each movement begins. Motor preparation can be studied in many different ways. We have elected to adopt a dynamical systems perspective in order to facilitate the construction of hypotheses, and set about testing their predictions. The optimal subspace hypothesis has led to seven tested predictions. It appears that this dynamical systems perspective, and closely associated state-space diagrams, is helping to generate an ongoing series of testable predictions. As a result of the recent studies reviewed above, numerous questions are now more apparent and remain to be addressed as described in the Future Directions. We believe that the dynamical systems perspective should continue to help generate new and testable ideas and lead to deeper insights for both basic and applied neuroscience.

Acknowledgment This work was supported by Burroughs Wellcome Fund Career Awards in the Biomedical Sciences (K. V. S and M. M. C.), DARPA REPAIR N66001-10-C-2010 and NIH-NINDS CRCNS R01-NS-054283 (K. V. S. and M. S.), an NIH

54

Director's Pioneer Award 1DP1OD006409 (K. V. S.), a National Science Foundation graduate research fellowship (M. T. K.), and the Gatsby Charitable Foundation (M. S.). References Achtman, N., Afshar, A., Santhanam, G., Yu, B. M., Ryu, S. I., & Shenoy, K. V. (2007). Free-paced high-performance braincomputer interfaces. Journal of Neuroengineering, 4, 336–347. Afshar, A., Santhanam, G., Yu, B. M., Ryu, S. I., Sahani, M., & Shenoy, K. V. (2011). Single-trial neural correlates of arm movement preparation. Neuron. In press. Andersen, R. A., Hwang, E. J., & Mulliken, G. H. (2010). Cognitive neural prosthetics. Annual Review of Psychology, 61, 169–190. Anderson, K. D. (2004). Targeting recovery: Priorities of the spinal cord-injured population. Journal of Neurotrauma, 21, 1371–1383. Borton, D. A., Song, Y.-K., Patterson, W. R., Bull, C. W., Park, S., Laiwalla, F., Donoghue, J. P., & Nurmikko, A. V. (2009). Wireless, high-bandwidth recordings from nonhuman primate motor cortex using a scalable 16-ch implantable microsystem. In: Conference proceedings: IEEE engineering in medicine and biology society, pp. 5531–5534. Briggman, K. L., Abarbanel, H. D., & Kristan, W. B. (2005). Optical imaging of neuronal populations during decisionmaking. Science, 307, 896–901. Carmena, J. M., Lebedev, M. A., Crist, R. E., O'Doherty, J. E., Santucci, D. M., Dimitrov, D. F., et al. (2003). Learning to control a brain-machine interface for reaching and grasping by primates. PLoS Biology, 1, 193–208. Chapin, J. K., Moxon, K. A., Markowitz, R. S., & Nicolelis, M. A. L. (1999). Real-time control of a robot arm using simultaneously recorded neurons in the motor cortex. Nature Neuroscience, 2, 664–670. Chase, S. M., Schwartz, A. B., & Kass, R. E. (2009). Bias, optimal linear estimation, and the differences between openloop simulation and closed-loop performance of spikingbased brain-computer interface algorithms. Neural Networks, 22, 1203–1213. Chestek, C. A., Cunningham, J. P., Gilja, V., Nuyujukian, P., Ryu, S. I., & Shenoy, K. V. (2009a). Neural prosthetic systems: Current problems and future directions. In: Proceedings of the 31st annual conference of IEEE EMBS, pp. 3369–3375. Chestek, C. A., Gilja, V., Nuyujukian, P., Foster, J. D., Fan, J. M., Kaufman, M. T., et al. (2011). Long-term stability of neural prosthetic control signals from silicon cortical arrays in rhesus macaque motor cortex. Journal of Neural Engineering.

Chestek, C. A., Gilja, V., Nuyujukian, P., Kier, R., Solzbacher, F., Ryu, S. I., et al. (2009b). HermesC: Lowpower wireless neural recording system for freely moving primates. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 17, 330–338. Churchland, M. M., Afshar, A., & Shenoy, K. V. (2006a). A central source of movement variability. Neuron, 52, 1085–1096. Churchland, M. M., Cunningham, J. P., Kaufman, M. T., Ryu, S. I., & Shenoy, K. V. (2010a). Cortical preparatory activity: Representation of movement or first cog in a dynamical machine? Neuron, 68, 387–400. Churchland, M. M., Cunningham, J. P., Kaufman, M. T., Ryu, S. I., & Shenoy, K. V. (2011). Firing rate oscillations underlie motor cortex responses during reaching in monkey. Frontiers in Neuroscience. Conference Abstract. Computational and Systems Neuroscience (COSYNE), Salt Lake City, UT: III-32. Churchland, M. M., Kaufman, M. T., Cunningham, J. P., Ryu, S. I., & Shenoy, K. V. (2010b). Some basic features of the neural response in motor and premotor cortex. In Program No. 382.2. Neuroscience meeting planner. San Diego, CA: Society for Neuroscience Online. Churchland, M. M., Kaufman, M. T., & Shenoy, K. V. (2008). A novel maze task for the study of motor preparation. In Program No. 182.14. Neuroscience meeting planner. Washington, DC: Society for Neuroscience. Churchland, M. M., Santhanam, G., & Shenoy, K. V. (2006b). Preparatory activity in premotor and motor cortex reflects the speed of the upcoming reach. Journal of Neurophysiology, 96, 3130–3146. Churchland, M. M., & Shenoy, K. V. (2007a). Delay of movement caused by disruption of cortical preparatory activity. Journal of Neurophysiology, 97, 348–359. Churchland, M. M., & Shenoy, K. V. (2007b). Temporal complexity and heterogeneity of single-neuron activity in premotor and motor cortex. Journal of Neurophysiology, 97, 4235–4257. Churchland, M. M., Yu, B. M., Cunningham, J. P., Sugrue, L. P., Cohen, M. R., Corrado, G. S., et al. (2010c). Stimulus onset quenches neural variability: A widespread cortical phenomenon. Nature Neuroscience, 13, 369–378. Churchland, M. M., Yu, B. M., Ryu, S. I., Santhanam, G., & Shenoy, K. V. (2006c). Neural variability in premotor cortex provides a signature of motor preparation. The Journal of Neuroscience, 26, 3697–3712. Churchland, M. M., Yu, B. M., Sahani, M., & Shenoy, K. V. (2007). Techniques for extracting single-trial activity patterns from large-scale neural recordings. Current Opinion in Neurobiology, 17, 609–618. Cisek, P. (2006). Preparing for speed. Focus on “preparatory activity in premotor and motor cortex reflects the speed of the upcoming reach” Journal of Neurophysiology, 96, 2842–2843.

55 Crammond, D. J., & Kalaska, J. F. (2000). Prior information in motor and premotor cortex: Activity during the delay period and effect on pre-movement activity. Journal of Neurophysiology, 84, 986–1005. Cunningham, J. P., Churchland, M. M., Kaufman, M. T., & Shenoy, K. V. (2011). Extracting rotational structure from motor cortical data. Frontiers in Neuroscience. Conference Abstract. Computational and Systems Neuroscience (COSYNE), Salt Lake City, UT: II-33. Cunningham, J. P., Nuyujukian, P., Gilja, V., Chestek, C. A., Ryu, S. I., & Shenoy, K. V. (2010). A closed-loop human simulator for investigating the role of feedback-control in brain-machine interfaces. Journal of Neurophysiology, 105, 1932–1949. Cunningham, J., Yu, B., Gilja, V., Ryu, S., & Shenoy, K. (2008). Toward optimal target placement for neural prosthetic devices. Journal of Neurophysiology, 100, 3445–3457. Day, B. L., Rothwell, J. C., Thompson, P. D., de Noordhout, A. M., Nakashima, K., Shannon, K., et al. (1989). Delay in the execution of voluntary movement by electrical or magnetic brain stimulation in intact man. Brain, 112, 649–663. Millan, J.del R., & Carmena, J. M. (2010). Invasive or noninvasive: Understanding brain-machine interface technology. IEEE Engineering in Medicine and Biology Magazine, 29, 16–22. Diester, I., Kaufman, M. T., Mogri, M., Pashaie, R., Goo, W., Yizhar, O., et al. (2011). An optogenetic toolbox designed for primates. Nature Neuroscience, 14, 387–397. Donoghue, J. P. (2008). Bridging the brain to the world: A perspective on neural interface systems. Neuron, 60, 511–521. Donoghue, J. P., Nurmikko, A., Black, M. J., & Hochberg, L. R. (2007). Assistive technology and robotic control using motor cortex ensemble-based neural interface systems in humans with tetraplegia. Journal of Physiology, 579, 603–611. Evarts, E. V. (1964). Temporal patterns of discharge of pyramidal tract neurons during sleep and waking in the monkey. Journal of Neurophysiology, 27, 152–171. Fetz, E. E. (1969). Operant conditioning of cortical unit activity. Science, 163, 955–957. Fetz, E. E. (1992). Are movement parameters recognizably coded in the activity of single neurons? The Behavioral and Brain Sciences, 15, 679–690. Fetz, E. E. (2007). Volitional control of neural activity: Implications for brain-computer interfaces. Journal of Physiology, 579, 571–579. Fraser, G. W., Chase, S. M., Whitford, A., & Schwartz, A. B. (2009). Control of a brain-computer interface without spike sorting. Journal of Neural Engineering, 6, 055004. Ganguly, K., & Carmena, J. M. (2009). Emergence of a stable cortical map for neuroprosthetic control. PLoS Biology, 7, e1000153.

Georgopoulos, A. P., Kalaska, J. F., Caminiti, R., & Massey, J. T. (1982). On the relations between the direction of two-dimensional arm movements and cell discharge in primate motor cortex. The Journal of Neuroscience, 2, 1527–1537. Georgopoulos, A. P., Schwartz, A. B., & Kettner, R. E. (1986). Neuronal population coding of movement direction. Science, 233, 1416–1419. Ghez, C., Favilla, M., Ghilardi, M. F., Gordon, J., Bermejo, R., & Pullman, S. (1997). Discrete and continuous planning of hand movements and isometric force trajectories. Experimental Brain Research, 115, 217–233. Gilja, V., Chestek, C. A., Diester, I., Henderson, J. M., Deisseroth, K., & Shenoy, K. V. (2011). Challenges and opportunities for next generation intra-cortically based neural prostheses. IEEE Transactions on Biomedical Engineering, doi:10.1109/TBME.2011.2107553. Gilja, V., Chestek, C. A., Nuyujukian, P., Foster, J., & Shenoy, K. V. (2010a). Autonomous head-mounted electrophysiology systems for freely-behaving primates. Current Opinion in Neurobiology, 20, 676–686. Gilja, V., Nuyujukian, P., Chestek, C. A., Cunningham, J. P., Yu, B. M., Ryu, S. I., et al. (2010b). A high-performance continuous cortically-controlled prosthesis enabled by feedback control design. In Program No. 20.6. Neuroscience meeting planner. San Diego, CA: Society for Neuroscience Online. Gilja, V., Nuyujukian, P., Chestek, C. A., Cunningham, J. P., Yu, B. M., Ryu, S. I., et al. (2010c). High-performance continuous neural cursor control enabled by a feedback control perspective. Frontiers in Neuroscience, doi:10.3389/conf. fnins.2010.03.00249. Godschalk, M., Lemon, R. N., Kuypers, H. G., & van der Steen, J. (1985). The involvement of monkey premotor cortex neurones in preparation of visually cued arm movements. Behavioural Brain Research, 18, 143–157. Green, A. M., & Kalaska, J. F. (2010). Learning to move machines with the mind. Trends in Neurosciences, 34, 61–75. Grillner, S. (2006). Biological pattern generation: The cellular and computational logic of networks in motion. Neuron, 52, 751–766. Han, X., Qian, X., Bernstein, J. G., Zhou, H.-H., Franzesi, G. T., Stern, P., et al. (2009). Millisecond-timescale optical control of neural dynamics in the nonhuman primate brain. Neuron, 62, 191–198. Harrison, R. R. (2008). The design of integrated circuits to observe brain activity. Proceedings of the IEEE, 96, 1203–1216. Harrison, R. R., Kier, R. J., Chestek, C. A., Gilja, V., Nuyujukian, P., Ryu, S. I., et al. (2009). Wireless neural recording with single low-power integrated circuit. IEEE Transactions on Neural and Rehabilitation Systems Engineering, 17, 322–329.

56 Harrison, R. R., Watkins, P. T., Kier, R. J., Lovejoy, R. O., Black, D. J., Greger, B., et al. (2007). A low-power integrated circuit for a wireless 100-electrode neural recording system. IEEE Journal of Solid-State Circuits, 42, 123–133. Hatsopoulos, N. G., & Donoghue, J. P. (2009). The science of neural interface systems. Annual Review of Neuroscience, 32, 249–266. Heliot, R., Ganguly, K., Jimenez, J., & Carmena, J. M. (2009). Learning in closed-loop brain-machine interfaces: Modeling and experimental validation. IEEE Transactions on Systems, Man, and Cybernetics. Part B: Cybernetics, 6, 3710–3715. Hochberg, L. R. (2008). Turning thought into action. The New England Journal of Medicine, 359, 1175–1177. Hochberg, L. R., Serruya, M. D., Friehs, G. M., Mukand, J. A., Saleh, M., Caplan, A. H., et al. (2006). Neuronal ensemble control of prosthetic devices by a human with tetraplegia. Nature, 442, 164–171. Hochberg, L. R., & Taylor, D. M. (2007). Intuitive prosthetic limb control. Lancet, 369, 345–346. Humphrey, D. R., Schmidt, E. M., & Thompson, W. D. (1970). Predicting measures of motor performance from multiple cortical spike trains. Science, 170, 758–762. Isaacs, R. E., Weber, D. J., & Schwartz, A. B. (2000). Work toward real-time control of a cortical neural prosthesis. IEEE Transactions on Rehabilitation Engineering, 8, 196–198. Jackson, A., & Fetz, E. E. (2007). Compact movable microwire array for long-term chronic unit recording in cerebral cortex of primates. Journal of Neurophysiology, 98, 3109–3118. Jackson, A., Mavoori, J., & Fetz, E. E. (2006). Long-term motor cortex plasticity induced by an electronic neural implant. Nature, 444, 56–60. Jarosiewicz, B., Chase, S. M., Fraser, G. W., Velliste, M., Kass, R. E., & Schwartz, A. B. (2008). Functional network reorganization during learning in a brain-computer interface paradigm. Proceedings of the National Academy of Sciences of the United States of America, 105, 19486–19491. Kalaska, J. F. (2009). From intention to action: Motor cortex and the control of reaching movements. Advances in Experimental Medicine and Biology, 629, 139–178. Kalaska, J. F., Scott, S. H., Cisek, P., & Sergio, L. E. (1997). Cortical control of reaching movements. Current Opinion in Neurobiology, 7, 849–859. Kalmar, R., Reppas, J., Ryu, S. I., Shenoy, K. V., & Newsome, W. T. (2010). Ensemble activity underlying movement preparation in prearcuate cortex. In: Frontiers in neuroscience. Conference abstract: Computational and systems neuroscience (COSYNE), Salt Lake City, UT, doi:10.3389/ conf.fnins.2010.03.00228. Kaufman, M. T., Churchland, M. M., Santhanam, G., Yu, B. M., Afshar, A., Ryu, S. I., et al. (2010a). The roles

of monkey premotor neuron classes in movement preparation and execution. Journal of Neurophysiology, 104, 799–810. Kaufman, M. T., Churchland, M. M., & Shenoy, K. V. (2009). Putative interneurons respond more rapidly than pyramidal cells in monkey premotor cortex. In Program No. 463.11. Neuroscience Meeting Planner. Chicago, IL: Society for Neuroscience. Kaufman, M. T., Churchland, M. M., Shenoy, K. V. (2010b, nanosymposium) Lack of evidence for inhibitory gating in monkey M1. Program No. 732.2. 2010. Neuroscience Meeting Planner. San Diego, CA: Society for Neuroscience. Online. Kaufman, M. T., Churchland, M. M., & Shenoy, K. V. (2011). Cortical preparatory activity avoids causing movement by remaining in a muscle-neutral space. Frontiers in Neuroscience. Conference Abstract. Computational and Systems Neuroscience. (COSYNE), Salt Lake City, UT: II-61. Keele, S. W. (1968). Movement control in skilled motor performance. Psychological Bulletin, 70, 387–403. Kemere, C., Santhanam, G., Yu, B., Afshar, A., Ryu, S., Meng, T., et al. (2008). Detecting neural-state transitions using hidden Markov models for motor cortical prostheses. Journal of Neurophysiology, 100, 2441–2452. Kemere, C. T., Shenoy, K. V., & Meng, T. H. (2004). Modelbased neural decoding of reaching movements: A maximum likelihood approach. IEEE Transactions on Biomedical Engineering, 51, 925–932. Kim, S.-P., Simeral, J. D., Hochberg, L. R., Donoghue, J. P., & Black, M. J. (2008). Neural control of computer cursor velocity by decoding motor cortical spiking activity in humans with tetraplegia. Journal of NeuroEngineering, 5, 455–476. Kristan, W. B., & Calabrese, R. L. (1976). Rhythmic swimming activity in neurones of the isolated nerve cord of the leech. The Journal of Experimental Biology, 65, 643–686. Kurata, K. (1989). Distribution of neurons with set- and movement-related activity before hand and foot movements in the premotor cortex of rhesus monkeys. Experimental Brain Research, 77, 245–256. Kutas, M., & Donchin, E. (1974). Studies of squeezing: Handedness, responding hand, response force, and asymmetry of readiness potential. Science, 186, 545–548. Lerchner, A., Shenoy, K. V., & Sahani, M. (2011). The role of horizontal long-range connections in shaping the dynamics of multi-electrode array data. Frontiers in Neuroscience. Conference Abstract. Computational and Systems Neuroscience, (COSYNE), Salt Lake City, UT: 1–27. Linderman, M. D., Santhanam, G., Kemere, C. T., Gilja, V., O'Driscoll, S., Yu, B. M., et al. (2008). Signal processing challenges for neural prostheses. IEEE Signal Processing Magazine, 25, 18–28. Macke, J., Busing, L., Cunningham, J. P., Yu, B. M., Shenoy, K. V., & Sahani, M. (2011). Modelling lowdimensional dynamics in recorded spiking populations.

57 In: Frontiers in Neuroscience. Conference Abstract. Computational and Systems Neuroscience, (COSYNE 2011), Salt Lake City, UT:I–34. Mandelblat-Cerf, Y., Paz, R., & Vaadia, E. (2009). Trial-totrial variability of single cells in motor cortices is dynamically modified during visuomotor adaptation. The Journal of Neuroscience, 29, 15053–15062. Mavoori, J., Jackson, A., Diorio, C., & Fetz, E. (2005). An autonomous implantable computer for neural recording and stimulation in unrestrained primates. Journal of Neuroscience Methods, 148, 71–77. Messier, J., & Kalaska, J. F. (2000). Covariation of primate dorsal premotor cell activity with direction and amplitude during a memorized-delay reaching task. Journal of Neurophysiology, 84, 152–165. Moritz, C. T., Perlmutter, S. I., & Fetz, E. E. (2008). Direct control of paralysed muscles by cortical neurons. Nature, 456, 639–642. Mulliken, G. H., Musallam, S., & Andersen, R. A. (2008). Decoding trajectories from posterior parietal cortex ensembles. The Journal of Neuroscience, 28, 12913–12926. Musallam, S., Corneil, B. D., Greger, B., Scherberger, H., & Andersen, R. A. (2004). Cognitive control signals for neural prosthetics. Science, 305, 258–262. Nicolelis, M. A. L., & Lebedev, M. A. (2009). Principles of neural ensemble physiology underlying the operation of brainmachine interfaces. Nature Reviews Neuroscience, 10, 530–540. Nurmikko, A. V., Donoghue, J. P., Hochberg, L., Patterson, W. R., Song, Y.-K., Bull, C., et al. (2010). Listening to brain microcircuits for interfacing with external world—Progress in wireless implantable microelectronic neuroengineering devices. Proceedings of the IEEE, 98, 375. Nuyujukian, P., Gilja, V., Chestek, C. A., Cunningham, J. P., Fan, J. M., Yu, B. M., et al. (2010). Generalization and robustness of a continuous cortically-controlled prosthesis enabled by feedback control design. In Program No. 20.7. Neuroscience Meeting Planner. San Diego, CA: Society for Neuroscience Online. O'Driscoll, S., Meng, T. H., Shenoy, K. V., & Kemere, C. (2006). In: Neurons to silicon: Implantable prosthesis processor. International solid state circuits conference (ISSCC) 552–553, 672. Petreska B, Cunningham JP, Santhanam G, Yu BM, Ryu SI, Shenoy KV, Sahani M (2011). Detecting changes in neural dynamics within single trials. Frontiers in Neuroscience. Conference Abstract. Computational and Systems Neuroscience (COSYNE), Salt Lake City, UT: I-33. Rickert, J., Riehle, A., Aertsen, A., Rotter, S., & Nawrot, M. P. (2009). Dynamic encoding of movement direction in motor cortical neurons. The Journal of Neuroscience, 29, 13870–13882. Riehle, A., & Requin, J. (1989). Monkey primary motor and premotor cortex: Single-cell activity related to prior

information about direction and extent of an intended movement. Journal of Neurophysiology, 61, 534–549. Riehle, A., & Requin, J. (1993). The predictive value for performance speed of preparatory changes in neuronal activity of the monkey motor and premotor cortex. Behavioural Brain Research, 53, 35–49. Rivera-Alvidrez, Z., Kalmar, R., Afshar, A., Santhanam, G., Yu, B. M., Ryu, S. I., et al. (2008). Single-trial representation of uncertainty about reach goals in macaque PMd. In Presentation III-52. Computational and Systems Neuroscience (COSYNE) meeting. Salt Lake City, UT. Rivera-Alvidrez, Z., Kalmar, R., Ryu, S. I., & Shenoy, K. V. (2010a). In: Low-dimensional neural features predict muscle EMG signals (pp. 6027–6033). Proceedings of the 32nd annual international conference of the IEEE EMBS. Rivera-Alvidrez, Z., Kalmar, R., Ryu, S. I., & Shenoy, K. V. (2010b). Low dimensional neural features predict specific muscle EMG signals. In Program No. 732.4. Neuroscience Meeting Planner. San Diego, CA: Society for Neuroscience. Program No. 463.4, Neuroscience Meeting Planner. (2009) Chicago, IL: Society for Neuroscience. Rosenbaum, D. A. (1980). Human movement initiation: Specification of arm, direction, and extent. Journal of Experimental Psychology. General, 109, 444–474. Ryu, S. I., & Shenoy, K. V. (2009). Human cortical prostheses: Lost in translation? Neurosurgical Focus, 27, E5. Santhanam, G., Linderman, M. D., Gilja, V., Afshar, A., Ryu, S. I., Meng, T. H., et al. (2007). HermesB: A continuous neural recording system for freely behaving primates. IEEE Transactions on Biomedical Engineering, 54, 2037–2050. Santhanam, G., Ryu, S. I., Yu, B. M., Afshar, A., & Shenoy, K. V. (2006). A high-performance brain-computer interface. Nature, 442, 195–198. Santhanam, G., Sahani, M., Ryu, S. I., & Shenoy, K. V. (2004). In: An extensible infrastructure for fully automated spike sorting during online experiments (pp. 4380–4384). Proceedings of the 26th annual international conference of IEEE EMBS, San Francisco, CA. Santhanam, G., Yu, B. M., Gilja, V., Ryu, S. I., Afshar, A., Sahani, M., et al. (2009). Factor-analysis methods for higher-performance neural prostheses. Journal of Neurophysiology, 102, 1315–1330. Scherberger, H. (2009). Neural control of motor prostheses. Current Opinion in Neurobiology, 19, 629–633. Schwartz, A. B. (1994). Direct cortical representation of drawing. Science, 265, 540–542. Schwartz, A. B. (2007). Useful signals from motor cortex. Journal of Physiology, 579, 581–601. Scott, S. H. (2004). Optimal feedback control and the neural basis of volitional motor control. Nature Reviews Neuroscience, 7, 524–546.

58 Scott, S. H. (2008). Inconvenient truths about neural processing in primary motor cortex. Journal of Physiology, 586, 1217–1224. Serruya, M. D., Hatsopoulos, N. G., Paninski, L., Fellows, M. R., & Donoghue, J. (2002). Instant neural control of a movement signal. Nature, 416, 141–142. Shenoy, K. V., Meeker, D., Cao, S., Kureshi, S. A., Pesaran, B., Mitra, P., et al. (2003). Neural prosthetic control signals from plan activity. Neuroreport, 14, 591–596. Snyder, L. H., Batista, A. P., & Andersen, R. A. (1997). Coding of intention in the posterior parietal cortex. Nature, 386, 167–170. Steinmetz, N. A., & Moore, T. (2010). Changes in the response rate and response variability of area v4 neurons during the preparation of saccadic eye movements. Journal of Neurophysiology, 103, 1171–1178. Stopfer, M., Jayaraman, V., & Laurent, G. (2003). Intensity versus identity coding in an olfactory system. Neuron, 39, 991–1004. Tanji, J., & Evarts, E. V. (1976). Anticipatory activity of motor cortex neurons in relation to direction of an intended movement. Journal of Neurophysiology, 39, 1062–1068. Taylor, D. M., Tillery, S. I. H., & Schwartz, A. B. (2002). Direct cortical control of 3D neuroprosthetic devices. Science, 296, 1829–1832. Todorov, E. (2000). Direct cortical control of muscle activation in voluntary arm movements: A model. Nature Neuroscience, 3, 391–398. Truccolo, W., Friehs, G. M., Donoghue, J. P., & Hochberg, L. R. (2008). Primary motor cortex tuning to intended movement kinematics in humans with tetraplegia. The Journal of Neuroscience, 28, 1163–1178. Truccolo, W., Hochberg, L. R., & Donoghue, J. P. (2010). Collective dynamics in human and monkey sensorimotor cortex: Predicting single neuron spikes. Nature Neuroscience, 13, 105–111. Velliste, M., Perel, S., Spalding, M. C., Whitford, A. S., & Schwartz, A. B. (2008). Cortical control of a prosthetic arm for self-feeding. Nature, 453, 1098–1101.

Weinrich, M., Wise, S. P., & Mauritz, K. H. (1984). A neurophysiological study of the premotor cortex in the rhesus monkey. Brain, 107, 385–414. Wessberg, J., Stambaugh, C. R., Kralik, J. D., Beck, P. D., Laubach, M., Chapin, J. K., et al. (2000). Real-time prediction of hand trajectory by ensembles of cortical neurons in primates. Nature, 408, 361–365. Wise, S. P. (1985). The primate premotor cortex: Past, present and preparatory. The Annual Review of Neuroscience, 8, 1–19. Wu, W., Gao, Y., Bienenstock, E., Donoghue, J. P., & Black, M. J. (2006). Bayesian population decoding of motor cortical activity using a kalman filter. Neural Computation, 18, 80–118. Wu, W., Shaikhouni, A., Donoghue, J. P., & Black, M. J. (2004). Closed-loop neural control of cursor motion using a kalman filter. In: Proceedings of the 26th annual international conference of the IEEE, engineering in medicine and biology society, Vol. 2, pp. 4126–4129. Yu, B. M., Afshar, A., Santhanam, G., Ryu, S. I., Shenoy, K. V., & Sahani, M. (2006). Extracting dynamical structure embedded in neural activity. In Y. Weiss, B. Schölkopf & J. Platt (Eds.), Advances in neural information processing systems, (Vol. 18, pp. 1545–1552). Cambridge, MA: MIT Press. Yu, B. M., Cunningham, J. P., Santhanam, G., Ryu, S. I., Shenoy, K. V., & Sahani, M. (2009). Gaussian-process factor analysis for low-dimensional single-trial analysis of neural population activity. Journal of Neurophysiology, 102, 614–635. Yu, B. M., Kemere, C., Santhanam, G., Afshar, A., Ryu, S. I., Meng, T. H., et al. (2007). Mixture of trajectory models for neural decoding of goal-directed movements. Journal of Neurophysiology, 97, 3763–3780. Yu, B. M., Santhanam, G., Sahani, M., & Shenoy, K. V. (2010). Neural decoding for motor and communication prostheses. In K. G. Oweiss (Ed.), Statistical signal processing for neuroscience (pp. 219–263). Elsevier.

A. M. Green, C. E. Chapman, J. F. Kalaska and F. Lepore (Eds.) Progress in Brain Research, Vol. 192 ISSN: 0079-6123 Copyright Ó 2011 Elsevier B.V. All rights reserved.

CHAPTER 4

Physically interactive robotic technology for neuromotor rehabilitation Neville Hogan{,{,* and Hermano I. Krebs{,} { {

Department of Mechanical Engineering, Massachusetts Institute of Technology, Cambridge, Massachusetts, USA Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, Massachusetts, USA } Department of Neurology, University of Maryland School of Medicine, Maryland, USA

Abstract: Robotic technology can provide innovative responses to the severe challenges of providing cost-effective care to restore sensory-motor function following neurological and biomechanical injury. It may be deployed at several points on a continuum of care, to provide precisely controlled sensorymotor therapy to ameliorate disability and promote recovery of function, or to provide assistance to compensate for functions that cannot be recovered, or to replace limbs lost irretrievably. This chapter reviews recent progress using robotic technology to capitalize on neural plasticity and promote recovery after neurological injury such as stroke (cerebral vascular accident), research on brain–computer interfaces as a source of control signals for assistive technologies, and research on high-performance multiple-degree-of-freedom upper-extremity prosthetic limbs. Keywords: robotics; sensory; motor; rehabilitation; therapy; neurorehabilitation; prosthetics; assistive technology.

imposed increasingly restrictive reimbursement guidelines, allotting fewer reimbursement dollars per patient. Providers have responded primarily by shortening inpatient length of stay (LOS) but this encourages greater reliance on home health visits for wound care, pharmacological monitoring, physical therapy, and more. In the 1990s, reimbursement for home health care and rehabilitation services increased (Grimaldi, 1998), but this changed in 1998 when CMS adopted a goal of significantly cutting rehabilitation and home health care spending.

Introduction The cost of healthcare in the United States, which surpassed 16.6% of the total 2007 Gross National Production ($2.1 trillion) continues to shape the landscape of healthcare delivery. According to the Centers for Medicare and Medicaid Services (CMS), Medicare and other payor programs have *Corresponding author. Tel.: þ1-617-253-8117; Fax: þ1-671-258-7018. E-mail: [email protected] DOI: 10.1016/B978-0-444-53355-5.00004-X

59

60

This played out against a backdrop of looming increases in the number of persons requiring rehabilitation services. Stroke (cerebral vascular accident) provides an informative example. Already the foremost cause of permanent motor disability in the United States, a substantial increase in the number of stroke cases, is projected to accompany the “graying” of the population. The leading edge of the baby-boom generation has already passed age 55, after which the relative incidence of stroke doubles in every succeeding decade. In the United States, the annual incidence of stroke has steadily increased over the past decade and is presently in excess of 795,000 occurrences annually (AHA, 2010). Similar trends are evident worldwide. Successfully improving the quality of care while reducing per-patient costs cries out for innovative methods. Fortunately, the decades that have seen increasing demands on rehabilitation services have also seen spectacular improvements in technology, a trend that shows little sign of abatement. Advances have been made in computation, communication, control, measurement, actuation—all of which are combined in the field of robotic engineering. Robotic technology can contribute at many points along the continuum of care: delivering sensory-motor training to capitalize on activity-dependent neural plasticity (Nudo, 2007) and maximize recovery, providing functional assistance to compensate for incomplete recovery, or replacing entire limbs or functions that have been lost irretrievably. This presents an opportunity but also a challenge: how to deploy these technological advances to achieve real improvements in rehabilitation. It is not simply a matter of “If you build it, they will come” (Robinson, 1989); the technology must augment the neuromechanical and biomechanical factors underlying recovery and demonstrably be appropriate and cost-effective. This chapter reviews some recent advances in the deployment of robotic technology for rehabilitation. It necessarily presents a personal view, and inevitably, some important developments

have been overlooked. This is not meant to dismiss or diminish that work but reflects the limitations of the authors’ experience and the difficulty of covering the breadth of this burgeoning field.

Robotic therapy The greatest impact of robotics on rehabilitation may be anticipated by attempting to address the greatest societal need. That was the vision motivating our earliest attempts to use robots to support recovery of sensory-motor function following after stroke—in the United States, the leading cause of permanent motor disability. Because of a recovering stroke survivor's obvious motor impairment, it was essential that the technology should be physically interactive, able to cooperate with the patient (and therapist, if needed) without suppressing or discouraging any residual movement ability, allowing her to express what appropriate movement she could, and assisting as necessary to achieve functional goals (Hogan et al., 1995). This presented a formidable technical challenge because physical interaction or even contact between a robot and any object not characterized by the robot's control system frequently evokes unstable behavior (Paul, 1987). This is obviously unacceptable, but ensuring stability is not the only challenge: the robot must also be gentle. A typical stroke survivor is an older adult who may be frail, with compromised musculo-skeletal integrity. For example, the biomechanical integrity of the shoulder depends in large part on the activity of muscles in the shoulder girdle. Weakness or inadequate activation of those muscles is common after stroke which makes the shoulder joint especially vulnerable in these patients. This may account, in part, for the prevalence of joint pain and may contribute to “shoulder-hand syndrome” in patients recovering after stroke. It is therefore essential that a therapy robot should not exert excessive force on a patient. To achieve

61

guaranteed stability and gentleness, we adopted a control strategy specifically designed to enable physical interaction: that is, impedance control (Hogan, 1980, 1985; Hogan and Buerger, 2004). The essential point is that the robot neither imposes a preprogrammed motion, nor does it impose a specified force. Instead, it establishes a relation between forces and motions which tends to achieve the task goal. This simple change of perspective admits strong guarantees of stable behavior (Hogan, 1988) and encourages cooperation between human and machine, the latter compensating for deficiencies of the former (Krebs et al., 2003).

Does robotic therapy work in practice? Even with a technical solution available, there remained the difficult challenge of testing whether the form of therapy that might be delivered by a robot would be effective or useful. At the time of writing, more than a decade of studies conducted by many different research groups (studies by the authors’ collaborators alone have involved more than 600 patients) have established the benefits of robot-mediated therapy in stroke rehabilitation, consistently reducing impairment by a factor of two or more (Aisen et al., 1997; Ferraro et al., 2003; Krebs et al., 1998, 2000, 2008; Krebs and Hogan, 2006; Kwakkel et al., 2008; Volpe et al., 1999, 2000, 2001). Acknowledging this record, the American Heart Association (AHA) issued “The Comprehensive Overview of Nursing and Interdisciplinary Rehabilitation Care of the Stroke Patient: A Scientific Statement from the American Heart Association” (Miller et al., 2010). It recommended that: “Robot-assisted therapy offers the amount of motor practice needed to relearn motor skills with less therapist assistance. Most robots for motor rehabilitation not only allow for robot assistance in movement initiation and guidance but also provide accurate feedback; some robots additionally provide movement resistance. Most trials of

robot-assisted motor rehabilitation concern the upper extremity (UE), with robotics for the lower extremity (LE) still in its infancy.” According to this AHA report, UE robotassisted therapy has already achieved Class I, Level of Evidence A for stroke care in the outpatient and chronic care settings; and it has achieved Class IIa, Level of Evidence A for stroke care in the inpatient setting. To explain, Class I is defined as: “Benefit ⋙ Risk. Procedure/Treatment SHOULD be performed/administered”; Class IIa is defined as: “Benefit  Risk, IT IS REASONABLE to perform procedure/administer treatment”; and Level of Evidence A is defined as “Multiple populations evaluated: Data derived from multiple randomized clinical trials (RCTs) or meta-analysis.”

CSP-558 Much of the evidence cited in that report is derived from a landmark study led by the Veterans Administration (VA) Cooperative Studies Program (CSP). CSP-558 was the first multisite RCT of UE robotic therapy with a population of Veterans in the chronic phase of poststroke recovery. Recently published in the New England Journal of Medicine (Lo et al., 2010), CSP-558 was compared to a combination of phase 2 and 3 studies for FDA (food and drug administration) approval of pharmacological agents. The study established the safety of UE rehabilitation robots; there were no serious adverse events in the robot-treated group. The study also evaluated efficacy and cost. By far, the most important finding of CSP-558 was that usual care (three sessions per week from therapists delivering treatment as they saw fit for the UE) did not reduce impairment, disability, or improve quality of life in chronic stroke survivors. The usual-care intervention had no measureable impact, and to conserve financial resources, it was discontinued as futile midway through the study (Krebs, 2010).

62

The trial continued for about another year and compared UE robotic therapy for the shoulder, elbow, wrist, and hand (which delivered 1024 movements in 1-hour sessions three times per week) with an intensive comparison training (ICT) protocol. The ICT protocol is an unusual form of human-administered therapy, specifically formulated to serve as a meaningful control for comparison with robotic therapy (Volpe et al., 2008): a therapist is required to deliver movement of intensity and repetition comparable to robotic therapy during the same period. It was anticipated that this control would offer benefits comparable to the robot. Recovery depends on neural plasticity and to the extent that intense movement practice shepherds that plasticity to aid recovery, whether that practice is administered by a robot or a human should make little difference, and that was confirmed by CSP-558. However, it must be emphasized that ICT is not conventional therapy. It employed manual techniques but at an intensity far beyond usual care and under a time–pressure that cannot feasibly be implemented as standard care in a clinical setting. As one therapist involved in the study quipped, the ICT therapist is the equivalent of Charlie Chaplin's hapless assembly-line worker in “Modern Times” (Chaplin, 1936). Other comparisons enabled by that study are revealing. Lo et al. (2010) compared the robot group with usual care at 12 weeks but of greater importance is whether any observed changes were robust and durable. At 12 weeks, the difference between the first half of the robotic treatment group and the usual-care group was slightly over two Fugl-Meyer points (The FuglMeyer assessment measures impairment). However, the difference between the second half of the robotic treatment group and the usual-care group was almost eight Fugl-Meyer points. Overall, at the end of treatment, the robotic group showed a total five-point advantage over usual care. In addition, 6 months following completion of treatment, robotic therapy was statistically superior to usual care on the Stroke Impact Scale

(which measures quality of life), on the FuglMeyer assessment, and on the Wolf Motor Function test (an assessment of function).

Applicability Another important point about CSP-558 is that these groups of patients with chronic stroke disability were moderately to severely impaired and over 30% had multiple strokes. The study groups represented a spectrum of disability burden that many previous studies have avoided. Yet, robotic therapy is applicable to this population, and 65% of the 197 volunteers who were screened for this study were enrolled. For comparison, an RCT of constraint-induced movement therapy (CIMT) reported durable statistically significant and clinically relevant improvements in arm motor function (Wolf et al., 2006). However, because that protocol restrains the less-affected arm to emphasize intensive practice with the affected arm, it is only applicable to patients with relatively mild impairment. In fact, that study enrolled only 6% of the 3626 volunteers who were screened. That is not to say that robotics and CIMT are competitive protocols; on the contrary, they are complementary alternatives. To illustrate, consider the effect of robot therapy for the wrist shown in Fig. 1. Panel A shows a stroke survivor engaged in robotic therapy for the wrist. Panel B shows wrist movements made by a chronicphase stroke survivor with an impaired left wrist before robotic treatment. Movements are irregular and uncoordinated and extremely limited in extension. Consequently, this patient would not qualify for CIMT. Panel C shows wrist movements made by the same patient after robotic treatment (1-hour sessions on 3 alternate days per week for 6 weeks, each session comprising 1024 pose-to-pose movements to visually presented targets in flexion–extension and abduction–adduction with the robot assisting only as needed to ensure reasonable completion of the task). After robotic therapy, movements are

63 A

C

15 10

Abduction Angle [degrees]

Abduction Angle [degrees]

B

5 0 −5 −10 −15 −20 −25 −10 −5

0

5 10 15 20 25 30 Flexion angle [degrees]

35

40

15 10 5 0 −5 −10 −15 −20 −25 −25 −20 −15 −10 −5

0

5

10

15

20

25

Flexion angle [degrees]

Fig. 1. Panel A shows a stroke survivor engaged in robotic therapy for the wrist. Therapy consisted of 1-hour sessions on 3 alternate days per week for 6 weeks in which the robot assisted with pose-to-pose movements to visually presented targets in flexion–extension and abduction–adduction (radial–ulnar deviation). Panel B shows wrist movements made by a chronic-phase stroke survivor with an impaired left wrist before robotic treatment. The horizontal axis shows flexion–extension; the vertical axis shows abduction–adduction. Movements are irregular and uncoordinated and extremely limited in extension. Panel C shows wrist movements made by the same patient after robotic treatment. Movements are visibly more coordinated, and the range of extension is substantially increased.

visibly more coordinated and the range of extension is substantially increased. This patient would now qualify for either CIMT or robotic therapy if further treatment was deemed appropriate. Even if the benefits of robotic therapy appear modest, they can be functionally significant, especially for severely to moderately impaired

patients. For example, an improvement of about three points on the Fugl-Meyer scale for a severely affected patient would enable her to raise the arm and bathe independently, or stretch the formerly paralyzed arm so that independent dressing could be accomplished. A more moderately affected patient might acquire the ability to

64

tuck in a shirt independently or reach overhead to grasp an object. Further, the persistent improvement at the 6-month follow-up evaluation suggests the tantalizing possibility that the impairment reduction due to robot therapy was sufficiently robust to enable use of the limb to be incorporated into activities of daily living, not only maintaining the incremental improvement but also potentially evoking further activitydependent neural plasticity and affording further improvement without continued therapeutic intervention.

vulnerable neuromechanical system. Further, therapy based on close physical contact and interaction can lead to durable improvements in neuromotor performance. Yet much remains to be done. Adapting the existing technology to treat other patient populations is an active area of research (Krebs et al., 2009), and in addition, the technology is by no means mature. As the AHA report notes “robotics for the LE (is) still in its infancy” (Miller et al., 2010). While some recent results justify a guarded optimism (Forrester et al., 2010), significant further work is required to make robotic technology an effective treatment for LE disorders.

Cost-effectiveness Establishing effectiveness is important but not enough for successful deployment. In this era of stringent cost-containment, it is also important to establish cost-effectiveness. In that regard, the results of CSP-558 were extremely encouraging. As expected, the active interventions (robotic therapy and ICT) incurred added cost—for example, the robotic equipment and an additional therapist cost the VA about $10,000 per patient for 36 months. Remarkably, when the total costs were compared—which includes the clinical care needed to take care of these Veterans—there were no differences between the active interventions and usual care. That was because patients in the usual-care group used the rest of the VA health care system three times more often than the active intervention groups. For 36 weeks of care, each patient in the robotic group cost the VA $10,000 for robotic therapy and $5000 for clinical care. For 36 weeks of care, each patient in the usual-care group cost the VA approximately $15,000 for clinical care. Though a detailed cost–benefit analysis remains to be published, this suggests better care for the same total cost. Summarizing briefly, evidence to date indicates that suitably configured robotic technology can operate safely in close physical contact with humans, even patients with a compromised and

Assistive technologies Even the most optimistic proponent of robotic therapy would be foolish to claim that it offers a complete cure for neuromotor disorders. Even with the best of treatment, it seems likely that fully functional recovery of unimpaired ability will be rare. For example, spinal cord injury (SCI) is especially challenging as the neural plasticity that appears to underlie recovery after stroke (Nudo, 2007) appears to be much more limited in the peripheral nervous system, though recent advances in pharmacological and cellular therapies may change this picture (Edgerton et al., 2006). Extrapolating this line of thought, the prospects for limb regrowth after amputation seem even more remote at this time. Fortunately, robotic technology is sufficiently versatile that it may be applied in many ways. It may initially provide therapy to enable a patient to achieve a “personal best” recovery of function, however, limited that may be. It may subsequently provide assistance to compensate for functions that cannot be recovered; or it may be used to replace lost limbs, the domain of orthotic, and prosthetic devices. All these technologies should be developed to become items available in the “toolbox” for technology-based rehabilitation.

65

Unfortunately, compensatory and assistive technologies can present surprisingly difficult challenges. Some assistive technologies have become commonplace; for example, powered wheelchairs to compensate for locomotor limitations. Though this is largely a mature technology, there remains room for significant innovation and development. However, successful deployment appears to be limited more by market factors rather than technology. For example, the iBOT (offered by Independence Technology, L.L.C. a Johnson & Johnson company) enabled upright balancing as well as stair ascent and descent but was recently withdrawn from the market (Associated Press, 2009). Compensation for reach and grasp limitations has been addressed by mounting multiaxis robots on a wheelchair; the most notable example is the MANUS robot (Verburg et al., 1996). This technology reveals one of the most difficult technical challenges: how to provide the sufficient control signals to coordinate multiple degrees of freedom. The usual approach relies on a manual interface such as a joystick but that typically makes the robot operation slow and difficult to master.

Brain–computer interfaces Research on so-called brain–computer interfaces (BCIs)—also termed brain–machine interfaces (BMIs)—appears to offer the promise of providing control signals derived from the rich communication between the cerebral cortex and the neuromuscular periphery (Cohen and Birbaumer, 2007; Hatsopoulos and Donoghue, 2009). Although research on BCIs dates to the late 1960s (Fetz, 1969), recent progress has demonstrated control of multiple degree-of-freedom robots by real-time recordings from neurons in the primate cortex (Carmena et al., 2003; Taylor et al., 2002). A similar approach has been used to derive control signals from the brain of a human subject with tetraplegia (Hochberg et al., 2006).

These interfaces are highly invasive, not only passing through the skull but also penetrating the meninges protecting the brain and posing a substantial risk of infection and tissue damage, and that may severely limit their applicability. Noninvasive approaches based on Electroencephalographic (EEG) recordings have been demonstrated (Wolpaw and McFarland, 2004) but are generally perceived as offering more limited control. Electrocorticographic (ECoG) recordings (which use subdural electrodes that do not penetrate the pia mater) may provide an appropriate compromise (Schalk et al., 2007). However, to the best of the authors’ knowledge, the communication rates (bits/second) reported to date in humans with any of these approaches have been extremely slow, substantially inferior to performance with a manual joystick (Wolpaw, 2007). Future research may be expected to overcome some of these technical difficulties, but at present, this approach has yet to prove its value in a practical rehabilitation context.

Amputation prosthetics Primarily due to highly publicized projects heavily funded by the U.S. Defense Advanced Research Projects Agency (DARPA), prosthetic arms based on advanced robotic technology have recently received substantial media attention (Adee, 2009). This is the most difficult technical challenge of all. Because the machine must be “worn” by the amputee, its design is dominated by severe constraints on safety, weight, power, and durability. Control also presents greater challenges, partly because of the large number of degrees of freedom that must be coordinated (22 in the DARPA arm). BCIs would seem to be an obvious candidate to provide the required control signals, but an ingenious alternative— targeted muscle reinnervation (TMR)—has already been demonstrated in clinical applications (Kuiken et al., 2009). The surviving nerves that supplied the lost limb are transferred to residual

66

chest or upper arm muscles that are no longer biomechanically functional due to loss of the limb. Remarkably, the transferred nerves reinnervate these muscles, whose Electromyographic activity (EMG) serves to amplify motor commands from these nerves, thereby providing control signals for a prosthetic arm. The TMR technique has been successfully performed in patients with transhumeral and shoulder-disarticulation amputations and has markedly improved their functional use of prostheses (Kuiken et al., 2009). But the challenges of arm prosthesis control are not met simply by acquiring control signals; how the machine responds to those signals is a critical concern. Whereas a therapy robot must ensure safe and gentle interaction primarily at one point of contact—between the robot and the human—a prosthetic arm must manage at least two: contact and physical interaction with objects to be manipulated as well as contact and physical interaction with the rest of the amputee. To achieve natural control, the prosthesis joints must be seamlessly coordinated with the (residual) joints of the arm or thorax to which the prosthesis is attached. One way this may be accomplished is by designing the prosthesis control system to emulate the controllable mechanical impedance that is achieved in the natural limbs by co-contraction of antagonist muscles (Hogan, 1984). A comparison of this control system with the more conventional motion control schemes commonly used in motorized amputation prostheses demonstrated superior performance and better synergy between natural and artificial joints (Abul-Haj and Hogan, 1990). To the best of the authors’ knowledge, a comparable control architecture has yet to be implemented on the newer generation of arm prostheses, but we expect exciting developments in the future.

Conclusion This brief survey of robotic technology may serve to illustrate its general applicability to neuromotor rehabilitation. Physically-interactive

robotics uniquely provides a means to manipulate and supplement the functional actions that result from neural activity, providing an ideal vehicle to translate cutting-edge motor neuroscience into real societal benefit.

Acknowledgment Portions of this chapter have been excerpted from previous publications. Neville Hogan is supported in part by Toyota Motor Corporation's Partner Robot Division and by the Eric P. and Evelyn E. Newman Fund. Hermano I. Krebs is supported in part by the Department of Veterans Affairs Rehabilitation Research and Development Service (VA RR&D) “Center of Excellence on Task-Oriented Exercise and Robotics in Neurological Diseases,” B3688R and NIH R01-HD045343. Conflict of Interest Statement: N. Hogan and H. I. Krebs are coinventors of MIT-owned patents for interactive therapeutic robotic devices and hold equity positions in Interactive Motion Technologies, Inc., a company that manufactures and distributes this technology under license to MIT. References Abul-Haj, C. J., & Hogan, N. (1990). Functional assessment of control-systems for cybernetic elbow prostheses. 2. Application of the technique. IEEE Transactions on Biomedical Engineering, 37, 1037–1047. Adee, S. (2009). The revolution will be prosthetized. IEEE Spectrum, 45–48, January. Aisen, M. L., Krebs, H. I., Hogan, N., McDowell, F., & Volpe, B. T. (1997). The effect of robot-assisted therapy and rehabilitative training on motor recovery following stroke. Archives of Neurology, 54, 443–446. American Heart Association. Heart disease and stroke statistics—2010 Update. (online at http://circ.ahajournals.org/). The Associated Press, 5/25/2009, “Stair-climbing wheelchair comes to a halt.” Carmena, J. M., Lebedev, M. A., Crist, R. E., O'Doherty, J. E., Santucci, D. M., Dimitrov, D. F., et al. (2003). Learning to control a brain–machine interface for

67 reaching and grasping by primates. PLoS Biology, 1, 001–016. Chaplin, C. (1936). Modern Times. United Artists. Cohen, L. G., & Birbaumer, N. (2007). The physiology of brain–computer interfaces. The Journal of Physiology, 579, 570. Edgerton, V. R., Kim, S. J., Ichiyama, R. M., Gerasimenko, Y. P., & Roy, R. R. (2006). Rehabilitative therapies after spinal cord injury. Journal of Neurotrauma, 23, 560–570. Ferraro, M., Palazzolo, J. J., Krol, J., Krebs, H. I., Hogan, N., & Volpe, B. T. (2003). Robot aided sensorimotor arm training improves outcome in patients with chronic stroke. Neurology, 61, 1604–1607. Fetz, E. E. (1969). Operant conditioning of cortical unit activity. Science, 163(870), 955. Forrester, L. W., Roy, A., Krebs, H. I., & Macko, R. F. (2011). Ankle training with a robotic device improves hemiparetic gait after a stroke. Neurorehabilitation and Neural Repair, 25(4), 369–377. Grimaldi, P. L. (1998). Medicare imposes new caps on postacute care. Nursing Management, 10–12. Hatsopoulos, N. G., & Donoghue, J. P. (2009). The science of neural interface systems. Annual Review of Neuroscience, 32, 249–266. Hochberg, L. R., Serruya, M. D., Friehs, G. M., Mukand, J. A., Saleh, M., Caplan, A. H., et al. (2006). Neuronal ensemble control of prosthetic devices by a human with tetraplegia. Nature, 442, 164–171. Hogan, N. (1980). Mechanical impedance control in assistive devices and manipulators. Proceedings of the Joint Automatic Control Conference, San Francisco, CA , paper TA-10-B. Hogan, N. (1984). Adaptive-control of mechanical impedance by coactivation of antagonist muscles. IEEE Transactions on Automatic Control, 29, 681–690. Hogan, N. (1985). Impedance control: An approach to manipulation. ASME Journal of Dynamic Systems, Measurement and Control, 107, 1–24. Hogan, N. (1988). On the stability of manipulators performing contact tasks. IEEE Journal of Robotics and Automation, 4, 677–686. Hogan, N., & Buerger, S. P. (2004). Impedance and interaction control. In T. R. Kurfess (Ed.), Robotics and automation handbook (pp. 19-11–19-24). Boca Raton: CRC Press. Hogan, N., Krebs, H. I., Sharon, A., & Charnnarong, J. (1995). Interactive robotic therapist. US Patent Number 5,466,213, issued November 14. Krebs, H. I. (2010). Rehabilitation robotics: 2010 and the new decade. WFNR World Federation for Neurorehabilitation. Newsletter Autumn. Krebs, H. I., Dipietro, L., Levy-Tzedek, S., Fasoli, S., Rykman, A., Zipse, J., et al. (2008). A paradigm shift for rehabilitation robotics. IEEE-EMBS Magazine, 61–70.

Krebs, H. I., & Hogan, N. (2006). Therapeutic robotics: A technology push. Proceedings of IEEE: Special Issue on Rehabilitation Robotics, 94(9), 1727–1738. Krebs, H. I., Hogan, N., Aisen, M. L., & Volpe, B. T. (1998). Robot-aided neurorehabilitation. IEEE Transactions on Rehabilitation Engineering, 6, 75–87. Krebs, H. I., Ladenheim, B., Hippolyte, C., Monterroso, L., & Mast, J. (2009). Robot-assisted task-specific training in cerebral palsy. Developmental Medicine and Child Neurology, 51, 140–145. Krebs, H. I., Palazzolo, J. J., Dipietro, L., Ferraro, M., Krol, J., Rannekleiv, K., Volpe, B. T., & Hogan, N. (2003). Rehabilitation robotics: Performance-based progressive robotassisted therapy. Autonomous Robots, 15, 7–20. Kluwer Academics. Krebs, H. I., Volpe, B. T., Aisen, M. L., & Hogan, N. (2000). Increasing productivity and quality of care: Robot-aided neurorehabilitation. VA Journal of Rehabilitation Research and Development, 37(6), 639–652. Kuiken, T. A., Li, G., Lock, B. A., Lipschutz, R. D., Miller, L. A., Stubblefield, K. A., et al. (2009). Targeted muscle reinnervation for real-time myoelectric control of multifunction artificial arms. Journal of the American Medical Association, 301, 619–628. Kwakkel, G., Kollen, B. J., & Krebs, H. I. (2008). Effects of robot-assisted therapy on upper limb recovery after stroke: A systematic review. Neurorehabilitation and Neural Repair, 22(2), 111–121. Lo, A., Guarino, P. D., Richards, L. G., Haselkorn, J. K., Wittenberg, G. F., Federman, D. G., et al. (2010). Robotassisted therapy for long-term upper-limb impairment after stroke. The New England Journal of Medicine, 362, 1772–1783. Miller, E. L., Murray, L., Richards, L., Zorowitz, R. D., Bakas, T., Clarck, P., et al. (2010). The comprehensive overview of nursing and interdisciplinary rehabilitation care of the stroke patient: A scientific statement from the American Heart Association. Stroke, 41, 2402–2448. Nudo, R. J. (2007). Postinfarct cortical plasticity and behavioral recovery. Stroke, 38, 840–845. Paul, R. P. (1987). Problems and research issues associated with the hybrid control of force and displacement. Proceedings IEEE Conference on Robotics and Automation. pp. 741–750. Robinson, P. A. (1989). Field of Dreams Universal Studios. adapted by from W. P. Kinsella (1982) Shoeless Joe, Houghton-Mifflin. The correct quotation is “If you build it, he will come.” Schalk, G., Kub´anek, J., Miller, K. J., Anderson, N. R., Leuthardt, E. C., Ojemann, J. G., et al. (2007). Decoding two-dimensional movement trajectories using electrocorticographic signals in humans. Journal of Neural Engineering, 4, 264–275.

68 Taylor, D. M., Tillery, S. I. H., & Schwartz, A. B. (2002). Direct cortical control of 3D neuroprosthetic devices. Science, 296, 1829–1832. Verburg, G., Kwee, H., Wisaksana, A., Cheetham, A., & van Woerden, J. (1996). Manus: The evolution of an assistive technology. Technology and Disability, 5. Volpe, B. T., Krebs, H. I., & Hogan, N. (2001). Is robot-aided sensorimotor training in stroke rehabilitation a realistic option? Current Opinion in Neurology, 14, 745–752. Volpe, B. T., Krebs, H. I., Hogan, N., Edelstein, O. L., Diels, C., & Aisen, M. (2000). A novel approach to stroke rehabilitation: Robot-aided sensorimotor stimulation. Neurology, 54, 1938–1944. Volpe, B. T., Krebs, H. I., Hogan, N., Edelsteinn, L., Diels, C. M., & Aisen, M. L. (1999). Robot training enhanced motor outcome in patients with stroke maintained over 3 years. Neurology, 53, 1874–1876.

Volpe, B. T., Lynch, D., Ferraro, M., Galgano, M., Hogan, N., & Krebs, H. I. (2008). Intensive sensorimotor arm training improves hemiparesis in patients with chronic stroke. Neurorehabilitation Neural Repair, 22(3), 305–310. Wolf, S. L., Winstein, C. J., Miller, J. P., Taub, E., Uswatte, G., Morris, D., et al. (2006). Effect of constraint-induced movement therapy on upper extremity function 3 to 9 months after stroke: The EXCITE randomized clinical trial. Journal of the American Medical Association, 296, 2095–2104. Wolpaw, J. R. (2007). Brain–computer interfaces as new brain output pathways. The Journal of Physiology, 579, 613–619. Wolpaw, J. R., & McFarland, D. J. (2004). Control of a twodimensional movement signal by a noninvasive brain–computer interface in humans. Proceedings of the National Academy of Sciences of the United States of America, 101, 17849–17854.

A. M. Green, C. E. Chapman, J. F. Kalaska and F. Lepore (Eds.) Progress in Brain Research, Vol. 192 ISSN: 0079-6123 Copyright Ó 2011 Elsevier B.V. All rights reserved.

CHAPTER 5

Sensory feedback for upper limb prostheses Steven S. Hsiao{,*, Michael Fettiplace{,1 and Bejan Darbandi{,2 { {

Department of Neuroscience and the Krieger Mind/Brain Institute, The Johns Hopkins University, Baltimore, Maryland, USA Department of Biomedical Engineering, The Johns Hopkins University, Baltimore, Maryland, USA

Abstract: In this chapter, we discuss the neurophysiological basis of how to provide sensory feedback to users with an upper limb prosthesis and discuss some of the theoretical issues that need to be considered when directly stimulating neurons in the somatosensory system. We focus on technologies that are currently available and discuss approaches that are most likely to succeed in providing natural perception from the artificial hand to the user. First, we discuss the advantages and disadvantages of providing feedback by stimulating directly the remaining afferents that originally innervated the arm and hand. In particular, we pay close attention to the normal functional roles that the peripheral afferents play in perception. What are the consequences and implications of stimulating these afferents? We then discuss whether it is reasonable to stimulate neurons in the ascending pathways that carry the information from the afferents to the cortex or directly in neurons in the primary somatosensory cortex. We show that for some modalities there are advantages for stimulating in the spinal cord, while for others it is advantageous to stimulate directly in the somatosensory cortex. Finally, we discuss results from a current experiment in which we used electrical stimuli in primary somatosensory cortex to restore the percept of the intensity of a mechanical probe indented into the hand. The results suggest that the simple percept of stimulus intensity can be provided to the animal from a single finger using four electrodes. We propose that significantly more electrodes will be needed to reproduce more complex aspects of tactile perception. Keywords: Sensory feedback; electrical stimulation; Somatosensory organization; Neural code. Introduction *Corresponding author. Tel.: þ1-410-516-6409; Fax: þ1-410-516-8648 E-mail: [email protected]

Often science fiction paves the way toward innovation and discovery. An example is the prosthetic limb Luke Skywalker uses in the movie “The Empire Strikes Back”. In the movie, Luke not only moves his prosthetic hand with the dexterity and strength of his original but also

1

Currently a MD/PhD candidate at University of Illinois at Chicago. 2 Currently an engineer at Medtronic Corporation. DOI: 10.1016/B978-0-444-53355-5.00005-1

69

70

receives robust sensory information that allows him to effortlessly grasp and manipulate objects (including his light saber) and perceive sensory inputs related to pain. While current technologies are far from achieving such lofty goals, we are starting to realize how to turn this fantasy into reality. Thus far, the most exciting advances in prosthesis research have come in the area of motor control (e.g., Velliste et al., 2008). These studies demonstrate that it is highly feasible for animals to accurately control movements not only of a prosthetic arm but also of the individual fingers of a prosthetic hand (Acharya et al., 2008). In these motor studies, single unit responses or local field potentials from populations of individual neurons, or the recordings from ECoG recordings from chronically implanted neurointerface chips, or arrays of microelectrodes located in motor cortex are decoded to determine the desired movement of the user. These decoded signals are then used to drive electrical motors that move the prosthesis. The current challenge in this area of prosthesis research is to determine how to use these cortical signals to give users fine coordinated movements of the prosthetic hand. However, dexterous control of the prosthesis is only half of what is needed to achieve the ideal prosthesis. What is also needed is high-quality sensory feedback that will enable users to know where their hand and arm is in space and to let them feel what the prosthetic hand is touching. It is only when both flexible motor control and sensory feedback are integrated will the upper limb prosthesis be considered a complete success. In this chapter, we focus on what is needed to provide robust sensory feedback to prosthetic arm users. While current methods of providing feedback are relatively crude, it is clear that they demonstrate the fundamental issues that need to be solved. Currently, three approaches are used to provide feedback. All three approaches assume that there is a fully instrumented hand with sensors that mimic or are capable of capturing information that was encoded by the original receptors in the skin. This information can then be processed to drive the appropriate sensory substitution device.

The first approach is targeted reinnervation (Kuiken et al., 2007), where the afferent fibers that once came from the hand are moved to target tissues in the upper chest. After the skin has become reinnervated, touching the skin evokes the perception that the missing hand is being touched. To produce percepts that are useful to the user, the skin on the chest can then be stimulated using mechanical stimulators that target orphaned afferent fibers. While the approach has been successful in giving prosthesis users some feedback, it has limited potential since there is only partial reinnervation of the skin and as such the information that can be delivered to the user is restricted to the afferent fibers that sparsely activate random patches of skin that map on to skin that use to be on the hand. The second approach is to use sensory substitution methods whereby signals from the prosthesis are used to activate a sensory substitution device that the user then learns to interpret as a sensory signal related to the prosthesis. For example, one such device stretches the intact skin somewhere else on the body back and forth using skin contactors that rotate on the forearm. Changes in hand position of the prosthetic are encoded as changes in the amount that the skin is stretched by the device. Another example would be a set of vibrators located on the forearm or back. Activating different combinations of vibrators are then discriminated as different components of the sensory input (see Jones, 2011). This approach is limited by the number of independent signals that can be processed and perceived by user. The third kind of sensory feedback, which holds the most promise and is the focus of this chapter, is to stimulate directly either the afferent fibers in the arm or spinal cord or neurons in the somatosensory cortex. While current technologies use electrical stimulation, it is highly likely that in the near future, neurons will be stimulated using optical methods. The reason why this approach has the greatest potential for giving users natural sensory feedback is because it has an unlimited potential to stimulate large populations of neurons and it takes advantage of the inherent modality specificity of the underlying neurons.

71

Below we first describe the roles and functions of the sensory afferents that innervate the hand with an eye toward determining how to restore these sensory functions to the prosthesis user. We then discuss a theoretical framework for providing sensory feedback to users and finish with a discussion showing the results from current experiments in our lab in which we use direct stimulation of neurons in the somatosensory cortex to give animals the perception of a mechanical probe being indented into the skin at different intensities.

Functional roles of the peripheral afferents for feedback for action and perception Sensory feedback from the receptors in the arm and hand play two important roles. The first is to provide feedback for producing controlled action or movements of the hand and arm and the second is to provide inputs that give users sensory experience. The functions of sensory feedback for action

and perception are overlapping since under normal circumstances we need to move our hands to manipulate and explore objects. However, it is useful to conceptually separate these functions to see what sensory information is needed to restore specific functions. Table 1 gives a summary of the 13 kinds of afferents that innervate the hand. It can be seen from this table that the nervous system solves the problem of sensing its environment by having an initial set of afferent fibers that are selectively sensitive to different features of external stimuli and to different components of the internal state of the hand. An important principle that can be taken from this table is that the somatosensory system is not a single system but instead is composed of multiple parallel processing streams with each stream originating from a distributed set of specialized receptors and terminating in parallel in central and subcortical regions in the brain. Reproducing the information from these parallel input streams, each of which is receiving

Table 1. Peripheral receptor types in the primate hand (glabrous skin only for cutaneous mechanoreceptors)

Receptor

Fiber group

Receptors respond to

Cutaneous, low-threshold mechanoreceptors Merkel (SA1) Ab Steady deformation and motion Ruffini (SA2) Meissner (RA)

Ab Ab

Pacinian (PC)

Ab

Skin stretch Skin movement (glabrous skin only) High-frequency vibration

Function

Local form (e.g., Braille) texture (roughness, hardness, etc.) Skin stretch, digit/hand conformation Light touch, local movement, slip (for grip control) Distant events (hand-held tools)

Proprioceptors Muscle spindle (Ia) Golgi tendon organ (Ib) Muscle spindle (II) Joint

I I

Muscle length and velocity Muscle force

Position, movement Muscle force

II II

Muscle length Joint angle, movement

Position? Unclear (sensitive to extreme joint angles)

Thermoreceptors Cold

Ad

Drop in skin temperature

Warm

C

Warmth

Cold (temperature of object relative to skin temperature) Warmth

Nociceptors Small myelinated Unmyelinated Itch

Ad C C

Noxious stimuli Noxious stimuli Pruritic stimuli

Sharp, pricking pain Dull, burning pain Itch receptors

72

information from thousands of afferent fibers, using electrical or optical stimulation methods is not a simple problem. Sensory feedback for action is necessary since it gives users the feedback required to control the movements and forces needed to grasp and manipulate objects. In this role, the proprioceptive feedback gives the user information about the positions, movements, and forces of the limb, hand, and fingers. The afferent fibers that provide the feedback for action are the four kinds of proprioceptive afferents that provide information about the positions, velocities, and forces of the arm and hand. As shown in the table, information about muscle force is carried by the Golgi tendon organs, which are located in the tendons of the muscles. These receptors are in series with the muscle fibers and along with an efferent copy of the motor command signal that is generated in the cortex, is the basis for the perception of muscle force. Information about muscle force plays an important role in everyday life. For example, you need to know how heavy objects are to smoothly grasp and lift them. The sense of force could be given to the user with a sensory substitution device or by stimulating the afferent fibers directly in the remaining nerve or spinal cord. Stimulating directly in the cortex is currently not an option since the central projections of these afferent fibers is not known. The second kind of proprioceptive afferents are joint afferents which include a variety of afferent types that end in free nerve endings and paciniform corpuscles located in the joint capsules. Originally, it was thought that these afferents were responsible for conveying information about joint angle; however, this was ruled out because neurophysiological recordings showed that these afferents respond only at the extremes of joint flexion and extension and thus gave poor representations of precise joint angle. It is currently thought that these afferents function as limit detectors and give users a perception of when joints are extended to their extremes. These signals are useful but not essential in everyday

function since humans with joint replacement surgery maintain normal functionality. Thus, replacing the function of these afferents appears to be minimally important and can be built directly into the motors. The third and fourth kinds of proprioceptive afferents receive their inputs from the two kinds of muscle spindle afferents located in the intrafusal muscles. While these afferents are clearly needed for controlling muscle length, their role in perception is less clear since they do not seem to carry accurate measurements of joint angle and velocity (Dimitriou and Edin, 2008). However, there is convincing evidence that some aspect of joint angle is carried by these afferent types. One demonstration is the Pinocchio effect in which subjects are asked to place their finger on their nose. The belly of the muscle is then vibrated, which activates the muscle spindle afferents which in turn evokes the perception that the joint angle of the arm is increasing. Since the finger is still in contact with the nose, subjects perceive that their nose must be growing! However, it is doubtful that the signals provided by these afferents are sufficient to convey fine joint angle. If joint angle is not carried by these muscle spindle afferents then the question arises as to which afferents convey information about joint angle. The answer to this question appears to come from the cutaneous afferents that innervate the skin. Edin and Johansson (1995) have provided strong evidence that the slowly adapting type 2 (SA2) afferents in the skin contribute strongly to the sense of joint angle and they propose that the pattern of neural activity across the population of SA2 afferents conveys information about fine joint movements to the central nervous system (Dimitriou and Edin, 2008; Edin and Johansson, 1995). Given our current understanding of how joint angle is coded and represented in the peripheral afferents, it appears as if separate populations of SA2 afferent and muscle spindle afferents must be stimulated in a coordinated fashion to produce veridical percepts of joint angle. This approach is theoretically possible,

73

however, we do not currently have a sufficient understanding of how joint angle is coded in the peripheral afferents to produce the necessary pattern of stimulation. As described later, a more reasonable candidate is to directly stimulate neurons in cortex where the integration between the different afferent types encoding joint angle has already occurred. Besides the SA2 afferents there are three other kinds of peripheral mechanoreceptive afferents that innervate the skin. The Pacinian (PC) afferents have very large receptive fields and innervate the skin sparsely. These afferents are responsible for conveying information about vibration and play a key role in conveying information about events distant from the hand—as when we use tools. In intact humans, as exemplified by a blind person using a cane, these afferents in combination with inputs from the proprioceptive afferents convey information about the form and texture of objects. With the vibratory sense, a blind person can construct an internal representation of his or her environment based solely on inputs that they receive through the cane. This suggests that providing prosthesis users with the sense of vibration through the PC system is a viable way to give sensory feedback about their environment and is critically important when the prosthesis is used to manipulate tools. Providing vibratory input requires that the outputs of accelerometers imbedded in the prosthesis are decoded and turned into electrical signals that mimic the temporal pattern of what a PC afferent would have experienced under the same circumstances. That is, to be interpreted properly by the brain, the signals must carry the same temporal information that the central nervous system uses when extracting information about textures with tools (Yoshioka et al., 2007). Although we still do not understand how complex vibratory inputs related to generalized texture perception are represented in the PC system, this is a solvable problem. The other two afferent types, the rapidly adapting (RA) and slowly adapting type 1 (SA1) afferents, convey information to the central nervous system about motion (RA) and two-dimensional (2D) form

and texture (SA1). The RA afferents are highly sensitive to minute movements on the skin, sense when objects begin to slip in the hand, and provide sensory feedback to the user about increasing grip force. Clearly, this afferent system is important if the prosthesis is to be used to grasp and lift objects without crushing or dropping them. Under normal circumstances, the skin is densely innervated by the RA afferents which allows for users to make rapid fine adjustments in grip force as objects slip between the fingers. Thus providing feedback to replace these functions requires that slip sensors be built into the prosthesis with the outputs decoded and used to activate the relevant motors involved in grasping the object. It is not obvious that these signals for grip control need to be fed back directly to the RA afferents since information needed for fine grip control occurs rapidly (most likely in the form of a spinal reflex) which occurs before the information about the object slipping reaches consciousness. RA afferents are also important for the perception of tactile motion and are used when the hand scans surfaces, but these functions are better recreated by stimulating neurons in the cortex. It is important to note that volitional control of the prosthesis is important and that automated grasping functions of the prosthesis could be disturbing to the user. The image that comes to mind comes is Dr Strangeglove, from the movie of the same name, whose artificial hand seems to act of its own accord. In summary, providing the user with inputs from the RA system at the peripheral level may not be critically important for providing feedback for grasping. The fourth kind of mechanoreceptive system is based on inputs from the SA1 afferents. These afferents have been shown to convey information about 2D form and texture (Hsiao and Bensmaia, 2008; Hsiao et al., 1996; Johnson et al., 2002) and play an important role in encoding mechanical intensity (Hsiao et al., 1996; Muniak et al., 2006). In combination with inputs from the proprioceptive afferents, the SA1 afferents are also critically important for coding object size and shape (Hsiao, 2008). Thus activating the SA1 system is critical for a prosthesis to be successful

74

since these afferents are the equivalent of the tactile visual system and convey spatial information about the distribution of stimuli on the skin to the central nervous system (Hsiao, 1998). Information about edges, roughness, or softness are normally conveyed to the central nervous system by these afferents. Effective stimulation of this spatial system requires that a large number of afferent fibers be activated to convey an effective 2D image of the spatial distribution of the patterns on the skin. While activating enough fibers to give a sense of pressure on the skin is possible, selectively activating enough peripheral SA1 afferent fibers to convey information about 2D form and texture is not realistic given our current technology. However, it is important to note that an alternative possibility described later is to stimulate neurons directly in cortex that are already coding for 2D stimulus features. Another class of peripheral receptors are the nociceptors which are the afferents responsible for conveying information about itch or painful events that result in damage to the skin. The ability to convey pain information leaves open the potential of giving users the perception of when the prosthesis itself is in danger of being damaged. For example, if a sensor on the prosthesis “perceives” that it is about to be damaged then an electrical pulse could signal the user by activating a pain afferent. Restoring the sense of pain is not essential for a prosthetic hand to be considered a success. The last kind of input fiber encodes the perception of temperature. These afferents provide information about both warm and cold and although potentially not critical for action, the sense of temperature plays an important role if the hand is to be used to provide affective percepts. For example, the pleasure of holding someone's hand at least partially comes about because one can sense the warmth of the receiver. Sensing temperature and stimulating the thermal afferents is probably important if the prosthesis is to feel like a natural hand. As can be seen from the above discussion there are 13 aspects of hand function that need to be restored for a prosthesis to replace normal

sensory inputs. In addition, for each afferent type, a large population of afferents need to be simultaneously stimulated for the prosthesis to completely replace the normal hand. One possible method to reduce the dimensional explosion that is required to achieve this goal is to activate neurons further up the processing pathways where the information has already been segregated into different pathways and has been integrated to extract out selective features of the external world. In the next section, we review the ascending and central pathways that underlie tactile perception (Fig. 1).

Ascending pathways to perception There are numerous potential places in the parallel ascending pathways that carry the inputs from the peripheral afferents to the cortex where electrical (or optical) stimulation could be performed. These include stimulating directly in the spinal cord, in the dorsal horn, dorsal column nuclei, medial leminiscus, the ventroposterior lateral nucleus (VPL) of the thalamus. Each of these sites has potential strengths and weaknesses. The major advantage is that each of these sites are organized somatotopically, with afferents carrying information about mechanoreception and proprioception ascending in the dorsal column medial-lemiscal pathway and information concerning pain and temperature ascending in the anterolateral pathway. The segregation of function along with the fibers being anatomically organized has the advantage of allowing one to specifically target afferents related to the desired body part (hand) and sensory modality (mechanoreception, proprioception, or temperature). Furthermore, stimulating in the dorsal horn (Luo et al., 2009) or ascending spinalthalamic track may be the optimal way to artificially evoke the percepts of pain and temperature since currently the central projections for these afferent fibers has not been clearly established. There is evidence that stimulating along the dorsal column pathway may be a viable approach in restoring mechanoreceptive

75

Insula

7b

Ri

SII SIIa

SIIc

SIIp 5

SI

3a

3b

2

mechano-

proprio-

Thalamus

1

VPI

Dorsal Horn

proprio-

VPL

VPS

DCN

nocithermomechano-

mechanoproprio-

Fig. 1. Block diagram of the somatosensory pathways. DCN, dorsal column nuclei; VPI, ventroposterior inferior; VPL, ventroposterior lateral; VPS, ventroposterior superior; SI, primary somatosensory cortex; SII, second somatosensory cortex (a, anterior; c, central; p, posterior), Ri, retroinsular.

function (Gaunt et al., 2009). The major drawback in stimulating the spinal cord is that it is difficult to implant and target these structures and further, stimulating brain stem regions has the potential of creating unwanted side effects. Primary somatosensory cortex The next logical place to evoke somatosensory percepts is in primary somatosensory cortex. During the early part of the last century, Penfield

and Jasper found that stimulation of the postcentral gyrus in human patients evoked systematic patterns of sensations of the body as they moved the electrical stimulus to different locations on the postcentral gyrus. Using this technique, they uncovered a representation of the body, or homunculus, in primary somatosensory cortex (Penfield and Boldrey, 1937). Further experimentation with electrical brain stimulation in the sensory cortex and other areas has demonstrated that electrical stimulation in specific brain regions can convey a specific percept associated with those regions in a behavioral task. Romo et al. (1998, 2000) used rhythmic electrical stimulation at a number of different frequencies to replace the mechanical vibrations of the tactile probe (Romo et al., 1998, 2000). The electrical stimulation was performed in a specific region of cortex which has been shown to be sensitive to flutter discrimination frequencies (Carli et al., 1971). More recently, it has been shown that nonhuman primates can discriminate spatial and temporal patterns of direct stimulation of primary somatosensory cortex (Fitzsimmons et al., 2007). Previous attempts at stimulation have not only demonstrated that the sensations of the body are closely associated with the activity of neurons in primary somatosensory cortex but also that the evoked percepts correlate closely with the modality specificity of neurons located in specific cortical columns. These results suggest that it may be possible to give patients with upper limb prostheses the natural perception from the prosthetic limb if neurons in the cortex are properly activated. The question then arises as to what it means for the cortex to be “properly activated.” As a first step in addressing this question, it must be noted that primary somatosensory cortex is composed of four distinct areas that are called areas 3a, 3b, 1, and 2. Each of these areas has been shown to (1) have a unique cytoarchitecture, (2) have a unique set of input and output projections, and (3) respond differently to somatosensory stimulation. Furthermore, studies in nonhuman primates show that selective ablations of these areas produce unique deficits in the ability

76

of the animals to perform tactile tasks (Randolph and Semmes, 1974). These simple findings suggest that the selective activation of neurons in each of these areas should evoke selective percepts. Thus stimulation of neurons in area 3a, which is composed of neurons that respond to movement of the joints, should produce percepts related to proprioception, stimulation of neurons in areas 3b and 1, which is composed of neurons that respond to cutaneous input, should evoke percepts related to cutaneous input, and stimulation of neurons in area 2, which contain neurons that receive both cutaneous and proprioceptive input should evoke perception of three-dimensional (3D) objects. But randomly stimulating neurons in these areas is not sufficient to produce natural percepts since each of these areas are composed of columns of neurons that are body location and modality specific. It is precisely for this reason that Romo finds that he can only evoke the percept of flutter when he selectively activates neurons in area 3b that respond to RA-like input (Romo et al., 1998). The ultimate goal for producing natural percepts with a prosthesis is to reproduce as closely as possible the normal patterns of neural activity that are produced by the natural arm and hand. To rephrase this statement, the goal is to understand how somatosensory information from the hand is coded and represented in the cortex and to artificially produce those representations using artificial stimulation. Using the natural underlying neural code to restore sensory function is the approach that has successfully been used in cochlear implant patients. In the cochlea, sounds are laid out along the cochlear membrane as a tonotopic representation. In these patients, a linear array of a dozen or so electrodes are inserted along the cochlea and electrical stimuli are used to selectively activate the membrane in a manner that best simulates the natural pattern of activation during speech. Cochlear implants have been a huge success in restoring hearing to a large number of patients with peripheral hearing loss. The success of the cochlear implant shows that a similar approach

of exploiting natural neural codes should be used for patients with upper limb prosthesis to restore normal hand function. The question then arises as to what neural code(s) is used by neurons in somatosensory cortex when performing tactile tasks. As discussed earlier, the hand plays many roles in everyday life. We use our hands to perceive properties of objects such as their size, shape, and texture (smooth, rough or hard or soft) when we directly contact and explore objects with our hands. We also use inputs from our hands to explore the environment indirectly through tools that we hold in our hands. Finally, we use our hands to interact with our environment, for example, when grasping and manipulating objects. Recent studies in the Hsiao lab have shown that the neural coding mechanisms employed by touch are highly similar to the ones used by the visual system. In particular, there is now strong evidence that the orientation of a bar indented in the skin is coded by a population of orientation tuned cells in area 3b that have receptive fields consisting of oriented bands of excitation- and inhibition-like neurons in primary visual cortex (Hsiao et al., 2002). Furthermore, the representation of stimulus motion on the skin appears to be processed by populations of neurons in area 1 that respond most effectively to pattern motion rather than to component motion (Pei et al., 2010). Finally, it has been shown that neurons in area 2 and SII cortex respond to stimulus curvature in a way that is highly similar to the tuning that is observed in area V4 in the visual system (Yau et al., 2009). Together, these results suggest that form, texture, and motion are represented by neurons in primary somatosensory cortex that are highly selective to features of stimuli on the skin. These results suggest that to achieve a robust prosthesis that has true sensory feedback requires that populations of neurons in S1 cortex be selectively activated in a manner that is consistent with the natural underlying neural codes that are normally used by the somatosensory system to code for features such as motion, form, and vibration.

77

An advantage of stimulating neurons directly in primary somatosensory cortex is that it takes advantage of the existing cortical machinery that currently exists for extracting information about the environment. Thus one does not need to stimulate entire populations of neurons from each of the peripheral afferent types but instead all that is needed is to stimulate columns of neurons that code for a specific feature such as orientation to evoke the percept of the orientation of an indented bar. In the next section, we discuss results from experiments that we performed in somatosensory cortex to simulate the perception of mechanical intensity.

Using electrical stimulation to produce the percept of mechanical intensity The aim of the experiment was to train a nonhuman primate (Macaca mulatta) to discriminate the intensity of a mechanical probe indented into

the skin on the hand. In the study, the animal sat in a chair with its hand restrained and facing upward. The stimuli consisted of a small 1-mm probe, mounted on a NorMag linear motor, that could be positioned anywhere over the animal's hand using a platen–forcer system. The animal was then trained to perform two tasks. In the first, the animal performed a two alterative forced choice (2AFC) task whereby the probe was indented into the skin during one of the two intervals. The animal's task was to report using a foot switch whether the stimulus occurred during the first or the second interval. The second task was similar to the first except that the animal was given two mechanical stimuli and was required to report whether the more intense stimulus occurred in the first or second interval. The animal was trained to perform the two tasks at several locations on its hand to ensure that it could generalize across skin locations (Fig. 2). Once the animal learned to perform the two tasks, a Utah electrode array (UEA) from Base intensity = 700 mm

Mechanical threshold detection task 1 0.9 P (Comparison judged higher)

0.9

P (detected)

0.8 0.7 0.6 0.5 0.4

−50

0

50 100 150 Indent amplitude (µm)

200

0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1

0

0.2

0.4 0.6 0.8 1 Comparison indent (mm)

1.2

1.4

Fig. 2. Left graph. Detection of mechanical indentation of varying amplitudes around threshold. Each point represents 120 trials; error bars are based on four trials of 30 data points each. Right graph. Discrimination of mechanical indentations against a comparison amplitude of 700 mm (each data point is composed of 256 trials; error bars are based on eight experiments of 32 trials each). The indentation threshold was similar to earlier reported detection and neural thresholds in humans (Johansson and Vallbo, 1979a,b).

78

D5 D4 D3 D2 D1 Face

Area 2-Proprioception

Central Sulcus

Blackrock Microsystems was implanted into the hand region of the somatosensory cortex and recordings were made using the Cerebus recording system. Figure 3 shows a somatotopic map of the recording locations of the 100 electrodes in the UEA. As can be seen, the array spanned the cortical map corresponding to the face and digits 1 and 2 of the animal. Because the length of the array was limited to 1.5 mm, it was unable to record from deep structures and as such the responses were restricted to a single layer of cortex in area 1. Because of the difficulty in presenting mechanical stimuli to digit 1 or the face, we chose to concentrate the experiment on digit 2.

Stimulation details Electrical stimuli were delivered with a custom built current-regulated electrical stimulator that had the ability to stimulate different patterns simultaneously on four output channels. The stimulator could generate stepwise arbitrary current waveforms with a memory capacity of 256 steps (per channel). The output current amplitude resolution was 30 pA, with a maximum output to  0.98 mA. The time resolution of the output was 1 ms; with a minimum of 3 ms for each time-step and a maximum of 65 ms between each time-step. The stimulator was optically isolated and was driven with custom-written Matlab code.

100

X

D2m

D2m

D2d

D2d

X

D2d

X

W2

X

10

99

X

D2m

D2m

D2p

D2m

D2d

D2d

D1

X

X

9

98

W2

W2

D2p

D2d

W2

W2

X

X

X

8

97

W1

P1

W2

D1

D1d

P1

W1

X

X

X

7

96

W1

W1

W1

W1

X

X

D1d

X

X

X

6

95

Thenar

W1

Thenar

W1

W1

W1

D1d

D1

X

5

94

D1d

D1d

Thenar

Thenar

D1d

D1d

D1d

X

Thenar

4

93

Face

D1d

Face

Thenar

X

Thenar

Thenar

D1,hair y

X

3

92

Face

Face

Face

Face

Thenar

Face

D1,hair y

Thenar

Face

X

2

91

X

Face

Face

Face

Face

Face

Face

Face

Face

X

1

91

81

71

61

51

41

31

21

11

1

X

X Face X

–Indicates sites of electrical stimulation –D1, D2, D3 represent the index, middle and, ring fingers –W1, W2 represent the palmar whorls –x represents nonactive electrodes

Fig. 3. Somatotopic map of the responses evoked from the Utah electrode array.

79

Electrical substitution task

Electrical detection task 1 0.9

Response rate

All stimulation pulses were symmetric biphasic pulses with pulse widths of 100 ms and no delay between anodic and cathodic phases; each biphasic pulse lasted for a total of 200 ms. All pulse trains were contained within a 400-ms pulse window. High-frequency stimulations (200 Hz) used 80 pulses during that 400 ms window with constant interpulse intervals. We were limited to a maximum of  83 biphasic pulses based on the stimulator. Low frequency stimulation used eight pulses during the 400-ms pulse window.

0.8 0.7 0.6 0.5 0.4 0

10

20

30

40

Current amplitude (µA)

We choose four stimulation sites (electrodes) for stimulating the cortex using electrical stimuli (Fig. 3). The sites were chosen based on recordings using the mechanical stimulator that these neurons were highly rate sensitive to the intensity of the stimuli. We first determined the electrical threshold for perception. In this experiment, the monkey reported whether he perceived the electrical stimulation in the same manner as the mechanical threshold task described above (Fig. 4). The results for the electrical detection experiment are shown in Fig. 4. In this experiment, the animal received a high-frequency (200 Hz) stimulus of bipolar electrical pulses of varying amplitude. The hypothesis behind the experiment is that increasing the current amplitude should cause a systematic spread of neurons that are activated by the stimulus and that it should be easier for the animal to detect this increase in neural activity as the current amplitude increases. The systematic increase in detection with increases in current amplitude suggests that the perceived amplitude of the stimulus also increased. That is, if the stimulus was perceived as an artificial unnatural sensation, which we call an “electrical buzz”, then we would have expected a step-like detection threshold instead of the smooth psychometric function. When a lower frequency stimulation was used, we observed that detection threshold rises more

Fig. 4. Detection of a high-frequency electrical stimulus with regular interspike interval spacing.

rapidly with high frequency than with low frequency stimulation as the current amplitude increases. These data demonstrate that when giving animals a perception of stimulus intensity, a wider range of intensity values (i.e., current levels) are available when low frequencies rather than high frequencies are used as the base frequency.

Electrical/mechanical intensity discrimination As a final set of experiments to test whether the patterns of electrical stimulation evoke natural percepts of stimulus intensity, we asked the animal to perform the mechanical intensity discrimination task with some of the comparison stimuli being a range of high-frequency electrical test stimuli. We hypothesized that the psychometric functions should be similar to the mechanical–mechanical trials if the electrical stimuli produce veridical intensity percepts and that the animal should not be able to perform the task if the percepts evoked by the two kinds of stimulation are completely different. To ensure

80

P (judged higher than comparison)

200 mm indent comparison with 200 Hz stimulation 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0

0

20 40 60 80 100 120 140 Electrical comparison amplitude (µA)

160

Fig. 5. Discrimination of high-frequency electrical stimuli of varying amplitudes against a mechanical comparison indent of 200 mm.

Estimated indentation depth evoked by different currents 300 250 Indentation depth (µm)

that the animal did not learn to make this association, we interleaved the mechanical–mechanical trials with the mechanical–electrical trials. We used the threshold value of 24 mA from the high-frequency electrical detection task as a guide and tested current amplitude values from 0 to 160 mA against a comparison mechanical stimulus of 200 mm (Fig. 5). From these data, we estimate that a 200-mm mechanical indent corresponded to a current amplitude of about 55 mA. Figure 6 shows the estimated relationship between the amplitude of the stimulation current and the perceived indentation depth. The data suggests that the relationship is not linear, which is reasonable since the current spread from a point source is not linear and one would expect that proportionally more neurons would be activated as the current level increase. The results from these experiments clearly show that the intensity of a mechanical stimulus can be produced by increasing the intensity of the current used to drive the neurons. A confounding issue with these results is that Romo reported that increasing current in an RA column also produces the perceived increase in vibratory

200 150 100 50

0

10

20

30 40 50 Pulse current (µA)

60

Fig. 6. Estimated relationship between the amplitude of the electrical stimulation and the perceived probe indentation depth.

frequency in the flutter range. More studies are clearly needed to understand how tactile information is coded and represented in somatosensory cortex.

Summary In this chapter, we summarize what is the current state-of-the-art in using electrical stimuli to provide somatosensory feedback to patients with upper limb prosthesis. Evidence suggests that electrical stimuli can be delivered at several sites along the pathways leading to perception. Each site has its advantages and disadvantages; however, it is clear that what is limiting progress in this field is a fundamental lack of knowledge of how information is encoded in the somatosensory cortex and a need for more precise ways to stimulate multiple neurons in the cortex. It is not until large populations of neurons in cortex can be selectively activated using specified patterns of activity that the dream of an ideal prosthetic hand will be achieved.

81

References Acharya, S., Tenore, F., Aggarwal, V., Etienne-Cummings, R., Schieber, M. H., & Thakor, N. V. (2008). Decoding individuated finger movements using volume-constrained neuronal ensembles in the M1 hand area. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 16, 15–23. Carli, G., LaMotte, R. H., & Mountcastle, V. B. (1971). A comparison of sensory behavior and activity of postcentral cortical neurons, observed simultaneously, elicited by oscillatory mechanical stimuli delivered to the contralateral hand in monkeys. International Union of Physiological Sciences, 25, 100. Dimitriou, M., & Edin, B. B. (2008). Discharges in human muscle receptor afferents during block grasping. The Journal of Neuroscience, 28(48), 12632–12642. Edin, B. B., & Johansson, N. (1995). Skin strain patterns provide kinaesthetic information to the human central nervous system. The Journal of Physiology, 487, 243–251. Fitzsimmons, N. A., Drake, W., Hanson, T. L., Lebedev, M. A., & Nicolelis, M. A. (2007). Primate reaching cued by multichannel spatiotemporal cortical microstimulation. The Journal of Neuroscience, 27(21), 5593–5602. Gaunt, R. A., Hokanson, J. A., & Weber, D. J. (2009). Microstimulation of primary afferent neurons in the L7 dorsal root ganglia using multielectrode arrays in anesthetized cats: Thresholds and recruitment properties. Journal of Neural Engineering, 6(5), 055009. Hsiao, S. S. (1998). Similarities between touch and vision. In J. W. Morley (Ed.), Neural aspects of tactile sensation (pp. 131–165). (127th ed.). Advances in psychology. Amsterdam: Elsevier. Hsiao, S. S. (2008). Central mechanisms of tactile shape perception. Current Opinion in Neurobiology, 18, 418–424. Hsiao, S. S., & Bensmaia, S. (2008). Coding of object shape and texture. In A. I. Basbaum, A. Kaneko, G. M. Shepherd & G. Westheimer (Eds.), Somatosensation volume 6 of the handbook of the senses (pp. 55–66). (6th ed.). Oxford: Academic Press/Elsevier. Hsiao, S. S., Johnson, K. O., Twombly, I. A., & DiCarlo, J. J. (1996). Form processing and attention effects in the somatosensory system. In O. Franzén, R. S. Johansson & L. Terenius (Eds.), Somesthesis and the neurobiology of the somatosensory cortex (pp. 229–247). Basel: Birkhäuser. Hsiao, S. S., Lane, J. W., & Fitzgerald, P. (2002). Representation of orientation in the somatosensory system. Behavioural Brain Research, 135, 93–103. Johansson, R. S., & Vallbo, Å.B. (1979a). Detection of tactile stimuli. Thresholds of afferent units related to psychophysical thresholds in the human hand. Journal of Physiology, 297, 405–422. Johansson, R. S., & Vallbo, Å.B. (1979b). Tactile sensibility in the human hand: Relative and absolute densities of four

types of mechanoreceptive units in glabrous skin. The Journal of Physiology, 286, 283–300. Johnson, K. O., Hsiao, S. S., & Yoshioka, T. (2002). Neural coding and the basic law of psychophysics. The Neuroscientist, 8, 111–121. Jones, L. A. (2011). Tactile communication systems: Optimizing the display of information. In A. Green & G. Venkatasamy (Eds.), Progress in brain research: Enhancing performance for action and perception—Multisensory integration. neuroplasticity and neuroprosthetics. Oxford: Elsevier, Vol. 192. Kuiken, T. A., Marasco, P. D., Lock, B. A., Harden, R. N., & Dewald, P. A. (2007). Redirection of cutaneous sensation from the hand to the chest skin of human amputees with targeted reinnervation PNAS, 104(50), 20061–20066. Luo, W., Enomoto, H., Rice, F. L., Milbrandt, J., & Ginty, D. D. (2009). Molecular identification of rapidly adapting mechanoreceptors and their developmental dependence on ret signaling. Neuron, 64(6), 841–856. Muniak, M. A., Hsiao, S. S., Dammann, J. F., Yoshioka, T., & Bensmaia, S. (2006). The peripheral representation of vibrotactile intensity: Correlating psychophysics with neurophysiology. Society for Neuroscience Abstracts. Pei, Y. C., Hsiao, S. S., Craig, J. C., & Bensmaia, S. J. (2010). Shape invariant coding of motion direction in somatosensory cortex. PLoS Biology, 8(2), e1000305. Penfield, W., & Boldrey, E. (1937). Somatic motor and sensory representation in the cerebral cortex of man as studied by electrical stimulation. Brain, 60, 389–443. Randolph, M., & Semmes, J. (1974). Behavioral consequences of selective ablations in the postcentral gyrus of Macaca mulatta. Brain Research, 70, 55–70. Romo, R., Hernandez, A., Zainos, A., Brody, C. D., & Lemus, L. (2000). Sensing without touching: Psychophysical performance based on cortical microstimulation. Neuron, 26, 273–278. Romo, R., Hernández, A., Zainos, A., & Salinas, E. (1998). Somatosensory discrimination based on cortical microstimulation. Nature, 392, 387–390. Velliste, M., Perel, S., Spalding, M. C., Whitford, A. S., & Schwartz, A. B. (2008). Cortical control of a prosthetic arm for self-feeding. Nature, 453, 1098–1101. Yau, J. M., Pasupathy, A., Fitzgerald, P. J., Hsiao, S. S., & Connor, C. E. (2009). Analogous intermediate shape coding in vision and touch. Proceedings of the National Academy of Science United States of America, 106(38), 16457–16462. Yoshioka, T., Bensmaia, S. J., Craig, J. C., & Hsiao, S. S. (2007). Texture perception through direct and indirect touch: An analysis of perceptual space for tactile textures in two modes of exploration. Somatosensory and Motor Research, 24, 53–70.

A. M. Green, C. E. Chapman, J. F. Kalaska and F. Lepore (Eds.) Progress in Brain Research, Vol. 192 ISSN: 0079-6123 Copyright Ó 2011 Elsevier B.V. All rights reserved.

CHAPTER 6

Stimulus-driven changes in sensorimotor behavior and neuronal functional connectivity: application to brain–machine interfaces and neurorehabilitation James M. Rebesco{ and Lee E. Miller{,{,},* { {

Department of Physiology, Feinberg School of Medicine, Northwestern University, Chicago, Illinois, USA Department of Physical Medicine and Rehabilitation, Feinberg School of Medicine, Northwestern University, Chicago, Illinois, USA } Department of Biomedical Engineering, Northwestern University, Evanston, Illinois, USA

Abstract: Normal brain function requires constant adaptation as an organism interacts with the environment and learns to associate important sensory stimuli with appropriate motor actions. Neurological disorders may disrupt these learned associations, potentially requiring new functional pathways to be formed to replace the lost function. As a consequence, neural plasticity is a critical aspect of both normal brain function as well as the response to neurological injury. A brain–machine interface (BMI) represents a unique adaptive challenge to the nervous system. Efferent BMIs have been developed, which harness signals recorded from a tiny proportion of the motor cortex (M1) to effect control of an external device. There is also interest in the development of an afferent BMI that would supply information directly to the brain (e.g., the somatosensory cortex—S1) via electrical stimulation. If a bidirectional BMI that combined these interfaces were to be successful, new functional pathways would be necessary between the artificial inputs and outputs. Indeed, stimulation of S1 that is contingent upon the consequences of motor command signals recorded from M1 might form the basis for artificial Hebbian associations not unlike those driving learning in the normal brain. In this chapter, we review recent developments in both efferent and afferent BMIs, as well as experimental attempts to understand and mimic the Hebbian processes that give rise to plastic changes within the cortex. We have used a rat model to develop the computational and experimental tools necessary to describe changes in the way small networks of sensorimotor neurons interact and process information. We show that by repetitively pairing the recorded spikes of one neuron with electrical stimulation of another or by repetitively pairing electrical stimulation of two neurons, we can strengthen the inferred functional *Corresponding author. Tel.: þ1-312-503-8677; Fax: þ1-312-503-5101 E-mail: [email protected] DOI: 10.1016/B978-0-444-53355-5.00006-3

83

84

connection between the pair of neurons. We have also used the dual-stimulation protocol to enhance the ability of a trained rat to detect intracortical microstimulation behavioral cues. These results provide an important proof of concept, demonstrating the feasibility of Hebbian conditioning protocols to alter information flow in the brain. In addition to their possible application to BMI research, techniques like this may improve the efficacy of traditional rehabilitation for patients with neurologic injury. Keywords: brain–machine interface; neural rehabilitation; cortical stimulation; functional connectivity; sensorimotor cortex; Hebbian association.

Introduction

The efferent brain–machine interface

Plasticity is a crucial component of normal brain function and a critical mechanism for recovery from neurological injury. Numerous experimental studies have attempted to elucidate its underlying mechanisms under both in vitro and in vivo conditions. Short latency, associative pairing of presynaptic “trigger” spiking with stimulus-induced activity of a “target” neuron has been shown to lead to changes in the effectiveness by which the postsynaptic neuron is activated by the presynaptic neuron. We have used similar conditioning methods to demonstrate changes in the statistically inferred functional connectivity (IFC) among small groups of recorded neurons in rat sensorimotor cortex. However, unlike methods that rely on a stimulusinduced measure of synaptic efficacy, this computational approach to network connectivity reflects changes in the patterns of the cells’ discharge during spontaneous motor behaviors. In separate experiments, we have shown that the same conditioning methods can be used to cause an increase in the rat's sensitivity to intracortical stimulation used to trigger a simple sensorimotor behavior. The latency dependence and the timecourse of this effect were very similar to the corresponding parameters of the inferred connectivity changes in the first experiment. Such targeted connectivity changes may provide a tool for rerouting the flow of information through a cortical network, with potentially profound implications for both neural rehabilitation and brain–machine interface (BMI) applications.

The advent of multielectrode recording and analysis methods during the past decade has enabled researchers to expand their focus from the coding properties of single neurons to that of large ensembles of simultaneously recorded neurons. In addition to the insights into the nature of neural coding and information storage that these advances have provided, the further development of microelectrode technology, computational hardware, and analytical methods has led to the ability to interface devices directly to the brain. Within the past decade, a great amount of experimental work has been devoted to the development of BMIs that use information extracted from the brain to control the movement of an external device. The “device” might be as simple as a cursor, moving to acquire a target displayed on a computer screen (Serruya et al., 2002), or a multidegree of freedom robot commanded to move in three dimensions to grasp a piece of food (Velliste et al., 2008) or even the subject's own paralyzed limb, reanimated by functional electrical stimulation (FES; Moritz et al., 2008; Pohlmeyer et al., 2009). BMIs have already been used to restore a limited ability to interact with the external world to the small number of human patients that have received such intracortical implants (Hochberg et al., 2006; Kennedy and Bakay, 1998). Although much of this work has focused on the primary motor cortex (M1), there is also interest in using inputs from higher order motor areas.

85

One might expect fewer cortical changes in a higher order region following spinal cord injury or amputation. The information content of these areas may also be advantageous for certain applications. While M1 may encode the ongoing movement trajectory or the dynamics of movement (Moran and Schwartz, 1999; Serruya et al., 2002; Wessberg et al., 2000), premotor cortical activity is thought to represent the endpoint, or goal of the movement (Hatsopoulos et al., 2004; Santhanam et al., 2006). This information may be the result of the area's strong inputs from the prefrontal cortex (Haggard, 2008), as well as inputs from the posterior parietal cortex. This latter area, in particular, the area around the lateral intraparietal sulcus, is also a target of interest for BMI applications. Eye and arm movements are both represented there, with arm-related signals predominating in the anterior intraparietal (AIP) and medial intraparietal (MIP) areas, including the so called, parietal reach region (PRR; Snyder et al., 1997). Movement representation is more complex in PRR because many neurons are influenced by eye position as well as arm movement (Batista et al., 1999). There is also interest in moving from M1 toward the periphery for a source of control signals that is more distinctly related to the actions of particular parts of the limb or individual muscles. While such an approach is not an option for a patient with a spinal cord injury or amyotrophic lateral sclerosis (ALS), it may be appropriate for a patient with an upper limb prosthesis. Myoelectric signals have been used to control transradial and even transhumeral prostheses for 50 years (Northmore-Ball et al., 1980). Until recently, however, such devices have been limited to one or two sequentially controlled degrees of freedom because of the relatively poor access in these patients to adequate control signals. However, using a relatively new surgical procedure in which the cut proximal ends of motor nerves are transferred to regions of surgically denervated muscles, the number of independent control signals can be increased. The reinnervated muscle

serves as a biological amplifier, making it possible to record signals related to the patient's attempted movements of the missing limb. This targeted muscle reinnervation (TMR) procedure has been successfully used in several patients, providing the means to control a prosthetic arm with as many as 6 degrees of freedom in a patient with a bilateral disarticulation-level shoulder amputation (Kuiken et al., 2009; Laura et al., 2008).

The afferent brain–machine interface The efferent BMI design uses neural signals recorded from the brain to control an external device; feedback information about the state of the external device comes simply through vision. This situation closely mimics that of human patients who have lost somatosensation. These patients, even if otherwise healthy, display significant movement deficits (Ghez et al., 1995; Gordon et al., 1995), likely due to the long (150 ms) latency of visual feedback (Thorpe et al., 1996). There is also significantly increased cognitive burden associated with these patients' attempted movements. It is worth noting that virtually all of the BMIs that have been tested to date extract only kinematic control information from the brain, for which visual feedback is at least nominally useful. Vision is much less useful to judge endpoint or grip forces. As BMIs are increasingly used to control the kinetics of movement, some type of somatosensory feedback will become even more important. The goal of providing short-latency feedback to BMI users that mimics natural somatosensation has only recently been undertaken. Proof of concept was demonstrated through experiments in which the decoded limb position was used to control not only the position of a cursor, but also the monkey's own limb via a mechanical exoskeleton (Suminski et al., 2010). Consequently, after learning to relax its arm, the monkey received proprioceptive feedback of the decoded limb position. This natural proprioception combined

86

with vision allowed the monkey to make braincontrolled movements that were faster, straighter, and more successful than with vision alone. A related example of feedback for mechanical limb prosthesis users came about through an unanticipated effect of the TMR procedure described above: the reinnervation of tactile receptors in the skin. Tactile stimulation of the reinnervated area caused the sensation of pressure in some patients’ phantom hand. This sense was exploited to provide an indirect sense of grip force. Force sensors in the prosthesis were used to drive mechanical stimulation of the reinnervated skin and thereby provide feedback to the patient about the state of the prosthesis (Marasco et al., 2009; Schultz et al., 2009). For patients lacking these natural afferent pathways, some type of electrical stimulation is likely to be necessary in order to replace natural sensation. The cochlear implant, which converts sound into electrical stimuli applied via a multielectrode array to the cochlear nucleus, is by far the most successful prosthesis used to supply afferent information via electrical stimulation. Cochlear implants have been used to restore hearing to nearly 200,000 deaf persons worldwide (http://www.nidcd.nih.gov/health/ hearing/coch.asp). However, the cochlear implant is appropriate only for patients for whom the auditory nerve and more central structures are intact. Consequently, considerable effort has also been devoted to the development of brainstem and cortical auditory prostheses, unfortunately, with considerably less success (Deliano et al., 2009; Otto et al., 2005; Rousche and Normann, 1999). Applications of peripheral stimulation outside of the auditory system have been much more limited. Electrical stimulation of single afferent fibers from skin and muscle receptors in human subjects has been shown to evoke well-localized sensations of flutter-vibration and pressure (Macefield et al., 1990; Ochoa and Torebjork, 1983; Torebjork et al., 1984). Likewise, stimulation of electrodes implanted in the remaining peripheral nerve of long-term amputees causes sensations of touch, joint movement, and position

(Dhillon and Horch, 2005; Dhillon et al., 2004). Stimulation of axons signaling a single modality related to a specific body part would require as simple an encoding scheme as might be imagined. However, despite the potential advantages of peripheral stimulation in terms of coding simplicity, the loss of ascending dorsal column input to the central nervous system would preclude its use for spinal cord injured patients. Intracortical microstimulation (ICMS) of the somatosensory cortex (S1) has been proposed as a method of providing short-latency feedback to these patients (Fagg et al., 2007; Fitzsimmons et al., 2007; London et al., 2008). The relative lack of success in achieving realistic auditory perception through cortical (as opposed to more peripheral) stimulation is cause for concern, but a study done by Romo and colleagues is encouraging. Monkeys were initially trained to indicate of which two mechanical vibrations applied sequentially to the fingertips was higher in frequency (Romo et al., 1998). Subsequently, one or both of the stimuli was replaced with ICMS trains in area 3b, a tactile region of S1. Without additional training, the monkeys were able to discriminate any combination of electrical or mechanical stimuli with essentially equivalent psychometric curves and reaction times (Fig. 1a). This study makes a compelling argument that the electrical stimulation induced a perception that was similar to that of the natural stimulation. There is some concern that the vibration sense tested by Romo could be a special case, as the electrical stimulation caused phase-locking of the neurons’ discharge at the frequency of the mechanical stimulation. However, another experiment done in the visual motion processing areas, MT and MST, gave further evidence that electrical stimulation of cortex might be used to mimic the effect of natural afferent input. In that study, monkeys were trained to discriminate the direction of motion of a moving dot display. The discharge rate of these neurons is modulated by the component of motion in a particular, “preferred” direction. The electrical stimulation biased the

87 (b)

xyz

100

1

4 3b

75

1.0

Mechanical Electrical

Proportion PD

cs

% called lower

(a)

50 25

x

0

2 mm

0.8

0.8

0.6

0.6

0.4

0.4 0.2

0.2 js25a

12 16 20 24 28

es2a 0.0

0.0

Frequency (Hz)

−20

−10

0

10

−20

20

1.0

1.0

0.8

0.8

0.6

0.6

0.4

0.4

−10

0

10

20

50 y

25

% called lower

75

75 50 z

25 0

0

Proportion PD

100

100 % called lower

1.0

No stimulation Stimulation

0.2

0.2 es5b

0.0

12 16 20 24 28

12 16 20 24 28

Frequency (Hz)

Frequency (Hz)

−20

−10

0

10

Correlation (%)

js21a

0.0 20

−20

−10

0

10

20

Correlation (%)

Fig. 1. (a) Psychophysical performance elicited by microstimulation at three sites along a cortical column in area 3b. The three microelectrodes (labeled x, y, and z, relating to the corresponding subpanels) recorded units with quickly adapting cutaneous RFs on the finger tip. CS denotes central sulcus; 4, 3b, and 1 denote corresponding Brodmann areas. Subpanels x–z display psychometric curves relating the monkey's ability to distinguish between a 20-Hz base frequency stimulus and a comparison stimulus delivered 1.5–3.5 s later. Mechanical and microstimulation trials were randomly intermixed. Reproduced from Fig. 4 of Romo et al. (2000). (b) Effect of microstimulation on psychophysical performance for four stimulation sites in MT. Dots were made to move with varying amount of mutual correlation either parallel or antiparallel (indicated as a negative correlation) to the preferred direction of neurons near the electrode tip (horizontal axis). Each psychometric function shows the proportion of trials in which the monkey indicated that the dots appeared to be moving in the preferred direction (vertical axis). Stimulation biased the monkey's judgment in each of these examples, ranging from an effect equivalent to 5% correlation (upper left subpanel) through 30% (lower right). Reproduced from Fig. 4 of Salzman et al. (1992).

monkeys’ judgment of the direction of dot motion toward the preferred direction of the cluster of cells surrounding the stimulation electrode (Celebrini and Newsome, 1995; Salzman et al., 1990; Fig. 1b). It is worth noting that the motion processing characteristics of these neurons are similar in some respects to the discharge properties of neurons within proprioceptive areas 3a and 2 that respond most strongly to motion of the hand in a particular direction (Costanzo and Gardner, 1980; Prud'homme and Kalaska, 1994).

The critical role of adaptation Optimizing normal motor behavior involves repeated practice that results in improved (if not

always perfect) performance. Viewed at the most reductionist level, learning a sensorimotor skill, whether it is hitting a baseball or controlling a BMI, requires a change in the interactions between sensory and motor neurons such that a particular input generates the desired output; ideally, seeing a baseball headed to the strike zone elicits the motor pattern required to hit it solidly. Whether in baseball or BMIs, it remains something of a challenge to discover the optimal ways to enhance this process of sensorimotor learning. Nearly all efferent BMIs begin with an estimate of the optimal mapping between neuronal state and limb state obtained during normal movement. A similar approach is the likely first step in development of afferent BMIs. Inevitably, whether efferent or afferent, this estimate actually

88

represents a very limited and artificial mapping. Yet, presumably, the closer the neuron's natural role is to that given to it in the BMI, the better the control. Massive discrepancies, for instance, attempting to encode motor outputs with a somatosensory neuron, may retard or entirely forestall a patient's ability to learn control of the device. Fortunately, work with efferent BMIs has demonstrated the ability of animal subjects to adapt fairly quickly to a given map, and to improve behavioral performance. Even under conditions in which the mappings for a subset of neurons is randomly scrambled with those of other neurons, the animal's behavioral performance improves, appearing to reach a plateau after only several sessions (Ganguly and Carmena, 2009; Jarosiewicz et al., 2008; Taylor et al., 2002). Rather than simply imitating the natural coding as closely as possible, an alternate approach may be to attempt to augment the natural process of adaptation. Typically, the information carried by populations of neurons is quite redundant; the best 10–20% of neurons may provide most of the model's performance. The vast majority of additional neurons provide little additional information that is useful for the decoding (Hatsopoulos et al., 2004; Pohlmeyer et al., 2007; Wessberg et al., 2000). If it were possible to alter the behavior of uninformative cells such that they would encode information not contained in the rest of the network, perhaps performance could be improved. For instance, an array placed in the arm area of M1 may contain significant redundant information about the activation of proximal arm muscles, but little about the distal muscles of the hand. Remapping the activity of some redundant neurons to the activation of these distal muscles might create an ensemble of neurons that encodes both hand and arm without the need for additional implanted arrays. Afferent BMIs potentially have the opposite problem. Because it is currently not feasible to stimulate individual neurons, the stimulus trains delivered to sensory cortex will activate a group of neurons that may be quite heterogeneous in

their physiological properties. The most successful MT stimulation experiments described above were those in which the neurons that were encountered along the electrode track 100 mm above and below the stimulus site shared a common visual motion preference (Celebrini and Newsome, 1995). For the afferent interface, it might be effective to remap the sensory representation of the stimulated neurons so that they become more, rather than less uniform. Hebbian association, the near-coincident timing of preand postsynaptic activity may provide a tool for this kind of conditioning.

Hebbian association and cortical plasticity Consideration of the possibility of changing neuronal connectivity in vivo extends back at least to the work of Donald Hebb, who postulated that if the activity of one neuron consistently preceded that of another, the connection from the first to the second neuron would be strengthened (Hebb, 1949). The discovery that persistent changes in synaptic strength can be artificially induced, a phenomenon called long-term potentiation (LTP), initiated the study of plasticity in neural circuits. The initial experiments were done in the dentate gyrus of the rabbit (Bliss and Gardner-Medwin, 1973; Bliss and Lomo, 1973). Repeated trains of tetanizing stimulation delivered to the perforant path caused long-lasting strengthening of synaptic connections onto dentate granule cells. Change in synaptic efficacy was evaluated in terms of the size of the population spike evoked within the dentate gyrus by single shocks of the perforant path. It is believed that this potentiation of synapses resulting from tetanic stimulation is an emergent effect of a phenomenon called spike-timing dependent plasticity (STDP) first seen in recordings from monosynaptically connected pairs of pyramidal cells in cortical slices (Markram et al., 1997). These, and later experiments confirmed not only the effect that Hebb postulated, but also its dependence on the

89

precise timing of the pre- and postsynaptic activity. The strength of synaptic inputs tends to weaken if they are reliably activated after postsynaptic activity. The details of this relation were determined experimentally using multiple, wholecell perforated-patch recording and stimulation from cultures of dissociated embryonic rat hippocampal neurons (Bi and Poo, 2001). The relation can be expressed as a relatively simple, asymmetric function of the relative timing between the activity in pre- and postsynaptic sites, provided the evoked EPSPs are not significantly overlapping in time (Fig. 2). More recent studies have directly mimicked the STDP model in awake animals by using electrical stimulation to induce activity at a postsynaptic target at a short latency beyond measured or evoked presynaptic activity at another site. This type of Hebbian, or paired associative stimulation has been used to drive the remapping of sensory representations within somatosensory (Jenkins et al., 1990; Recanzone et al., 1992; Wang et al., Δt < 0

1995), auditory (Recanzone et al., 1993), barrel (Diamond et al., 1993), and visual (Yao and Dan, 2001) cortices. Of particular interest is a pair of studies done in anesthetized adult cat and humans. In the cats, appropriately timed visual stimuli were used to alter the receptive fields (RFs) of V1 neurons (Fu et al., 2002). Stimuli were presented first in the RF of the trigger neuron, then 10 ms later, in the target RF. After presentation of roughly 1000 pairs of stimuli, the RF of the target neuron shifted toward that of the trigger. When the same paired-stimulus protocol was applied to human subjects, they reported shifts in visual perception that mirrored the RF shifts in the cat experiments. Within the motor system, analogous pairedstimulation protocols have been used in human subjects to increase the size of the potentials evoked by transcranial magnetic stimulation (TMS) delivered to M1. The paradigm typically involves stimulating a peripheral nerve (e.g., the median nerve for the arm; Stefan et al., 2000), Δt > 0 10 mv 10 ms

Change in EPSC amplitude (%)

100 80 60 40 20 0 −20 −40 −60

−80

40 −40 0 Interstimulation delay (ms)

80

Fig. 2. Timing window for the induction of synaptic potentiation and depression through paired stimulation. Percentage change in the EPSC amplitude (vertical axis) as a function of the timing between pre- and postsynaptic activity induced by the stimulation (horizontal axis). The effect was measured 20–30 min after the conditioning stimulation (60 pulses at 1 Hz). Reproduced from Fig. 7 of Bi and Poo (1998).

90

or the common peroneal nerve for the leg (Mrachacz-Kersting et al., 2007). The peripheral activation is followed by a single TMS pulse timed to activate the cortex approximately synchronously with the arrival of the afferent discharge. Such repeated pairings for tens of minutes causes an increase in the motor potentials evoked by a test TMS pulse. Changing the timing slightly, so that the TMS pulse precedes the afferent volley by 5–10 ms, causes the effectiveness of the test pulse to decrease relative to baseline, rather than increase (Wolters et al., 2003). Several lines of evidence suggest that these changes are due to an LTP-like process occurring in the cortex (Stefan et al., 2002). Spike-triggered electrical stimulation has been used within M1 to change the output properties of primate M1 neurons that project to muscles in the wrist (Jackson et al., 2006). The output properties of a pair of sites were initially characterized in terms of the direction of torque about the wrist produced by brief ICMS trains. Next, spikes of a trigger cell at one site were paired with electrical stimulation of a target electrode. After approximately 2 days of such pairing, the torque response of the trigger cell rotated in the direction of that of the target cell. The interpretation of these results was that connections between the trigger and target were strengthened such that stimulation of the trigger now activated the target neuron and its output muscles (Jackson et al., 2006). The potential that specifically targeted changes of this type, or those evoked in sensory cortices, might be used to improve the performance of BMIs is an exciting prospect.

Inferred functional connectivity among small networks of neurons We have reviewed a variety of stimulus-based methods of determining the apparent connectivity between pairs of neurons or between stimulus and recording sites. While widely used, these methods have the drawback that they are inherently limited to pairwise comparisons, and they do not

necessarily reflect the behavior of the neurons under more natural conditions. Furthermore, unless it is feasible to use intracellular stimulation, the effects are necessarily due to an indeterminate set of neurons activated by the stimulating electrode. A primary objective in our work was to develop methods of quantifying the IFC among a set of extracellularly recorded neurons while avoiding the limitations of the pairwise stimulusbased methods. Ultimately, our objective was to change the strength of particular connections using paired associative stimulation, and to document these changes in terms of altered IFC. To this end, we developed a method to determine the minimal set of kernels interconnecting a set of recorded neurons that are best able to account for all of the neurons’ measured spiking statistics. Each kernel is simply a set of time-dependent coefficients that describe the timecourse of the influence of one neuron's discharge on another neuron. The full set of kernels describes the firing of one neuron as a function of the past firing history of all observed neurons. Hence, we modeled the activation of a given neuron as a weighted sum of inputs from all other observed neurons in the network, the cell's own firing history (modeled as a self-connection), and a baseline firing rate (Fig. 3a). The self-connections were used to represent intrinsic spiking characteristics such as burstiness and refractoriness. The summed activation was mapped into a firing rate, and spikes were ultimately generated by a stochastic Poisson process controlled by the firing rate. The IFC algorithm we used to find the kernels is based on a Bayesian approach that has been explored in a number of previously published studies (Okatan et al., 2005; Paninski, 2004; Pillow et al., 2008; Rigat et al., 2006; Stevenson et al., 2009; Truccolo et al., 2005). It is a pointprocess, regularized, generalized linear model that extracts the interactions within a population of neurons. In this model, each neuron is labeled by an index i and described through its instantaneous firing rate li(t|ai, Ht). Here, ai are network

91 (a)

(b)

(c)

Fig. 3. Diagram of the inferred functional connectivity (IFC) model. (a) Schematic representation of the model: each neuron's firing is driven by the firing of all observed neurons, convolved with individual temporal kernels. The convolved inputs are then summed and transformed to a firing rate. Spikes are generated stochastically with Poisson statistics from the firings rates. (b) Recorded spikes are used to fit a maximum a posteriori estimate of the temporal kernels interconnecting each pair of neurons. (c) Kernels are integrated to produce a scalar connection weight Wij that represents the net effect of neuron j on neuron i. A gray scale represents the strength of these connections.

parameters associated with neuron i, and Ht is the history of network spiking up to time t. The effect on neuron i at time t of a spike emitted by neuron j at time t0 is quantified by aij(m), where m ¼ t  t0 is the discretized time lag. This expression formalizes the model described above. The baseline firing rate of the neuron is ai0. The connections among neurons are thus described by a set of timedependent kernels {aij(m)}. The observed spikes are assumed to be generated by a stochastic Poisson process with instantaneous firing rate: ! XX aij ðmÞIj ðt  mÞ li ðtjai ; Ht Þ ¼ exp ai0 þ j

m

Here Ij(t  m) is an indicator function that takes the value of 1 if a spike is emitted by neuron j at time (t  m), and 0 otherwise. Inferred kernels for a four-neuron network, including neurons 1 and 2 from Fig. 3a, are illustrated in Fig. 3b. We implemented a maximum a posteriori (MAP)

evaluation of the kernels {aij(m)}, using the loglikelihood of the observed spiking activity li(t|ai, Ht), combined with a prior favoring sparse timedependent kernels (Pillow et al., 2008). Each of the alphas expresses the influence of neuron j on neuron i at a particular time lag. The net effect of neuron j on neuron i can be quantified by a single scalar connectivity parameter Wij obtained by summing the elements of the corresponding kernel aij overall time lags: X Wij ¼ aij ðmÞ m

The matrix W ¼ {Wij} quantifies the full network connectivity (Fig. 3c). Wij can be thought of as the net influence of one observed neuron on another. However, this measure loses some sensitivity; a connection that contains strong, but balanced excitatory and inhibitory components at different times would yield very small W. See for example, the kernel connecting neuron 2 to

92

neuron 4 in Fig. 3b. Consequently, an experimental manipulation that had a multiplicative effect on functional connectivity would not generate a corresponding change in W. However, expressing the weight as a scalar greatly simplified the analysis. We have previously tested this algorithm in simulation, using it to determine the functional connectivity among a small group of neurons chosen from among a much larger network (Rebesco et al., 2010). The R2 between the inferred connectivity and that of the actual network averaged 0.61 despite the fact that the activity of fewer than 1 in 1000 simulated neurons was actually sampled. Furthermore, the accuracy of inferred changes within the subsampled networks averaged R2 ¼ 0.81.

Altered IFC through Hebbian conditioning The initial goal of our in vivo experiments was to induce changes in functional connectivity using paired associative stimulation paradigms like those that have been shown to produce changes in the magnitude of stimulus-evoked activity (Fu et al., 2002; Jackson et al., 2006; Jacob et al., 2007). We used chronically implanted fine-wire electrode arrays to record the activity of four to nine neurons in rat forelimb sensorimotor cortex during several-day long experiments while the animal was allowed to behave freely, with no trained sensorimotor task. All of the animalrelated procedures were approved by the Northwestern University Animal Care and Use Committee. Data acquisition and stimulation control was done using a 16-channel recording and stimulating system (Tucker-Davis Technologies, Alachua FL and Triangle BioSystems, Inc., Durham, NC). Trigger and target neurons were selected from among the recorded neurons on the basis of their good recording stability and large signal-to-noise ratios. We used one of two different Hebbian paradigms in each of the experiments. In the spike-triggered stimulus paradigm, action potentials from the trigger neuron were passed through a window discriminator. At

a fixed delay (either 5 or 500 ms) after each action potential, a current pulse was sent to the target electrode (30 mA, 200 ms, biphasic pulses). In the dual-stimulation paradigm, 5-pulse stimulus trains (1000 Hz; 20 mA, 200 ms, biphasic pulses) were delivered at a rate of 5 Hz through both the trigger and target electrodes. There was a lag of either 5 or 100 ms between the trigger and target stimulation. The stimulation, which ran nearly continuously for 2–3 days in each experiment, was interrupted at intervals of 6–24 h to record from the entire network. These recordings were used to monitor the timecourse of connectivity changes. In order to establish a baseline measure of connectivity, we made recordings of the network activity 24 h before and immediately prior to the beginning of the stimulation paradigm. To determine the degree of persistence of the plastic changes, we recorded again at 12 and 24 h after conditioning stimulation had ended. The results of two of the spike-triggered stimulus experiments are summarized in Fig. 4a and b. As anticipated, there was no significant change in functional connectivity of the trigger-to-target pair during the 24 h interval prior to stimulation onset (Fig. 4a). However, with the onset of 5 ms latency stimulation there were significant changes in the network spiking statistics during the subsequent 48 h. These changes reflected a progressive increase in the strength of the IFC from trigger to target, which persisted for several hours beyond the end of the stimulation. This potentiation in IFC was dependent on the timing of postsynaptic stimulation; a separate experiment using 500 ms latency stimulation resulted in no potentiation (Fig. 4b). The marked difference in connectivity change, DW, between the 5-ms and the 500ms latency protocols is consistent with studies of STDP (Bi and Poo, 2001). As further confirmation of this difference we performed several experiments in which we used a single neuron to trigger stimulation in two target electrodes, one at a 5-ms latency, the other at 500 ms. As in the single-electrode stimulation experiments, only

93

1

15

10

5

2 3 4 0 5

0 −24 −12

(b)

0

12

24

36

48

60

1

72

2

3

4

5

Connection from (d) 6

500 ms latency

Targeted Nontargeted

15 4 10

Counts

Inferred weight change (ΔW )

16

(c)

5 ms latency

Connection to

Inferred weight change (ΔW )

(a)

5

2

0 0 12 24 36 48 60 −24 −12 0 Time from stimulation onset (h)

72

–4

0

4 8 12 Weight change (ΔW )

16

Fig. 4. Spike-triggered potentiation. Stimulation took place during the 48-h period shown by the gray shading. (a) Latency between trigger spike and target stimulation was 5 ms and led to a progressive increase in the inferred weight of the trigger-to-target connection strength, DW. The error bars on DW here and in later figures were generated from the uncertainty in model parameters. For more details, see Rebesco et al. (2010). (b) Latency between trigger spike and target stimulation was 500 ms, resulting in no change in DW. (c) Matrix of all connectivity changes DW for the stimulation experiment in (a) across the 48-h stimulation period. The trigger-to-target connection (3–4) is highlighted. (d) Histogram of weight changes.

the connection associated with the 5-ms latency was significantly strengthened following the onset of stimulation. Figure 4a and b summarize the changes in IFC for the trigger-to-target pair, but in addition, our calculation provides a measure of DW between all combinations of the five neurons that were recorded stably for the full duration of the experiment. Provided that the stimulation latency was

5 ms, connections beyond that from trigger-to-target were affected. Figure 4c summarizes all the inferred connectivity changes that occurred during the 48-h period from stimulus onset to offset, and reveals a striking example of this nontargeted potentiation. The largest increase was that of the trigger-to-target connection (from neuron three to four), but several other connections were strengthened nearly as much (Fig. 4d).

94

This occurrence of both targeted and nontargeted potentiation was an unexpected but consistent effect in all 5 ms latency experiments, whether using the spike-triggered (Fig. 5a) or dual-stimulation (Fig. 5b) paradigm. However, unlike the significant potentiation observed at 5 ms latency, stimulation at long latency (either 500 or 100 ms) induced no substantial changes in either targeted or nontargeted connections. The IFC is not intended to describe the precise anatomical connectivity among the observed neurons. Rather, it describes a system that approximates the statistics and dynamics of the observed network of neurons. We consider it to be unlikely that the detected connectivity changes were the result simply of the potentiation of monosynaptic connections. At distances corresponding to the interelectrode spacing, the probability of a monosynaptic connection can be conservatively estimated to be less than 1% (Braitenberg and Schuz, 1998). Consistent with this interpretation, we found that the IFC kernels lacked the fine, highly stereotypic timecourse of the typical EPSP, instead having a (a)

Spike-triggered stimulation (n = 8) 5 ms 500 ms

Inferred weight change (ΔW)

10

rather broad and heterogeneous timecourse that might be expected from structurally varied oligiosynaptic connections between observed neurons.

Altered behavior through Hebbian conditioning The changes in spiking statistics apparently caused by the Hebbian conditioning should also have some effect on the rat's behavior, if they genuinely represent altered functional connections within sensorimotor cortex. We tested this possibility by attempting to alter rats’ perceptual thresholds in an ICMS cue-detection task. The rats learned to recognize the ICMS as a cue to initiate a learned behavior. Importantly, we also used the cue electrode as the trigger electrode for conditioning stimulation. If the paired stimulation caused a behaviorally relevant increase in the strength of connection from trigger to target (as well as any other nontargeted increases), we should see a corresponding increase in the rat's sensitivity to the ICMS cue. (b)

Paired stimulation (n = 5) 5 ms 100 ms

12

8 5 4

0

0 Targeted

Non-targeted

Targeted

Non-targeted

Fig. 5. Summary of changes in inferred functional connectivity. (a) Spike-triggered stimulation. (b) Paired stimulation. The average weight change for targeted and nontargeted connections is shown for short latency (black) and long latency (gray) experiments in both panels.

95

The task required that the rat initiate trials in a self-paced manner by pressing a cue lever. Initially in training, this caused an auditory cue to be presented between 1 and 8 s later. Following presentation of the cue, the rat had 2 s to press a reward lever in order to receive a juice reward. After learning to perform more than 70% of trials successfully, the auditory cue was combined with an ICMS cue. During training, the intensity of the ICMS cue was fixed, consisting of a 330-ms train of 25 mA biphasic pulses, each of 200 ms duration, delivered at 300 Hz. The intensity of the auditory cue was gradually reduced over several sessions until the rat was able to perform the task using only the ICMS cue. The rats were then trained until performance reached an asymptote. On subsequent experimental sessions, the intensity of the ICMS cue was drawn randomly on each trial from the range 0 to 24 mA. The probability of success or failure as a function of stimulus current (I) was determined by accumulating the data into 4 mA bins. To allow for apparent variations in the animal's motivational level, we scaled the data for a given session by the minimum (pmin) and maximum (pmax) success rates. This normalized success probability was then fit to a sigmoid function: p max  p min 1 þ ebðII0 Þ A slope, b, and a detection threshold, I0, were determined from the fit, and a Laplace approximation was used to estimate error bounds on the fit of the threshold. During these behavioral experiments, as in the IFC experiments, the rats received nearly continuous, paired stimulation for 48–72 h, except during a 30–45 min period during behavioral task that was used to assess the rat's sensitivity to the ICMS cue. Increased sensitivity would be expressed as a decrease in the rat's perceptual threshold, the current at which the rat's normalized performance was 50%. We computed the change in perceptual threshold by calculating the difference in threshold from the beginning to the pðI Þ ¼ p min þ

end of the paired stimulation. With this convention, a reduction of threshold, equivalent to the animal becoming more sensitive to the ICMS cue, has a negative sign. Figure 6a shows the psychophysical curve computed 24 h before the onset of conditioning stimulation (day -1) for one experiment. The threshold determined from the fitted sigmoid was 13 mA, which represents the midpoint of the fitted sigmoid. With the onset of the paired-stimulation paradigm at day 0, the fitted sigmoids shifted progressively to the left, toward lower thresholds and greater sensitivity (Fig. 6b). The overall effect of the stimulation was a progressive decrease in threshold during the period of stimulation, followed by a rebound effect in the 24 hours after stimulation ended (Fig. 6c). There was a similar reduction of the perceptual threshold in nearly all experiments in which the animal received short latency, paired stimulation (Fig. 7a, black bars). The mean change in threshold across all experiments was significant ( 4.9  1.9 mA; p < 0.01, n ¼ 8, Wilcoxson signed rank test). The rebound from short-latency stimulation (3.3  2.7 mA) was larger than the change that occurred in the absence of stimulation (1.3  2.9 mA), but the difference was not significant (p ¼ 0.17, n ¼ 6, Wilcoxson rank-sum test). Importantly, the perceptual threshold did not change during the long latency stimulation (Fig. 7a; gray bars;  1.5  1.1 mA; p ¼ 0.25, n ¼ 4, Wilcoxson signed rank test). The difference between short and long latency stimulation was significant (p < 0.01, n ¼ 4, Wilcoxson rank-sum test). In order to examine the timecourses of the different stimulation effects with greater precision, we averaged the psychometric curves across sessions and animals (Fig. 7b). The change in threshold for the short-latency experiments was nearly linear, suggesting that 72 h may not have been sufficient to reach full effect. The average rate of change during the rebound from 5 ms stimulation was approximately twice that during the stimulation period itself. This may suggest a

96 (a) 1.0

(b) 1.0

Day -1

(c)

Day 0

5 ms latency

20

Day 1

0.6

0.2

Threshold (μA)

Normalized fraction correct

Fraction correct

Day 2

0.5

10

0.0 0

5

10 15 20 Current (mA)

25

15

5 0

5

10 15 20 Current (mA)

25

−1

0

1 Day

2

3

Fig. 6. Effect of paired stimulation on perceptual threshold. (a) Sigmoid fit to the data from one experimental session, 1 day prior to onset of conditioning stimulation. Threshold was approximately 13 mA. (b) Psychometric curves for the 3 days of stimulation. Hebbian conditioning started immediately after the day 0 data were taken, and ended immediately prior to the day 2 session. (c) Change in perceptual threshold from Hebbian conditioning. The conditioning period is shown by the gray rectangle.

more rapid timecourse for the return to baseline, which would be consistent with many other behavioral adaptation studies (Smith et al., 2006). Although the dependence of perceptual threshold changes on stimulation latency was significant, there was considerable noise in the individual measurements. While some of this noise may have been due to variation in the efficacy of the paired-stimulation protocol, much of it was also probably due to variation in the level of the animal's motivation and attention, some of which could have been due to uncontrolled environmental factors. We ran several experiments with pairs of animals cued by ICMS, only one of which received the conditioning stimulation. The second rat was used as a control for any nonspecific environmental effects that might have systematically altered both animals’ perceptual thresholds or rate of learning. We computed a mean threshold for each stimulus condition for each rat, and the daily variations or residuals from this mean. Presumably, external factors affecting the experimental rat's motivation or learning would have also

affected the paired animal, yielding a positive correlation in the residuals. There was, however, no such correlation (r ¼  0.15, p ¼ 0.41), suggesting that there was little or no nonspecific effect of the environment on the rats’ performance.

Potential applications to neural rehabilitation In addition to the potential use in BMI applications, the cortical plasticity evoked by Hebbian association methods may also have profound impact on recovery from neurological injury. It is well known that severe neurologic injury can cause massive functional reorganization of the brain following stroke (Nudo and Milliken, 1996) spinal cord or peripheral nerve injury (Sanes et al., 1988; Topka et al., 1991), or in patients with amputated limbs (Aglioti et al., 1994; Ramachandran et al., 1992). Following stroke, a series of genetic pathways are activated that lead to increased cortical plasticity in and around the injured area (Murphy and Corbett,

97 (a)

15

Change in threshold (μA)

10 5 0 −5 −10

5 ms stim

post 5 ms stim

100 ms stim

no stim

−15

Change in threshold (μA)

(b)

5 ms

100 ms

post 5 ms

no stim

8

0

–8 0

1

2

3

0 1 2 3 0 1 2 3 0 Time course of experiment (days)

1

2

3

Fig. 7. Summary of changes in perceptual threshold across all experiments. (a) Changes in threshold resulting from short and long latency-paired stimulation (black and gray bars, respectively) as well as rebound in threshold following the end of short-latency stimulation (hashed bars). White bars show the changes occurring during control periods when there had been no stimulation in the preceding 3 days. (b) Average timecourse of changes in threshold across all experiments for each of the defined experimental periods.

2009; Nudo, 2006). Despite this, only about half of the stroke patients suffering hemiplegia affecting an arm are able to regain functional use of the limb. Fewer than one-fifth of the patients make an essentially complete recovery (Kwakkel et al., 2003). Recovery is critically dependent on the patient's active attempts to use the affected limb (Nudo et al., 1996). Recovery of upper limb function tends to be more limited than that of the lower limb. This may occur, in part, because patients quickly learn to reach and grasp using only the unaffected limb, while standing and walking requires the use of both limbs (Feys et al., 1998). In addition to the adoption of alternate motor strategies, the functional recovery that does occur is associated with plastic changes in surrounding cortical structures and the recruitment of new areas within both ipsi- and contralesional cortex (Plow et al., 2009).

Targeted application of paired associative stimulation might be used to enhance and direct the formation of new pathways to redirect afferent input to unaffected motor areas or to provide alternate output pathways from higher order motor areas. FES has been used to restore grasp function in patients with spinal cord injury for over 30 years (Merletti et al., 1975; Peckham et al., 1980). The paralyzed muscles are made to contract by the application of electrical current, typically controlled by use of a switch or single degree of freedom proportional control. FES has also been used to improve grasping behavior in stroke patients, although in this application, it has been used for its therapeutic effect. A number of studies have examined the use of FES in combination with standard therapy (Cauraugh et al., 2000; Francisco et al., 1998; Popovic et al., 2002; Thrasher et al., 2008). The stimulation was either

98

initiated manually by the patient or therapist, or triggered by EMG recordings, and used to allow the patient to make a greater range of voluntary movement. Following treatment periods ranging from 2 to 16 weeks, most patients showed greater improvement with the combined therapy than with traditional methods alone. Improvements were typically greatest in acute stroke patients but also present in some patients at periods greater than 1 year beyond the initial injury. Some of the effect may be due simply to the patients’ improved motivation and ability to perform the rehabilitation therapy. However, it is also thought that the increased sensory input from the limb, including that evoked directly by the electrical stimulation, may be of benefit. Plastic changes may occur within the cortex as a result of the Hebbian association between this increased afferent input and the patient's volitional efforts (Cauraugh et al., 2000; Popovic et al., 2002). It is also possible that antidromic activation of motor axons could lead to similar changes at the spinal level between descending corticospinal axons and motor neurons (Rushton, 2003). In addition to the abundant animal literature showing the effects of paired associative stimulation, there is evidence that cortical stimulation can improve outcomes in animal models of stroke (Adkins-Muir and Jones, 2003; Kimberley et al., 2004; Kleim et al., 2003; Plautz et al., 2003) and there have been several encouraging small-scale human studies of cortical stimulation following stroke. These included five studies of repetitive transcranial magnetic stimulation (rTMS) and two that used transcranial direct current stimulation (tDCS; Harvey and Nudo, 2007). This evidence was encouraging enough to lead to the Everest clinical trial, a large scale, phase III randomized trial of 146 participants with hemiplegia at least 4 months after ischemic stroke, from 21 different centers (Harvey and Nudo, 2007). The trial tested a subdural stimulation system manufactured by Northstar Neuroscience, Inc. (Seattle, WA). Unfortunately, the study showed no advantage for the patients who underwent

the combination of cortical stimulation and rehabilitation compared to those in the rehabilitation alone group (Plow et al., 2009). Among the factors that may have diminished the odds of a successful outcome is the grid placement, which was determined solely on the basis of the representation of hand and finger movements determined by fMRI. Intraoperative mapping and attention to more proximal limb representation may have led to better outcomes. Variation in grid placement, as well as in the extent of remaining viable output pathways caused only 16% of patients to have movements evoked intraoperatively, a much smaller fraction than in either the Phase I or II studies. Among this subset of patients, the effect of the combined stimulation and rehabilitation was significant (Plow et al., 2009). Other factors related to the timing of stimulation, both in terms of its relation to the attempted movements, as well as to fluctuations in overall cortical excitability and the patient's level of motivation may also have been important.

Summary and conclusions We have summarized a wide range of literature that potentially impacts the use of paired associative stimulation methods to develop more effective BMI applications, as well as to augment standard rehabilitation approaches to the treatment of stroke and other neurological disorders. A number of different methods have now been used both to induce and to infer changes in the apparent strength of synaptic connections. These methods have in common, the dependence on a short latency between pre- and postsynaptic evoked activity. Application of these approaches to the development of new sensorimotor associations, or to the modulation of the strength of existing pathways, is likely to represent an exciting new component of BMI research. Likewise, new therapeutic approaches are possible, that would couple precisely timed and

99

targeted stimulation of particular cortical areas to the patient's attempted movements during rehabilitation. Our demonstration of changes in functional connectivity inferred from patterns of spike discharge that parallel in several respects, changes in the rat's learned sensorimotor behavior are quite encouraging. It is apparent that we must better understand the mechanisms leading to the nontargeted changes in IFC that accompanied the targeted changes. We also need to determine the factors that will allow these effects to be extended beyond the current 24 h. It is possible that the period of heightened plasticity following stroke will naturally lead to greater efficacy. There is also evidence that intermittent, paired stimulation over the course of an entire month leads to greater persistence than the effects we have described here following 3–4 days of continuous stimulation (Racine et al., 1995). We anticipate that as these methods are refined, targeted, paired associative stimulation will become a valuable scientific and clinical tool.

Acknowledgments The authors gratefully acknowledge the contributions of Matt Bauman, and Rebecca Friesen, who assisted with the care and training of the animal subjects. Drs. Sandro Mussa-Ivaldi, Konrad Kording, and Yang Dan made important contributions to the experimental design, data analysis, and interpretation. This work was supported by a grant from NINDS (R01 NS048845), and an NIH/NINDS fellowship (F31NS062552).

References Adkins-Muir, D. L., & Jones, T. A. (2003). Cortical electrical stimulation combined with rehabilitative training: Enhanced functional recovery and dendritic plasticity following focal cortical ischemia in rats. Neurological Research, 25, 780–788.

Aglioti, S., Bonazzi, A., & Cortese, F. (1994). Phantom lower limb as a perceptual marker of neural plasticity in the mature human brain. Proceedings of the Royal Society B: Biological Sciences, 255, 273–278. Batista, A. P., Buneo, C. A., Snyder, L. H., & Andersen, R. A. (1999). Reach plans in eye-centered coordinates. Science (New York, NY), 285, 257–260. Bi, G. Q., & Poo, M. M. (1998). Synaptic modifications in cultured hippocampal neurons: Dependence on spike timing, synaptic strength, and postsynaptic cell type. The Journal of Neuroscience, 18, 10464–10472. Bi, G., & Poo, M. (2001). Synaptic modification by correlated activity: Hebb's postulate revisited. Annual Review of Neuroscience, 24, 139–166. Bliss, T. V., & Gardner-Medwin, A. R. (1973). Long-lasting potentiation of synaptic transmission in the dentate area of the unanaesthetized rabbit following stimulation of the perforant path. The Journal of Physiology, 232, 357–374. Bliss, T. V., & Lomo, T. (1973). Long-lasting potentiation of synaptic transmission in the dentate area of the anaesthetized rabbit following stimulation of the perforant path. The Journal of Physiology, 232, 331–356. Braitenberg, V., & Schuz, A. (1998). Cortex: Statistics and geometry of neuronal connectivty. Berlin: Springer. Cauraugh, J., Light, K., Kim, S., Thigpen, M., & Behrman, A. (2000). Chronic motor dysfunction after stroke: Recovering wrist and finger extension by electromyography-triggered neuromuscular stimulation. Stroke, 31, 1360–1364. Celebrini, S., & Newsome, W. (1995). Microstimulation of extrastriate area MST influences preformance on a direction discrimination task. Journal of Neurophysiology, 73, 437–448. Costanzo, R. M., & Gardner, E. P. (1980). A quantitative analysis of responses of direction-sensitive neurons in somatosensory cortex of awake monkeys. Journal of Neurophysiology, 43, 1319–1341. Deliano, M., Scheich, H., & Ohl, F. W. (2009). Auditory cortical activity after intracortical microstimulation and its role for sensory processing and learning. The Journal of Neuroscience, 29, 15898–15909. Dhillon, G. S., & Horch, K. W. (2005). Direct neural sensory feedback and control of a prosthetic arm. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 13, 468–472. Dhillon, G. S., Lawrence, S. M., Hutchinson, D. T., & Horch, K. W. (2004). Residual function in peripheral nerve stumps of amputees: Implications for neural control of artificial limbs. Journal of Hand Surgery. American Volume, 29, 605–615, discussion 616–618. Diamond, M. E., Armstrong-James, M., & Ebner, F. F. (1993). Experience-dependent plasticity in adult rat barrel cortex. Proceedings of the National Academy of Sciences of the United States of America, 90, 2082–2086.

100 Fagg, A. H., Hatsopoulos, N. G., de Lafuente, V., Moxon, K. A., Nemati, S., Rebesco, J. M., et al. (2007). Biomimetic brain machine interfaces for the control of movement. The Journal of Neuroscience, 27, 11842–11846. Feys, H. M., De Weerdt, W. J., Selz, B. E., Cox Steck, G. A., Spichiger, R., Vereeck, L. E., et al. (1998). Effect of a therapeutic intervention for the hemiplegic upper limb in the acute phase after stroke: A single-blind, randomized, controlled multicenter trial. Stroke, 29, 785–792. Fitzsimmons, N. A., Drake, W., Hanson, T. L., Lebedev, M. A., & Nicolelis, M. A. (2007). Primate reaching cued by multichannel spatiotemporal cortical microstimulation. The Journal of Neuroscience, 27, 5593–5602. Francisco, G., Chae, J., Chawla, H., Kirshblum, S., Zorowitz, R., Lewis, G., et al. (1998). Electromyogramtriggered neuromuscular stimulation for improving the arm function of acute stroke survivors: A randomized pilot study. Archives of Physical Medicine and Rehabilitation, 79, 570–575. Fu, Y. X., Djupsund, K., Gao, H., Hayden, B., Shen, K., & Dan, Y. (2002). Temporal specificity in the cortical plasticity of visual space representation. Science (New York, NY), 296, 1999–2003. Ganguly, K., & Carmena, J. M. (2009). Emergence of a stable cortical map for neuroprosthetic control. PLoS Biology, 7, e1000153. Ghez, C., Gordon, J., & Ghilardi, M. F. (1995). Impairments of reaching movements in patients without proprioception. II. Effects of visual information on accuracy. Journal of Neurophysiology, 73, 361–372. Gordon, J., Ghilardi, M. F., & Ghez, C. (1995). Impairments of reaching movements in patients without proprioception. I. Spatial errors. Journal of Neurophysiology, 73, 347–360. Haggard, P. (2008). Human volition: Towards a neuroscience of will. Nature Reviews, 9, 934–946. Harvey, R. L., & Nudo, R. J. (2007). Cortical brain stimulation: A potential therapeutic agent for upper limb motor recovery following stroke. Topics in Stroke Rehabilitation, 14, 54–67. Hatsopoulos, N., Joshi, J., & O'Leary, J. G. (2004). Decoding continuous and discrete motor behaviors using motor and premotor cortical ensembles. Journal of Neurophysiology, 92, 1165–1174. Hebb, D. (1949). The organization of behavior. New York: Wiley. Hochberg, L. R., Serruya, M. D., Friehs, G. M., Mukand, J. A., Saleh, M., Caplan, A. H., et al. (2006). Neuronal ensemble control of prosthetic devices by a human with tetraplegia. Nature, 442, 164–171. Jackson, A., Mavoori, J., & Fetz, E. E. (2006). Long-term motor cortex plasticity induced by an electronic neural implant. Nature, 444, 56–60.

Jacob, V., Brasier, D. J., Erchova, I., Feldman, D., & Shulz, D. E. (2007). Spike timing-dependent synaptic depression in the in vivo barrel cortex of the rat. The Journal of Neuroscience, 27, 1271–1284. Jarosiewicz, B., Chase, S. M., Fraser, G. W., Velliste, M., Kass, R. E., & Schwartz, A. B. (2008). Functional network reorganization during learning in a brain–computer interface paradigm. Proceedings of the National Academy of Sciences of the United States of America, 105, 19486–19491. Jenkins, W. M., Merzenich, M. M., Ochs, M. T., Allard, T., & Guic-Robles, E. (1990). Functional reorganization of primary somatosensory cortex in adult owl monkeys after behaviorally controlled tactile stimulation. Journal of Neurophysiology, 63, 82–104. Kennedy, P. R., & Bakay, R. A. (1998). Restoration of neural output from a paralyzed patient by a direct brain connection. Neuroreport, 9, 1707–1711. Kimberley, T. J., Lewis, S. M., Auerbach, E. J., Dorsey, L. L., Lojovich, J. M., & Carey, J. R. (2004). Electrical stimulation driving functional improvements and cortical changes in subjects with stroke. Experimental Brain Research. Experimentelle Hirnforschung, 154, 450–460. Kleim, J. A., Bruneau, R., VandenBerg, P., MacDonald, E., Mulrooney, R., & Pocock, D. (2003). Motor cortex stimulation enhances motor recovery and reduces peri-infarct dysfunction following ischemic insult. Neurological Research, 25, 789–793. Kuiken, T. A., Li, G., Lock, B. A., Lipschutz, R. D., Miller, L. A., Stubblefield, K. A., et al. (2009). Targeted muscle reinnervation for real-time myoelectric control of multifunction artificial arms. JAMA, 301, 619–628. Kwakkel, G., Kollen, B. J., van der Grond, J., & Prevo, A. J. (2003). Probability of regaining dexterity in the flaccid upper limb: Impact of severity of paresis and time since onset in acute stroke. Stroke, 34, 2181–2186. Laura, A. M., Robert, D. L., Kathy, A. S., Blair, A. L., He, H., Williams, T. W., et al. (2008). Control of a six degree of freedom prosthetic arm after targeted muscle reinnervation surgery. Archives of Physical Medicine and Rehabilitation, 89, 2057–2065. London, B. M., Jordan, L. R., Jackson, C. R., & Miller, L. E. (2008). Electrical stimulation of the proprioceptive cortex (area 3a) used to instruct a behaving monkey. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 16, 32–36. Macefield, G., Gandevia, S. C., & Burke, D. (1990). Perceptual responses to microstimulation of single afferents innervating joints, muscles and skin of the human hand. The Journal of Physiology, 429, 113–129. Marasco, P. D., Schultz, A. E., & Kuiken, T. A. (2009). Sensory capacity of reinnervated skin after redirection of amputated upper limb nerves to the chest. Brain, 132, 1441–1448.

101 Markram, H., Lubke, J., Frotscher, M., & Sakmann, B. (1997). Regulation of synaptic efficacy by coincidence of postsynaptic APs and EPSPs. Science (New York, NY), 275, 213–215. Merletti, R., Acimovic, R., Grobelnik, S., & Cvilak, G. (1975). Electrophysiological orthosis for the upper extremity in hemiplegia: Feasibility study. Archives of Physical Medicine and Rehabilitation, 56, 507–513. Moran, D. W., & Schwartz, A. B. (1999). Motor cortical representation of speed and direction during reaching. Journal of Neurophysiology, 82, 2676–2692. Moritz, C. T., Perlmutter, S. I., & Fetz, E. E. (2008). Direct control of paralysed muscles by cortical neurons. Nature, 456, 639–642. Mrachacz-Kersting, N., Fong, M., Murphy, B. A., & Sinkjaer, T. (2007). Changes in excitability of the cortical projections to the human tibialis anterior after paired associative stimulation. Journal of Neurophysiology, 97, 1951–1958. Murphy, T. H., & Corbett, D. (2009). Plasticity during stroke recovery: From synapse to behaviour. Nature Reviews, 10, 861–872. Northmore-Ball, M. D., Heger, H., & Hunter, G. A. (1980). The below-elbow myo-electric prosthesis. A comparison of the Otto Bock myo-electric prosthesis with the hook and functional hand. Journal of Bone and Joint Surgery. British Volume, 62, 363–367. Nudo, R. J. (2006). Mechanisms for recovery of motor function following cortical damage. Current Opinion in Neurobiology, 16, 638–644. Nudo, R. J., & Milliken, G. W. (1996). Reorganization of movement representations in primary motor cortex following focal ischemic infarcts in adult squirrel monkeys. Journal of Neurophysiology, 75, 2144–2149. Nudo, R. J., Wise, B. M., SiFuentes, F., & Milliken, G. W. (1996). Neural substrates for the effects of rehabilitative training on motor recovery after ischemic infarct. Science (New York, NY), 272, 1791–1794 (see comments). Ochoa, J., & Torebjork, E. (1983). Sensations evoked by intraneural microstimulation of single mechanoreceptor units innervating the human hand. The Journal of Physiology, 342, 633–654. Okatan, M., Wilson, M. A., & Brown, E. N. (2005). Analyzing functional connectivity using a network likelihood model of ensemble neural spiking activity. Neural Computation, 17, 1927–1961. Otto, K. J., Rousche, P. J., & Kipke, D. R. (2005). Cortical microstimulation in auditory cortex of rat elicits best-frequency dependent behaviors. Journal of Neural Engineering, 2, 42–51. Paninski, L. (2004). Maximum likelihood estimation of cascade point-process neural encoding models. Network: Computation in Neural Systems, 15, 243–262.

Peckham, P. H., Marsolais, E. B., & Mortimer, J. T. (1980). Restoration of key grip and release in the C6 tetraplegic patient through functional electrical stimulation. Journal of Hand Surgery. American Volume, 5, 462–469. Pillow, J. W., Shlens, J., Paninski, L., Sher, A., Litke, A. M., Chichilnisky, E. J., et al. (2008). Spatio-temporal correlations and visual signalling in a complete neuronal population. Nature, 454, 995–999. Plautz, E. J., Barbay, S., Frost, S. B., Friel, K. M., Dancause, N., Zoubina, E. V., et al. (2003). Post-infarct cortical plasticity and behavioral recovery using concurrent cortical stimulation and rehabilitative training: A feasibility study in primates. Neurological Research, 25, 801–810. Plow, E. B., Carey, J. R., Nudo, R. J., & Pascual-Leone, A. (2009). Invasive cortical stimulation to promote recovery of function after stroke: A critical appraisal. Stroke, 40, 1926–1931. Pohlmeyer, E. A., Oby, E. R., Perreault, E. J., Solla, S. A., Kilgore, K. L., Kirsch, R. F., et al. (2009). Toward the restoration of hand use to a paralyzed monkey: Brain-controlled functional electrical stimulation of forearm muscles. PLoS ONE, 4, e5924. Pohlmeyer, E. A., Solla, S. A., Perreault, E. J., & Miller, L. E. (2007). Prediction of upper limb muscle activity from motor cortical discharge during reaching. Journal of Neural Engineering, 4, 369–379. Popovic, M. B., Popovic, D. B., Sinkjaer, T., Stefanovic, A., & Schwirtlich, L. (2002). Restitution of reaching and grasping promoted by functional electrical therapy. Artificial Organs, 26, 271–275. Prud'homme, M. J. L., & Kalaska, J. F. (1994). Proprioceptive activity in primate primary somatosensory cortex during active arm reaching movements. Journal of Neurophysiology, 72, 2280–2301. Racine, R. J., Chapman, C. A., Trepel, C., Teskey, G. C., & Milgram, N. W. (1995). Post-activation potentiation in the neocortex. IV. Multiple sessions required for induction of long-term potentiation in the chronic preparation. Brain Research, 702, 87–93. Ramachandran, V. S., Rogers-Ramachandran, D., & Stewart, M. (1992). Perceptual correlates of massive cortical reorganization. Science (New York, N.Y), 258, 1159–1160. Rebesco, J. M., Stevenson, I. H., Koerding, K., Solla, S. A., & Miller, L. E. (2010). Rewiring neural interactions by microstimulation. Frontiers in Systems Neuroscience, 4, 39. Recanzone, G., Jenkins, W., Hradek, G., & Merzenich, M. (1992). Progressive improvement in disciminative abilites in adult owl monkeys performing a tactile frequency discrimination task. Journal of Neurophysiology, 67, 1015–1030. Recanzone, G. H., Schreiner, C. E., & Merzenich, M. M. (1993). Plasticity in the frequency representation of primary

102 auditory cortex following discrimination training in adult owl monkeys. The Journal of Neuroscience, 13, 87–103. Rigat, F., de Gunst, M., & van Pelt, J. (2006). Bayesian modeling and analysis of spatio-temporal neuronal networks. Bayesian Analysis, 1, 733–764. Romo, R., Hernandez, A., Zainos, A., Brody, C. D., & Lemus, L. (2000). Sensing without touching: Psychophysical performance based on cortical microstimulation. Neuron, 26, 273–278. Romo, R., Hernandez, A., Zainos, A., & Salinas, E. (1998). Somatosensory discrimination based on cortical microstimulation. Nature, 392, 387–390. Rousche, P. J., & Normann, R. A. (1999). Chronic intracortical microstimulation (ICMS) of cat sensory cortex using the Utah Intracortical Electrode Array. IEEE Transactions on Rehabilitation Engineering, 7, 56–68. Rushton, D. N. (2003). Functional electrical stimulation and rehabilitation–An hypothesis. Medical Engineering and Physics, 25, 75–78. Salzman, C. D., Britten, K. H., & Newsome, W. T. (1990). Cortical microstimulation influences perceptual judgements of motion direction. Nature, 346, 174–177. Salzman, C. D., Murasugi, C. M., Britten, K. H., & Newsome, W. T. (1992). Microstimulation in visual area MT: Effects on direction discrimination performance. The Journal of Neuroscience, 12, 2331–2355. Sanes, J. N., Suner, S., Lando, J. F., & Donoghue, J. P. (1988). Rapid reorganization of adult rat motor cortex somatic representation patterns after motor nerve injury. Proceedings of the National Academy of Sciences of the United States of America, 85, 2003–2007. Santhanam, G., Ryu, S. I., Yu, B. M., Afshar, A., & Shenoy, K. V. (2006). A high-performance brain–computer interface. Nature, 442, 195–198. Schultz, A. E., Marasco, P. D., & Kuiken, T. A. (2009). Vibrotactile detection thresholds for chest skin of amputees following targeted reinnervation surgery. Brain Research, 1251, 121–129. Serruya, M. D., Hatsopoulos, N. G., Paninski, L., Fellows, M. R., & Donoghue, J. P. (2002). Instant neural control of a movement signal. Nature, 416, 141–142. Smith, M. A., Ghazizadeh, A., & Shadmehr, R. (2006). Interacting adaptive processes with different timescales underlie short-term motor learning. PLoS Biology, 4, e179. Snyder, L. H., Batista, A. P., & Andersen, R. A. (1997). Coding of intention in the posterior parietal cortex. Nature, 386, 167–170. Stefan, K., Kunesch, E., Benecke, R., Cohen, L. G., & Classen, J. (2002). Mechanisms of enhancement of human motor cortex excitability induced by interventional paired associative stimulation. The Journal of Physiology, 543, 699–708.

Stefan, K., Kunesch, E., Cohen, L. G., Benecke, R., & Classen, J. (2000). Induction of plasticity in the human motor cortex by paired associative stimulation. Brain, 123 (Pt 3), 572–584. Stevenson, I. H., Rebesco, J. M., Hatsopoulos, N. G., Haga, Z., Miller, L. E., & Kording, K. P. (2009). Bayesian inference of functional connectivity and network structure from spikes. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 17, 203–213. Suminski, A. J., Tkach, D. C., Fagg, A. H., & Hatsopoulos, N. G. (2010). Incorporating feedback from multiple sensory modalities enhances brain-machine interface control. J Neurosci. 30, 16777–16787. Taylor, D. M., Tillery, S. I., & Schwartz, A. B. (2002). Direct cortical control of 3D neuroprosthetic devices. Science (New York, NY), 296, 1829–1832. Thorpe, S., Fize, D., & Marlot, C. (1996). Speed of processing in the human visual system. Nature, 381, 520–522. Thrasher, T. A., Zivanovic, V., McIlroy, W., & Popovic, M. R. (2008). Rehabilitation of reaching and grasping function in severe hemiplegic patients using functional electrical stimulation therapy. Neurorehabilitation and Neural Repair, 22, 706–714. Topka, H., Cohen, L. G., Cole, R. A., & Hallett, M. (1991). Reorganization of corticospinal pathways following spinal cord injury. Neurology, 41, 1276–1283. Torebjork, H. E., Schady, W., & Ochoa, J. (1984). Sensory correlates of somatic afferent fibre activation. Human Neurobiology, 3, 15–20. Truccolo, W., Eden, U. T., Fellows, M. R., Donoghue, J. P., & Brown, E. N. (2005). A point process framework for relating neural spiking activity to spiking history, neural ensemble, and extrinsic covariate effects. Journal of Neurophysiology, 93, 1074–1089. Velliste, M., Perel, S., Spalding, M. C., Whitford, A. S., & Schwartz, A. B. (2008). Cortical control of a prosthetic arm for self-feeding. Nature, 453, 1098–1101. Wang, X., Merzenich, M., Sameshima, K., & Jenkins, W. (1995). Remodelling hand representation in adult cortex determined by timing of tactile stimulation. Nature, 378, 71–76. Wessberg, J., Stambaugh, C. R., Kralik, J. D., Beck, P. D., Laubach, M., Chapin, J. K., et al. (2000). Real-time prediction of hand trajectory by ensembles of cortical neurons in primates. Nature, 408, 361–365. Wolters, A., Sandbrink, F., Schlottmann, A., Kunesch, E., Stefan, K., Cohen, L. G., et al. (2003). A temporally asymmetric Hebbian rule governing plasticity in the human motor cortex. Journal of Neurophysiology, 89, 2339–2345. Yao, H., & Dan, Y. (2001). Stimulus timing-dependent plasticity in cortical processing of orientation. Neuron, 32, 315–323.

A. M. Green, C. E. Chapman, J. F. Kalaska and F. Lepore (Eds.) Progress in Brain Research, Vol. 192 ISSN: 0079-6123 Copyright Ó 2011 Elsevier B.V. All rights reserved.

CHAPTER 7

Inference from populations: going beyond models Steven M. Chase and Andrew B. Schwartz* Department of Neurobiology and the Center for the Neural Basis of Cognition, University of Pittsburgh, Pittsburgh, Pennsylvania, USA

Abstract: How are abstract signals, like intent, represented in neural populations? By creating a direct link between neural activity and behavior, brain–computer interfaces (BCIs) can help answer this question. Early instantiations of these devices sought mainly to mimic arm movements: by building models of arm tuning for the neurons, desired arm movements could be read out and used to control various prosthetic devices. However, as the functionality of these devices increases, a more general approach that relies less on endogenous control signals may be required. Here we review some of the current, model-based approaches for finding volitional control signals for spiking-based BCIs, and present some new approaches for finding control signals without resorting to parametric models of neural activity. Keywords: brain–machine interface; motor cortex; decoding; cosine tuning.

et al., 1999; Velliste et al., 2008; Wessberg et al., 2000), or even a muscle stimulator (Moritz et al., 2008; Pohlmeyer et al., 2009), that could help paralyzed patients regain the ability to move. In each case, the goal of these devices is the same: translate the intent of the user, encoded in trains of action potentials, into the desired action of the device. At first blush, this seems quite simple. If the user wants the computer cursor to move upward, he simply imagines the cursor moving upward: somewhere in the brain, this process of imagination causes changes in the firing rates of neurons, and the BCI needs only to decode these changes into upward cursor movement. The reality,

Spiking-based brain–computer interfaces (BCIs) map the activity of dozens to hundreds of recorded neurons to the control of some device. This device could be something fairly simple, like a spelling tool that can help locked-in patients communicate with the outside world (Musallam et al., 2004; Santhanam et al., 2006), or it could be something more complex, like a cursor on a computer screen (Hochberg et al., 2006; Kennedy et al., 2000; Mulliken et al., 2008; Serruya et al., 2002; Taylor et al., 2002), a robotic arm (Chapin *Corresponding author. Tel.: +412-383-7021; Fax: +412-383-5460 E-mail: [email protected] DOI: 10.1016/B978-0-444-53355-5.00007-5

103

104

however, is probably quite a bit different. Because the neural processes related to imagination and imagery are not well understood, they are hard to tap into with any degree of reliability. Further, the process of imagining typing the letter “p” seems fundamentally different from the process of imagining “grab the cup.” Does this imply that a BCI that controls a spelling device must use a different set of neurons than a BCI that controls a robotic hand? The answer, of course, is no. Most BCIs that rely on single-neuron recordings extract them from the primary motor cortex (e.g., Hochberg et al., 2006; Serruya et al., 2002; Taylor et al., 2002; Velliste et al., 2008). Since this area is a major source of “output” from the brain, it is a logical place to find signals that might be used to control an external device. The idea is that by tapping into endogenous control signals related to some observable, quantifiable behavior, like arm movement, the subject could eventually learn to associate those behaviors with certain actions of the device. This has proven to be a successful tactic: signals from the proximal arm area of primary motor cortex have successfully been used to control computer cursors moving in two (Carmena et al., 2003; Hochberg et al., 2006) and three (Jarosiewicz et al., 2008; Taylor et al., 2002) dimensions, and a monkey was even successfully trained to feed itself with a 4 degree of freedom robotic arm (Velliste et al., 2008). However, the human hand alone contains
20 independently controllable degrees-of-freedom. To further complicate things, the neural representation of hand shape is only beginning to be worked out (see, e.g., Hendrix et al., 2009; Lemon, 1993; Vargas-Irwin et al., 2010). There is still a long way to go before prosthetic arms can replace the capabilities of a lost limb. In this chapter, we review some of the approaches taken toward extracting potential control signals for BCI devices, and argue that to achieve the next generation of prosthetic devices we will need a new approach that does not rely on assumptions about how arm

movements might actually be encoded. By going beyond these models, we can use the neural population responses to infer the existence of potential control signals, without making particular reference to what those control signals might be. We postulate that an understanding of these control signals will not only facilitate the design of high functioning prosthetic devices, but will also help answer basic questions about the neural correlates of intent.

Prostheses based on arm movements In the 1980s, Georgopoulos and colleagues released a series of papers detailing how populations of neurons in the proximal arm area of motor cortex might represent arm movements (Georgopoulos et al., 1982; Schwartz et al., 1988), and further how this information could be read out from simultaneously recorded neurons (Georgopoulos et al., 1986, 1988; Kettner et al., 1988). These papers detailed an approach to neural decoding which has become fairly common in the BCI field: first perform a series of experiments with natural arm movements to build an encoding model detailing how firing rates depend on arm movements, then invert this encoding model and use decoded arm movements to drive a remote effector (e.g., a computer cursor; Taylor et al., 2002). Alternately, specification of the encoding model could be skipped entirely, and instead the arm movement data could be used to train an algorithm to extract kinematic data directly from the spike trains (e.g., Mulliken et al., 2008; Wessberg et al., 2000). In either case, this approach relies extensively on arm movement data to calibrate the decoding device. Apart from the obvious clinical problem of using arm movement data to train a device that is supposed to replace the functionality of a lost arm, this approach may not be ideal because it assumes the control signals appropriate for an arm are the same as the signals appropriate for the device. A number of studies now suggest that

105

this is not the case. First, it has been observed that as the subjects learn to modulate their cortical signals to control the cursor directly, they stop moving their arms (Carmena et al., 2003; Chapin et al., 1999; Taylor et al., 2002). This indicates that there must be at least some change in the neural signal that is specific to brain-control. Second, tuning curves recorded during the arm control session have been found to be different from the tuning curves recorded during the brain-control session (Carmena et al., 2003; Taylor et al., 2002); in fact, these tuning curve differences tend to increase with training (Ganguly and Carmena, 2009; Taylor et al., 2002). Taylor and colleagues investigated this issue in detail by recording from neurons in the primary motor cortex of monkeys while they performed center-out reaching movements under both handand brain-control. On the first day, they found that neurons changed their preferred directions between the two sessions by an average of
65 , and these tuning curve differences increased as the subject trained with the BCI (Fig. 1). This change in tuning is indicative of a brain-control specific change in the control signals. It is still an open question as to why the tuning differs between hand-control and brain-control. It could be that the motor cortex is sensitive to the differences in the dynamic properties of the remote effector and the limb. Another possibility is that the neural activity is influenced by the proprioceptive feedback, which changes between the hand-control and brain-control tasks because the subjects are no longer moving their arms. In fact, Hatsopoulos and colleagues have shown that when the arm is passively moved in concert with a brain-controlled cursor, decoding performance improves (Suminski et al., 2010). Since they calibrated their decoder with arm movement data, this could be an indication that the tuning curves change less between hand-control and brain-control when the arm is kept moving in the brain-control condition. An interesting implication of this work is that it suggests that providing proper proprioceptive feedback to

patients might improve their ability to use a neural prosthetic device. To date, however, providing realistic proprioceptive feedback to patients remains a challenge.

Prostheses based on cursor imagery Another, related approach does not rely on arm movements themselves, but rather on motor imagery. In work with human subjects, Hochberg and colleagues found that many cells in primary motor cortex could be driven by imagined movements, such as imagined wrist or elbow flexion (2006). Presumably, the neural tuning to these “imagined” movements is similar to the tuning these neurons would have had to natural arm movements, and studies in monkeys comparing the neural tuning during active movement with that during passive observation seem to bear this out (Tkach et al., 2007). Electrocorticographic signals recorded from humans have shown that the modulations induced by motor imagery have a similar spatial distribution to the modulations produced by actual movement, although they are weaker. Providing visual feedback about the modulation, however, can increase the magnitude of the modulations to be the same as or greater than with overt movement (Miller et al., 2010). While these findings allow researchers to overcome the clinical problem of relying on arm movement data to build their decoder, as with the arm movement data there is evidence that these imagery-based neural tunings may still not be capturing all of the neural activity that could be used to drive a prosthetic device (Wahnoun et al., 2006). Several researchers have explored iterative approaches to try to uncover the right set of control signals. Although details can vary from lab to lab, an example of the procedure used by Schwartz and colleagues is as follows (Chase et al., 2009; Fraser et al., 2009; Jarosiewicz et al., 2008). First, the decoding parameters are initialized, by assigning values at random to every neuron. Alternate approaches involve initializing

250 μV

(a)

(d) HC 1

0

ΔPD day B

106 X component

0 700 μs

−250

−1 ΔPD day A

0

−250

(b) 100

(c)

ΔPD day B

BC

Y component

ΔPD day B

250 μV

Z component

0 −1 ΔPD day A

Mean angle (°)

90 80 1 70 0 60 −1 50

0

10

20 Day

30

40

ΔPD day A −1

0

1

Fig. 1. Changes in cortical activity between hand-control (HC) and brain-control (BC) tasks in subject M. (a) Cell with a 107 change in tuning direction (the unit waveform is shown in black). Each dot is the mean firing rate during one movement. HC rates are in the right column and BC rates are in the left column of each square. The eight squares correspond to the eight target directions (center four ¼ distal; outer four ¼ proximal). (b) Daily mean angles (thick lines) between HC and BC preferred directions for all cells significantly tuned during both tasks (black, contralateral; gray, ipsilateral to the arm moved during the HC task). (c) Lines connecting HC preferred directions with BC preferred directions (circle ends) projected onto a unit sphere (day 8, only cells significantly tuned in both tasks; black, contra; dotted, ipsi). (d) Change in the X, Y, and Z components of the preferred direction unit vectors between the HC and BC tasks plotted day-against-day for 8 random pairs of days (day 27 or later, only units that were significantly tuned in both tasks on both days; 35  3 units per pair of days). With permission from Taylor et al. (2002).

with parameters from arm data (as in Taylor et al., 2002) or from a motor observation or imagery session (as in Hochberg et al., 2006; Wahnoun et al., 2006). Targets are then presented, one at a time in random order, and left on the screen until a movement time-out period elapses. Subjects are instructed to attempt to hit the target, and cells in the motor cortex tend to modulate as a function of target direction, indicating that an attempt is

being made. Depending on the initialization, however, the attempt may or may not be successful; with the random assignment initialization, the cursor hardly moves at all (Chase et al., 2009). Regardless of success or failure, after each target has been presented, firing rates are regressed against target direction to build an encoding model describing each cell's tuning to desired movement, and these tuning curves are used to

107

recompute decoding parameters. Then another round of targets is presented. It typically takes only a few minutes or so to converge to a stable set of decoding parameters. Once convergence is reached, the decoding parameters are fixed. Not only does this iterative procedure result in tuning curves that are substantially different from those observed during arm movement tasks (Taylor et al., 2002) or passive observation (Wahnoun et al., 2006), it also results in better control. Wahnoun and colleagues characterized various parameters of control during the iterative procedure (Fig. 2). In less than 5 min, they found substantial straightening of the trajectories, and the average time to hit the targets decreased by about 20%. Further, their improvements in

control were accompanied by a decrease in the average firing rates of the cells. This might indicate that the iterative procedure resulted in control that required less effort, an important consideration for a device which might need to be used by patients for long periods of time. Although motor imagery approaches have been successfully applied to prosthetic control, as with arm movement approaches they still suffer from one major failing: they do not account for types of neuronal modulation that might not be included in the tested range of movements. For example, if some of the recorded neurons happen to encode neck movement, and neck movements are not included in the set of instructed movements or imagined movements,

(a)

Down I

Left

II

Front

(c) Reach duration (s)

Trajectory ratio

I 8 II

6 4 1

2

3 Time (min)

4

5

(d)

2.5

Average FR (spk/s)

(b)

2

1.5

1

2

3 Time (min)

4

5

36.5 35.0 33.5 31.5 1

2

3 Time (min)

4

5

Fig. 2. Change in performance as parameters for the control algorithm are tuned. (a) illustrates individual cortically controlled cursor movements to each of the eight targets at two times following the beginning of tuning: (I) at the end of the visual following task; (II) after 60 s of additional data accumulation. Figures are drawn as shown on the screen prior to reflection in the mirror, leading to a left-handed coordinate system. (b, c, and d) show general characteristics of the movements and neural activity. (b) Ratio of the summed path length to the direct distance from the center to the target. Smaller values indicate straighter, more direct movements. (c) Time in seconds taken to reach the target. (d) Average firing rate of the entire neural ensemble as a function of time. With permission from Wahnoun et al. (2006).

108

this potential source of control signal might be overlooked. This could in turn limit device performance, as this extra control signal could have been co-opted to extend the number of controllable degrees of freedom of the effector. If we are to achieve control of a device that can replace the functionality of a limb, we will need to utilize every available control signal.

A more general approach To take advantage of all of the potential control signals that can influence the neural activity, it is necessary to make no a priori assumptions about what those control signals might actually be. Instead of assuming that the neurons will respond to a particular type of motor imagery, it is possible to use operant conditioning to allow the subject to discover the underlying control signal by trial and error. In the late 1960s and early 1970s, work by Dr. Eberhard Fetz and colleagues demonstrated a technique by which one could train monkeys to gain volitional control over the firing rates of individual neurons in primary motor cortex (Fetz, 1969; Fetz and Baker, 1973). By providing feedback only about the firing rate of a target neuron, the subjects quickly (typically, within 10 min or so) learned to ramp the firing rate of that neuron up and down to achieve reward. These firing rate increases and decreases were often, though not always, associated with overt movements. Even when the firing rate of the neuron was consistently related to muscle activity, however, the activity could typically be dissociated by properly conditioning the feedback signal (Fetz and Finocchio, 1971, 1975). In a follow on to this work, they were even able to train monkeys to use single neurons to control a muscle stimulation device (Moritz et al., 2008). In a sense, Fetz's technique could be considered as a way of identifying the volitional control signals that affect single neurons in motor cortex, without resorting to any kind of explicit motor task. In fact, this ability is not unique to motor cortex; there are now numerous examples of

subjects using operant feedback to gain volitional control over single neurons in a number of cortical areas (for review, see Fetz, 2007). Can this technique be extended to populations of neurons? We have recorded from groups of neurons while using operant conditioning to train monkeys to gain control of the firing rates of single cells. Invariably, we find that even when only one neuron is being used to provide feedback, there are typically large correlations in the firing rates of many of the other simultaneously recorded cells (Fig. 3). Clearly, these correlations indicate that the volitional control signal used to drive the target neuron also influences the firing rates of other neurons. In essence, the operantconditioning task has uncovered a controllable pattern of neural activity. We have found that by performing the single-neuron operant-conditioning task with multiple target neurons, a variety of controllable patterns can be uncovered. Of course, the patterns of activity uncovered in these operant-conditioning tasks may themselves represent collections of correlated volitional control signals. Neurons in motor cortex tend to be tuned to multiple parameters (for review, see Scott, 2003), and so may be best driven by multiple control signals. Further, control signals used to drive two different neurons may be partially overlapping. Using dimensionality reduction techniques, like principal component analysis or independent component analysis, it may be possible to resolve the effect of individual volitional drivers on the neural population response. Recently, a new technique was developed that combined dimensionality reduction with temporal smoothing to infer the presence of latent driving signals from populations of neurons recorded during an arm movement task (Yu et al., 2009). Called Gaussian process factor analysis, the method allows one to disentangle the multiple driving signals inducing correlations across the population; a description of the technique is given in Fig. 4. These factor analysis methods have already been shown to account better for correlated noise that might otherwise degrade prosthetic decode performance (Santhanam et al., 2009).

Neuron

Radius

109

0

2

4

6 Time (s)

8

10

–1

0 CC

1

Fig. 3. Correlations in neural activity observed during an operant-conditioning task. Top: radius of a ring controlled by the firing rate of one neuron. The subject had to oscillate the firing rate to hit target ring sizes located at the dashed lines. Vertical lines denote successful completions. Bottom, left: rasters of 18 simultaneously recorded neurons during this task. The black raster is the neuron being used to control the ring. Bottom, right: correlation coefficient of each neuron with the neuron controlling the ring.

(a)

(b)

(c)

N3

(d)

N3

S1

time

Neuron 1 Neuron 2 Neuron 3

S2 S1

time N2 Spike trains

Noisy time series

N1

N2

N1

Denoised time series

S2 Low-dimensional time series

Fig. 4. Extracting a neural trajectory from multiple spike trains. For clarity, the activity of only three neurons are considered in this illustration. (a) Spike trains recorded simultaneously from three neurons. (b) The time evolution of the recorded neural activity plotted in a three-dimensional space, where each axis measures the instantaneous firing rate of a neuron (e.g., N1 refers to neuron 1). (c) The neural trajectory (a “denoised” version of the trajectory in (b) is shown to lie within a two-dimensional space with coordinates S1 and S2. (d) The neural trajectory can be directly visualized in the low-dimensional space and be referred to using its low-dimensional coordinates (S1, S2). With permission from Yu et al. (2009).

110

One of the real benefits of these dimensionality reduction methods is that they can recover a description of the latent driving signals using only the correlations they induce in the population, without reference to any kind of external, movement-related parameters. In essence, these techniques make no assumptions about what these driving signals might actually encode. This is especially useful since it has been shown that, in motor cortex, firing rate predictions based only on the firing rates of simultaneously recorded neurons tend to outperform predictions based on external parameters, when recording from 50 or more cells (Stevenson and Kording, 2011). We have begun to apply these dimensionality reduction techniques in the operant-conditioning framework. After performing conditioning on several single neurons and observing the correlations in the recorded population, we apply the dimensionality reduction methods to recover canonical patterns of population activity. We then perform the same ring-control task described in Fig. 3, but instead of reinforcing the firing rate of a single neuron, we condition on the overlap of the population activity with a particular target pattern. Eventually, we hope to use these techniques to identify putative control signals for a prosthetic device. By marrying operant-conditioning experimental procedures with nonparametric, correlationbased techniques for source identification, it is possible to develop a fully nonparametric description of the volitional control signals that influence a neural population. These signals have the potential to extend the capability of prosthetic devices to enable more complicated grasping and hand-shaping movements, even if we do not understand how these movements are represented during natural behavior. At the very least, they ought to be useful in identifying volitionally controllable signals inherent in the population that might not otherwise be found. They might also serve to facilitate learning. Current methods of prosthetic decoding try to identify patterns neural activity that covary together while the subjects are attempting to learn how

to control the various degrees-of-freedom of the prosthetic device. With these new techniques, it may be possible to separate this into two sequential processes: first identify the patterns of activity that naturally covary, then apply them to decoding. For example, after the patterns are identified, they could be applied to decoding by actuating each degree-of-freedom according to the projection of the neural activity along each pattern. The subject must then solve the easier problem of associating each controllable pattern with a particular device function. It will be interesting to see how BCI control based on volitional signals identified through operant conditioning will compare with BCI control based on standard, parametric encoding models. Ultimately, we hope that synergizing these approaches will allow us both to develop better prosthetic devices, and gain insight into the cognitive mechanisms of volition.

References Carmena, J. M., Lebedev, M. A., Crist, R. E., O'Doherty, J. E., Santucci, D. M., Dimitrov, D. F., et al. (2003). Learning to control a brain-machine interface for reaching and grasping by primates. PLoS Biology, 1, E42. Chapin, J. K., Moxon, K. A., Markowitz, R. S., & Nicolelis, M. A. (1999). Real-time control of a robot arm using simultaneously recorded neurons in the motor cortex. Nature Neuroscience, 2, 664–670. Chase, S. M., Schwartz, A. B., & Kass, R. E. (2009). Bias, optimal linear estimation, and the differences between openloop simulation and closed-loop performance of spikingbased brain-computer interface algorithms. Neural Networks, 22, 1203–1213. Fetz, E. E. (1969). Operant conditioning of cortical unit activity. Science, 163, 955–958. Fetz, E. E. (2007). Volitional control of neural activity: Implications for brain-computer interfaces. Journal of Physiology, 579, 571–579. Fetz, E. E., & Baker, M. A. (1973). Operantly conditioned patterns on precentral unit activity and correlated responses in adjacent cells and contralateral muscles. Journal of Neurophysiology, 36, 179–204. Fetz, E. E., & Finocchio, D. V. (1971). Operant conditioning of specific patterns of neural and muscular activity. Science, 174, 431–435.

111 Fetz, E. E., & Finocchio, D. V. (1975). Correlations between activity of motor cortex cells and arm muscles during operantly conditioned response patterns. Experimental Brain Research, 23, 217–240. Fraser, G. W., Chase, S. M., Whitford, A., & Schwartz, A. B. (2009). Control of a brain-computer interface without spike sorting. Journal of Neural Engineering, 6, 055004. Ganguly, K., & Carmena, J. M. (2009). Emergence of a stable cortical map for neuroprosthetic control. PLoS Biology, 7, e1000153. Georgopoulos, A. P., Kalaska, J. F., Caminiti, R., & Massey, J. T. (1982). On the relations between the direction of two-dimensional arm movements and cell discharge in primate motor cortex. The Journal of Neuroscience, 2, 1527–1537. Georgopoulos, A. P., Kettner, R. E., & Schwartz, A. B. (1988). Primate motor cortex and free arm movements to visual targets in three-dimensional space. II. Coding of the direction of movement by a neuronal population. Journal of Neuroscience, 8, 2928–2937. Georgopoulos, A. P., Schwartz, A. B., & Kettner, R. E. (1986). Neuronal population coding of movement direction. Science, 233, 1416–1419. Hendrix, C. M., Mason, C. R., & Ebner, T. J. (2009). Signaling of grasp dimension and grasp force in dorsal premotor cortex and primary motor cortex neurons during reach to grasp in the monkey. Journal of Neurophysiology, 102, 132–145. Hochberg, L. R., Serruya, M. D., Friehs, G. M., Mukand, J. A., Saleh, M., Caplan, A. H., et al. (2006). Neuronal ensemble control of prosthetic devices by a human with tetraplegia. Nature, 442, 164–171. Jarosiewicz, B., Chase, S. M., Fraser, G. W., Velliste, M., Kass, R. E., & Schwartz, A. B. (2008). Functional network reorganization during learning in a brain-computer interface paradigm. Proceedings of the National Academy of Sciences of the United States of America, 105, 19486–19491. Kennedy, P. R., Bakay, R. A., Moore, M. M., Adams, K., & Goldwaithe, J. (2000). Direct control of a computer from the human central nervous system. IEEE Transactions on Rehabilitation Engineering, 8, 198–202. Kettner, R. E., Schwartz, A. B., & Georgopoulos, A. P. (1988). Primate motor cortex and free arm movements to visual targets in three-dimensional space. III. Positional gradients and population coding of movement direction from various movement origins. Journal of Neuroscience, 8, 2938–2947. Lemon, R. N. (1993). The GL. Brown Prize Lecture. Cortical control of the primate hand. Experimental Physiology, 78, 263–301. Miller, K. J., Schalk, G., Fetz, E. E., den Nijs, M., Ojemann, J. G., & Rao, R. P. (2010). Cortical activity during motor execution, motor imagery, and imagery-based online feedback. Proceedings of the National Academy of Sciences of the United States of America, 107, 4430–4435.

Moritz, C. T., Perlmutter, S. I., & Fetz, E. E. (2008). Direct control of paralysed muscles by cortical neurons. Nature, 456, 639–642. Mulliken, G. H., Musallam, S., & Andersen, R. A. (2008). Decoding trajectories from posterior parietal cortex ensembles. The Journal of Neuroscience, 28, 12913–12926. Musallam, S., Corneil, B. D., Greger, B., Scherberger, H., & Andersen, R. A. (2004). Cognitive control signals for neural prosthetics. Science, 305, 258–262. Pohlmeyer, E. A., Oby, E. R., Perreault, E. J., Solla, S. A., Kilgore, K. L., Kirsch, R. F., et al. (2009). Toward the restoration of hand use to a paralyzed monkey: Brain-controlled functional electrical stimulation of forearm muscles. PLoS ONE, 4, e5924. Santhanam, G., Ryu, S. I., Yu, B. M., Afshar, A., & Shenoy, K. V. (2006). A high-performance brain-computer interface. Nature, 442, 195–198. Santhanam, G., Yu, B. M., Gilja, V., Ryu, S. I., Afshar, A., Sahani, M., et al. (2009). Factor-analysis methods for higher-performance neural prostheses. Journal of Neurophysiology, 102, 1315–1330. Schwartz, A. B., Kettner, R. E., & Georgopoulos, A. P. (1988). Primate motor cortex and free arm movements to visual targets in three-dimensional space. I. Relations between single cell discharge and direction of movement. Journal of Neuroscience, 8, 2913–2927. Scott, S. H. (2003). The role of primary motor cortex in goaldirected movements: Insights from neurophysiological studies on non-human primates. Current Opinion in Neurobiology, 13, 671–677. Serruya, M. D., Hatsopoulos, N. G., Paninski, L., Fellows, M. R., & Donoghue, J. P. (2002). Instant neural control of a movement signal. Nature, 416, 141–142. Stevenson, I. H., & Kording, K. P. (2011). How advances in neural recording affect data analysis. Nature Neuroscience, 14, 139–142. Suminski, A. J., Tkach, D. C., Fagg, A. H., & Hatsopoulos, N. G. (2010). Incorporating feedback from multiple sensory modalities enhances brain-machine interface control. The Journal of Neuroscience, 30, 16777–16787. Taylor, D. M., Tillery, S. I., & Schwartz, A. B. (2002). Direct cortical control of 3D neuroprosthetic devices. Science, 296, 1829–1832. Tkach, D., Reimer, J., & Hatsopoulos, N. G. (2007). Congruent activity during action and action observation in motor cortex. The Journal of Neuroscience, 27, 13241–13250. Vargas-Irwin, C. E., Shakhnarovich, G., Yadollahpour, P., Mislow, J. M., Black, M. J., & Donoghue, J. P. (2010). Decoding complete reach and grasp actions from local primary motor cortex populations. The Journal of Neuroscience, 30, 9659–9669.

112 Velliste, M., Perel, S., Spalding, M. C., Whitford, A. S., & Schwartz, A. B. (2008). Cortical control of a prosthetic arm for self-feeding. Nature, 453, 1098–1101. Wahnoun, R., He, J., & Helms Tillery, S. I. (2006). Selection and parameterization of cortical neurons for neuroprosthetic control. Journal of Neural Engineering, 3, 162–171. Wessberg, J., Stambaugh, C. R., Kralik, J. D., Beck, P. D., Laubach, M., Chapin, J. K., et al. (2000). Real-time

prediction of hand trajectory by ensembles of cortical neurons in primates. Nature, 408, 361–365. Yu, B. M., Cunningham, J. P., Santhanam, G., Ryu, S. I., Shenoy, K. V., & Sahani, M. (2009). Gaussian-process factor analysis for low-dimensional single-trial analysis of neural population activity. Journal of Neurophysiology, 102, 614–635.

A. M. Green, C. E. Chapman, J. F. Kalaska and F. Lepore (Eds.) Progress in Brain Research, Vol. 192 ISSN: 0079-6123 Copyright Ó 2011 Elsevier B.V. All rights reserved.

CHAPTER 8

Tactile communication systems: optimizing the display of information Lynette A. Jones* Department of Mechanical Engineering, Massachusetts Institute of Technology, Cambridge, Massachusetts, USA

Abstract: Tactile communication systems based on vibrotactile signals have been developed as sensory substitution devices for those with visual, auditory, or vestibular impairments and to assist users in spatial orientation and navigation in unfamiliar environments. One of the main challenges in using tactile displays to compensate for sensory loss in other modalities or to overcome the limitations of visual and auditory information overload is in determining what type of information can be presented tactually and which parameters of stimulation can be used to convey these messages effectively. Psychophysical studies of vibrotactile perception provide a framework that assists in determining which stimulus dimensions and ranges of values can be used to create tactile patterns, known as tactons. A number of experiments have been conducted in which the ability of participants to identify tactons presented at different sites on the body has been measured. The results from this research indicate that tactons created by varying the spatial location, number, and temporal sequence of activation of motors in a tactile display can be accurately identified. They further demonstrate the potential of using two-dimensional tactile displays to present information, and the feasibility of creating tactile communication systems that are easily learned. Keywords: communication systems; skin tactile displays; touch vibration. those with visual, auditory, or vestibular impairments (Kaczmarek and Bach-y-Rita, 1995; Kaczmarek et al., 1991; Reed and Delhorne, 1995; Wall and Weinberg, 2003). In many of these applications, the devices have been developed to be worn on areas such as the torso or arm, so that the hands are free to carry on daily activities. This has obviously necessitated taking into account the reduced sensitivity of hairy as compared to glabrous skin (Bolanowski et al., 1994; Gescheider

Introduction Tactile communication systems have been developed for a number of applications, including spatial orientation and navigation, notifications and alerts, and as sensory substitution devices for

*Corresponding author. Tel.: þ1-617-253-3973; Fax: þ1-617-253-2218 E-mail: [email protected] DOI: 10.1016/B978-0-444-53355-5.00008-7

113

114

et al., 2002; Jones and Lederman, 2006), a deficiency that is often compensated for by the considerably larger areas of hairy skin available to present information. Some of the first tactile displays used to compensate for sensory loss were developed in the 1960s to present tactile cues to the visually impaired. These systems used a camera to capture the visual information which was then presented pictorially via a tactile array on the back (Bachy-Rita et al., 1969) or the fingertip (Bliss et al., 1970). More recently, tactile displays have shown promise as balance prostheses for people with vestibular impairments. In this application, angular position and velocity of body sway is measured and a tactile display is used to present cues about the direction and magnitude of body tilt to improve postural stability and so prevent falls (Peterka et al., 2006; Wall and Weinberg, 2003). In these more recent applications, the opportunity to use the sense of touch as a medium of communication has benefited from advancements in tactile display technologies which have become more sophisticated and less intrusive, with the result that they are more effective and acceptable to users (Jones and Sarter, 2008). An additional factor that has stimulated interest in the possibilities of tactile communication has been the widespread use of mobile devices and wearable computers with limited screen space to display visual information (MacLean, 2009). Tactile displays encompass a spectrum that ranges from devices affixed to chairs or seats in vehicles that stimulate the back to provide spatial cues about the environment (Lindeman and Yanagida, 2003; Tan et al., 2003), to more dense arrays that are used to present tactile cues to the fingertips, such as virtual Braille displays (Lévesque et al., 2005, 2007). One of the challenges in using tactile displays to compensate for sensory loss in other modalities or to overcome the limitations of visual and auditory information overload is in determining what type of information can be presented tactually and which

parameters of stimulation can be used to convey these messages effectively. At present, most tactile displays deliver rather simple vibrotactile inputs at single frequencies that are within the range of maximal sensitivities of the skin, that is between 100 and 300 Hz. Devices have also been developed that use electrotactile inputs (Kaczmarek and Haase, 2003) or lateral skin deformation (Lévesque et al., 2007) to stimulate the skin. The predominance of displays based on vibrating motors reflects the fewer safety and comfort considerations associated with their use (as compared to electrotactile displays), and the greater dynamic range of stimulus parameters available to communicate information (cf. skin stretch and electrotactile displays). Vibrotactile inputs have been shown to be effective warning signals that alert the user and provide robust spatial cues that assist in navigation in unfamiliar environments (Ho et al., 2005, 2006; Scott and Gray, 2008). In addition, tactile signals are effective at directing the user's attention to a particular area in space. Van Erp (2001) has shown that it appears to be very intuitive to perceive an external direction emanating from a single point of stimulation on the body. However, when messages of some complexity need to be transmitted tactually, it is not clear how these are best communicated. Neurophysiological and psychophysical studies of vibrotactile perception (Bolanowski et al., 1994; Gescheider et al., 2009) provide a framework that assists in determining which stimulus dimensions and ranges of values can be used to create tactile patterns. Vibrotactile signals vary along five basic dimensions: amplitude, frequency, waveform, duration, and location, each of which can be varied to create tactile patterns as illustrated in Fig. 1. To date, variations in stimulus duration and the location on the body stimulated have been used most frequently to generate different tactile patterns. The selection of these two parameters reflects the skin's capacity to encode the spatial coordinates of tactile stimulation accurately and its sensitivity to the

115

1

Amplitude

Amplitude

1

0

−1

−1 0

5

10 Time (ms)

15

20

0

1

5

15

10 Time (ms)

20

1

Amplitude

Amplitude

0

0

−1

0

−1 0

5

10 Time (ms)

15

20

0

100

200

300

Time (ms)

Fig. 1. Parameters of vibrotactile signals that can be used to create tactons. Variations in frequency (upper left), amplitude (upper right), waveform (lower left), and duration (lower right) have all been used to create tactile patterns.

changes in the temporal properties of stimuli, which is inferior to that of the ear but superior to the eye (Geldard, 1960). There are challenges associated with using some of the other dimensions of vibrotactile signals to create different patterns. For example, the frequency and amplitude of vibration are not orthogonal: when the frequency of the signal changes so too does its perceived amplitude (Bolanowski et al., 1994; Morley and Rowe, 1990). An additional factor that influences the perception of vibrotactile frequency is that perceived frequency changes as a function of the site on the body is stimulated. In regions with higher densities of mechanoreceptors, such as the fingertips, perceived frequency

increases more rapidly with increases in the frequency of vibrotactile signals than areas with lower innervation densities such as the forearm (von Békésy, 1962). These differences in perceived frequency presumably reflect not only variations in the density of mechanoreceptors but also the damping effects of the underlying soft tissue. As noted above, changes in the amplitude or intensity of vibrotactile signals can also affect their perceived frequency (Verrillo et al., 1969). An additional factor that needs to be considered in using amplitude to encode information is that, in contrast to many stimulus dimensions, the perception of vibrotactile amplitude does not follow Weber's law. The change in intensity that human

116

observers reliably discriminate (i.e., difference threshold) varies as a function of the amplitude of vibration. The threshold is smallest at moderate to high intensities of vibration, where it averages approximately 16% (Craig, 1972). For stimuli above threshold, as the amplitude of the vibrotactile stimulus increases its perceived intensity also increases. However, this function varies at different sites on the body. In less-sensitive areas, such as the torso or forearm, the perceived magnitude of vibration increases more rapidly with increases in amplitude, than in more sensitive areas such as the fingers (Verrillo, 1973; Verrillo and Chamberlain, 1972). In contrast to the ear, the skin is relatively insensitive to changes in the waveform of vibrotactile signals, which suggests that the perception of changes in frequency is based largely on temporal rather than spectral information (Summers et al., 1997). However, by using frequency modulation of sinusoids the “roughness” of vibrotactile signals is perceived to vary, and

variations along this dimension are reliably encoded (Brown et al., 2005). In creating patterns for tactile communication systems based on any of these vibrotactile parameters, stimuli must be selected judiciously and carefully evaluated before they are implemented (Jones and Sarter, 2008).

Tactile displays Tactile displays that mechanically stimulate the skin use an actuator that converts electrical energy into a mechanical displacement of either the whole actuator or a contactor pad at frequencies typically ranging from 50 to 300 Hz (Mortimer et al., 2007). Different actuator technologies have been used to create these displays, with small inertial motors being the most common due to their size, availability, and low power requirements (see Fig. 2). The latter has been an important consideration for applications

Fig. 2. Electromechanical actuators used in tactile displays. Upper row from left to right: C2 tactor (Engineering Acoustics Inc.), Tactaid tactor (Audiological Engineering Corp.); Lower row: pancake motor (Sanko Electric): eccentric mass exposed (left), intact motor (middle), encased motor (right).

117

involving mobile users of tactile communication systems, such as the visually impaired, and for mobile communication devices. A number of different actuator technologies have been used in vibrotactile displays ranging from electromagnetic motors to arrays of pins that indent the skin when activated by piezoelectric bimorphs. Over the past decade, a variety of small DC motors has been used in tactile displays developed for spatial orientation, navigation, and to facilitate motor skills learning (Jones et al., 2006; Lindeman et al., 2006; Spelmezan et al., 2009). These motors are activated at a fixed frequency (usually between 80 and 250 Hz) and amplitude and vibrate using an off-axis weight on their rotor. The frequency of vibration is directly proportional to the motor's speed which is a function of the driving voltage (Jones and Held, 2008). The advantages of these motors are that they are simple to control, and can produce vibrations on the skin that are readily perceptible. They do, however, have limited power-to-mass ratios, and in general the frequency and amplitude of the vibration cannot be independently controlled.

Tactons Tactile patterns that represent abstract concepts are often referred to as tactons or tactile icons, by analogy to icons and earcons in the visual and auditory modalities, respectively (Brewster and Brown, 2004; MacLean and Enriquez, 2003; Roberts and Franklin, 2005). Tactons are created by varying one or a number of the parameters of vibrotactile stimuli (see Fig. 1) and can represent actions, objects, or concepts. Graphically, tactons can be represented in a number of formats ranging from musical scores to stimulus waveforms, as illustrated in Fig. 3. There has been a tendency to assume that the ear and the development of earcons provide a reasonable model of how the skin responds to the variables manipulated to create tactons (Brewster and Brown, 2004). However, even for basic dimensions such as stimulus

frequency, there is considerable variation in the estimates of how much the frequency of a vibrotactile stimulus must change for an observer to perceive the difference (i.e., the differential threshold), with estimates ranging from 18% to 50% (Jones and Sarter, 2008; Mahns et al., 2006). Early research on tactons focused primarily on first-order dimensions of vibrotactile signals. In the first tactile language developed by Geldard (1957) called Vibratese, 45 basic elements that varied with respect to amplitude, duration, and location were created. More recently, variables such as temporal variations in the tactile signals and frequency modulation of a base signal have been used to create patterns (Brown et al., 2005; Ternes and MacLean, 2008). A critical aspect of designing tactons is selecting parameters that result in patterns which are easy to identify and learn. Although tactons may have intuitive meaning, for example, sequentially activating a row of motors from left to right across the back to indicate turn or attend to the right, the association between the tactile signal and the concept it represents must be learned. These arbitrary associations are readily acquired and subjects typically learn small sets of tactons (up to 10 patterns) in less than 10 min (Enriquez et al., 2006; Jones et al., 2009). As the size of the tactile vocabulary increases, so too does the time required to learn each tacton. Geldard (1957) reported that 12 h of training with the 45 elements of his tactile alphabetic code (Vibratese) were required for the three individuals whom he tested to reach a plateau in comprehension (at a rate of 38 words/min). However, with a considerably smaller “vocabulary” of 15 elements, Jones et al. (2007) found that subjects could accurately recognize the tactons with over 90% accuracy, after 10 min of training. Vibrotactile stimuli that vary along several dimensions appear to hold most promise in developing tactile vocabularies, although how these dimensions can be optimally combined is still unclear. There has been some research using multidimensional scaling techniques (Cox, 2001) to

118

Amplitude

1

0

-1 0

0.02

0.04 0.06 Time (s)

0.08

0.1

“Roughness” tacton Vibrotactile score Front

I

G

C

in

R

W

O

1

7

3

Intensity

2

8

9

4

6 5 Turn around

Duration

Navigation tacton

Language tacton

Fig. 3. Graphical depictions of tactons. Upper left: frequency modulated sinusoid (250 Hz modulated by 20 Hz) used to vary the “roughness” of tactile patterns (Brown et al., 2005); upper right: the vibrotactile score uses the metaphor of a musical score to represent the duration and frequency of elements in the tacton (Lee et al., 2009); lower left: communication tacton in which the pattern of activation of the motors represents the direction that the user should take (Jones et al., 2006); lower right: representation of nine elements of the coding of the Vibratese language. Each set of nine symbols was presented at a single vibrator whose output varied in intensity and duration (redrawn from Geldard, 1957).

determine how haptic and tactile phonemes could be created from signal parameters such as frequency, force, or waveform (e.g., Enriquez et al., 2006; MacLean and Enriquez, 2003). Much of this research is empirical in nature; as yet there is no a priori basis for combining certain elements serially or in parallel to create new tactons. However, the results from several experiments suggest that variations in the temporal parameters of a single tacton or group of tactons, for example, by decreasing the interstimulus interval, are easily encoded and recognized by subjects as an

indicator of urgency or proximity (Brewster and King, 2005; Ternes and MacLean, 2008). When combined with another tacton, such as a directional signal that indicates an upcoming turn to the left, it is possible to indicate proximity of the turn by decreasing the interval between successive presentations. The hierarchical structure that is optimal for merging different types of cues such as these is not known. In addition, it is not clear whether tactile information presented at different sites on the body can be encoded independently. Certainly, the ability to perceive tactile patterns

119

presented simultaneously at different sites deteriorates as a function of the number of sites stimulated (Geldard, 1968). However, it is the degree of overlap between patterns, known as communality, rather than the number of sites stimulated, that has the greater impact on the ability to perceive differences between patterns (Cholewiak and Collins, 1995; Geldard, 1968; Jones et al., 2009). This suggests that tactons will always need to be evaluated in the context of the vocabulary in which they will be used.

Tactons for spatial cuing Localization One of the more successful applications of tactile display technology has been in using the location of stimulation on the body as a cue about the external environment (Van Erp, 2005; Van Erp et al., 2005). Anatomical landmarks can facilitate identifying the site of the tactile input (e.g., on the elbow, near the spine) and so the number of sites available for communication is potentially quite large. Most tactile displays used for spatial cuing have been mounted on small defined areas of the body such as the forearm or lower back. At these locations, the ability to localize a point of stimulation in an array of motors depends on the distance between motors and the number of motors in the display (Chen et al., 2008; Cholewiak and Collins, 2003; Jones and Ray, 2008). These two variables are often coupled in that as the number of motors increases, the distance between them decreases. Cholewiak et al. (2004) found that when the number of motors in a belt worn around the waist increased from six (with spacing between the motors of 140 mm) to 12 (spacing of 72 mm), the ability of subjects to localize accurately the point of stimulation decreased from 97% to 74% correct. Most of the decrement in performance occurred when the number of motors exceeded eight. In these localization experiments, the frequencies of

stimulation have typically been between 80 and 250 Hz. Over this range, there has been no effect of frequency on the ability of subjects to localize a point of stimulation. It is also relevant to note in this context that vibrotactile sensitivity is not related to the ability to localize a point of stimulation. Vibrotactile thresholds have been determined to be quite uniform across a skin surface with marked variation in vibrotactile localization (Cholewiak and Collins, 2003). In a number of contexts, two-dimensional vibrotactile displays have been used for tactile cuing and the results from these experiments have demonstrated that the ability to localize a point of stimulation critically depends on the spacing between motors (Chen et al., 2008; Lindeman and Yanagida, 2003). When the spacing between motors mounted in a 3  3 array on the back is 60 mm, participants can identify the location of a vibrotactile stimulus on 84% of the trials (Lindeman and Yanagida, 2003). However, with a denser array of motors arranged in a 4  4 configuration with a distance between motors of 60 mm in the horizontal direction and 40 mm in the vertical direction, Jones and Ray (2008) found that participants identified the location of a single vibrotactile stimulus on only 59% of the trials. Most errors involved mislocalization by a single motor, and so when the responses were coded in terms of localizing stimulation to within one motor of the one activated, the overall response rate was 95% correct. In that experiment, participants were more accurate in identifying the correct column of activation (87% correct) than the row (68% correct), which presumably reflected the closer spacing of motors in the vertical as compared to horizontal direction. It appears that the increase in the number of motors in the display and the smaller distances between motors both contributed to the decline in performance when compared to that of Lindeman and Yanagida (2003). The ability to localize a point of vibrotactile stimulation on the back does not appear to reflect limitations in its dynamic spatial acuity. The two-point threshold

120

for discriminating vibratory stimuli on the torso is 10–11 mm, and is the same independent of whether the stimuli are delivered simultaneously or successively (Eskildsen et al., 1969). One factor that may influence the ability to localize precisely a vibrotactile stimulus on the skin is the surface wave created by activation of the motor. Skin is a viscoelastic material that absorbs some of the energy imposed on it during vibration and transmits some energy in the form of a surface wave. The shearing forces produced diminish at a rate proportional to the inverse square of the distance from the vibration source. Stroboscopic illumination has been used to measure the velocity of the surface waves created by vibrotactile stimulation and has been estimated to range from 2 to 40 m/s (Franke, 1951). The exact velocity depends on a number of variables including the frequency of vibration, skin temperature, and the constituency of the underlying tissue. Even though the intensity of these waves diminishes with distance, vibrations applied to the finger can sometimes be seen traveling up the arm many centimeters from their source

(Cholewiak and Collins, 2003). This means that surface waves could excite afferent units some distance from the site of skin stimulation, possibly contributing to errors in tactile localization. Attempts to reduce this effect have involved placing a rigid surround around the moving contactor (e.g., see C2 tactor in Fig. 2) to dampen the traveling waves. Because surface waves may contribute to errors in localization and in perceiving tactile patterns, characterizing their properties is important to the design of tactile communication systems. An experiment was conducted to measure the properties of traveling waves evoked by vibrotactile stimulation so that the optimal spacing of motors in a display used for localization could be estimated. The pancake motors used in tactile displays were glued (liquid bandage) to the skin on the forearm (170 mm from wrist crease) and a 3-axis accelerometer (Analog Devices ADXL335) was glued at 30-mm intervals from the motor (Fig. 4). Accelerations were sampled at 10 kHz (by a NI 9215 16-bit ADC) while the motor was activated at 3.3 V which produced 100

Acceleration (m/s2)

10

1

0.1

3 axis accelerometer 0.01 0

10

20

30

40

50

60

70

80

90 100

Distance from motor (mm)

Fig. 4. (a) Motor glued on the forearm with 3-axis accelerometer mounted 30 mm from its center. (b) Acceleration measured on the motor and at 30 mm intervals in the x (red), y (blue), and z (green) axes (circles), together with the noise levels of each axis measured (squares).

121

a vibration of approximately 90 Hz. Activation of the motor created a surface wave on the skin primarily in the plane of vibration of the motor as illustrated in Fig. 4. The acceleration decreased in amplitude with distance, and at 60 mm from the source, it was attenuated to less than 10% of the initial value. For this type of motor, a distance of at least 60 mm between motors would be required for precise localization of the vibrotactile input. The results from this experiment indicate that surface waves produced by vibrotactile stimulation travel over considerable distances when considered in terms of the spatial configuration of tactile displays. For this type of motor, the amplitude of vibration is such that it could be mislocalized as arising from a source up to 40 mm from its origin. It seems plausible that much of the difficulty encountered in identifying the locus of a vibrotactile stimulus in a two-dimensional tactile display results from the spacing between motors. If spatial cues are to be presented on large sensory surfaces such as the back or abdomen, then the distance between motors should be maximized.

Tactile pattern identification Spatial cuing is one of a number of functions for which tactile communication systems have been shown to be effective at different sites on the body (Cholewiak and Collins, 2003; Jones and Ray, 2008; Van Erp, 2005, 2008). Tactile displays also provide the opportunity to present more complex cues to the user. There has been a considerable number of experiments that have examined how tactile vocabularies based on tactons can be developed (see section “Tactons”). In our laboratory, a series of experiments has been conducted using tactile displays based on small inertial actuators (pancake motors) to evaluate the ability of subjects to identify tactile patterns presented around the waist, and on the forearm and lower back. The objectives of this research

were to compare tactile pattern recognition at three sites on the body that could be used to provide tactile cues, and to ascertain the characteristics of patterns that were more difficult to identify by examining the confusion matrices of the participants’ responses. The tactile displays were based on pancake motors that vibrate by rotating a mass in a plane parallel to the surface on which the motor is mounted (Fig. 2). The motors are encased in plastic (Smooth-On, Inc.) to make them more robust and increase the contact area (300 mm2) between the skin and motor. A wireless tactile control unit (WTCU) was designed and fabricated to control the motors. The WTCU has two main components, a wireless transceiver module for communication with a notebook computer and a microcontroller that receives commands from the wireless module (Bluetooth) and translates them into sequences of motor actuation. Each pulse activation of the motors lasts 500 ms, followed by a delay of 500 ms. Two motor driver integrated circuits in the WTCU can be programmed to drive up to 16 motors in a display. The circuit was designed to make the most efficient use of the energy available to it (Jones et al., 2006). A Visual Basic.NET interface is used to send signals from the notebook computer to the WTCU. The tactile patterns presented at each site varied with respect to the spatial location and temporal sequence of stimulation, and the number of motors concurrently active. They were designed to represent directional cues that could be used for navigation and simple instructions as illustrated in Fig. 5. The tactile display for the waist comprised eight encased motors that were mounted on a band with a strip of Velcro sewn along its length so that their positions could be adjusted for each participant (see Fig. 5). The tactile array was placed tightly around the waist with the motors positioned over the spine, navel, above the hip on the left and right side of the torso, and at the midpoints between these four locations. The intermotor spacing ranged from

122 4 3

3

2

2 1

1

1

1

1

2

2

2

3

3

3

1

2

3

4

1

2

3

4

1

2

3

4

1

2

3

4

Fig. 5. Tactile displays and schematic representation of tactons presented on the waist, forearm, and back. In each tacton, the shading, numbers, and arrows all indicate the spatial and temporal sequence of activation of the motors.

80 to 100 mm across subjects. The forearm display comprised a 3  3 array of motors that was mounted on the volar surface of the forearm with an intermotor distance of 24 mm in both the longitudinal (proximal–distal) and transverse (mediolateral) directions (Fig. 5). A 4  4 array of motors mounted on a spandex waist band was used for the back display (Fig. 5); the intermotor spacing was 40 mm vertically and 60 mm horizontally. Eight different patterns were presented five times in a random order on the arm and back, and five patterns were presented five times around the waist. Each of the three experiments was conducted with a different group of 10 participants. Prior to the experiment, participants were shown a visual representation of the patterns that could be presented (similar to those depicted in

Fig. 5) and were told that the numbers, colors, and arrows all represented ways of illustrating the spatial and temporal sequence of motor activation. The experimental protocol was explained and they were then familiarized with the tactile patterns, each of which was presented three times during the training period. After the third presentation of each pattern, participants could ask for any pattern to be repeated. The visual representation of the patterns could be viewed at all times during the experiment. Using this template, participants indicated which pattern was presented tactually on each trial. The results were analyzed in terms of the percentage of correct responses for each pattern as a function of site stimulated and in terms of static information transfer (IT). The latter measures the increase in information about a signal transmitted that results from knowledge of the received signal and so provides a quantitative measure of the amount by which uncertainty has been reduced (Tan et al., 1999, 2010). Metrics based on IT such as static IT or IT rate have the advantage of providing unit free measures of human performance and so can be compared meaningfully across different displays. There was a ceiling effect in the ability of participants to identify patterns in this series of experiments with the group mean correct responses averaging 99% on the waist, 89% on the forearm, and 99% on the back. Further analyses of the results from the experiments involving the forearm for which the greatest number of errors occurred, revealed that tactile patterns that involved sequential activation of the motors up or down the arm in a longitudinal (distal–proximal) direction were harder to identify (63% correct) than those that involved sequential activation in the transverse (mediolateral) direction (89% correct). This difference occurred even though the distances between the motors were the same in both directions. Moreover, if the tactile pattern involved two directions, for example, across and up the arm, performance was even poorer as shown in Fig. 6.

123

100

3

2

1

3

2

1

3

2

1

Percent correct

80 60 40

1

1

1

2

2

2

3

3

3

3

3

2

2

2

1

2

1

1

20 0

Transverse

Longitudinal Combined Tactile pattern direction

Fig. 6. Group mean percentage of correct responses in identifying tactile patterns as a function of the direction that the motors were activated on the forearm. Representative tactile patterns are illustrated above each bar.

These findings indicate that tactile patterns that limit the course of motor activation within a pattern to a single direction will be more accurately identified. They further demonstrate the anisotropies of tactile perception on the arm that have been observed in other tasks and may explain the present findings. In their experiments on tactile spatial acuity and anisotropy on the hand and arm, Gibson and Craig (2005) found that anisotropy occurred only on tasks that relied in spatial cues (e.g., gap detection thresholds) and that it was most prominent on the forearm and upper arm, where gap detection thresholds were 2.35 (forearm) and 1.64 (upper arm) times higher in the mediolateral as compared to the proximal–distal orientation. Suprathreshold stimuli, such as the perception of distance on the skin, are also susceptible to anisotropic effects. Green (1982) reported that the distance between two points of static tactile stimulation on the forearm is perceived to be greater (by about 70%) when presented transversely as compared to longitudinally. In addition, he noted that judgments of distances presented in the transverse orientation approached veridicality whereas there was spatial compression of distance in the longitudinal

direction. The difficulty in perceiving the features of tactile patterns presented along the forearm in the present experiment may result from the spatial compression of distances which makes it harder to discern the spatial characteristics of the patterns. It is of interest to note that there is no effect of the direction of sequential activation of motors on pattern recognition on the back, even when a larger set of tactile patterns (15) is presented (Jones et al., 2007). A reanalysis of the results from that experiment indicated that there was no difference in the performance of participants when identifying patterns in which the spatial sequence of motor activation was horizontal (96% correct) as compared to those that were vertical (95% correct), even though the distance between the motors was smaller in the vertical direction. Consistent with these results is Green's (1982) finding that the orientation of the tactile stimulus had no effect on perceived distance on the stomach (and presumably the back), in contrast to its effect on distance perception on the arm and thigh. The effect of stimulus orientation on spatial acuity and the perception of distance have been hypothesized to reflect a number of

124

factors including asymmetries in receptive fields, the presence of orientation-selective neurons in somatosensory cortex, and the contribution of anatomical landmarks such as joints to perceived distance (Cholewiak, 1999; Essock et al., 1997). Tactile anisotropies do not appear to be a consequence of dermatomal boundaries as they occur within regions such as on the index finger or thenar eminence which are encompassed within one dermatome (Gibson and Craig, 2005). An additional factor that may influence the perception of tactile patterns presented at different sites on the body is the spatial acuity of the site. The distance between two points of stimulation is perceived to be greater on areas of the body with higher spatial acuity, such as the hand, as compared to those with lower acuity, for example, the arm, an effect known as Weber's illusion (Anema et al., 2008; Cholewiak, 1999; Green, 1982). Tactile communication systems are often evaluated in terms of their information transmission capabilities or communication efficiency (Rabinowitz et al., 1987; Summers et al., 1997, 2005). One limitation of measures such as task completion time, percent correct, or discrimination thresholds is the inability to use them to compare performance with different devices. IT measures are usually independent of task conditions (e.g., number of stimuli in an identification task) and indicate the attainable information transmission as the dimensionality of a tactile or haptic display increases. The maximum likelihood estimate of IT, ITest, can be calculated by approximating underlying probabilities with frequencies of occurrence: ITest ¼

k X k  X ni;j i¼1 j¼1

n

 log2

nni;j ni nj



where n is the total number of trials in the experiment, nij is the number of times the joint stimulus–response pair P P (SiRj) occurs, and ni ¼ kj¼1 nij and nj ¼ ki¼1 nij are the row and column sums, respectively. These quantities are

calculated from the confusion matrices of participants’ responses (Tan et al., 1999). The IT measured using the above equation was calculated for the tactile pattern identification experiments with displays mounted around the waist and on the forearm and back. The average IT was 2.21, 2.15, and 2.68 bits for the waist, forearm, and back, respectively. An entity related to IT is 2IT which can be interpreted as indicating the number of tactile patterns that can be identified. For these experiments, 2IT was 4.63, 4.44, and 6.41 patterns, respectively. The IT value measured with the display mounted on the back (2.68) is very similar to the 2.78 bits reported by Cholewiak and McGrath (2006) who used a 24-motor display on the abdomen. The similarity in these values suggests that these IT estimates represent the attainable information transmission capabilities for the torso. They are considerably smaller than the bits of information potentially available which is 4.9 bits for a 24-motor array. Estimates of IT for displays on the hand are generally larger, although the higher spatial and temporal acuity of the hand does not translate into remarkably superior performance. For vibrotactile stimuli presented to a single digit, Rabinowitz et al. (1987) measured an IT of 1–2 bits for each dimension of the stimulus (i.e., intensity, frequency, and stimulator area) that they varied, with an IT of 4–5 bits for all three dimensions. Lower IT values were reported by Summers et al. (2005) for vibrotactile stimuli presented to the wrist that were based on timevarying sequences which were presumably more difficult to perceive. In the latter experiment, IT values of 0.6 and 1.05 bits were measured for stimuli created using frequency modulation alone or frequency plus amplitude modulation. The stimuli used in the present experiments may be considered two-dimensional, that is, they varied with respect to spatial sequence of motor activation and the number of motors activated. Higher IT values may possibly be achieved by increasing the dimensionality of the stimuli and the number of tactile patterns presented.

125

Conclusion The development of effective tactile communication systems involves identifying what types of signals can be recognized with minimal training and determining how information that has traditionally been conveyed visually or aurally can be represented tactually. It is also important to determine the optimal configuration and location of the displays used to present this information. Some of these issues have been addressed in the context of evaluating tactile patterns (tactons) that provide spatial information that could be used to assist in navigation (e.g., for the visually impaired) and to provide instructions to individuals working in hazardous environments or under degraded visual conditions (e.g., fire fighting, undersea diving, during military operations). Tactons presented on the waist, forearm, and back are easily learned and participants experience little difficulty in associating a pattern with an arbitrary meaning (Jones et al., 2009; MacLean and Enriquez, 2003). The present experiments revealed that tactile patterns created by varying the spatial location and temporal sequence of motor activation, and the number of motors concurrently active can be accurately identified and interpreted. Although two-dimensional displays have greater bandwidth for communication in that there are more dimensions available to present information, one-dimensional displays are effective for simple instructions and for providing spatial cues about the environment. Studies of tactile pattern perception reveal the importance of understanding the properties of the somatosensory system and their marked variation across the body. Features of tactile processing such as spatial compression, skin anisotropies, and spatial-temporal interactions must be considered carefully in designing and evaluating tactons. Tactile stimuli presented at one site and in one orientation (e.g., the arm) will not necessarily be encoded in a similar manner at another site (e.g., the back) or in another orientation (Green, 1982). In addition, the perceived

distance between two points on skin can vary dramatically across the body, both as a function of variations in spatial acuity and when the temporal parameters of stimulation change. In concluding, it is important to note that tactile communication systems have the potential to have an impact well beyond the provision of spatial information. Over the past decade, the availability of small, low-cost, low-power vibrating motors has led to renewed interest in developing wearable technology for those with sensory impairments (for review, see Jones and Sarter, 2008). This includes the visually impaired (tactile displays for navigation and graphical data visualization), the hearing impaired (tactile displays to aid speech comprehension), and those with vestibular dysfunction (tactile displays to provide feedback about body tilt and posture). Each of these application domains benefits from a better understanding of the tactile sensory communication capabilities of human users and how multidimensional tactile signals are processed. However, many questions remain unanswered about the most effective way of communicating via the skin. For the visual and auditory senses, it is clear that with increased dimensionality of a display there is an increase in the amount of information that can be received by the user (Lockhead, 1972). Much less is known about multidimensional tactile displays, and how different cues can be effectively combined to create tactons from tactile phonemes.

Acknowledgments This research was supported by a grant to the author from the National Science Foundation (NSF). References Anema, H. A., Wolswijk, V. W. J., Ruis, C., & Dijkerman, H. C. (2008). Grasping Weber's illusion: The effect of receptor density differences on grasping and matching. Cognitive Neuropsychology, 25, 951–967.

126 Bach-y-Rita, P., Collins, C. C., Saunders, F. A., White, B., & Scadden, L. (1969). Vision substitution by tactile image projection. Nature, 221, 963–964. Bliss, J. C., Katcher, M. H., Rogers, C. H., & Shepard, R. P. (1970). Optical-to-tactile image conversion for the blind. IEEE Transactions on Man–Machine Systems, MMS-11, 58–65. Bolanowski, S. J., Gescheider, G. A., & Verrillo, R. T. (1994). Hairy skin: Psychophysical channels and their physiological substrates. Somatosensory and Motor Research, 11, 279–290. Brewster, S. A., & Brown, L. M. (2004). Tactons: Structured tactile messages for non-visual information display. Proceedings of the fifth Australian user interface (AUIC2004) (pp. 15–23). Brewster, S. A., & King, A. (2005). The design and evaluation of a vibrotactile progress bar. Proceedings on first joint EuroHaptics conference and symposium on haptic interfaces virtual environment teleoperator systems (pp. 499–500). Brown, L. M., Brewster, S. A., & Purchase, H. C. (2005). A first investigation into the effectiveness of tactons. Proceedings of first joint EuroHaptics conference and symposium on haptic interfaces for virtual environment teleoperator systems, (pp. 167–176). Chen, H.-Y., Santos, J., Graves, M., Kim, K., & Tan, H. Z. (2008). Tactor localization at the wrist. Proceedings of EuroHaptics, (pp. 209–218). Cholewiak, R. W. (1999). The perception of tactile distance: Influence of body site, space, and time. Perception, 28, 851–875. Cholewiak, R. W., Brill, J. C., & Schwab, A. (2004). Vibrotactile localization on the abdomen: Effects of place and space. Perception & Psychophysics, 66, 970–987. Cholewiak, R. W., & Collins, A. A. (1995). Vibrotactile pattern discrimination and communality at several body sites. Perception & Psychophysics, 57, 724–737. Cholewiak, R. W., & Collins, A. A. (2003). Vibrotactile localization on the arm: Effects of place, space, and age. Perception & Psychophysics, 65, 1058–1077. Cholewiak, R. W., & McGrath, C. (2006). Vibrotactile targeting in multimodal systems: Accuracy and interaction. Proceedings of symposium on haptic interfaces for virtual environment teleoperator systems (pp. 413–420). Cox, T. F. (2001). Multidimensional scaling. Boca Raton, FL: Chapman and Hall. Craig, J. C. (1972). Difference threshold for intensity of tactile stimuli. Perception & Psychophysics, 11, 150–152. Enriquez, M., MacLean, K. E., & Chita, C. (2006). Haptic phonemes: Basic building blocks of haptic communication. Proceedings of the eighth international conference on multimodal interfaces, (pp. 150–152). Eskildsen, P., Morris, A., Collins, C. C., & Bach-Y-Rita, P. (1969). Simultaneous and successive cutaneous two-point threshold for vibration. Psychonomic Science, 14, 146–147.

Essock, E. A., Krebs, W. K., & Prather, J. R. (1997). Superior sensitivity for tactile stimuli oriented proximally-distally on the finger: Implications for mixed class 1 and class 2 anisotropies. Journal of Experimental Psychology: Human Perception and Performance, 23, 515–527. Franke, E. K. (1951). Mechanical impedance measurements of the human body surface. Dayton, OH: Wright-Patterson Air Force Base Air Force Technical Report No. 6469. Geldard, F. A. (1957). Adventures in tactile literacy. The American Psychologist, 12, 115–124. Geldard, F. A. (1960). Some neglected possibilities of communication. Science, 131, 1583–1588. Geldard, F. A. (1968). Pattern perception by the skin. D. R. Kenshalo (Ed.), The skin senses (pp. 304–321). Springfield, IL: C.C. Thomas. Gescheider, G. A., Bolanowski, S. J., Pope, J. V., & Verrillo, R. T. (2002). A four-channel analysis of the tactile sensitivity of the fingertip: Frequency selectivity, spatial summation, and temporal summation. Somatosensory and Motor Research, 19, 114–124. Gescheider, G. A., Wright, J. H., & Verrillo, R. T. (2009). Information-processing channels in the tactile sensory system. New York: Taylor & Francis. Gibson, G. O., & Craig, J. C. (2005). Tactile spatial sensitivity and anisotrophy. Perception & Psychophysics, 67, 1061–1079. Green, B. G. (1982). The perception of distance and location for dual tactile pressures. Perception & Psychophysics, 31, 315–323. Ho, C., Reed, N., & Spence, C. (2006). Assessing the effectiveness of “intuitive” vibrotactile warning signals in preventing front-to-rear-end collisions in a driving simulator. Accident Analysis & Prevention, 38, 989–997. Ho, C., Tan, H. Z., & Spence, C. (2005). Using spatial vibrotactile cues to direct visual attention in driving scenes. Transportation Research Part F: Traffic Psychology and Behaviour, 8, 397–412. Jones, L. A., & Held, D. (2008). Characterization of tactors used in vibrotactile displays. Journal of Computing and Information Sciences in Engineering, 8, 044501-1–044501-5. Jones, L. A., Kunkel, J., & Piateski, E. (2009). Vibrotactile pattern recognition on the arm and back. Perception, 38, 52–68. Jones, L. A., Kunkel, J., & Torres, E. (2007). Tactile vocabulary for tactile displays. Proceedings on second joint EuroHaptics conference and symposium on haptic interfaces virtual environment and teleoperator systems (pp. 574–575). Jones, L. A., & Lederman, S. J. (2006). Human hand function. New York: Oxford University Press. Jones, L. A., Lockyer, B., & Piateski, E. (2006). Tactile display and vibrotactile recognition on the torso. Advanced Robotics, 20, 1359–1374. Jones, L. A., & Ray, K. (2008). Localization and pattern recognition with tactile displays. Proceedings of symposium on

127 haptic interfaces for virtual environment teleoperator systems (pp. 33–39). Jones, L. A., & Sarter, N. B. (2008). Tactile displays: Guidance for their design and application. Human Factors, 50, 90–111. Kaczmarek, K. A., & Bach-y-Rita, P. (1995). Tactile displays. W. Barfield & T. A. Furness (Eds.), Virtual environments and advanced interface design (pp. 349–414). New York: Oxford University Press. Kaczmarek, K. A., & Haase, S. J. (2003). Pattern identification as a function of stimulation current on a fingertip-scanned electrotactile display. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 11, 269–275. Kaczmarek, K. A., Webster, J. G., Bach-y-Rita, P., & Tompkins, W. J. (1991). Electrotactile and vibrotactile displays for sensory substitution systems. IEEE Transactions on Biomedical Engineering, 38, 1–16. Lee, J., Ryu, J., & Choi, S. (2009). Vibrotactile score: A metaphor for designing vibrotactile patterns. Proceedings on third joint EuroHaptics conference and symposium on haptic interfaces virtual environment and teleoperator systems (pp. 302–307). Lévesque, V., Pasquero, J., & Hayward, V. (2007). Braille display by lateral skin deformation with the STReSS2 tactile transducer. Proceedings on second joint EuroHaptics conference and symposium on haptic interfaces virtual environment teleoperator system (pp. 115–120). Lévesque, V., Pasquero, J., Hayward, V., & Legault, M. (2005). Display of virtual Braille dots by lateral skin deformation: Feasibility study. ACM Transactions on Applied Perception, 2, 132–149. Lindeman, R. W., & Yanagida, Y. (2003). Empirical studies for effective near-field haptics in virtual environments. Proceedings on IEEE virtual reality conference (pp. 287–288). Lindeman, R. W., Yanagida, Y., Noma, H., & Hosaka, K. (2006). Wearable vibrotactile systems for virtual contact and information display. Virtual Reality, 9, 203–213. Lockhead, G. R. (1972). Processing dimensional stimuli: A note. Psychological Review, 79, 410–419. MacLean, K. E. (2009). Putting haptics into the ambience. IEEE Transactions on Haptics, 2, 123–135. MacLean, K., & Enriquez, M. (2003). Perceptual design of haptic icons. Proceedings of EuroHaptics (pp. 351–363). Mahns, D. A., Perkins, N. M., Sahai, V., Robinson, L., & Rowe, M. J. (2006). Vibrotactile frequency discrimination in human hairy skin. Journal of Neurophysiology, 95, 1442–1450. Morley, J. W., & Rowe, M. J. (1990). Perceived pitch of vibrotactile stimuli: Effects of vibration amplitude, and implications for vibration frequency coding. Journal of Physiology, 431, 403–416. Mortimer, B. J. P., Zets, G. A., & Cholewiak, R. W. (2007). Vibrotactile transduction and transducers. The Journal of the Acoustical Society of America, 121, 2970–2977.

Peterka, R. J., Wall, C., & Kentala, E. (2006). Determining the effectiveness of a vibrotactile balance prosthesis. Journal of Vestibular Research, 16, 45–56. Rabinowitz, W. M., Houtsma, A. J. M., Durlach, N. I., & Delhorne, L. A. (1987). Multidimensional tactile displays: Identification of vibratory intensity, frequency, and contactor area. The Journal of the Acoustical Society of America, 82, 1243–1252. Reed, C. M., & Delhorne, L. A. (1995). Current results of a field study of adult users of tactile aids. Seminars in Hearing, 16, 305–315. Roberts, J. C., & Franklin, K. (2005). Haptic glyphs (Hlyphs)—Structured haptic objects for haptic visualization. Proceedings of first joint EuroHaptics conference symposium on haptic interfaces for virtual environment and teleoperator systems (pp. 369–374). Scott, J. J., & Gray, R. (2008). A comparison of tactile, visual and auditory warnings for rear-end collision prevention in simulated driving. Human Factors, 50, 264–275. Spelmezan, D., Jacobs, M., Hilgers, A., & Borchers, J. (2009). Tactile motion instructions for physical activities. ACM Computer Human Interaction Conference (CHI) (pp. 1–10). Summers, I. R., Cooper, P. G., Wright, P., Gratton, D. A., Milnes, P., & Brown, B. H. (1997). Information from timevarying vibrotactile stimuli. The Journal of the Acoustical Society of America, 102, 3686–3696. Summers, I. R., Whybrow, J. J., Gratton, D. A., Milnes, P., Brown, B. H., & Stevens, J. C. (2005). Tactile information transfer: A comparison of two stimulation types. The Journal of the Acoustical Society of America, 118, 2527–2534. Tan, H. Z., Durlach, N. I., Reed, C. M., & Rabinowitz, W. M. (1999). Information transmission with a multifinger tactual display. Perception & Psychophysics, 61, 993–1008. Tan, H., Gray, R., Young, J. J., & Traylor, R. (2003). A haptic back display for attentional and directional cueing. Hapticse: The Electronic Journal of Haptics Research, 3, 11 June. Tan, H. Z., Reed, C. M., & Durlach, N. I. (2010). Optimum information-transfer rates for communication through haptic and other sensory modalities. IEEE Transactions on Haptics, 3, 98–108. Ternes, D., & MacLean, K. (2008). Designing large sets of haptic icons with rhythm. Proceedings of EuroHaptics (pp. 199–208). Van Erp, J. B. F. (2001). Tactile navigation display. S. Brewster & R. Murray-Smith (Eds.), Haptic HCI 2000 (pp. 165–173). Berlin: Springer-Verlag. Van Erp, J. B. F. (2005). Presenting directions with a vibrotactile torso display. Ergonomics, 48, 302–313. Van Erp, J. B. F. (2008). Absolute localization of vibrotactile stimulation on the torso. Perception & Psychophysics, 70, 1016–1023. Van Erp, J. B. F., van Veen, A. H. C., Jansen, C., & Dobbins, T. (2005). Waypoint navigation with a vibrotactile

128 waist belt. ACM Transactions on Applied Perception, 2, 106–117. Verrillo, R. T. (1973). Vibrotactile intensity scaling at several body sites. F. Geldard (Ed.), Cutaneous communication systems and devices (pp. 9–14). Austin, TX: Psychonomic Society. Verrillo, R. T., & Chamberlain, S. C. (1972). The effect of neural density and contactor surround on vibrotactile sensation magnitude. Perception & Psychophysics, 11, 117–120.

Verrillo, R. T., Fraioli, A. J., & Smith, R. L. (1969). Sensation magnitude of vibrotactile stimuli. Perception & Psychophysics, 6, 366–372. von Békésy, G. (1962). Can we feel the nervous discharges of the end organs during vibratory stimulation of the skin? The Journal of the Acoustical Society of America, 34, 850–856. Wall, C., III& Weinberg, M. S. (2003). Balance prostheses for postural control. IEEE Engineering in Medicine and Biology Magazine, March/April, 84–90.

A. M. Green, C. E. Chapman, J. F. Kalaska and F. Lepore (Eds.) Progress in Brain Research, Vol. 192 ISSN: 0079-6123 Copyright Ó 2011 Elsevier B.V. All rights reserved.

CHAPTER 9

Understanding haptics by evolving mechatronic systems Gerald E. Loeb{,*, George A. Tsianos{, Jeremy A. Fishel{, Nicholas Wettels{ and Stefan Schaal{,{ {

Department of Biomedical Engineering, University of Southern California, Los Angeles, California, USA { Department of Computer Science, University of Southern California, Los Angeles, California, USA

Abstract: Haptics can be defined as the characterization and identification of objects by voluntary exploration and somatosensory feedback. It requires multimodal sensing, motor dexterity, and high levels of cognitive integration with prior experience and fundamental concepts of self versus external world. Humans have unique haptic capabilities that enable tool use. Experimental animals have much poorer capabilities that are difficult to train and even more difficult to study because they involve rapid, subtle, and variable movements. Robots can now be constructed with biomimetic sensing and dexterity, so they may provide a suitable platform on which to test theories of haptics. Robots will need to embody such theories if they are ever going to realize the long-standing dream of working alongside humans using the same tools and objects. Keywords: tactile sensing; haptics; robots; perception; cognition; learning; sensorimotor control; dexterity. denote the intersection of these usages as “the identification of properties of objects via voluntary exploration and somatosensory feedback.” The entity doing the identification can be human or machine, the properties of the object can be physical or aesthetic, somatosensory includes both tactile and proprioceptive modalities from any body part (or their machine equivalents), and the interaction must involve active movement. Natural scientists usually seek to understand a phenomenon by reducing it to its component parts. Psychologists have catalogued the

Introduction The term “haptics” has been used variously by psychologists to mean the science of the sense of touch, by computer technologists for tactile feedback from an electronic device, and by aesthetic philosophers to denote affective responses to manual exploration. We shall use it here to *Corresponding author. Tel.: þ213-821-5311 E-mail: [email protected] DOI: 10.1016/B978-0-444-53355-5.00009-9

129

130

exploratory behaviors made by human adults and infants when confronted by a novel object and neuroscientists have identified and characterized the receptors activated by such behaviors (Jones and Lederman, 2006). Nevertheless, we are lacking what David Marr called a “theory of computation” (Marr, 1982). In recent years, fairly strong theories of computation have been developed for many aspects of perceptual and even cognitive behavior. These usually derive from systematic recordings of neural activity from various brain regions of highly trained animals performing tasks and correlations of such activity with various parameters of the performance. This strategy is difficult to apply to haptics because few animals have anything like the manual dexterity of humans. For haptic behaviors that are feasible for animals, the movements and forces between the digits and objects are difficult to capture and the steps in the complex sequences tend to be variable and uncontrollable by the experimentalist. The difficulties of a reductionist approach to haptics can be appreciated by comparison with locomotion, another class of sensorimotor behaviors. The mechanics and control of walking in animals are now fairly well understood because reproducible behavior is easily generated and measured in both intact and reduced preparations (McCrea and Rybak, 2008). Some of the principles have been successfully incorporated into robots that perform reasonably well (Buchli and Ijspeert, 2004). The process of building machines that incorporate principles of operation of living organisms is called biomimetic design. We propose that a theory of haptics can be developed and tested by starting with biomimetic robots and attempting to emulate the behaviors and capabilities of human subjects. Recent advances in mechatronics (engineered systems combining mechanical components with electronic control) have made it possible to emulate the mechanical behavior of biological hands and limbs (Delcomyn, 2007). Sensors built into the actuators and mechanical linkages provide the equivalent of proprioceptive

information. Impedance control of the actuators can be used to emulate the natural compliance of biological limbs (Hogan, 1984; Pratt et al., 1996). Multimodal tactile information can be provided by a new sensor array that mimics the mechanical properties and robustness of a human fingertip (Wettels et al., 2008). It may be possible and perhaps even necessary to emulate the hierarchical structure of the biological nervous system, in which the brain formulates high-level strategies and tactics and the spinal cord coordinates the action of muscles and regulates the interactions with external objects (Raphael et al., 2010). If the controller of a mechatronic equivalent of a biological system could use a theory of computation to achieve humanlike haptic performance, this would be suggestive that the brain may be using a similar theory of computation. It should also be possible to apply such knowledge to the design of neural prosthetic systems to restore dexterity to patients with paralysis or amputation of their arms (Fig. 1).

Components of haptic behavior The motor strategies that humans use to explore, characterize, and identify objects have been catalogued by psychologists (Jones and Lederman, 2006; Lederman and Klatzky, 1987). These strategies seem reasonable in view of the various somatosensory modalities that have been identified neurophysiologically, but they fall far short of a theory of computation for haptic behavior and they provide no insight into how they developed in the first place.

Development of internal representations of objects We shall start with the assumption that the developing nervous system has little or no preconceived information about itself or the other entities that comprise its world and must instead

131 Dexterous neural prosthetic systems Patient

Control system

Motor command interface

Sensory perception interface

Signal processing

coordination algorithms

Paralyzed limb

BIONic spindles

Neurogram

NMES

Prosthetic limb Actuators

the development of internal representations is known to depend on actual experience, often associated with “critical periods” of neonatal development and plasticity (Crair et al., 1998; Hubel and Wiesel, 1970). We shall here assume that the objects to be represented centrally have an identity that is static, but this identity must be built up from highly dynamic interactions and it includes information required to predict dynamic behavior of the object.

Identification of self

Mechanical linkage

Tactile sensors

Fig. 1. Restoration of haptic function in patients with paralysis or amputation of the upper extremity requires both bidirectional interfaces with sensory and motor signals and a control system that can integrate those signals with command signals from the patient to provide coordination and rapid adjustments. Neuromuscular electrical stimulation (NMES) interfaces in the paralyzed limbs could utilize injectable, wireless microdevices called BIONs (Loeb et al., 2001; Sachs and Loeb, 2007).

self-organize its internal representations by detecting patterns of coherence in the incoming sensory information and its own outputs. This is the starting assumption for the unsupervised neural networks that were modeled in the early days of artificial intelligence (Haykin, 1999; Hebb, 1949). It is now clear that a certain amount of signal processing is genetically hardwired into the physical structure of primary sensory receptors and their associated local circuitry, which presumably predisposes the perceptual nervous system to start with specific salient features. Nevertheless,

The first thing that a self-organizing brain is likely to recognize is that some, but not all, of the sensory information coming back is contingent on motor signals that the brain sends out, initially randomly and eventually purposefully. The contingent sensory data are associated with the existence of a self; sensory data that are constant or inconsistently modified by efferent signals constitutes evidence of external entities, which are defined later. The representation of self is essentially a mapping between efferent signals and proprioceptive and visual afferent signals that represents the set of movements that the organism can make. Efferent and afferent feedback In order to recognize the correlations between efferent and afferent signals, both must be available as inputs to a perceptual center. This explains why efferent projections are accompanied by recurrent projections that are organized similarly to afferent sources. Internal reference frames Much has been written about the coordinate frames used by various parts of the brain to represent the relationships of the self to the external world. Some of them appear to be inherent in the structure of sense organs (e.g., retinotopic and cutaneous maps), others are likely to arise as the CNS detects robust correlations among the senses (e.g., extrapersonal

132

space as fusion of visual information with gaze direction and proprioceptive feedback of posture), and some are purely mathematical creations designed to account for psychophysical data (e.g., shoulder-centered kinesthesia; Soechting and Flanders, 1989). One likely outcome of adaptive neural networks is that they can extract more-or-less orthogonal representations of whatever principal components are present in their input data, but this begs the question of what the extracted coordinates might represent. If a given dataset can be adequately represented by a particular orthogonal coordinate frame (an eigenvector), it can also be equally well represented by other coordinate frames that are rotations of the first in eigenspace. At the early stages of sensory information processing, the coordinate frames of the neural representations may bear a strong resemblance to the physical structure of the sense organs from which the input signals are derived, but they will become increasingly abstract at higher levels where inputs are multimodal. The motor cortex integrates highly abstracted representations from many areas of association cortex plus relatively low-level direct input from proprioceptors. Thus, it is not surprising that simple attempts to correlate its output signals or its topography to simple physical coordinates based on either sensory input (extrapersonal space) or motor output (muscles) have been frustrating (Churchland and Shenoy, 2007). The notion of first identifying “self” suggests that the internal coordinates of many, if not most, cortical areas will be different from each other but will reflect a combination of both afferent and efferent information. Such hybrid coordinate frames do not have simple physical or mathematical analogs and may vary from subject to subject (see below), so we have no guesses to correlate with neural activity. If they exist, we must first identify the neural processes and hierarchies that lead to their emergence.

Definition of surfaces Once the organism has a repertoire of movements in extrapersonal space, it is in a position to recognize visual signals that do not change as a result of such movement (except occlusions by self) but that give rise to somatosensory and perhaps auditory signals when the movement reaches the place of those visual signals. Thus, the first externality to be represented in the brain seems likely to be the notion of surfaces that obstruct otherwise free limb movements.

Definition of rigid objects Once the organism knows how to make contact with surfaces in extrapersonal space, it is in a position to recognize that contact between itself and a subset of those surfaces can result in changes in subsequent visual appearance and somatosensory feedback from those surfaces. This is the definition of a movable object as opposed to a fixed surface. By systematically reorienting its view of a rigid object, the brain can associate all the different patterns of sensory feedback that can be produced by that object, forming a fused percept. By systematically grasping and exploring the object, the brain can include in this percept information about weight, inertia, friction, thermal properties, etc.

Definition of deformable objects Once the organism can recognize and manipulate rigid objects, it can appreciate that certain types of object have an even richer set of visual and somatosensory feedback depending on which of a limited set of states they occupy. Those states, in turn, may depend on the history of interactions with the object (e.g., hingelike motion) or the forces being applied (e.g., elastic deformation).

133

Definition of tools Once the organism can recognize and manipulate any object, it is in a position to cause interactions between objects and surfaces, for example, using one object to hit and cause noise and/or movement of another object or surface. This is the definition of rudimentary tool use.

Definition of materials An organism that can recognize objects that have a closed set of possible appearances can then recognize interactions that cause an object to change its appearance permanently. This is the definition of breaking or otherwise permanently changing an object, which renders it a material that can be used to create other objects (e.g., flaking stone tools, molding clay pots). Infants commonly handle known objects aggressively, as if they are fascinated by whether or not they fall into the category of things that can be broken.

Integrating exteroceptive senses In the developmental sequence hypothesized above, visual information is only one of many types of sensory and efference copy information from which the structure of the self and the external world must be defined. By contrast, most roboticists start with sophisticated machine vision and use that as the basis for motor planning and sensory fusion. Infants are relatively slow to develop visual acuity, relying instead on tactile information from both fingers and mouth (Gibson, 1988). Adults who are blind from birth develop sophisticated representations of objects and dexterity in manipulating and characterizing them (Marks, 1983; Smitsman and Schellingerhout, 2000). What they do lack is a sense of how the visual appearance of objects changes with point of view (Heller et al., 1996). This is actually a problem rather than an attribute

of vision, and much of machine vision and presumably human vision is concerned with overcoming this problem so as to maintain a veridically fused representation of individual objects. Thus, it seems both easier and more appropriate for these high-level visual capabilities to emerge spontaneously based on the perceived utility of visual information rather than as a preordained organizing principle for the brain. The sounds made by objects and their interactions are rarely considered in robotics but they appear to be of great interest to infants. Sound contains valuable information that may be difficult to extract from other sensory modalities and it tends to be integrated centrally with other sensory modalities including touch (Bresciani and Ernst, 2007). Most obviously, sound provides precise temporal information, which can be useful for understanding kinetic interactions during touch. It also provides a valuable hint about the mutability of objects—articulated objects often change shape silently or with repeatable sounds whereas breaking an object is often accompanied by a nonrecurring sound. Attributes such as hollowness may be impossible to perceive in any other way. The vibrotactile spectrum (up to  800 Hz) substantially overlaps the acoustic spectrum (20–20,000 Hz). Both may use temporospatial cross-correlation to extract useful information (Johansson and Flanagan, 2009; Loeb et al., 1983), but neither biological mechanism is well understood. Nevertheless, acoustic information is easy to acquire electronically and to preprocess in a way that conveys at least some of its biological saliency. Testing new entities against internal representations Once the organism has defined all major classes of entities in its world and has a reasonable library of sensorimotor patterns associated with previously encountered entities in each class, then its daily existence and continued development depend on two complementary capabilities:

134 



Recognizing when a newly encountered entity is sufficiently similar to a previously known entity to treat them as identical. Recognizing when a newly encountered entity is sufficiently dissimilar to all previously known entities to warrant creation of a new item in the library.

These decisions can be seen as forced-choice outcomes of Bayesian decision making, in which the probability of picking one or the other usually depends on fragmentary data and a complex set of prior information about the probability of a known entity being present, the importance of not making a mistake, and the cost of obtaining additional information. This cost has at least two dimensions, reflecting the energetic cost of the exploratory movement and its execution time. For many of the haptic discrimination tasks facing a hunter-gatherer, execution time will be critical—foraging time is limited and prey whose own motor behaviors cannot be identified and

countered tends to escape. Bayes’ theorem is useful for decision making once the new data are in hand (Kording, 2007), but it must be extended to account for the decision about which data to pursue based on the relative costs and the expected discriminative value of the new data. This depends on having an accurate and immediately accessible representation of all the possible associations between exploratory movements and the sensory data that they are likely to yield (Fig. 2). This suggests that the internal representation of the properties of objects in the brain is not in the usual canonical coordinates that we define in physics (e.g., mass, rotational inertia, hardness, friction, etc.), but rather in the associational coordinates of learned exploratory movements and the critical raw sensory signals associated with them. For example, the representation of mass of a given object is the association between the parameters of a hefting exploratory movement and the proprioceptive and tactile data resulting from hefting that object. In fact, it is possible that the goal of hefting

Fig. 2. Haptic exploration to identify an object can be conceptualized as an interactive sequence of hypothesis testing designed to collect specifically those data Di that are most likely to shift the prior probabilities P(A), P(B) sufficiently to conclude that the object matches an object that produces similar data when subjected to those exploratory strategies.

135

is to obtain similar sensory feedback from all objects, in which case the discriminative information is actually in the parameters of the hefting movement itself (e.g., amount of motor effort required to produce a given acceleration of the object). Similarly, the shape of an object obtained from contour following may be reflected in the sequence of exploratory movements required to keep the fingertip following a feature and thereby generating constant tactile feedback. The choice of exploratory movement also involves the decision to stop exploration and settle on whatever is the currently most likely identification. There are many circumstances when erroneous “lumping” with previous experience is “good enough,” which accounts for a wide range of illusions, beloved of psychophysicists, magicians, and trial lawyers. When the brain insists that all is not familiar even after exhaustive exploration, the problem becomes one of creating a new percept that is associated with and based upon the closest known match. This leads to both a parsimonious extension of the coordinates of the existing internal representations and associations that facilitate access to previously developed strategies for handling similar but different entities. It is important to recognize that this incremental learning about the world is likely to give rise to internal representations of the same entity that are quite different among individuals, because they depend on the sequence of all other entities to which each individual was exposed during its prior experience.

Hierarchical system of sensorimotor control The nervous system may start out “tabula rasa” but it is not free from genetically determined structures that evolved because they promote and accelerate the functionality that must develop ontogenetically. Each part of the nervous system tends to start with certain types of computational elements (neurons) whose various properties are

specified. The initial interconnections among them appear to follow certain general rules and the rules for modifying that connectivity as a result of experience appear to be quite specific. In sensory systems, the sequence of embryological development gives rise to topographic gradients that are related to physical coherence in the signals that will later be experienced (e.g., retinotopic, cochleotopic, and somatotopic maps), so it has been natural (but not necessarily useful; see below) to interpret the higher levels of perceptual processing in those same coordinate frames. Organizing the motor system from periphery upward is appealing because the peripheral elements effectively define the computational problem that must be solved at higher levels of motor planning, but the major information flow is necessarily in the opposite direction from sensory systems. This discussion considers what is known about the structure and relationships from the bottom up but from the perspective of motor function, which is top-down.

Design of the spinal regulator Industrial robots use hierarchical control that is usually divided into a movement planning level and a servocontrol level. For the last 50 years, sensorimotor neurophysiologists have tried to interpret the brain and spinal cord as analogous to those respective engineering subsystems, with little success. As details of the anatomical circuitry and physiological signals have been elucidated, it has become clear that they are not consistent with such an interpretation. The circuitry in the spinal cord is much more complex and integrative than an engineering servocontrol and the signals in the brain correlate only loosely with many different aspects of the output behavior. The spinal cord may function more like another, more sophisticated engineered system— a programmable regulator (He et al., 1991; Loeb al., 1990). Recently, models of et

136

neuromusculoskeletal systems based on the known circuitry of the spinal cord have been shown to have highly useful emergent properties that were completely unexpected (Raphael et al., 2010). The biological regulator consists of component circuits with specific combinations of input and output signals that have evolved and remained stable over many millions of years of vertebrate evolution. The brain can “program” the gains of these elements via relatively simple descending command signals so that the highly phasic and coordinated sequences of muscle activations required to perform each task are generated by the nonlinear combinations of these commands and ongoing sensory feedback. Despite the large number of gains that must be set, simple gradient descent algorithms converge rapidly and inevitably to stable solutions that perform similarly to human subjects doing the same tasks. The set of interneurons defines a high dimensional space that is rich in “good enough” local minima. When started from random gains, the controller quickly “discovers” a nearby local minimum defined by the cost function used for training. Because the output is generated by circuits that include rich combinations of sensory feedback, the solutions automatically handle whatever types of noise, perturbations, or complex loads were included in the training set. Since the pioneering work of Nicolas Bernstein in the 1930s, both motor psychologists and roboticists have been concerned with resolving the problem of “redundancy,” in which there are more articulations or actuators than necessary to perform a given task (Bernstein, 1967). The programmable regulator turns this problem around, greatly expanding the redundancy at the interneuronal level to ensure that a single system can generate a virtually unlimited repertoire of desirable behaviors depending on the goals at hand. Evolutionary success is defined by rapidly finding good enough solutions rather than meticulously computing globally optimal solutions. We intend to demonstrate that robots controlled by similar programmable regulators can achieve similarly

robust functionality. We can compare their strengths and weaknesses with state-of-the-art compliant controllers based on more traditional trajectory planning and servocontrol. The problem with applying a regulator to a robot is that we do not have any blueprints. The properties of most mechatronic systems are so different from biological musculoskeletal systems that the patterns of interneurons known in the spinal cord cannot be applied to the robot. So we will have to recapitulate the evolution of the spinal regulator by using a genetic algorithm (Ijspeert, 2001). Genetic algorithms create and evolve structures incrementally, keeping components that improve performance and rejecting those that do not. In this case, the performance criterion is the speed and security with which a simple gradient descent controller can learn to perform a repertoire of diverse tasks using each new generation of regulator. Because the mechanical dynamics and sensors of the robot are actually substantially simpler to model than their biological counterparts, it should be feasible to evolve a fairly sophisticated regulator using a fast PC.

Learning to perform tasks The redundancy problem is not really a problem for the nervous system because, unlike most robotic controllers, it does not try to compute analytical solutions to new problems. Rather the brain starts with one of the motor programs that it already knows (or a random output if an infant) and gradually refines that program until it gets what it wants. Given a reasonable repertoire of motor skills, most of the problem posed by a new task is the perceptual one of recognizing similarities to something already in the repertoire. The process of refinement by trial-and-error is made efficient and reliable by the many, good enough local minima of the regulator. This is distinctly different from servocontrollers, whose gains and input commands are more critical and are better set by analytical solutions for optimal control (Todorov and Jordan,

137

2002). Given a suitable regulator and a suitable cost function defining a task, it should be straightforward to train a model of cortical motor control. To achieve performance similar to a human, the cost function should probably include both kinematic terms related to accuracy and energetic terms related to effort. Interestingly, the spinal-like regulator seems predisposed to identifying motor strategies that result in minimal coactivation of actuators (Raphael et al., 2010), similar to the patterns seen in well-learned tasks performed by biological subjects.

Cortical consolidation Surprisingly, after adequate performance has been achieved in a motor task, the controller in the brain continues to introduce large, apparently random fluctuations in its behavior (Churchland et al., 2010). We can surmise that the next level of controller upward in the hierarchy (e.g., premotor cortex) is performing its own trial-anderror learning to minimize some cost function. What might that cost be? From the well-described plasticity of cortical representations, we can surmise that the computational resources of the cortex are finite and perhaps a limiting factor in the repertoire and resolution of motor tasks that can be learned. When good enough performance is first achieved, the number of cortical motoneurons contributing to the net gains in the spinal regulator is likely to be much larger than necessary. This is because those net gains are the result of diverse excitatory and inhibitory functions controlled directly by each corticospinal motoneuron as well as indirectly via extrapyramidal subsystems that are also ultimately under the control of the cerebral cortex. By gradually adjusting the commands to the motor cortex, the premotor cortex should be able to reduce overlapping and conflicting activity. Shrinking the cortical neural representation of a given task frees up computational machinery to learn another task. The robustness of the spinal regulator, with its

many good enough local minima, makes it relatively easy and safe to employ trial-and-error learning in all stages.

Cortical representations Because there are many good enough solutions to common motor tasks, the repertoire of solutions that a given brain has at its disposal is likely to be rather different from another brain that has learned to achieve similar nominal performance. If a previously learned task is complicated by new loads, perturbations, or accuracy requirements, those different starting strategies may have different utilities and consequences. Psychophysicists studying motor learning usually create learning curves by averaging the data from many different subjects with similar starting skills, but these smooth, averaged curves do not reflect the apparently random, stepwise processes that actually occur in the individual subjects (Gallistel et al., 2004). They also fail to account for sport coaching practices that are required to force athletes out of idiosyncratic, well learned but ultimately suboptimal habits. By utilizing trial-and-error learning of good enough behaviors instead of analytical optimal control, biological systems have accepted a Faustian bargain that roboticists have traditionally rejected: biological systems can rapidly achieve acceptable and stable performance with noisy components but are unlikely ever to achieve globally optimal performance. One consequence of that bargain is that the signals that can be recorded from behaving biological systems can never be fully understood by correlating them with the predictions of analytical engineering tools (e.g., Churchland and Shenoy, 2007). The deeper one dives from the unavoidable requirements of physics into the more arbitrary details of neural representations, the more the data become colored by the unique and essentially random experiences of the individual subject. This fundamental limit to experimental reductionism provides the rationale for our alternative approach

138

based on simulation and synthesis. We choose haptics over other motor behaviors that have already been studied reductionistically because haptics requires rich sensorimotor integration and it has little reductionist baggage to discard.

Development and testing of Haptic robots Mechatronic platform A great variety of robotic arms and hands for both research and industrial applications have been developed over many years (Table 1). The more anthropomorphic systems of late tend to be more expensive and more fragile, both because of their complexity and their limited production for research. It is not clear what level of verisimilitude is necessary to generalize lessons learned from a haptic robot to a haptic human or vice versa.

Biomimetic tactile sensing A variety of technologies have been used in tactile sensors (Table 2), but commercially available tactile sensors tend to be limited to relatively coarse arrays of normal-force sensors based on compression of elastic materials. In fact, most of the commercially available hands listed in Table 1 are not supplied with any tactile sensing. Many technologies are difficult to apply to the curved, deformable “skin” that facilitates grip and few are able to resist damage in the electromechanically hostile environments in which hands are often used (moisture, grit, sharp edges, etc.). One promising new candidate is the BioTacÒ, a biomimetically designed, multimodal array that provides most of the dynamic range of human tactile sensing for location, magnitude, and vector direction of contact forces, microvibrations associated with slip and textures, and thermal flux resulting from contact with objects that differ in

thermal effusivity (Lin et al., 2009; Raphael et al., 2010; Wettels et al., 2008, 2009) (Fig. 3).

Compliant control algorithms Many of the exploratory movements that underlie human haptics involve force or impedance control rather than position control, meaning that the trajectory of the limb results from a dynamic interaction between the robot and objects that it encounters. Most robots use highly geared DC and stepping motors that generate whatever torque is required to produce precise movements. When external forces are applied to them, they behave in a stiff rather than a compliant manner. In order to behave compliantly, these motors must be fitted with torque or force sensors that actively modify the position commands to which they are responding. If these control loops and motors are sufficiently fast, the system can achieve a reasonable approximation of the compliant control that arises naturally from biological muscles, which generate forces that depend on position and velocity of movement. Recently, such control principles have been organized into Dynamic Movement Primitives (DMP; Schaal, 2006), a system of nonlinear differential equations whose parameters correspond to the speed and accuracy criteria that are typically applied to biological affordances (Pastor et al., 2009; Schaal, 2007).

Visual targeting Much haptic behavior in humans tends to start with visual information about an object of interest and its location in extrapersonal space. If the object is familiar, this information alone may be sufficient to identify the object and its expected handling properties. If not, it may provide convenient, albeit occasionally misleading, starting assumptions (e.g., large objects tend to be heavy, shiny surfaces tend to be slippery, etc.). Location,

139 Table 1. Summary of mechatronic hands Research group

Tactile sensing technology

DOF

Utah/MIT hand USC/Belgrade hand Honda hand Hirzinger hand NASA robonaut 2 GiFu III hand Southampton hand Stanford/JPL hand UB III hand Smart hand Dist hand DLR II hand Shadowhand Yokoi III iLIMB pulse LMS hand BUAA hand Zurich/Tokyo hand Torino hand RCH-I hand MA-I hand SARAH hand RTR II hand Vanderbilt hand ACT hand Barrett hand Vecna HG2 Heidelberg fluid hand LMS hand Anybots Monty hand Tuat/Karlsruhe hand Ultralight Elumotion–Sheffield hand Orebro University Sweden Manus Colobi TBM hand Otto Bock Michelangelo 2

Capacitive normal force Normal-force-sensitive resistors Normal-force-sensitive resistors Normal-force-sensitive resistors 6-DOF load cells in fingertip Pressure-sensitive conductive ink Piezoelectric polymer film 6-axis strain gauge 6-axis IT sensors Triaxial MEMS array Triaxial force, slip sensors Conductive polymer grid Quantum tunneling composite Force sensitive resistors None Unknown Unknown Force sensitive resistors Unknown Unknown None None None None None None None None None None None None None None None None None

16 20 2 12 24 16 4 9 16 16 16 13 24 13 16 16 13 Unknown Unknown 16 10 9 9 24 4 6 Unknown 16 18 20 13 20 12 3 Unknown 2

* denotes currently available commercially; þ denotes currently under development by research teams. http://www.davidbuckley.net/RS/HandResearch.htm http://asimo.honda.com/asimo-specs/ http://robonaut.jsc.nasa.gov/default.asp http://www-arts.sssup.it/newCyberhand/smarthand/index.htm http://www.dlr.de/rm/en/desktopdefault.aspx/tabid-3802/6102_read-8923/ http://www-lar.deis.unibo.it/activities/dexterousmanipulation/UBHandIII/index.html http://robot.gmc.ulaval.ca/en/research/theme304.html http://www.touchbionics.com/Pulse http://www.vanderbilt.edu/exploration/stories/bionicarm.html http://www.vecnarobotics.com/solutions/end_effectors/hg2.shtml http://haptic.buaa.edu.cn/English_dexteroushand.htm http://neurobotics.cs.washington.edu/projects.html http://www.barrett.com/robot/products-hand.htm http://www.elumotion.com/shefarm.html http://www.ottobock.com/cps/rde/xchg/ob_com_en/hs.xsl/32149.html?openteaser¼1

Comments

þ part of ASIMO robot þ þ þ Prosthetic prototype þ Follow-on from cyber hand þ þ * þ Tsukasa Kiko engineering * for prosthetic use þ þ þ

* þ Prosthetic prototype þ Pneumatic (peroxide) þ * * Hydraulic, hand camera þ þ Prosthetic prototype þ þ þ Prosthetic hand * for prosthetic use

140 Table 2. Summary of tactile sensors

Transduction method

No. of axis

Multimodal

Advantages

Disadvantages

Example

Capacitive

1

No

Small, very high resolution taxels, can be flexible, wide dynamic range, sensitive

Hysteresis, noise

Inductive

1

No

High sensitivity, repeatability

Resistive: deformable contact area

1

No

Flexible, thin

Complex, vulnerable construction, low spatial resolution Hysteresis

Pressure profile systems robotouch: http://www.pressureprofile. com/technology-capacitive.php Futai et al. (2003)

Resistive: conductive fabric Resistive: quantum tunneling composite Resistive: strain gauge

1

No

Flexible, robust, simple

1

No

Sensitive, wide dynamic range

Unable to resolve more than one contact point Hysteresis, gas absorption

6

No

Same as above

Bulky, expensive

Resistive: Piezoresistive conductive polymer Resistive: Piezo-MEMS Polymer-MEMs (multimodal)

1

No

Thin, low cost, simple

Hysteresis, stiff

6

Yes

Small, multielement

6

Yes

6-DOF force, temperature and heat flow, roughness

Piezoelectric

1

Yes

Detects dynamics for slip and texture

Optical: video processing

3

No

Very high resolution, sensitive

Optical: resistive Magneto-elastic

1

No

Flexible, low hysteresis

Large number of wires in workspace Large number of wires in workspace, wiring complexity Only detects dynamic events, thermal sensitivity Computationally intensive, sensitive to ambient light Complex fabrication

1

No

Very sensitive, low hysteresis

6

No

1

Yes

Robust, sensitive, low hysteresis Static and dynamic

Magnetoresistive Ultrasound

Sensitive to external magnetic fields Noisy High voltage, complex electronics

Inaba: Inastomer http://www. inaba-rubber.co.jp/en/ b_products/inastomer/index. html Pan and Zhu (2005)

QTC: http://www.peratech. com/

ATI: Nano 17 load cell: http:// www.ati-ia.com/products/ft/ sensors.aspx Tekscan Flexiforce: http:// www.tekscan.com/flexiforce. html Oddo et al. (2009), Beccai et al. (2005) Engel et al. (2006)

Dario et al. (1984), Howe and Cutkosky (1993) Hristu et al. (2000), Ohka et al. (2004) http://www.skilsens.com/index. html Mitchell et al. (1986) Hackwood et al. (1983) Brashford and Hutchins (1996), Grahn and Astle (1986)

141

Thermistor

External texture fingerprints

Rigid core

Impedance electrodes

Conductive fluid

Pressure sensor

Elastic skin

Fig. 3. A biomimetic, multimodal tactile sensor with the physical form and mechanical properties of a fused middle and distal phalanx consisting of a deformable skin inflated over a rigid core by a conductive fluid. Location, magnitude, and direction of contact forces can be extracted from changes in the pattern of electrical impedances measured through the conductive fluid by electrodes distributed over the surface of the core (visible through transparent skin in photo below). A pressure sensor connected to the fluid functions like a hydrophone to detect acoustic vibrations associated with slip or sliding over textured surfaces, which are enhanced by fingerprint-like ridges molded into the skin. The thermal material properties of objects contacting the finger can be assessed by a thermistor that measures heat flow from the heated core (Lin et al., 2009; Wettels et al., 2009). Sources: Upasani et al. (1999), Biagiotti et al. (2002), Puig et al. (2008).

size, and regions of interest for grasp or contour following can be extracted readily from stereovision. Machine vision has developed fairly sophisticated algorithms for all of these functions.

may need to be driven by received sensory data (e.g., contour following) or adjusted iteratively to fine-tune discrimination (e.g., repeated hefting to assess weight or stroking to assess texture).

Exploratory algorithms

Internal representations

As discussed above, the decision about which exploratory movement to employ at any given time depends on whatever prior information is currently available about the probable identity of an object and the property of the object that is most likely to distinguish it from other possible objects. After selecting and initiating a specific exploratory strategy, the details of the movement

In order to extract a typical canonical physical property of an object (e.g., weight, texture, etc.), any sensory data received during an exploratory movement would have to be deconvolved with the parameters of the exploratory movement in order to yield information specific to the object. In order to use canonical physical properties to inform the dexterous manipulation of that object,

142

the anticipated dynamic effects of those properties on the proposed manipulation would have to be computed, something that has been attributed to “internal dynamic models” (Imamizu et al., 2000; Kawato, 2008). Alternatively, the brain could represent objects as a set of learned associations in which each association includes both the output parameters of the exploratory movement (available from the many copies of efferent signals that project recurrently in the central nervous system) and the sensory data associated with the object being manipulated in that way (e.g., tactile, proprioceptive, visual, auditory; Pastor et al., 2009). This would facilitate the Bayesian strategy selection described above as well as the recall of appropriate motor strategies for dexterous manipulation of the object.

Conclusions Haptic behaviors do not lend themselves to the types of reductionist studies in animals that have been effective in revealing the neural computational algorithms that underlie other perceptual capabilities such as vision and hearing. Instead, it may be possible to develop and test theories of haptic computation by applying them to robotic platforms whose sensory and motor capabilities are increasingly biomimetic. For this to be effective, however, it may be necessary to recapitulate at least some of the early cognitive stages during which intelligent organisms develop representations of themselves and the external world. Those representations and strategies may be somewhat idiosyncratic, further emphasizing the importance of understanding the processes rather than the results. References Beccai, L., Roccella, S., Arena, A., Valvo, F., Valdastri, P., Menciassi, A., et al. (2005). Design and fabrication of a hybrid silicon three-axial force sensor for biomechanical applications. Elsevier, Sensors and Actuators, 120, 370–382.

Bernstein, N. (1967). The coordination and regulation of movement. New York: Pergamon. Biagiotti, L., Lotti, F., Melchiorri, C., & Vassura, G. (2008). How far is the human hand? A review on anthropomorphic robotic end-effectors. Bologna, Italy: University of Bologna. Brashford, A. G., & Hutchins, D. A. (1996). Characteristics of ultrasonic micromachined capacitance transducers in water. IEEE Ultrasonics, Ferroelectrics, and Frequency Control Society, 2, 955–958. Bresciani, J.-P., & Ernst, M. O. (2007). Signal reliability modulates auditory-tactile integration for event counting. Neuroreport, 18, 1157–1161. Buchli, J., & Ijspeert, A. J. (2004). Distributed central pattern generator model for robotics application based on phase sensitivity analysis. Biologically Inspired Approaches to Advanced Information Technology, 3141, 333–349. Churchland, M. M., Afshar, A., & Shenoy, K. V. (2010). A central source of movement variability. Neuron, 52(6), 1085–1096. Churchland, M. M., & Shenoy, K. V. (2007). Temporal complexity and heterogeneity of single-neuron activity in premotor and motor cortex. Journal of Neurophysiology, 97, 4235–4257. Crair, M. C., Gillespie, D. C., & Stryker, M. P. (1998). The role of visual experience in the development of columns in cat visual cortex. Science, 279, 566–570. Dario, P., De Rossi, D., Domenici, C., & Francesconi, R. (1984). Ferroelectric polymer tactile sensors with anthropomorphic features. IEEE International Conference on Robotics and Automation (1984), pp. 332–340. Delcomyn, F. (2007). Biologically inspired robots. Advanced Robotic Systems International and I-Tech, (in press). Engel, J., Chen, N., Tucker, C., Liu, C., Kim, S.-H., & Jones, D. (2006). Flexible multimodal tactile sensing system for object identification. 5th IEEE conference on sensors, pp. 563–566. Futai, N., Matsumoto, K., & Shimoyama, I. (2003). Simulation, fabrication and evaluation of microinductor-based artificial tactile mechanoreceptor embedded in PDMS. In IEEE MEMS, pp. 206–209. Gallistel, C. R., Fairhurst, S., & Balsam, P. (2004). The learning curve: Implications of a quantitative analysis. Proceedings of the National Academy of Sciences of the United States of America, 101(39), 13124–13131. Gibson, E. J. (1988). Exploratory behavior in the development of perceiving, acting, and the acquiring of knowledge. Annual Review of Psychology, 39, 1–41. Grahn, A. R., & Astle, L. (1986). Robotic ultrasonic force sensor arrays. Robot Sensors, 2, 297–315. Hackwood, G. B., Hornal, L. A., Wolfe, R., & Nelson, T. J. (1983). A torque sensitive tactile array for robotics. International Journal of Robotics Research, 2, 46–50.

143 Haykin, S. (1999). Neural networks: A comprehensive foundation. In (2nd ed.). Upper Saddle River, NJ: Prentice Hall. He, J., Levine, W. S., & Loeb, G. E. (1991). Feedback gains for correcting small perturbations to standing posture. IEEE Transactions on Automatic Control, 36(3), 322–332. Hebb, D. O. (1949). The organization of behavior. New York: Wiley. Heller, M. A., Calcaterra, J. A., Tyler, L. A., & Burson, L. L. (1996). Production and interpretation of perspective drawings by blind and sighted people. Perception, 25(3), 321–334. Hogan, N. (1984). An organising principle for a class of voluntary movements. Journal of Neuroscience, 4(11), 2745–2754. Howe, R. D., & Cutkosky, M. R. (1993). Dynamic tactile sensing: Perception of fine surface features with stress rate sensing. IEEE Transactions on Robotics and Animation, 9, 2. Hristu, D., Ferrier, N., & Brockett, R. W. (2000). The performance of a deformable-membrane tactile sensor: Basic results on geometrically-defined tasks. IEEE International Conf. on Robotics and Automation 1, pp. 508–513. Hubel, D. H., & Wiesel, J. N. (1970). The period of susceptibility to the physiological effects of unilateral eye closure in kittens. The Journal of Physiology, 206, 419–436. Ijspeert, A. J. (2001). A connectionist central pattern generator for the aquatic and terrestrial gaits of a simulated salamander. Biological Cybernetics, 84(5), 331–348. Imamizu, H., Miyauchi, S., Tamada, T., Sasaki, Y., Takino, R., Putz, B., et al. (2000). Human cerebellar activity reflecting an acquired internal model of a new tool. Nature (London), 403(6766), 192–195. Johansson, R. S., & Flanagan, J. R. (2009). Coding and use of tactile signals from the fingertips in object manipulation tasks. Nature Reviews. Neuroscience, 10, 345–359. Jones, L. A., & Lederman, S. J. (2006). Human hand function. New York: Oxford University Press. Kawato, M. (2008). From “Understanding the Brain by Creating the Brain” towards manipulative neuroscience. Philosophical Transactions of the Royal Society of London. Series B: Biological Sciences, 363, 2201–2214. Kording, K. (2007). Decision theory: What “should” the nervous system do? Science, 318, 606–610. Lederman, S. J., & Klatzky, R. L. (1987). Hand movements: A window into haptic object recognition. Cognitive Psychology, 19, 342–368. Lin, C.-H., Erickson, T. W., Fishel, J. A., Wettels, N., & Loeb, G. E. (2009). Signal processing and fabrication of a biomimetic tactile sensor array with thermal, force and microvibration modalities. ROBIO 2009 IEEE international conference on robotics and biomimetics. Loeb, G. E., Levine, W. S., & He, J. (1990). Understanding sensorimotor feedback through optimal control. Cold Spring Harbor Symposia on Quantitative Biology, 55, 791–803. Loeb, G. E., Peck, R. A., Moore, W. H., & Hood, K. (2001). BION system for distributed neural prosthetic interfaces. Medical Engineering & Physics, 23(1), 9–18.

Loeb, G. E., White, M. W., & Merzenich, M. M. (1983). Spatial cross-correlation. A proposed mechanism for acoustic pitch perception. Biological Cybernetics, 47(3), 149–163. Marks, L. E. (1983). Similarities and differences among the senses. The International Journal of Neuroscience, 19(1–4), 1–11. Marr, D. (1982). Vision. New York: W.H. Freeman & Co. McCrea, D. A., & Rybak, I. A. (2008). Organization of mammalian locomotor rhythm and pattern generation. Brain Research Reviews, 57(1), 134–146. Mitchell, E. E., DeMoyer, R., & Vranish, J. (1986). A new metglas sensor. IEEE Transactions on Industrial Electronics, IE-33, 166–170. Oddo, C. M., Beccai, L., Felder, M., Giovacchini, F., & Carrozza, M. C. (2009). Artificial roughness encoding with a bio-inspired MEMS-based tactile sensor array. Sensors, 9, 3161–3183. Ohka, M., Mitsuya, Y., & Takeuchi, S. (2004). Sensing characteristics of an optical three-axis tactile sensor under combined loading. Robotica, 22, 213–221. Pan, Z., & Zhu, Z. (2005). Flexible full-body tactile sensor of low cost and minimal output connections for service robot. Industrial Robot: An International Journal, 32, 485–491. Pastor, P., Hoffmann, H., Asfour, T., & Schaal, S. (2009). Learning and generalization of motor skills by learning from demonstration. IEEE International Conference on Robotics and Automation. Pratt, J., Torres, A., Dilworth, P., & Pratt, G. (1996). Virtual actuator control. IEEE/RSJ international conference on intelligent robots and systems. Puig, J. E. P., Nestor, E. N., Rodriguez, N., & Ceccarelli, M. (2008). A methodology for the design of robotic hands with multiple fingers. International Journal of Advanced Robotic Systems, 5(2), 177–184. Raphael, G., Tsianos, G., & Loeb, G. E. (2010). Spinal-like regulator facilitates control of a two-degree-of-freedom wrist. The Journal of Neuroscience, 30(28), 9431–9444. Sachs, N. A., & Loeb, G. E. (2007). Development of a BIONic muscle spindle for prosthetic proprioception. IEEE Transactions on Biomedical Engineering, 54(6), 1031–1041. Schaal, S. (2006). Dynamic movement primitives—A framework for motor control in humans and humanoid robotics. Adaptive Motion of Animals and Machines, 261–280 Part 6. Schaal, S. (2007). The new robotics—Towards human-centered machines. HFSP Journal, 1, 115–126. Smitsman, A. W., & Schellingerhout, R. (2000). Exploratory behavior in blind infants: How to improve touch? Infant Behavior & Development, 23, 485–511. Soechting, J. F., & Flanders, M. (1989). Sensorimotor representations for pointing to targets in three-dimensional space. Journal of Neurophysiology, 62(2), 582–594. Todorov, E., & Jordan, M. I. (2002). Optimal feedback control as a theory of motor coordination. Nature Neuroscience, 5 (11), 1226–1235.

144 Upasani, A. V., Kapoor, C., & Tesar, D. (1999). Survey of available sensor technology for robotic hands. Proceedings of the DETC 99, ASME design engineering technical conferences. Wettels, N., Fishel, J., Su, Z., Lin, C.-H., & Loeb, G. E. (2009). Multi-modal synergistic tactile sensing. Tactile sensing in

humanoids—Tactile sensors and beyond workshop, 9th IEEE-RAS international conference on humanoid robots. Wettels, N., Santos, V. J., Johansson, R. S., & Loeb, G. E. (2008). Biomimetic tactile sensor array. Advanced Robotics, 22(8), 829–849.

A. M. Green, C. E. Chapman, J. F. Kalaska and F. Lepore (Eds.) Progress in Brain Research, Vol. 192 ISSN: 0079-6123 Copyright Ó 2011 Elsevier B.V. All rights reserved.

CHAPTER 10

Technology improves upper extremity rehabilitation Jan Kowalczewski and Arthur Prochazka* Centre for Neuroscience, School of Molecular and Systems Medicine, University of Alberta, Edmonton, Alberta, Canada

Abstract: Stroke survivors with hemiparesis and spinal cord injury (SCI) survivors with tetraplegia find it difficult or impossible to perform many activities of daily life. There is growing evidence that intensive exercise therapy, especially when supplemented with functional electrical stimulation (FES), can improve upper extremity function, but delivering the treatment can be costly, particularly after recipients leave rehabilitation facilities. Recently, there has been a growing level of interest among researchers and healthcare policymakers to deliver upper extremity treatments to people in their homes using in-home teletherapy (IHT). The few studies that have been carried out so far have encountered a variety of logistical and technical problems, not least the difficulty of conducting properly controlled and blinded protocols that satisfy the requirements of high-level evidence-based research. In most cases, the equipment and communications technology were not designed for individuals with upper extremity disability. It is clear that exercise therapy combined with interventions such as FES, supervised over the Internet, will soon be adopted worldwide in one form or another. Therefore it is timely that researchers, clinicians, and healthcare planners interested in assessing IHT be aware of the pros and cons of the new technology and the factors involved in designing appropriate studies of it. It is crucial to understand the technical barriers, the role of telesupervisors, the motor improvements that participants can reasonably expect and the process of optimizing IHT-exercise therapy protocols to maximize the benefits of the emerging technology. Keywords: stroke; spinal cord injury; multiple sclerosis; upper extremity; telerehabilitation; in-home teletherapy; functional electrical stimulation; upper extremity rehabilitation. (Lloyd-Jones et al., 2009). Over 100,000 people living with spinal cord injury (SCI) have bilateral paresis or paralysis (NSCISC, 2010). People with tetraplegia due to SCI often depend on caregivers to perform the simplest manual tasks. Recovery of upper extremity function is their top priority, over all other disabilities (Anderson, 2004).

Introduction Three million stroke survivors in North America have unilateral paresis of the upper extremity *Corresponding author. Tel.: þ1-780-492-3783; Fax: þ1-780-492-1617 E-mail: [email protected] DOI: 10.1016/B978-0-444-53355-5.00010-5

147

148

A rigorous program of exercise therapy can improve upper extremity function after stroke and SCI (Drolet et al., 1999), and small improvements can make a large difference (Beekhuizen and Field-Fote, 2005). However, ensuring compliance to a regular exercise therapy program after people leave rehabilitation facilities is challenging. Clients may be given lists of exercises they should perform, but these tend to be boring and compliance drops off over time. Health-care systems cannot afford to pay for home visits by therapists to supervise exercise therapy. This unsatisfactory situation has given rise over the last few years to some new methods of delivering upper extremity rehabilitation. These include forced-use training, now known as constraint-induced movement therapy (Taub et al., 2006), computerized exercise devices such as the Nintendo Wii, robotic devices that apply forces to the arm to assist or resist movements (Volpe et al., 2009), therapeutic electrical stimulation and functional electrical stimulation (FES; Peckham and Knutson, 2005; Stein and Prochazka, 2009) and in-home teletherapy (IHT) supervised over the Internet (Gritsenko and Prochazka, 2004; Gritsenko et al., 2001; Kowalczewski et al., 2011; Krebs et al., 1998; Reinkensmeyer et al., 2011). IHT delivered to participants, particularly when combined with interventions such as FES, poses unique safety and legal challenges which must be resolved before the technology is made available to the larger population (Cooper et al., 2001). Nevertheless, with an aging population and everdecreasing technology costs, telesupervised rehabilitation will most likely provide an important alternative to traditional rehabilitation. Little information has been published on the obstacles that may arise when implementing IHT and how they can be overcome. In this chapter, we will discuss some of the technical problems we encountered in a recent clinical trial involving IHT and FES. The descriptions and opinions presented here are based on the experiences acquired in a

randomized clinical trial (NCT00656149, www. clinicaltrials.gov). In this trial, we compared two levels of FES-exercise therapy and IHT treatment in people with SCI. The main results of this study have been published elsewhere (Ellaway et al., 2010; Kowalczewski and University of Alberta, Centre for Neuroscience, 2009; Kowalczewski et al., 2011). General issues Clinical studies of rehabilitation treatments, particularly when these are telesupervised, differ from clinical trials in other fields. First, most researchers planning randomized controlled trials in rehabilitation, struggle with a suitable design, as in this field it is often difficult to provide quantitative outcome measures, blinded assessments, control treatments that are not obvious to the participants and therapists and even suitable randomization, given the variability between people with the same basic disability. It has been pointed out that in the whole field of rehabilitation, there has been only a handful of studies that have fulfilled all the criteria of high-level, evidence-based studies (Johnston et al., 2006). Second, the equipment required for IHT has to be designed in such a way as to withstand daily use for the duration of the trial, yet be intuitive and simple enough to be used by impaired participants without the constant presence of an able-bodied person to aid in the rehabilitation process. Further, IHT equipment linking a therapist to a participant has unique requirements compared to the commercial teleconferencing equipment primarily used for conference discussions. Third, if FES is involved, the FES equipment must be easy for the participant to don, doff, and control, and be safe and robust enough to survive rough handling in the home environment. Laboratory prototypes are generally not built with this in mind. Finally, if the therapy is performed at home in the absence of daily supervision, compliance over days and weeks is a major issue, requiring careful attention

149

to the entertainment value of the treatment. Therapists providing telesupervision should closely monitor fatigue as this tends to demoralize participants. Fatigue can easily be overlooked in a telerehabilitation setting. Evidence-based rehabilitation Only recently has the field of rehabilitation been examined according to the standards of evidencebased treatment through clinical meta-studies such as Spinal Cord Injury Rehabilitation Evidence (www.icord.org/scire) and the Evidence-Based Review of Stroke Rehabilitation (www.ebrsr. com). Rehabilitation therapists have been relatively slow in providing evidence-based care to clients compared to practitioners in other medical fields for a number of reasons. Rehabilitation relies heavily on customization; therapists frequently are faced with unique injuries and obstacles that require patient-specific adaptation of protocols and equipment. This customization is difficult to validate scientifically, unlike medical interventions in conditions with fewer variables. It has generally been difficult in rehabilitation to reach a consensus on the best course of a particular treatment, as it is often impossible to run properly blinded and controlled clinical trials. Unlike pharmacological trials, where placebos can be given to the control group of patients in a double-blind protocol, in rehabilitation trials, the rules of human experimentation require full disclosure to participants of the treatments to be compared (Boutron et al., 2007). If a given treatment is compared to “standard care” or a “placebo treatment,” participants quickly recognize whether they are in the treatment or control group and adjust their expectations accordingly. It is therefore preferable to compare different intensities of a given treatment, where the relative outcomes are genuinely not predictable (Kowalczewski et al., 2007a) or to compare two plausible treatments, for example, a novel intervention and a more conventional treatment that is additional to normal care (Mangold et al., 2009).

It is clear from the above that to run proper evidence-based rehabilitation trials, researchers have to be creative in designing and randomizing their trials (Komaroff and DeLisa, 2009). Rehabilitation also usually takes a substantial amount of time, so treating enough participants to achieve statistical significance can be extremely costly. It is therefore vital for the field that researchers concentrate on designing the best possible randomized controlled trials so as to maximize the chance of influencing clinical practice. Exercise equipment: efficacy, affordability, quantified outcomes Conventional exercise therapy has focused on the manipulation of simple objects such as blocks, stacking cones, therapy putty, and so on. Exercise therapy sessions are boring and in the absence of supervision, compliance falls off quickly, particularly at home. Performance is rarely if ever quantified. This began to change with the development of robotic rehabilitation devices instrumented with sensors (Krebs et al., 1998; Volpe et al., 2009). The simplest rehabilitation robots are motors that impose cyclical motion on extremities. They are commonly used in orthopedics (Salter, 1996) and occasionally in stroke and SCI (Dirette and Hinojosa, 1994). The MITManus (interactive-motion.com) is a robot that supports the arm and applies forces in the horizontal plane to assist or resist tracking of virtual objects on a computer monitor (Aisen et al., 1997; Hogan et al., 2006). A recent randomized controlled trial concluded that upper extremity function in chronic stroke subjects improved as much, though not more, after MIT-Manus robotic exercise therapy as after usual care by therapists (Lo et al., 2010). An editorial on the project concluded that the potential for robotic therapy after stroke was enormous (Cramer, 2010). The KINARM (bkintechnologies.com) is another example of a planar robotic device that supports the arm. The Motorika ReoGo (motorika.com)

150

is a telescopic device similar to a floor-shift gearstick, which applies forces to the hand in 3D space. None of these robots exercise dexterous hand movements. The Inmotion 3.0 wrist robot, the Inmotion 5.0 hand robot, and some experimental robots address this deficiency to some extent (Hesse et al., 2006; Lambercy et al., 2007; Popescu et al., 2000). The above devices all cost over $50,000 and so are unaffordable for IHT-exercise therapy. The only affordable robotic device, at around $7000, is the Columbia Scientific “Hand Mentor.” However, this device only exercises wrist and finger flexion–extension movements and ignores range of motion of the whole arm. Some groups have attempted to deliver motor rehabilitation in the home setting, with conventional exercises (Holden et al., 2007; Piron et al., 2004), therapeutic electrical stimulation (Alon et al., 2003; Sullivan and Hedman, 2007), or simple robotic devices (Johnson et al., 2008; Reinkensmeyer et al., 2002). It has become clear that great efforts must be made in designing usable rehabilitation equipment, as participants in such trials are not able-bodied. In two previous trials, we set out to design and test instrumented workstations suitable for in-home use (Gritsenko and Prochazka, 2004; Gritsenko et al., 2001; Kowalczewski et al., 2007a,b) but we found that there were significant deficiencies in the devices. The first device (Fig. 1a) comprised a desk with a number of instrumented objects chosen to represent household items: a spring-loaded doorknob, a handle attached via a cord and pulley to an adjustable set of weights, rectangular blocks and a cylinder designed to be transferred between two docking bays. This workstation had loose items that tended to be dropped, the layout made some of the items hard to reach, the device was bulky and not easily manufactured. In the next version (Fig. 1b), the items were mounted on a “Lazy Susan,” the idea being that the participant could rotate the device to bring a given item within easy reach. Unfortunately participants were not strong enough to rotate the assembly, loose objects still fell and the structure was even

more bulky. We concluded from these attempts that any equipment destined for use in a home setting would have to be of a size that suits the limited amount of space in participants’ homes; the motor tasks would have to take into account the participant's disabilities and residual motor skills and all items should be in easy reach. The next attempt comprised a table-mounted “suitcase model” comprising a set of “task modules” attached by compliant cables to docking ports (Fig. 1c). Each module could be pulled out of its docking port, positioned, and stabilized with the less affected hand. This version was significantly cheaper to manufacture and it was reasonably successful in tests on a number of stroke patients; however, a major limitation was the restriction of tasks to a horizontal plane. Further, the device was not suitable for people with bimanual deficits as it depended on tasks being stabilized with one hand while training the other. By now it had become clear that a portable device was needed that would present the user with tethered, instrumented objects that could move within the full 3D physiological workspace of the hand. Further, the device would ideally provide interesting games that would make exercise sessions enjoyable. We decided to adopt and extend the approach of Reinkensmeyer et al. (2002) who used a commercially available joystick in a computer gaming environment. The range of motion of consumer computer joysticks is very restricted. To extend the range, we fabricated a joystick that had a telescopic shaft and a gimble joint at its base (Fig. 1d). This combination allowed the top of the joystick, which held a number of manipulanda, to be moved through a larger volume. The joystick also contained a sliding card manipulandum designed to train lateral and palmar prehension and push–pull movements like those involved in inserting credit cards into automatic bank machines. Unfortunately, the volume of the workspace was restricted by the lengths of the lower and upper segments of the telescopic shaft, and thus remained significantly smaller than the

151 (a) (b)

(c)

Handle loading weight Docking bay

Blocks

Doorknob

Handle

Control box Cylinder

(d)

FES pushbuttons

(e)

(f)

Fig. 1. Six versions of an exercise therapy workstation developed for in-home teletherapy. (a) The device used in our first study in chronic stroke (Gritsenko and Prochazka, 2004; Gritsenko et al., 2001); (b) The device used in a study of people with subacute stroke; (c) A table-top system with task modules; (d) A telescopic joystick with attachments; (e) Prototype ReJoyce workstation used in a recent IHT study of people with SCI (Kowalczewski et al., 2011); (f) Final version of the ReJoyce system.

full physiological range of the hand of an ablebodied person. It was difficult to maintain the same grasp on the attachments at different shaft angles. Friction within the shaft at oblique angles resisted movement. The only dexterous task on this workstation was the card-sliding mechanism. Some users with restricted ROM could not reach the manipulanda at the top of the joystick, which at its shortest, was still 35 cm above the table surface. The main lesson learnt from this prototype was the importance of positioning the easiest tasks closest to the user so that they were accessible even to low-functioning users.

As a result of all these unsuccessful iterations, we finally settled on a workstation comprising a spring-loaded, segmented arm that presents the user with a variety of attachments representing activities of daily life. We called this the Rehabilitation Joystick for Computerized Exercise (“ReJoyce”). Ten prototype ReJoyce workstations were manufactured and used in our IHT study (Fig. 1e; Kowalczewski and Prochazka, 2010). Sensors in the arm and attachments of the ReJoyce provide signals that are used by the system's software to evaluate motor function and to control video games that exercise specific types

152

of hand movement. The device allows movement over the full physiological range of an able-bodied person. Each joint and attachment has a sensor that quantifies displacement or force. Signals from these sensors are used to control a suite of computer games which vary widely in subject matter and difficulty. They range from simple games that exercise whole-arm range of motion to games requiring coordinated movements of multiple joints, for example, grasp and squeeze, pinch and lift, grasp and rotate. The prototype was generally successful but not surprisingly, a number of hardware problems developed over time, requiring home visits for repairs or modifications. On two occasions, air freight was required and this involved assurances regarding the purpose and safety of the devices, which resulted in significant delivery delays. When devices were set up in participants’ homes, in some cases, the table clamps did not have a sufficient range of adjustment for different tables, finding space for the equipment could be problematic in small rooms and connecting to the Internet was occasionally challenging. The final implementation of the ReJoyce (Fig. 1f), its games and Internet software is the result of 6 years of experimentation. The system may be used with or without telesupervision (see below). FES equipment Early studies showed that therapeutic electrical stimulation can significantly reduce hypertonus and improve motor function in stroke survivors (Baker et al., 1979; Taylor et al., 1996; Waters et al., 1981). EMG-triggered FES with hand exercises has since been shown to have beneficial effects (Cauraugh and Kim, 2002; Chae, 2003; de Kroon et al., 2005; Francisco et al., 1998; Heckmann et al., 1997). FES-exercise therapy performed daily for several weeks has been shown to produce clinically significant improvements in hand function in subacute and

chronic stroke participants (Alon et al., 2007; Gritsenko and Prochazka, 2004; Kowalczewski et al., 2007a; Popovic et al., 2004, 2005). Several FES devices have been developed for foot-drop (Stein et al., 2006; Taylor et al., 1999; Vodovnik et al., 1981) and some hand FES devices have been developed recently (Hansen, 1979; Nathan, 1994; Prochazka, 1997; Prochazka et al., 1997; Weingarden et al., 1998). Currently, the only commercial FES hand stimulator is the Bioness H200 (Nathan, 1994; Weingarden et al., 1998). It comprises a hinged splint containing pad electrodes, and a separate stimulator triggered by push-button. It costs around $6000, which is out of the range of most potential IHT-exercise therapy clients; however, more affordable hand stimulators are on the horizon. In the IHT study upon which this chapter is based, two different stimulators were employed. The EMS 7500 surface stimulator was used for therapeutic electrical stimulation. It is an affordable, commercially available consumer device designed to deliver stimuli via self-adhesive gel electrodes placed over appropriate motor points. In our study, we found it necessary to mark the locations of the electrodes on the participants’ forearms to ensure accurate and repeatable placement. A permanent marker was used every 2 weeks to refresh the locations of the electrodes. The placement of the electrodes was most often performed by an aid or family member. After two or three sessions, the electrode gel lost its adherence and so participants were provided with neoprene straps that helped keep the electrodes in place after this occurred. The other stimulator was a new version of the “Bionic Glove” (Prochazka et al., 1997). The original device was a fingerless glove-like garment containing a built-in stimulator and wettable electrodes. It was controlled by wrist movements. The new version was controlled by the participant, who generated small tooth-clicks to advance the stimulator through a cyclical sequence of three states corresponding to hand opening, grasp, and relaxation (Simpson et al., 2008).

153

The tooth-clicks were detected by a wireless earpiece similar to a hearing aid. The hanger portion of the earpiece was looped over the ear and held a three-axis accelerometer that rested on the tragus, the small cartilage in front of the ear. When a tooth-click occurred, the earpiece sent a coded transmission to the stimulator in the participant's garment. This caused the stimulator to advance to the next state in the stimulation sequence. Initially the garment portion of the device was the same as in the Bionic Glove, covering the wrist, palm, and dorsal part of the hand. However, we found that even though the wrist section was compliant, it restricted hand movements. Because we had replaced the wrist movement sensor of the Bionic Glove with the earpiece controller, the palmar portion of the garment was no longer necessary and was replaced with a neoprene loop over the webspace between thumb and index finger which held the thumb adductor electrode in place. This was an improvement, as we have found that covering the palm, even with the use of high-friction materials, increases the likelihood of slip, for example, when grasping and pushing wheelchair rims. The electrodes were secured by Velcro to the inner surface of the garment, so that when the garment was donned, they were pressed on or close to the desired motor points. The electrode positions within the garment were reevaluated at every laboratory visit. The correct donning of the device required some practice as inappropriate positioning could lead to inadequate or ineffective stimulation. Unlike the adhesive electrodes used in the EMS 7500, the electrodes in the garment needed to be moistened with tap water before each use. Wetting and reconnecting the electrodes to the glove was most often performed by an aid or family member, especially in very low-functioning SCI participants. In contrast, most stroke participants have adequate function of their less affected hand to don and doff stimulator cuffs unaided. Some failures of the hand stimulator occurred initially because the connector between the stimulator and wristlet

was exposed and unreliable. This was solved by recessing the connector within a rubberized shoe or “galosh” attached to the wristlet. Teleconferencing equipment and software We found that one-on-one telesupervision using the Internet was possible with relatively modest software requirements on the supervisors’ and participants’ computers and recurring costs were negligible. Internet-based telerehabilitation requires a minimum of two computers connected reliably to the Internet, webcams, speakers, and microphones or headsets consisting of headphones and a microphone (recommended for echo-cancelation). Standard desktop or laptop computers were used in our study and they had sufficient processing power to handle the large video and audio streams and the custom processor-intensive games that were supplied with the ReJoyce system. We found that it was crucial to maintain a robust Internet connection. Most participants had a wireless router system in their homes. Remote access from the laboratory to these wireless routers was useful, as electrical storms, router resets and modem resets corrupted the wireless links on several occasions. Remote access generally allowed these problems to be overcome quickly, though on a few occasions a home visit was required. The majority of Internet-related difficulties emerged in the first few days of treatment. In our trial, we used two types of Virtual Network Computing software. The first, RealVNCÒ (realvnc.com), implemented a direct computerto-computer link. The second, LogmeinÒ (www. logmein.com), used a third party server to access the participant's computer, which was more convenient as it did not require the use and configuration of an Internet protocol address, nor the setting up of the participant's router to establish a connection. The drawback to using a third party server is the greater potential for a breach of security (see below).

154

Various teleconferencing software programs were available prior to this trial, but they were all designed for group interactions. None allowed a therapist to monitor a group of participants and to select particular participants for brief individualized discussions which the other participants did not hear. Software that did provide this ability was developed specifically for the ReJoyce system. It also allowed the therapist remotely to choose games, difficulty levels and the manipulanda involved, for each participant individually. It also allowed an automated, quantitative hand function test to be performed remotely and the data to be downloaded. Though data and audiovisual streams were encrypted, as with any telecommunication system, it was impossible completely to exclude the possibility of intruders intercepting audiovisual and data transmissions. However, because the custom software was very specialized, restricted to small groups and password-protected, the risk of interception was probably far less than that of public Internet telecommunications protocols, such as Skype, which is used by over 100 million people worldwide. Users and therapists alike were required to accept this risk. Precautions such as avoiding the use of names and places during teletherapy sessions were taken, and will probably remain advisable in all future systems. Not all computer games were of equal value in our rehabilitation trial. The types of computer games used in upper extremity rehabilitation vary from very simple games (Reinkensmeyer et al., 2002) to virtual reality simulation of real-world tasks with force feedback from haptic devices such as the Phantom robot (Adamovich et al., 2004; Boian et al., 2002). The primary role of a computer game in upper extremity rehabilitation is to increase compliance. Therefore, the games need to be entertaining. They also should ensure that the types of movements involved are beneficial and ideally the games should provide feedback to both the participants and therapists on performance. This feedback can be in the form of an overall score or the time taken to perform

a given task. An example of a successful device that provides numerous entertaining games is the Nintendo Wii, which has become popular in rehabilitation clinics for providing range of motion exercises of the whole upper body (Allen, 2007; Graves et al., 2008). However, the Wii was not designed for rehabilitation and its controller does not require or exercise manual dexterity. Nor are the movement signals that control the games available for analysis, though some researchers are working to change this. Games that provided this type of feedback to the users were among the most utilized by our telesupervisors and the most requested by the participants. Feedback on performance evidently provides a “hook” that plays on the user's competitive nature. Participants are more inclined to improve on their previous performance when they are provided with a measure of this performance and rewarded for a better outcome. Games that incorporated a reward mechanism had the highest rate of acceptance and usage. Our trial also suggested that improvements in upper extremity function did not require fully immersive 3D games. All the games developed for the ReJoyce were built with 2D game technology which avoids the need for 3D graphic acceleration, virtual reality displays, or haptics. It was clear, however, that variety in games was crucial. The more games the participant and supervisor could choose from, the higher the chance of the games continuing to appeal to the participants in many repeated sessions. Although preference in computer games has been related to gender in young adults (Lucas and Sherry, 2004) and children (Blumberg and Sokol, 2004), in our trial this was not an obvious outcome, though it would be interesting to study this more rigorously. Unlike many games designed only for entertainment, the games used in our trial were specifically designed for rehabilitating and retraining upper extremity movements. The advantages of using custom games included the ability to (1) train unique movements that could not be trained on preexisting consumer-oriented games; (2)

155

modify difficulty settings during the training process in order to optimize the therapy; (3) automatically compute difficulty settings on the basis of a hand function test score; (4) embed the games in the telerehabilitation software suite, minimizing the need to learn how to use noncustomized games; (5) design the games to be played with the various manipulanda on the workstation. Six games were developed and used in the trial: a car racing game, a gardening game, a boxing game, a timing game, a target shooting game, and a catching game. It has become clear in the course of our work that different treatment regimes are needed for different motor disorders. Thus, stroke survivors seemed to benefit most from the training of whole-arm range of motion and finger extension (hand opening) movements, whereas SCI participants benefitted most from training hand grasp and tenodesis pinch-grip. Outcome evaluation, effect of fatigue The results of the IHT study have been reported in detail elsewhere (Kowalczewski et al., 2011). The primary outcome measure was the widely used Action Research Arm Test (Lyle, 1981; Yozbatiran et al., 2008). Secondary outcome measures included (1) The ReJoyce Automated Hand Function Test; (2) Pinch force between thumb and fingers measured with a pinch gauge (B&L Engineering, Santa Ana, CA); (3) Grasp force measured with a rubberized, instrumented cylinder on the workstation. The ReJoyce Automated Hand Function Test was performed on the ReJoyce workstation with audiovisual prompts and reminders generated by interactive software. Sensors in the workstation provided signals that allowed quantitative scoring of the following variables: range of motion of the hand along three axes (in–out, up–down, left–right), the force of grasp and the amount of pronation and supination during grasp and key-grip. These variables were scored as a percentage of the mean

ranges achieved by able-bodied individuals. In addition, there were two functional placement tasks involving grasping, transferring, and releasing two manipulanda on the workstation, one mimicking a soft-drink can and the other a peg. The mean times taken to do several repetitions of these functional tasks were used to obtain a percentage score related to the performance of able-bodied individuals. The mean of all the scores obtained in the above tasks was computed, providing a single overall outcome score. The ReJoyce Automated Hand Function Test took about 5 min to perform. The scores from both the Action Research Arm Test and the ReJoyce Automated Hand Function Test improved significantly more during and after ReJoyce exercise therapy with FES than after conventional exercise therapy with therapeutic electrical stimulation (Fig. 2). Both protocols involved 6 weeks of 1 h/day IHT. Grasp force also increased more in the ReJoyce group. We concluded that FES-assisted exercise therapy on a ReJoyce workstation, supervised over the Internet with IHT, was feasible and effective. The improvements exceeded the minimal clinically important difference, which is often used as a criterion to introduce a treatment into best practice. We found that during IHT therapists had to be careful not to overexert participants involved in IHT-exercise therapy. In one case, a participant developed proximal muscle strain that required a rest period of 2 weeks for recovery. Fatigue can potentially aggravate preexisting pain and spasticity as well as cognitive and emotional disorders (Hammell et al., 2009). On the other hand, moderate physical activity in SCI has been shown to lower pain, fatigue, and depression (Tawashy et al., 2009). Therefore, it is important to judge the appropriate amount of exercise on an individual basis. Muscle fatigue was commented upon by all of the participants in our study. It is known that muscles lose fatigue resistance following SCI (Shields et al., 1997). The loss occurs fairly

156 (a) ARAT improvement (%)

25 20 15

ReJoyce Conventional ReJoyce (postexercise) Conventional (postexercise)

10 5 0 −5 Baseline

Week 2

Week 4

Week 6

Time (b)

RAHFT improvement (%)

30 25 20

ReJoyce Conventional ReJoyce (postexercise) Conventional (postexercise)

15 10 5 0 −5 Baseline

Week 2

Week 4

Week 6

Time Fig. 2. Hand function tests performed during biweekly laboratory visits by participants taking part in the IHT study (Kowalczewski et al., 2011). (a) Mean improvements over baseline in the Action Research Arm Test (ARAT). Solid and dashed lines: scores before and after 1-h of FES-ET performed in the laboratory. Notice that at 2 weeks, there was a significant difference (asterisk) between the pre-ET and post-ET scores, which we attribute to muscle fatigue. This was not seen in subsequent sessions at 4 and 6 weeks. (b) Mean improvements in the ReJoyce automated hand function test (RAHFT), showing a similar difference in preand post-ET scores at 2 weeks.

rapidly, is generally complete within 2 years of injury but is apparently not age-dependent (Shields et al., 2006). Loss of fatigue resistance can be reversed to some extent with repeated electrical stimulation commencing soon after injury (Shields and Dudley-Javoroski, 2006).

Traditionally, muscle fatigability is quantified by measuring the torque generated by an electrically stimulated muscle or group of muscles (Stein et al., 1992). In our IHT trial, no such quantitative measure was available, but it was commonly observed that most participants were unable to complete a full hour of FES-assisted exercise in the first week of ReJoyce training sessions. Initially, muscles would only respond to stimulation for a few minutes. This improved as the treatment progressed and by the second week all participants were able to generate functional movements with FES for the entire 1 h session. Interestingly, this effect was discernible in the scores of the Action Research Arm Test and ReJoyce Automated Hand Function Test when these were performed before and after 1 h exercise sessions (Fig. 2, solid lines: scores just before exercise therapy, dashed lines: scores just after exercise therapy). Paired Student's t-tests showed significant differences at week 2, but not thereafter, suggesting that the muscles had become more fatigue resistant by week 4. This correlated with spontaneous comments from the ReJoyce group in early training sessions that the games distracted them to the point that at the end of a session they felt that they had had a very vigorous workout. The control group rarely made this sort of comment. Experiencing rapid muscle fatigue following electrical stimulation in the initial period was discouraging for the participants. During the first 2 weeks, it is therefore important that participants be made aware that electrical stimulation can rapidly fatigue muscles but that with training the muscles build fatigue resistance. Conclusion Many recent studies have shown that exercise therapy can significantly improve upper extremity function after a stroke or SCI, which implies that currently the duration and intensity of exercise therapy provided to these individuals are insufficient. This is because of the cost and logistical

157

difficulties of providing such treatment, particularly after people rejoin their communities. Further, because of time and budgetary constraints, the main focus of many therapists is to teach basic adaptive strategies, self-care, and hygiene and to provide assistive devices and techniques for life at home. Improving hand function is often treated as a desirable but secondary aim. This state of affairs is driven by the increased pressure on healthcare systems and third-party payers to provide patients with only the most elementary coping skills to contain costs and allow for a large throughput. But as suggested by Dr. Kimberley Anderson in a study of SCI priorities (Anderson, 2004), this may be a false economy. The use of affordable technology that allows upper extremity rehabilitation to be performed at home with Internet-based telesupervision provides a promising solution to this dilemma. Acknowledgments This work was funded by the Canadian Institutes for Health Research, the Alberta Heritage for Medical Research (now Alberta Innovates Health Solutions), and the International Spinal Research Trust. References Adamovich, S. V., Merians, A. S., Boian, R., Tremaine, M., Burdea, G. S., Recce, M., et al. (2004). A virtual reality based exercise system for hand rehabilitation post-stroke: Transfer to function. Conference Proceedings: Annul International Conference of the IEEE Engineering in Medicine and Biology Society, 7, 4936–4939. Aisen, M. L., Krebs, H. I., Hogan, N., Mcdowell, F., & Volpe, B. T. (1997). The effect of robot-assisted therapy and rehabilitative training on motor recovery following stroke. Archives of Neurology, 54, 443–446. Allen, D. (2007). You're never too old for a Wii. Nursing Older People, 19, 8. Alon, G., Levitt, A. F., & Mccarthy, P. A. (2007). Functional electrical stimulation enhancement of upper extremity functional recovery during stroke rehabilitation: A pilot study. Neurorehabilitation and Neural Repair, 21, 207–215.

Alon, G., Sunnerhagen, K. S., Geurts, A. C., & Ohry, A. (2003). A home-based, self-administered stimulation program to improve selected hand functions of chronic stroke. Neuro Rehabilitation, 18, 215–225. Anderson, K. D. (2004). Targeting recovery: Priorities of the spinal cord-injured population. Journal of Neurotrauma, 21, 1371–1383. Baker, L. L., Yeh, C., Wilson, D., & Waters, R. L. (1979). Electrical stimulation of wrist and fingers for hemiplegic patients. Physical Therapy, 59, 1495–1499. Beekhuizen, K. S., & Field-Fote, E. C. (2005). Massed practice versus massed practice with stimulation: Effects on upper extremity function and cortical plasticity in individuals with incomplete cervical spinal cord injury. Neurorehabilitation and Neural Repair, 19, 33–45. Blumberg, F. C., & Sokol, L. M. (2004). Boys’ and girls’ use of cognitive strategy when learning to play video games. The Journal of General Psychology, 131, 151–158. Boian, R., Sharma, A., Han, C., Merians, A., Burdea, G., Adamovich, S., et al. (2002). Virtual reality-based poststroke hand rehabilitation. Studies in Health Technology and Informatics, 85, 64–70. Boutron, I., Guittet, L., Estellat, C., Moher, D., Hrobjartsson, A., & Ravaud, P. (2007). Reporting methods of blinding in randomized trials assessing nonpharmacological treatments. PLoS Medicine, 4, e61. Cauraugh, J. H., & Kim, S. (2002). Two coupled motor recovery protocols are better than one: Electromyogram-triggered neuromuscular stimulation and bilateral movements. Stroke, 33, 1589–1594. Chae, J. (2003). Neuromuscular electrical stimulation for motor relearning in hemiparesis. Physical Medicine and Rehabilitation Clinics of North America, 14, S93–S109. Cooper, R. A., Fitzgerald, S. G., Boninger, M. L., Brienza, D. M., Shapcott, N., Cooper, R., et al. (2001). Telerehabilitation: Expanding access to rehabilitation expertise. Proceedings of the IEEE, 89, 1174–1193. Cramer, S. C. (2010). Brain repair after stroke. The New England Journal of Medicine, 362, 1827–1829. De Kroon, J. R., Ijzerman, M. J., Chae, J., Lankhorst, G. J., & Zilvold, G. (2005). Relation between stimulation characteristics and clinical outcome in studies using electrical stimulation to improve motor control of the upper extremity in stroke. Journal of Rehabilitation Medicine, 37, 65–74. Dirette, D., & Hinojosa, J. (1994). Effects of continuous passive motion on the edematous hands of two persons with flaccid hemiplegia. The American Journal of Occupational Therapy, 48, 403–409. Drolet, M., Noreau, L., Vachon, J., & Moffet, H. (1999). Muscle strength changes as measured by dynamometry following functional rehabilitation in individuals with spinal cord injury. Archives of Physical Medicine and Rehabilitation, 80, 791–800.

158 Ellaway, P. H., Kuppuswamy, A., Balasubramaniam, A. V., Maksimovic, R., Gall, A., Craggs, M. D., et al. (2010). Development of quantitative and sensitive assessments of physiological and functional outcome during recovery from spinal cord injury: A clinical initiative. Brain Research Bulletin, 84, 343–357. Francisco, G., Chae, J., Chawla, H., Kirshblum, S., Zorowitz, R., Lewis, G., et al. (1998). Electromyogram-triggered neuromuscular stimulation for improving the arm function of acute stroke survivors: A randomized pilot study. Archives of Physical Medicine and Rehabilitation, 79, 570–575. Graves, L. E., Ridgers, N. D., & Stratton, G. (2008). The contribution of upper limb and total body movement to adolescents’ energy expenditure whilst playing Nintendo Wii. European Journal of Applied Physiology, 104, 617–623. Gritsenko, V., Chhibber, S., & Prochazka, A. (2001). Automated FES-assisted exercise therapy for hemiplegic hand function. Society for Neuroscience Abstracts, 27, 210–220. Gritsenko, V., & Prochazka, A. (2004). A functional electric stimulation-assisted exercise therapy system for hemiplegic hand function. Archives of Physical Medicine and Rehabilitation, 85, 881–885. Hammell, K. W., Miller, W. C., Forwell, S. J., Forman, B. E., & Jacobsen, B. A. (2009). Fatigue and spinal cord injury: A qualitative analysis. Spinal Cord, 47, 44–49. Hansen, G. V. O. (1979). EMG-controlled functional electrical stimulation of the paretic hand. Scandinavian Journal of Rehabilitation Medicine, 11, 189–193. Heckmann, J., Mokrusch, T., Kroeckel, A., Warnke, S., Von Stockert, T., & Neundoerfer, B. (1997). Electromyogramtriggered neuromuscular stimulation for improving the arm function of acute stroke survivors: A randomized pilot study. European Journal of Physical Medicine and Rehabilitation, 7, 138–141. Hesse, S., Schmidt, H., & Werner, C. (2006). Machines to support motor rehabilitation after stroke: 10 years of experience in Berlin. Journal of Rehabilitation Research and Development, 43, 671–678. Hogan, N., Krebs, H. I., Rohrer, B., Palazzolo, J. J., Dipietro, L., Fasoli, S. E., et al. (2006). Motions or muscles? Some behavioral factors underlying robotic assistance of motor recovery. Journal of Rehabilitation Research and Development, 43, 605–618. Holden, M. K., Dyar, T. A., & Dayan-Cimadoro, L. (2007). Tele-rehabilitation using a virtual environment improves upper extremity function in patients with stroke. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 15, 36–42. Johnson, M. J., Loureiro, R. C., & Harwin, W. S. (2008). Collaborative tele-rehabilitation and robot-mediated therapy for stroke rehabilitation at home or clinic. Intelligent Service Robotics, 1, 109–121. Johnston, M. V., Sherer, M., & Whyte, J. (2006). Applying evidence standards to rehabilitation research. American Journal of Physical Medicine and Rehabilitation, 85, 292–309.

Komaroff, E., & Delisa, J. A. (2009). Best research evidence for physical medicine and rehabilitation. The Journal of Spinal Cord Medicine, 32, 3–5. Kowalczewski, J., Chong, S. L., Galea, M., & Prochazka, A. (2011). In-home tele-rehabilitation improves tetraplegic hand function. Neurorehabilitation and Neural Repair, 25, 412–422. Kowalczewski, J., Gritsenko, V., Ashworth, N., Ellaway, P., & Prochazka, A. (2007). Upper-extremity functional electric stimulation-assisted exercises on a workstation in the subacute phase of stroke recovery. Archives of Physical Medicine and Rehabilitation, 88, 833–839. Kowalczewski, J., & Prochazka, A. (2010). Method and apparatus for automated delivery of therapeutic exercises of the upper extremity. USA patent application US2007265146. Kowalczewski, J. A., Prochazka, A., & Chong, S. L. (2007b). Home based upper extremity rehabilitation in spinal cord injured patients. In Society for Neuroscience Abstracts. San Diego, USA: 75.71/GG71. Kowalczewski, J.University of Alberta, Centre for Neuroscience. Upper extremity neurorehabilitation. (2009). [Online]. Available:http://hdl.handle.net/10048/755 (Accessed). Krebs, H. I., Hogan, N., Aisen, M. L., & Volpe, B. T. (1998). Robot-aided neurorehabilitation. IEEE Transactions on Rehabilitation Engineering, 6, 75–87. Lambercy, O., Dovat, L., Gassert, R., Burdet, E., Teo, C. L., & Milner, T. (2007). A haptic knob for rehabilitation of hand function. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 15, 356–366. Lloyd-Jones, D., Adams, R. J., Brown, T. M., Carnethon, M., Dai, S., De Simone, G., et al. (2009). Heart disease and stroke statistics–2010 update. A report from the American Heart Association. Circulation, 121, e46–e215. Lo, A. C., Guarino, P. D., Richards, L. G., Haselkorn, J. K., Wittenberg, G. F., Federman, D. G., et al. (2010). Robotassisted therapy for long-term upper-limb impairment after stroke. The New England Journal of Medicine, 362, 1772–1783. Lucas, K., & Sherry, J. L. (2004). Sex differences in video game play: A communication-based explanation. Communication Research, 31, 499–523. Lyle, R. C. (1981). A performance test for assessment of upper limb function in physical rehabilitation treatment and research. International Journal of Rehabilitation Research, 4, 483–492. Mangold, S., Schuster, C., Keller, T., Zimmermann-Schlatter, A., & Ettlin, T. (2009). Motor training of upper extremity with functional electrical stimulation in early stroke rehabilitation. Neurorehabilitation and Neural Repair, 23, 184–190. Nathan, R. H. (1994). Device for generating hand function. US Patent #5,330,516 US Patent Office, 15 claims, 6 drawing sheets. NSCISC. (2010). Spinal cord injury facts and figures at a glance Birmingham. Alabama: National Spinal Cord Injury Statistical Center.

159 Peckham, P. H., & Knutson, J. S. (2005). Functional electrical stimulation for neuromuscular applications. Annual Review of Biomedical Engineering, 7, 327–360. Piron, L., Tonin, P., Trivello, E., Battistin, L., & Dam, M. (2004). Motor tele-rehabilitation in post-stroke patients. Medical Informatics and the Internet in Medicine, 29, 119–125. Popescu, V. G., Burdea, G. C., Bouzit, M., & Hentz, V. R. (2000). A virtual-reality-based telerehabilitation system with force feedback. IEEE Transactions on Information Technology in Biomedicine, 4, 45–51. Popovic, D., Popovic, M., Sinkjaer, T., Stefanovic, A., & Schwirtlich, L. (2004). Therapy of paretic arm in hemiplegic subjects augmented with a neural prosthesis: A cross-over study. Canadian Journal of Physiology and Pharmacology, 82, 749–756. Popovic, M., Thrasher, T., Zivanovic, V., Takaki, J., & Hajek, V. (2005). Neuroprosthesis for retraining reaching and grasping functions in severe hemiplegic patients. Neuromodulation, 8, 58–72. Prochazka, A. (1997). Garment having controller that is activated by mechanical impact. Canada, WO 99/19019 patent application. Prochazka, A., Gauthier, M., Wieler, M., & Kenwell, Z. (1997). The bionic glove: An electrical stimulator garment that provides controlled grasp and hand opening in quadriplegia. Archives of Physical Medicine and Rehabilitation, 78, 608–614. Reinkensmeyer, D. J., Pang, C. T., Nessler, J. A., & Painter, C. C. (2002). Web-based telerehabilitation for the upper extremity after stroke. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 10, 102–108. Salter, R. B. (1996). History of rest and motion and the scientific basis for early continuous passive motion. Hand Clinics, 12, 1–11. Shields, R. K., Chang, Y. J., Dudley-Javoroski, S., & Lin, C. H. (2006). Predictive model of muscle fatigue after spinal cord injury in humans. Muscle & Nerve, 34, 84–91. Shields, R. K., & Dudley-Javoroski, S. (2006). Musculoskeletal plasticity after acute spinal cord injury: Effects of long-term neuromuscular electrical stimulation training. Journal of Neurophysiology, 95, 2380–2390. Shields, R. K., Law, L. F., Reiling, B., Sass, K., & Wilwert, J. (1997). Effects of electrically induced fatigue on the twitch and tetanus of paralyzed soleus muscle in humans. Journal of Applied Physiology, 82, 1499–1507. Simpson, T., Broughton, C., Gauthier, M. J., & Prochazka, A. (2008). Tooth-click control of a hands-free computer interface. IEEE Transactions on Biomedical Engineering, 55, 2050–2056. Stein, R. B., Chong, S., Everaert, D. G., Rolf, R., Thompson, A. K., Whittaker, M., et al. (2006). A multicenter trial of a footdrop stimulator controlled by a tilt sensor. Neurorehabilitation and Neural Repair, 20, 371–379.

Stein, R. B., Gordon, T., Jefferson, J., Sharfenberger, A., Yang, J. F., De Zepetnek, J. T., et al. (1992). Optimal stimulation of paralyzed muscle after human spinal cord injury. Journal of Applied Physiology, 72, 1393–1400. Stein, R. B., & Prochazka, A. (2009). Impaired motor function: Functional electrical stimulation. In A. M. Lozano, P. L. Gildenberg & R. R. Tasker (Eds.), Textbook of stereotactic and functional neurosurgery. Berlin: Springer. Sullivan, J. E., & Hedman, L. D. (2007). Effects of homebased sensory and motor amplitude electrical stimulation on arm dysfunction in chronic stroke. Clinical Rehabilitation, 21, 142–150. Taub, E., Uswatte, G., King, D. K., Morris, D., Crago, J. E., & Chatterjee, A. (2006). A placebo-controlled trial of constraint-induced movement therapy for upper extremity after stroke. Stroke, 37, 1045–1049. Tawashy, A. E., Eng, J. J., Lin, K. H., Tang, P. F., & Hung, C. (2009). Physical activity is related to lower levels of pain, fatigue and depression in individuals with spinal-cord injury: A correlational study. Spinal Cord, 47, 301–306. Taylor, P., Burridge, J., Dunkerley, A., Wood, D., Norton, J., Singleton, C., et al. (1999). Clinical audit of 5 years provision of the Odstock dropped foot stimulator. Artificial Organs, 23, 440–442. Taylor, P. N., Burridge, J. H., Hagan, S. A., Chapple, P., & Swain, I. D. (1996). Improvement in hand function and sensation in chronic stroke patients following electrical stimulation exercises. A retrospective clinical audit. In P. Hunter Peckham (Ed.), 1st Annual Conference of the International FES Society. Cleveland, OH. Vodovnik, L., Bajd, T., Kralj, A., Gracanin, F., & Strojnik, P. (1981). Functional electrical stimulation for control of locomotor systems. CRC Critical Reviews in Bioengineering, 6, 63–131. Volpe, B. T., Huerta, P. T., Zipse, J. L., Rykman, A., Edwards, D., Dipietro, L., et al. (2009). Robotic devices as therapeutic and diagnostic tools for stroke recovery. Archives of Neurology, 66, 1086–1090. Waters, R., Bowman, B., Baker, L., Benton, L., & Meadows, P. (1981). Treatment of hemiplegic upper extremity using electrical stimulation and biofeedback training. In D. Popovic (Ed.), Advances in external control of human extremities. Belgrade: Yugoslav Committee for Electronics and Automation. Weingarden, H. P., Zeilig, G., Heruti, R., Shemesh, Y., Ohry, A., Dar, A., et al. (1998). Hybrid functional electrical stimulation orthosis system for the upper limb: Effects on spasticity in chronic stable hemiplegia. American Journal of Physical Medicine and Rehabilitation, 77, 276–281. Yozbatiran, N., Der-Yeghiaian, L., & Cramer, S. C. (2008). A standardized approach to performing the action research arm test. Neurorehabilitation and Neural Repair, 22, 78–90.

A. M. Green, C. E. Chapman, J. F. Kalaska and F. Lepore (Eds.) Progress in Brain Research, Vol. 192 ISSN: 0079-6123 Copyright Ó 2011 Elsevier B.V. All rights reserved.

CHAPTER 11

Guiding task-oriented gait training after stroke or spinal cord injury by means of a biomechanical gait analysis Sylvie Nadeau{,{,},*, Cyril Duclos{,{,}, Laurent Bouyer},},k and Carol L. Richards},},k {

Pathokinesiology Laboratory, Centre for Interdisciplinary Research in Rehabilitation, Institut de réadaptation Gingras-Lindsay-de-Montréal, Montréal, Québec, Canada { School of Rehabilitation, Université de Montréal, Montréal, Québec, Canada } CIHR Multidisciplinary Team in Locomotor Rehabilitation, Québec, Canada } Centre for Interdisciplinary Research in Rehabilitation and Social Integration, IRDPQ, Université Laval, Québec, Canada k Department of Rehabilitation, Faculty of Medicine, Université Laval, Québec, Canada

Abstract: To recover the ability to walk is one of the most important goals of persons recovering from a stroke or spinal cord injury (SCI). While a task-oriented approach to gait training is recommended, randomized controlled trials or meta-analyses comparing different methods of delivering training have failed in general to demonstrate the superiority of one approach over the other. The large variations in the mean outcome gait measures reported in these studies reflect, at least in part, the heterogeneity of the sensorimotor impairments underlying the gait disability as well as variations in the therapeutic response. The purpose of this chapter is to demonstrate that biomechanical gait analysis can reveal information pertinent to the selection of a task-oriented approach to enhance gait training as well as the therapeutic response that clinical evaluations alone cannot provide. We first briefly review locomotor impairments underlying the gait disability after stroke and SCI as well as the effects of selected technological taskoriented gait training interventions. We then give examples that demonstrate the use of gait analysis to pinpoint underlying impairments that can guide the choice of sensorimotor therapy and then immediately identify responders to the intervention. Such an individualized approach should promote therapeutic efficacy while leading over time to the identification of clinical indices to guide therapy when gait analysis is not feasible. Given the requirements of a gait analysis laboratory and the qualified personnel to capture and interpret the data, future studies will need to demonstrate the feasibility of the technological proposed approach and assess the costs and benefits for the health care system. *Corresponding author. Tel.: þ514-343-2253; Fax: þ514-340-2154 E-mail: [email protected] DOI: 10.1016/B978-0-444-53355-5.00011-7

161

162

Keywords: Rehabilitation; gait training; therapeutic response; biomechanics; EMG; sensorimotor impairments.

Introduction Following a stroke or spinal cord injury (SCI), the gait disability is variable and ranges from very severe and unable to walk to very mild. The gait is usually slow, unstable, and asymmetric. The amplitude and coordination of the lower extremity, trunk, and arm movements differ from the normal to varying degrees even to the naked eye. In-depth biomechanical gait analyses of the interplay of internal and external forces (kinetics) that generate and control the gait movements have contributed to our understanding of these gait disorders. Studies of muscle activations and joint moment and power patterns during walking have revealed a constellation of sensorimotor impairments expressed to varying degrees in individual subjects. For example, in persons with chronic stroke, impairments in the amplitude and coordination of muscle activations (Knutsson and Richards, 1979; Lamontagne et al., 2000a, 2002; Shiavi et al., 1987) and reduced power generation of the hip extensors, hip flexors, and ankle plantarflexors required for propulsion result in slower gait speeds (Kim and Eng, 2004; Milot et al., 2008; Nadeau et al., 1999; TeixeiraSalmela et al., 2001). The elicitations of spastic reflexes as the gait movements stretch the muscles (Knutsson and Richards, 1979; Lamontagne et al., 2001) and hypoextensibility of the muscle and tendon complex (Lamontagne et al., 2000b) contribute to the disturbed locomotor control and performance. After stroke, walking endurance is reduced (Dean et al., 2001; Eng et al., 2002; Macko et al., 2005), anticipatory control may be poor (Chang et al., 2010; Ladouceur et al., 2003), orthoses and walking devices are used to substitute for loss of sensory and motor abilities, and there is an increased risk for falls (Campbell and Matthews, 2010; Divani et al.,

2009). Although less studied, after SCI, hyperactive stretch reflexes, weakness, excessive coactivation of antagonist muscles, and altered muscle mechanics have also been shown to contribute to the gait and postural movement disorder (Dietz, 2009; Dietz and Sinkjaer, 2007; Leroux et al., 2006; van Hedel et al., 2005). Given the importance of walking and mobility in activities of daily living, it is not surprising that a main goal of stroke or SCI survivors is to recover the ability to walk. Over the last 25 years, a task-oriented approach to the rehabilitation of gait has become recognized as the best practices approach after stroke (Carr and Shepherd, 1989; Malouin and Richards, 2005; Malouin et al., 1992; Richards et al., 2004) and SCI (Sunnerhagen, 2010; van Hedel and Dietz, 2010). The task-oriented approach to locomotor training is characterized by interactive mobility training which incorporates functional strengthening, balance, aerobic exercises, and the practice of a variety of walking tasks and subtasks. In this way, muscles are lengthened and contract in the modes and speeds of contraction (concentric, static, eccentric) specific to walking, and rhythm and coordination of muscle activations, anticipatory adjustments, endurance, and navigational skills are practiced during the locomotor related tasks. It involves a large number of repetitions, based on motor learning principles and may be offered by means of circuit training (Dean et al., 2000; English and Hillier, 2010; Salbach et al., 2004; Wevers et al., 2009), cardiorespiratory training (Macko et al., 2005; Saunders et al., 2009), mental practice (Dunsky et al., 2008; Malouin and Richards, 2010), and walking in virtual reality scenarios (Fung et al., 2006; Yang et al., 2008). Over the years, the task-oriented approach, initially developed to be practiced without technical aids other than those available in traditional

163

physiotherapy departments, has incorporated many new technologically driven means of delivering the training. Today, the approach includes the use of treadmills with or without body weight support, electrical stimulation while walking on a treadmill, split-belt treadmills, the use of robots to assist gait movements while walking on a treadmill, and the use of virtual scenes that simulate walking in different environments. These technological advances have created new interest and hope. They are stimulating and motivational for the subject and promote increased practice. In many centers, these technological advances are considered to be essential to the task-oriented approach even though, it has been shown that when the practice time is similar, the use of treadmills with or without weight support is not superior to task-oriented therapy offered by traditional methods (Dobkin et al., 2006; Nilsson et al., 2001; Richards et al., 2004). Therapists, trained in the delivery of task-oriented therapy, adapt their approach to meet the individual needs. They base their approach on: (1) observations of the locomotor pattern, (2) the results of clinical assessments, and (3) their skill in deriving the best approach for each individual. Clinical assessments inform on sensorimotor impairments such as sensation (pain, touch, and proprioception), muscle strength, range of motion, and the ability to perform voluntary movements. For example, the Fugl-Meyer Sensorimotor (Fugl-Meyer et al., 1975) assessment, based to a large extent on the ability to perform isolated voluntary movements at bedside, cannot predict the ability to control the complex semiautomatic movements and forces that generate walking or to discern underlying impairments during gait (Bowden et al., 2010; Knutsson and Richards, 1979). An in-depth biomechanical gait analysis, however, can reveal impairments underlying an abnormal movement pattern (Crenna et al., 1992; Knutsson and Richards, 1979; Krawetz and Nance, 1996; Ladouceur et al., 2003; Nadeau et al., 1999; Olney, 2005; Richards and Olney, 1996) and provide guidelines for

therapeutic interventions to improve gait function. Today, as opposed to 30 years ago, gait analysis laboratories are more common and technologically superior, making their clinical use more amenable. The main purpose of this chapter is thus to argue for a task-oriented gait training approach guided by underlying sensorimotor impairments revealed by gait analysis rather than a generic approach after stroke or SCI. We first briefly review the conclusions from systematic reviews and meta-analyses regarding selected task-oriented gait training interventions. Second, we describe important findings derived from biomechanical gait analyses. We then demonstrate the use of biomechanical gait analysis to promote improved locomotor training by pinpointing specific impairments and the response to selected training approaches. We believe that gait analysis can help clinicians learn to better match sensorimotor therapeutic approaches to the individual patient.

Effects of selected task-oriented gait training interventions After stroke, the task-oriented approach leads to improvements in outcome measures, such as gait speed, the distance covered in 6 min, or cardiorespiratory factors (English and Hillier, 2010; Richards et al., 2004; Salbach et al., 2004; States et al., 2009; van de Port et al., 2007; Wevers et al., 2009). Overall, studies that average the results of groups of subjects in randomized controlled trials (RCTs) and meta-analyses report small gains in the mean improvement in gait speed after stroke that rarely attain 0.16 m/s, recently determined to be the minimal clinically important difference associated with an improvement in the modified Rankin Scale, a global index of disability in subacute stroke (Tilson et al., 2010), or to a change in the category level of ambulation (Bowden et al., 2008; Perry et al., 1995).

164

There are fewer reports of trials on the effects of task-oriented interventions for SCI. The largest (n ¼ 148) reported trial (Dobkin, 2007; Dobkin et al., 2006), compared the efficacy of step training with body weight support on a treadmill (BWSTT) with over-ground practice to the efficacy of a defined over-ground mobility therapy (CONT) in patients with incomplete SCI who were unable to walk on admission for rehabilitation. This important trial, although unable to show the superiority of one approach over the other, described for the first time functional improvements that could be attained 6 months after SCI by means of the Functional Independence Measure for Locomotion (FIM-L) for ASIA B (see Table 1) and C subjects and walking speed for ASIA C and D subjects with both approaches. In the Dobkin et al. study, 35% of ASIA B, 92% of ASIA C, and all ASIA D subjects walked independently post-training with a mean gait speed for ASIA C and D subjects of 1.1 m/s. The use of BWSTT, developed with the spinalized cat model (Barbeau and Rossignol, 1987) and introduced in the rehabilitation of patients after SCI or stroke by Barbeau et al. (1987) or for treadmill training without weight support after stroke by Malouin et al. (1992), have since become favored forms of task-oriented gait training. BWSTT is provided with the assistance of therapists or robots. Although all types of treadmill training lead to beneficial results,

none has been shown to be superior after stroke (Moseley et al., 2005) or SCI (Dobkin et al., 2006; Mehrholz et al., 2008; Swinnen et al., 2010). In fact, despite a large number of clinical studies and meta-analyses over the past years, it has been remarkably difficult to demonstrate that one therapeutic approach is superior to another. In a way, this is not surprising, given that natural recovery occurs over the same period that rehabilitative therapy is instituted, the heterogeneity of the lesions and consequent sensorimotor impairments, and the life histories and motivations of those recovering from stroke or SCI. Also, one must consider whether the approaches that were compared were different enough (Dobkin et al., 2006) to promote measurable differences or whether the outcome measures used to document change were appropriate, reliable, valid, and responsive (Richards et al., 1999, 2006). The choice of subjects included in the studies also contributes to such results. RCTs recruit subjects that are as similar as possible, chosen according to specific inclusion and exclusion criteria based on clinical evaluations and walking ability. Unfortunately, the use of clinical assessments or walking speed as inclusion criteria will not discern underlying impairments leading to the walking disability and we cannot assume that these impairments will be equally distributed in the comparative groups. For example, the meta-analysis comparing the effectiveness of

Table 1. Neurological classification of injury proposed by the American Spinal Injury Association (ASIA) ASIA grade

Description

ASIA A ASIA B

Complete: No motor or sensory function is preserved in the sacral segments S4–S5. Incomplete: Sensory but no motor function is preserved below the neurological level and includes the sacral segments S4–S5. Incomplete: Motor function is preserved below the neurological level and the majority of key muscles below the neurological level have a motor score 10%.

(Buurke et al., 2008; Kautz et al., 2005; Richards et al., 2009). A lack of correlation between gains in gait speed and muscle strength as well as a variable response to a muscle strengthening intervention (Milot, 2007) are illustrated in Fig. 1. Training consisted of concentric contractions, performed on a Biodex dynamometer, that matched the range of movement and speeds of contractions during gait, to strengthen the energy generating muscles during gait (plantarflexors, hip flexor, and extensor muscles). As can be seen, some subjects had large improvements (> 10%) in both outcomes (strength of the three trained muscle groups and gait speed), others did not show any gain in strength and only a small change in gait speed despite intensive training (three times per week, during 6 weeks). Still others had no change in gait speed but a medium change in strength. Likewise, Wirz et al. (2005) reported that after robotic-assisted BWSTT after SCI, there was no correlation between improvements in walking speed or changes in muscle strength

166

measured by lower extremity motor scores (LEMS; manual muscle testing of five key muscle groups of both lower extremities performed according to ASIA guidelines) or spastic motor behaviors. Such findings demonstrate that subjects respond differently to muscle strengthening exercises. It could be that the key muscles in individuals were not targeted, but it is more likely due to the lack of task-specific strength training given the lack of carryover from improved strength acquired in sitting using traditional modes of strength training (Eng and Tang, 2007; Kim et al., 2001; Malouin and Richards, 2005; Milot et al., 2007; Wirz et al., 2005), even when matching contraction type, speed, and amplitude of motion to that used during gait (Milot, 2007). Task-oriented gait training, however, is associated with both increased gait speed and power generation during walking (Dean et al., 2000; Richards et al., 2004; Teixeira-Salmela et al., 1999). Specific dynamic strength impairments during gait can be pinpointed by means of a biomechanical gait analysis as outlined in the following section.

Biomechanical gait analysis Overview of the approach A gait laboratory capable of performing a complete gait biomechanical analysis (including an electromyographic (EMG) analysis of the muscle activations) is needed to quantify the locomotor pattern of persons with neurological impairments. A three-dimensional (3D) kinematic and kinetic analysis can reveal gait deficits and compensatory and/or adaptive strategies of patients walking at comfortable and fast gait speeds. In addition to quantifying the temporal distance measures (e.g., step length, cycle duration), the angular displacements and segment positions are captured using markers placed on the body segments. The 3D positions of the markers are tracked by a motion analysis system at a frequency chosen

according to the speed of motion. Then, the data is processed by commercial or custom software to provide the kinematics of gait that will be useful to characterize the locomotor movement pattern: “the way the patient is walking” and compare it to the “normal accepted pattern.” However, kinematic variables, when used alone, yield little information about the mechanisms underlying the abnormal movement patterns (Olney et al., 1986; Teixeira-Salmela et al., 2008; Winter, 1991). For example, a decreased plantar flexion angle at the transition from stance to swing cannot conclusively be associated to weak plantar flexor muscles or a lack of dorsiflexion at foot contact because it may be associated to several impairments such as weakness or spasticity of the plantarflexors, stiffness of the muscle–tendon complex, or an abnormal muscle synergy of the lower extremity. For further details on the data capture and analysis of kinematic and kinetic data during gait, the reader is referred to the work of Robertson et al. (2004) and Burden et al. (2003). To determine the optimal intervention for each patient, one must first elucidate the gait impairments to target and then to assess the response to the chosen therapeutic approach. The next section will provide evidence that the use of gait analysis can provide guidelines for a customized choice of therapeutic approach and its evaluation.

Biomechanical analysis of gait in persons after stroke The different types of disturbed motor control described by Knutsson and Richards (1979) inferred distinct therapeutic approaches. Thus, when hyperactive stretch reflexes predominate, antispastic medication may be the therapy of choice (Knutsson et al., 1974), conversely, when weakness is the key component antispastic medication may be contraindicated and instead methods promoting increased muscle activations and dynamic muscle strengthening are encouraged

167

(Knutsson and Richards, 1979). When overly sensitive exteroceptive reflexes lead to a gait pattern dominated by a flexor withdrawal type of movement at foot contact, reducing the painful stimulus with foam inserts may be beneficial (Richards and Knutsson, 1974). Gait analysis of various disturbed walking patterns may also provide clinical indications of the underlying sensorimotor impairments to guide the therapeutic approach when individual gait analysis is not feasible. For example, knee hyperextension in the stance phase on the affected leg may be associated with different types of disturbed motor control (Richards and Knutsson, 1974). When weakness is the dominant impairment, knee hyperextension tends to occur throughout the stance phase and is often accompanied by circumduction of the lower extremity to assist in its forward movement because of the lack of ankle dorsiflexion and knee flexion to insure a safe toe clearance in the swing phase. Weakness may also lead to another pattern of knee hyperextension that occurs at the end of the stance phase, likely for stability because the plantarflexors are too weak to stabilize the position of the shank on the foot as the knee flexes preparatory to swing initiation (Bogardh and Richards, 1981). Last, hyperactive stretch reflexes in the early stance phase are associated with knee hyperextension in early and mid stance. More recent task-oriented training studies have demonstrated the value of targeting specific impairments revealed by gait analysis. Providing biofeedback of the triceps surae when walking in persons with chronic stroke has been shown to lead to a faster gait speed related to a larger push-off impulse (Jonsdottir et al., 2010). Another example is the recent case study demonstrating that practice on a split-belt treadmill to target step length asymmetry combined with overground practice can lead to less step length asymmetry and a faster walking speed (Reisman et al., 2010). Another means of pinpointing the underlying impairments associated with disturbed locomotor

control is the study of the propulsive force needed for forward propulsion and generation of walking speed. Thus, the analysis of the muscle moments and powers of 30 persons with chronic stroke divided into slow, medium, and fast walkers (Olney and Richards, 1996; Olney et al., 1991) showed that the power generation is reduced on the affected side and is often greater than normal on the less affected side. They also found that the magnitude of the late stance power generation burst was significantly correlated to the gait speed and that a larger than normal hip flexor power generation burst in early swing was used in some subjects to compensate for a weak late stance plantarflexor burst (Nadeau et al., 1999). This work led to recommendations to emphasize strengthening of the hip extensors in early stance, the hip flexors in early swing, and the ankle plantarflexors in late stance to promote a faster gait speed (Olney, 2005). Thus, focusing on the hip and ankle power generating muscles by means of circuit training (Dean et al., 2000) or individualized training (Richards et al., 1999) has been shown to lead to larger power generation bursts and faster gait speeds after 9–12 h of therapy over 3–4 weeks. Richards et al. (2004) found that about 25% of the increase in gait speed after 2 months of task-oriented gait training during inpatient rehabilitation was related to an increase in the push-off power generation at the ankle. Gait analysis has also helped us to understand why persons poststroke are often able to walk faster than their preferred walking speed (Lamontagne and Fung, 2004; Milot et al., 2007) or with less asymmetry, indicating that they are able to produce more energy on demand. As illustrated in Fig. 2, the power curves, particularly those for the ankle (A2), are clearly asymmetric between sides for both subjects when they walk at preferred speed. Subject S204 see Figure 2 demonstrates a power reserve on the paretic side at his safe fastest speed attaining a level even superior to the mean curve produced by healthy subjects walking at their preferred speed. Since there is evidence that they have a power reserve,

168

Gait after stroke 2.5 2.0

1.2 S204

S313 0.8

1.5 1.0

H3 H1

0.4

0.5 0.0

0.0 –0.5 –1.0

–0.4 H2

Hip

–1.5 1.5

K2

1.0

1.0

Power (W/kg)

Hip

–0.8

0.5

0.5 0.0

0.0

–0.5

–0.5

–1.0

–1.0

–1.5

Knee

–2.0

K1

K4

K3

Knee

–1.5 –2.0

P at preferred speed: 0.8 m/s

3.5 3.0 2.5 2.0 1.5 1.0 0.5 0.0 –0.5 –1.0 –1.5

NP at preferred speed: 0.8 m/s P at fast speed: 1.65 m/s H at preferred speed: 1.25 m/s

Ankle 0

20

40

60

80

Gait cycle (%)

P at preferred speed: 0.6 m/s

–3.5 –3.0 2.5 2.0 1.5 1.0 0.5 0.0 –0.5 –1.0 –1.5 100

A2

NP at preferred speed: 0.6 m/s P at fast speed: 1.13 m/s H at preferred speed: 1.25 m/s

A1 0

20

Ankle 40

60

80

100

Gait cycle (%)

Fig. 2. Power data (W/kg) for normalized gait cycles at the hip, knee, and ankle after stroke in two subjects and mean results of 14 controls walking at preferred speed. Positive power values correspond to energy generation or concentric action of the muscle groups and negative values to mean energy absorption or eccentric action. At the hip, H1, H2, and H3 refer to power generation by the hip extensors, power absorption by the hip flexors, and power generation by the hip flexors, respectively. At the knee, K1, K3 represent energy absorption by the knee extensors; K2, energy generation by the knee extensors; and K4, energy absorption by the knee flexors. At the ankle, A1 denotes energy absorption and A2, energy generation produced by the ankle plantarflexors. Abbreviations: P, paretic side; NP, nonparetic side; H, healthy controls.

why do persons poststroke still walk at a lower speed and asymmetrically? In other words, persons poststroke could perform tasks symmetrically by reducing the force output from the nonparetic limb or by increasing the force output of the paretic limb. Importantly, they could walk at faster speeds. One can argue that subjects walk at this nonmaximal speed of walking and have an

asymmetric gait pattern because they aim to minimize the “costs” to the system (Jeng et al., 1996). Criteria such as minimization of the physiological energy, maximization of the mechanical energy conservation, and reduction of the amount of loading on the skeleton may in part explain the slower gait speed and adaptive pattern (Jeng et al., 1996). A study by Milot et al. (2006) may provide another explanation. They studied the

169

levels of effort (joint moment during gait/maximal moment that can be produced) in the muscle groups involved in energy generation during gait in healthy controls and stroke survivors. They found that stroke survivors used the same levels of effort on both sides when they walk at their preferred speed. Moreover, these levels of effort were similar to those of healthy subjects also walking at their preferred speed (not the same speed as those after stroke). When requested to walk at fast speed, they increased their speed by increasing the effort at the hip, possibly because the values at the ankle were relatively higher than those of the hip flexors and extensors at preferred speed. This effort-matching strategy may correspond to the bilateral force matching paradigm proposed by Bertrand et al. (2004) wherein the mechanical moment during gait is decreased to accommodate the reduced maximal strength in the paretic muscle. At preferred gait speed, this could be achieved by decreasing the gait speed until the level of effort of the paretic side is close to that on the nonparetic side and near normal values. The consequence is inevitably an asymmetrical gait pattern concomitant with a decreased speed. These findings, obtained by means of in-depth biomechanical gait analysis and clinical measures of strength, suggest that the level of use of the residual muscle strength during gait rather than the residual strength per se should guide therapy to promote gait recovery after stroke.

Biomechanical analysis of the gait after SCI Overall, the most common findings from EMG studies of the muscle activations during gait after incomplete SCI are excessive coactivation, spasticity, and alterations in the timing and amplitude of muscle activations (Barbeau et al., 2006). Clinically classified as having spastic pareses, they tend to walk slower, to have poor balance, and excessive hip and knee flexion in the stance

phase. Typically they make foot contact with the knee in a flexed position, and for some, the knee and ankle (dorsiflexion) remain flexed throughout the stance phase, so that the power generation capacity of the plantarflexors at push-off is affected (Ditunno and Scivoletto, 2009; Krawetz and Nance, 1996; Pépin et al., 2003). In general, the quality of gait in functional ambulators is related to the level and completeness of the injury and, as shown by Krawetz and Nance (1996), also on the degree of spasticity. BWSTT has been used for more than 20 years to promote walking post-SCI. Since Barbeau et al. (1987) showed that subjects with incomplete SCI were able to produce stepping movements when suspended over a treadmill, studies have reported improvements in walking speed (Dobkin et al., 2006) associated with improved postural alignment and muscle activations of the lower extremity (Wirz et al., 2005) after BWSTT. As mentioned previously, however, BWSTT has not been shown to be superior to conventional mobility training (Dobkin, 2007; Dobkin et al., 2006). The use of gait analysis after SCI, as after stroke, has guided the therapeutic approach. First, EMG recordings revealed that activations of the lower extremity muscles when walking on a treadmill with BWS could be elicited and retrained in persons with both complete and incomplete SCI (Dietz, 2009). The amplitude of the muscle activations was lower than in healthy controls and lower in complete than in incomplete SCI. Visintin and Barbeau (1994) studied the effects of first using the support provided by parallel bars and, second, the effects of different levels of BWS when walking on a treadmill after an incomplete SCI. When symmetrically involved subjects walked without parallel bar support, the EMG activity increased and was prolonged during the stance phase, especially in the distal muscles. In patients with an asymmetrical gait pattern, removing parallel bars decreased the use of compensations and thus allowed for the expression of a more normal gait pattern. The use of BWS (40%) with walking without parallel

170

bars facilitated gait and led to less clonus, especially in the asymmetrically or severely involved subjects who had marked difficulty at 0% BWS. Others (Barbeau et al., 2002; Leroux et al., 1999) have described postural adaptations of individuals with incomplete SCI when they walk on the level or up and down a ramp. As expected, some use adaptations very similar to those used by healthy controls, while others use different strategies. For example, hip hiking, one of the pelvic-trunk strategies used to bring the leg forward to compensate for insufficient ankle dorsiflexion during the swing phase, that is present during level walking, is increased when walking up a slope or during obstructed walking (Ladouceur et al., 2003). Gait analysis has also been used to assess the differential effects of pharmacological agents such as clonidine and baclofen after SCI (Norman et al., 1998; Stewart et al., 1991). It was clearly demonstrated in these studies that human spinal locomotor activity can be influenced pharmacologically. In addition, gait evaluation revealed that pharmacological agents that modulate reflexes when the subject is at rest do not necessarily produce the same effects on muscle activations during gait (Norman et al., 1998). In a recent study, Ivanenko et al. (2009) argue that with training after SCI, new motor patterns may be learned that are specific to the practiced task, rather than reactivating pre-injury motor patterns. Such compensatory equivalent solutions would be possible because of redundancy in the neuromuscular system or malfunctioning of injured “elements.” These reconstructed spatiotemporal maps of motor neuron activity could nevertheless produce similar joint kinematics, leading to the suggestion that training kinematics may be more successful than training based on reconstructing “normal” muscle activation patterns. Very little is known, however, about moments and powers produced during gait in persons after SCI. To our knowledge, the only published study is Gordon et al. (2009) that reported modulations

of hip kinetics with the addition of a load at the ankle. Garneau et al. (2006) have studied the kinematics and kinetics (moments and powers) of the lower extremities during gait in a large group of individuals post-SCI in comparison to a group of healthy controls walking at the same cadence. They found that, in comparison to healthy controls, SCI patients had a large increase in the extensor moment and hip power in the first part of the stance phase, more absorption at the knee at the end of the stance phase, and a reduction in the plantar flexor power at push-off. The overuse of the hip extensors to compensate for the poor ankle power at push-off is in agreement with the findings of Gorassini et al. (2009). They report that some persons with SCI respond to BWSTT by increasing their proximal muscle activity three- to fourfold greater than that in healthy control subjects walking at similar speeds and level of BWS. These few studies highlight the need for further studies quantifying the kinetics to better understand the mechanisms underlying the gait after SCI. Clinical measures of strength do not predict walking function after an incomplete SCI. As shown in Fig. 3, three subjects had marked differences in the preferred gait speeds (1.07 m/ s vs. 0.53 m/s vs. 0.34 m/s) even though their muscle strength assessed with LEMS was similar. Gait analysis, however, demonstrates large power differences associated with decreasing walking speeds at the ankle and hip among the subjects with almost normal (S01) to moderate (S02) and severe (S03) decreases of power generation and absorption relative to the normal values. These data again demonstrate the uncertainty of predicting walking speed from clinical measures of muscle strength as previously shown after stroke. After a stroke, the gait pattern is expected to be asymmetrical (Fig. 2). After a SCI, however, a lesion that is grossly symmetrical may result either in a symmetrical or an asymmetrical gait pattern (Fig. 3). From the assessment of gait asymmetry in 14 individuals with incomplete SCI

171

Gait after SCI

Power (W/kg)

S01-C4: LEMS (L/R): 23/22

S02-C4: LEMS (L/R): 23/22

S03-C4: LEMS (L/R): 21/20

3

3

2

2

2

1

1

1

0

0

−1 −2

Hip

0 SCIL—preferred speed: 0.53 m/s

−1

SCIL—preferred speed: 1.07 m/s SCIR—preferred speed: 1.07 m/s H—preferred speed: 1.25 m/s

SCIL—preferred speed: 0.24 m/s

−1

SCIR—preferred speed: 0.53 m/s H—preferred speed: 1.25 m/s

−2

1

3

SCIR—preferred speed: 0.24 m/s H—preferred speed: 1.25 m/s

−2 1

1

Power (W/kg)

Knee 0.5

0.5

0.5

0

0

0

−0.5

−0.5

−0.55

−1

−1

−1

−1.5

−1.5

−1.55

4

4

3

3

3

2

2

2

1

1

1

4

Power (W/kg)

Ankle

0

0

0

−1

−1

−1

−2

0

20

40

60

80

100

−2

0

20

Gait cycle (%)

40

60

Gait cycle (%)

80

100

−2

0

20

40

60

80

100

Gait cycle (%)

Fig. 3. Power data (W/kg) at the hip, knee, and ankle joints after SCI (S01, S02, and S03; ASIA score of D at level C4) on the right (dotted line) and left (gray line) sides and mean results of 14 healthy controls (bold black line). All subjects walked at their preferred speed. LEMS, lower extremity motor score; R, right; L, left; H, healthy controls.

walking at preferred speed, Nadeau et al. (2008) found differences between sides greater at the hip than the ankle. Moreover, in general, clinical asymmetry of motor and sensory scores did not predict the asymmetry of the gait parameters, except at the hip. Figure 4 illustrates such a case, where the ankle and hip power profiles of two persons (also ASIA D and C4 level) differ moderately at the ankle and greatly at the hip between sides. The large energy generation by the right hip extensors (H1) at the beginning of the gait cycle, which is larger than that for the

healthy controls walking at preferred speed, is likely a compensation for the low push-off power generation by the ankle plantarflexors at the end of the stance phase. Moreover, even though for S04, the LEMS are identical and reach normal values (maximal score is 25), the power profile at the ankle is drastically reduced, indicating that a normal clinical strength evaluation does not predict the capacity of the muscle groups to generate power during gait. Figure 4 also shows a power reserve after SCI, similar to that demonstrated after stroke, that can be exposed by asking

172

Gait after SCI S04-C4: LEMS (L/R): 25/25

S05-C4: LEMS (L/R): 22/25 3

3 2 Power (W/kg)

Hip

SCIL—preferred speed: 0.83 m/s SCIR—preferred speed: 0.83 m/s

SCIL—preferred speed: 1.05 m/s SCIR—preferred speed: 1.05 m/s SCIL—fast speed: 1.5 m/s H—preferred speed: 1.25 m/s

2

SCIL—fast speed: 1.15 m/s H—preferred speed: 1.25 m/s

1

1

0

0

−1

−1

−2

−2 4

4

Power (W/kg)

Ankle 3

3

2

2

1

1

0

0

−1

−1 −2

−2 0

20

40 60 Gait Cycle (%)

80

100

0

20

40 60 Gait Cycle (%)

80

100

Fig. 4. Power data (W/kg) at the hip and ankle joints after SCI (S04 and S05; ASIA score of D at level C4) and mean results of 14 healthy controls (bold black line) walking at preferred speed. Data on the left side of the SCI subject at a safe fast speed are also presented. Abbreviations are identical to those in Fig. 3.

the person to walk at their fastest safe speed (thin gray curves vs. dotted curves).

Use of in-depth biomechanical gait analysis for locomotor training In-depth biomechanical gait analysis can help to improve locomotor training in at least two ways. First, as presented in the previous section, it can determine impairments underlying the abnormal gait pattern and reduced gait speed. Thus, it provides the rationale for individualization or categorization of patients based on specific gait-related

findings. Second, it makes it possible to gauge the subject's response to the specific intervention. For example, it is possible to evaluate the immediate effects of adding a weight to the ankle. Such feedback of intervention response promotes an individualized choice of therapy and likely a more rapid recovery. In this section, examples from ongoing experimental work related to the development of task-oriented therapeutic approaches are presented to support this hypothesis. Interestingly, these examples demonstrate that some patients do not respond in the same way as healthy controls when they are exposed to an additional mechanical demand. Consequently,

173 Control

2

1 0.8 0.6 0.4 0.2 0 −0.2 −0.4 −0.6 −0.8 −1 Ankle 1.5

1.5 Power (W/kg)

1 0.5 0 −0.5 −1

Level Upward Downward

−1.5 −2 4 3 Power (W/kg)

SCI

Hip

1

2 0.5

1

0

0 −1

−0.5

−2

−1

−3

0

20

40 60 Gait cycle (%)

80

100

−1.5

0

20

40

60

80

100

Gait cycle (%)

Fig. 5. Hip and ankle mean power curves for control (left) and post-SCI subjects (ASIA D; right side) for level walking and walking up and down a ramp.

they can be trained week after week without ever achieving the training objectives. A person's responsiveness to the intervention likely explains a large part of the variability in therapy effectiveness. Usually, as observed in a healthy control (Fig. 5, left graph), walking up or down a slope increases the mechanical demand at the hip (up for extensors and down for flexors) in comparison to walking on level ground with only minor changes of the mechanical demand at the ankle. For the person with a SCI, negotiating the slope also changes the mechanical demand at the hip but has a different effect at the ankle. Among other things, plantarflexor work is abolished at the end of the stance phase when walking down the slope. If the therapist's objective is to strengthen the hip muscles and the plantarflexors

together by walking down a slope, this therapeutic strategy will not be effective in this case. A second example (Fig. 6) illustrates the different response of two individuals with CNS lesions (left: low thoracic incomplete SCI; right: ischemic stroke) when exposed to a force field applied by a robotic ankle foot orthosis (Noël et al., 2008). The plantarflexing force opposes the action of the dorsiflexor muscles in the mid-swing phase of gait, stimulating increased dorsiflexor activation to ensure foot clearance as the persons walk on a treadmill. For the person on the left, the EMG profiles reveal an appropriate reaction with a specific effect on the activation of the tibialis anterior (TA), while the EMG profiles on the right do not show a response in the TA but rather in the rectus femoris (RF), suggesting that the force field has an effect at the hip instead of at the ankle.

(a)

1

2

3

4

5

6

7

(b)

8 1

Force field

(c)

(d) iKnee (°)

20

0

0

20 10

20 10

0 −10

0 −10

10

10

40

0 −10

400

400

200

200

0

0

200

200

100

100

0

0

100

100

50

50

0

0

200

200

100

100

0

0

200

200

100

100

0

0 400

200

200

coTA (µV)

0 −10

400

0

0

coSOL (µV)

iST (µV)

iVL (µV)

iRF (µV)

iSOL (µV)

iTA (µV)

20

TORQUE (Nm) iANKLE (°)

P-ORT-C2: responsive subject

40

200

200

100

100

0

P-ORT-C7: nonresponsive subject

0 0

20

40

60

Gait cycle (%)

80

100

0

20

40

60

80

100

Gait cycle (%)

Fig. 6. Force field applied by a robotized ankle foot orthosis (AFO) during gait. (a) Stick figure illustrating the different phases of the gait cycle and the time of action of the force field (gray bar). (b) A picture of the robotized AFO. The cylinder located at the back of the device transfers the torque generated by an electric motor located beside the treadmill to the moving subject. (c) Kinematic and EMG recordings from a subject (low thoracic incomplete SCI) that responded positively to the force field. Control walking (AFO on, but no force field) is shown in black, and force field walking in gray. (d) Kinematic and EMG recordings from a subject (ischemic stroke) that did not respond to the force field. Abbreviations: i, ipsilateral limb; co, contralateral limb; TA, tibialis anterior; SOL, soleus; RF, rectus femoris; VL, vastus lateralis; ST, semitendinosus.

175

Thus, task-specific strength training with a force field will not likely result in improved ankle control in this person. Overloading a lower limb by attaching a load at the ankle while walking has been assessed in healthy controls and individuals after SCI (Lam et al., 2008) and stroke (Regnaux et al., 2008). This approach provides a means of practicing task-specific strength training. For example, a load placed distally to the shank increases the level of effort at the proximal joints (hip > knee) during the swing phase of gait by changing the mass and inertial properties of the lower limbs (Noble and Prentice, 2006). In Fig. 7, one subject after stroke (S11) increases the work of the hip flexors at the transition from stance to swing

when walking with a 1.0 kg load (H2 and H3 bursts), whereas the second subject (S10) does not show such a change. This subject (S10) uses an alternative strategy to increase the mechanical energy of the loaded lower limb to ensure adequate foot clearance as the loaded leg moves forward. After stroke, individuals might also use pelvic hiking and circumduction of the paretic limb (Chen et al., 2005; Kim and Eng, 2004) to compensate for the lack of response at the hip to the loaded paretic limb. Such compensatory strategies prevent the attainment of specific hip flexor strengthening during walking with this approach, highlighting the need to ensure an adequate response to the proposed therapeutic approach. Gait after stroke

0.3

S10

Hip

Power (W/kg)

0.2 0.1 0 −0.1 −0.2 −0.3

Without load at preferred speed: 0.5 m/s With load at preferred speed: 0.5 m/s

−0.4 0.3

S11

Hip

Power (W/kg)

0.2 0.1 0 −0.1 −0.2 −0.3

Without load at preferred speed: 0.5 m/s With load at preferred speed: 0.5 m/s 0

20

40 60 Gait Cycle (%)

80

100

Fig. 7. Hip mean power data for the paretic side after stroke in two subjects (S10 and S11) walking at their preferred speed with a load (gray line) and without a load (black line) attached to the paretic ankle. Only S11 showed an increase in the H2 and H3 power bursts with the load.

176

Conclusion This chapter has presented evidence supporting the use of gait analysis to guide task-oriented gait training to inform on underlying impairments and response to interventions rather than a generic task-oriented approach in persons recovering from stroke or SCI. We argue that such an approach could guide the choice of task-oriented gait training and enhance its efficacy by tailoring the approach to better meet the specific needs of the individual and to identify responders to the chosen approach. Given the requirements of a gait analysis laboratory and the qualified personnel to capture and interpret the data, future studies will need to demonstrate the feasibility of the proposed approach and assess the costs and benefits for the health care system. We believe that a better understanding of the gait impairments concomitantly with response to interventions can enhance the acquisition of motor and sensory skills and promote improved gait performance. Acknowledgments This work was supported by the Multidisciplinary Team in Locomotor Rehabilitation (Strategic initiative, CIHR and REPAR (Réseau provincial de recherche en adaptation-réadaptation/FRSQ)). S. Nadeau is a senior scientist supported by the “Fonds de la Recherche en Santé du Québec” (FRSQ) and C. L. Richards is the Holder of the Université Laval Research Chair in Cerebral Palsy. The authors thank Dr. Francine Malouin for her constructive comments. References Barbeau, H., Ladouceur, M., Mirbagheri, M. M., & Kearney, R. E. (2002). The effect of locomotor training combined with functional electrical stimulation in chronic spinal cord injured subjects: Walking and reflex studies. Brain Research. Brain Research Reviews, 40, 274–291.

Barbeau, H., Nadeau, S., & Garneau, C. (2006). Physical determinants, emerging concepts, and training approaches in gait of individuals with spinal cord injury. Journal of Neurotrauma, 23, 571–585. Barbeau, H., & Rossignol, S. (1987). Recovery of locomotion after chronic spinalization in the adult cat. Brain Research, 412, 84–95. Barbeau, H., & Visintin, M. (2003). Optimal outcomes obtained with body-weight support combined with treadmill training in stroke subjects. Archives of Physical Medicine and Rehabilitation, 84, 1458–1465. Barbeau, H., Wainberg, M., & Finch, L. (1987). Description and application of a system for locomotor rehabilitation. Medical & Biological Engineering & Computing, 25, 341–344. Bertrand, A. M., Mercier, C., Shun, P. L., Bourbonnais, D., & Desrosiers, J. (2004). Effects of weakness on symmetrical bilateral grip force exertion in subjects with hemiparesis. Journal of Neurophysiology, 91, 1579–1585. Bogardh, E., & Richards, C. L. (1981). Gait analysis and relearning of gait control in hemiplegic patients. Physiotherapy Canada, 33, 223–230. Bowden, M. G., Balasubramanian, C. K., Behrman, A. L., & Kautz, S. A. (2008). Validation of a speed-based classification system using quantitative measures of walking performance poststroke. Neurorehabilitation and Neural Repair, 22, 672–675. Bowden, M. G., Clark, D. J., & Kautz, S. A. (2010). Evaluation of abnormal synergy patterns poststroke: Relationship of the Fugl-Meyer assessment to hemiparetic locomotion. Neurorehabilitation and Neural Repair, 24, 328–337. Burden, A., Trew, M., & Baltzopoulos, V. (2003). Normalisation of gait EMGs: A re-examination. Journal of Electromyography and Kinesiology, 13, 519–532. Buurke, J. H., Nene, A. V., Kwakkel, G., Erren-Wolters, V., Ijzerman, M. J., & Hermens, H. J. (2008). Recovery of gait after stroke: What changes? Neurorehabilitation and Neural Repair, 22, 676–683. Campbell, G. B., & Matthews, J. T. (2010). An integrative review of factors associated with falls during post-stroke rehabilitation. Journal of Nursing Scholarship, 42, 395–404. Carr, J., & Shepherd, R. (1989). A motor learning model for stroke rehabilitation. Physiotherapy, 75, 372–380. Chang, W. H., Tang, P. F., Wang, Y. H., Lin, K. H., Chiu, M. J., & Chen, S. H. (2010). Role of the premotor cortex in leg selection and anticipatory postural adjustments associated with a rapid stepping task in patients with stroke. Gait & Posture, 32, 487–493. Chen, G., Patten, C., Kothari, D. H., & Zajac, F. E. (2005). Gait differences between individuals with post-stroke hemiparesis and non-disabled controls at matched speeds. Gait & Posture, 22, 51–56. Crenna, P., Inverno, M., Frigo, C., Palmieri, R., & Fedrizzi, E. (1992). Pathophysiological profile of gait in children with cerebral palsy. Medicine and Sport Science, 36, 186–198.

177 Danielsson, A., & Sunnerhagen, K. S. (2000). Oxygen consumption during treadmill walking with and without body weight support in patients with hemiparesis after stroke and in healthy subjects. Archives of Physical Medicine and Rehabilitation, 81, 953–957. Dean, C. M., Richards, C. L., & Malouin, F. (2000). Taskrelated circuit training improves performance of locomotor tasks in chronic stroke: A randomized, controlled pilot trial. Archives of Physical Medicine and Rehabilitation, 81, 409–417. Dean, C. M., Richards, C. L., & Malouin, F. (2001). Walking speed over 10 metres overestimates locomotor capacity after stroke. Clinical Rehabilitation, 15, 415–421. Dietz, V. (2009). Body weight supported gait training: From laboratory to clinical setting. Brain Research Bulletin, 78, 1–6. Dietz, V., & Sinkjaer, T. (2007). Spastic movement disorder: Impaired reflex function and altered muscle mechanics. Lancet Neurology, 6, 725–733. Ditunno, J., & Scivoletto, G. (2009). Clinical relevance of gait research applied to clinical trials in spinal cord injury. Brain Research Bulletin, 78, 35–42. Divani, A. A., Vazquez, G., Barrett, A. M., Asadollahi, M., & Luft, A. R. (2009). Risk factors associated with injury attributable to falling among elderly population with history of stroke. Stroke, 40, 3286–3292. Dobkin, B. H. (2007). Confounders in rehabilitation trials of task-oriented training: Lessons from the designs of the EXCITE and SCILT multicenter trials. Neurorehabilitation and Neural Repair, 21, 3–13. Dobkin, B., Apple, D., Barbeau, H., Basso, M., Behrman, A., Deforge, D., et al. (2006). Weight-supported treadmill vs over-ground training for walking after acute incomplete SCI. Neurology, 66, 484–493. Dunsky, A., Dickstein, R., Marcovitz, E., Levy, S., & Deutsch, J. E. (2008). Home-based motor imagery training for gait rehabilitation of people with chronic poststroke hemiparesis. Archives of Physical Medicine and Rehabilitation, 89, 1580–1588. Eng, J. J., Chu, K. S., Dawson, A. S., Kim, C. M., & Hepburn, K. E. (2002). Functional walk tests in individuals sith stroke: Relation to perceived exertion and myocardial exertion. Stroke, 33, 756–761. Eng, J. J., & Tang, P. F. (2007). Gait training strategies to optimize walking ability in people with troke: A synthesis of evidence. Expert Review of Neurotherapeutics, 7, 1417–1436. English, C., & Hillier, S. L. (2010). Circuit class therapy for improving mobility after stroke. Cochrane Database of Systematic Reviews, CD007513. Fugl-Meyer, A. R., Jääskö, L., Leyman, I., Olsson, S., & Steglind, S. (1975). The post-stroke hemiplegic patient. Scandinavian Journal of Rehabilitation Medicine, 7, 13–31. Fung, J., Richards, C. L., Malouin, F., Mcfadyen, B. J., & Lamontagne, A. (2006). A treadmill and motion coupled

virtual reality system for gait training post-stroke. Cyberpsychology & Behavior, 9, 157–162. Garneau, C., Nadeau, S., Barbeau, H., & Milot, M.-H. (2006). Biomechanical analysis of level walking in persons with incomplete spinal cord injury. The Journal of Spinal Cord Medicine, 29(3), 297–298. Gorassini, M., Norton, J., Nevett-Duchcherer, J., Roy, F., & Yang, J. (2009). Changes in locomotor muscle activity after treadmill training in subjects with incomplete spinal cord injury. Journal of Neurophysiology, 101, 969–979. Gordon, K. E., Wu, M., Kahn, J. H., Dhaher, Y. Y., & Schmit, B. D. (2009). Ankle load modulates hip kinetics and EMG during human locomotion. Journal of Neurophysiology, 101, 2062–2076. Hesse, S., Bertelt, C., Jahnke, M. T., Schaffrin, A., Baake, P., Malezic, M., et al. (1995). Treadmill training with partial body weight support compared with physiotherapy in nonambulatory hemiparetic patients. Stroke, 26, 976–981. Hesse, S., Konrad, M., & Uhlenbrock, D. (1999). Treadmill walking with partial body weight support versus floor walking in hemiparetic subjects. Archives of Physical Medicine and Rehabilitation, 80, 421–427. Ivanenko, Y., Poppele, R., & Lacquaniti, F. (2009). Distributed neural networks for controlling human locomotion: Lessons from normal and SCI subjects. Brain Research Bulletin, 78, 13–21. Jeng, S., Holt, K., Fetters, L., & Certo, C. (1996). Self-optimization of walking in nondisabled children and children with spastic hemiplegic cerebral. Journal of Motor Behavior, 28, 15–27. Jonsdottir, J., Cattaneo, D., Recalcati, M., Regola, A., Rabuffetti, M., Ferrarin, M., et al. (2010). Task-oriented biofeedback to improve gait in individuals with chronic stroke: Motor learning approach. Neurorehabilitation and Neural Repair, 24, 478–485. Kautz, S. A., Duncan, P. W., Perera, S., Neptune, R. R., & Studenski, S. A. (2005). Coordination of hemiparetic locomotion after stroke rehabilitation. Neurorehabilitation and Neural Repair, 19, 250–258. Kim, C. M., & Eng, J. J. (2004). Magnitude and pattern of 3D kinematic and kinetic gait profiles in persons with stroke: Relationship to walking speed. Gait & Posture, 20, 140–146. Kim, C. M., Eng, J. J., Macintyre, D. L., & Dawson, A. S. (2001). Effects of isokinetic strength training on walking in persons with stroke: A double-blind controlled pilot study. Journal of Stroke and Cerebrovascular Diseases, 10, 265–273. Knutsson, E., Lindblom, U., & Martensson, A. (1974). Plasma and cerebrospinal fluid levels of baclofen (Lioresal) at optimal therapeutic responses in spastic paresis. Journal of the Neurological Sciences, 23, 473–484. Knutsson, E., & Richards, C. L. (1979). Different types of disturbed motor control in gait of hemiparetic patients. Brain, 102, 405–430.

178 Krawetz, P., & Nance, P. (1996). Gait analysis of spinal cord injured subjects: Effects of injury level and spasticity. Archives of Physical Medicine and Rehabilitation, 77, 635–638. Ladouceur, M., Barbeau, H., & Mcfadyen, B. J. (2003). Kinematic adaptations of spinal cord-injured subjects during obstructed walking. Neurorehabilitation and Neural Repair, 17, 25–31. Lam, T., Wirz, M., Lünenburger, L., & Dietz, V. (2008). Swing phase resistance enhances flexor muscle activity during treadmill locomotion in incomplete spinal cord injury. Neurorehabilitation and Neural Repair, 22, 438–446. Lamontagne, A., & Fung, J. (2004). Faster is better: Implications for speed-intensive gait training after stroke. Stroke, 35, 2543–2548. Lamontagne, A., Malouin, F., & Richards, C. L. (2000a). Contribution of passive stiffness to ankle plantarflexor moment during gait after stroke. Archives of Physical Medicine and Rehabilitation, 81, 351–358. Lamontagne, A., Malouin, F., & Richards, C. L. (2001). Locomotor-specific measure of spasticity of plantarflexor muscles after stroke. Archives of Physical Medicine and Rehabilitation, 82, 1696–1704. Lamontagne, A., Malouin, F., Richards, C. L., & Dumas, F. (2002). Mechanisms of disturbed motor control in ankle weakness during gait after stroke. Gait & Posture, 15, 244–255. Lamontagne, A., Richards, C. L., & Malouin, F. (2000b). Coactivation during gait as an adaptive behavior after stroke. Journal of Electromyography and Kinesiology, 10, 407–415. Leroux, A., Fung, J., & Barbeau, H. (1999). Adaptation of the walking pattern to uphill walking in normal and spinal-cord injured subjects. Experimental Brain Research, 126, 359–368. Leroux, A., Pinet, H., & Nadeau, S. (2006). Task-oriented intervention in chronic stroke: Changes in clinical and laboratory measures of balance and mobility. American Journal of Physical Medicine & Rehabilitation, 85, 820–830. Macko, R. F., Ivey, F. M., Forrester, L. W., Hanley, D., Sorkin, J. D., Katzel, L. I., et al. (2005). Treadmill exercise rehabilitation improves ambulatory function and cardiovascular fitness in patients with chronic stroke: A randomized, controlled trial. Stroke, 36, 2206–2211. Malouin, F., Potvin, M., Prevost, J., Richards, C. L., & WoodDauphinee, S. (1992). Use of an intensive task-oriented gait training program in a series of patients with acute cerebrovascular accidents. Physical Therapy, 72, 781–789 discussion 789–793. Malouin, F., & Richards, C. L. (2005). Assessment and training of locomotion after stroke: Evolving concepts. In K. M. Refshauge, L. Ada & E. Ellis (Eds.), Science-based Rehabilitation: Theory into Practice (pp. 185–222). Sydney: Elsevier Science. Malouin, F., & Richards, C. L. (2010). Mental practice for relearning locomotor skills. Physical Therapy, 90, 240–251.

Marino, R. J., Barros, T., Biering-Sorensen, F., Burns, S. P., Donivan, W. H., Graves, D. E., et al. (2003). International standards for neurological classification of spinal cord injury. The Journal of Spinal Cord Medicine, 26, S50–S56. Mehrholz, J., Kugler, J., & Pohl, M. (2008). Locomotor training for walking after spinal cord injury. Spine (Phila Pa 1976), 33, E768–E777. Milot, M.-H. (2007). Effet du renforcement musculaire sur les niveaux d'effort des muscles de la cheville et de la hanche lors de la marche chez le sujet hémiparétique. Thèse (Ph.D.). Université de Montréal, Canada, 317 pp.; AAT NR45382. Milot, M. H., Nadeau, S., & Gravel, D. (2007). Muscular utilization of the plantarflexors, hip flexors and extensors in persons with hemiparesis walking at self-selected and maximal speeds. Journal of Electromyography and Kinesiology, 17, 184–193. Milot, M. H., Nadeau, S., Gravel, D., & Bourbonnais, D. (2008). Effect of increases in plantarflexor and hip flexor muscle strength on the levels of effort during gait in individuals with hemiparesis. Clinical Biomechanics (Bristol, Avon), 23, 415–423. Milot, M. H., Nadeau, S., Gravel, D., & Requiao, L. F. (2006). Bilateral level of effort of the plantar flexors, hip flexors, and extensors during gait in hemiparetic and healthy individuals. Stroke, 37, 2070–2075. Moseley, A. M., Stark, A., Cameron, I. D., & Pollock, A. (2005). Treadmill training and body weight support for walking after stroke. Cochrane Database of Systematic Reviews, 4. Art. No.: CD002840. DOI: 10.1002/14651858. CD002840.pub2. See: http://www2.cochrane.org/reviews/en/ ab002840.html. Nadeau, S., Barbeau, H., Garneau, C., & Duclos, C. (2008). Can we assume that the individuals with incomplete spinal cord injury have a symmetrical gait pattern?. North American Congress of Biomechanics (NACOB 2008) Ann Arbor, MI. Nadeau, S., Gravel, D., Arsenault, A. B., & Bourbonnais, D. (1999). Plantarflexor weakness as a limiting factor of gait speed in stroke subjects and the compensating role of hip flexors. Clinical Biomechanics (Bristol, Avon), 14, 125–135. Nilsson, L., Carlsson, J., Danielsson, A., Fugl-Meyer, A., Hellstrom, K., Kristensen, L., et al. (2001). Walking training of patients with hemiparesis at an early stage after stroke: A comparison of walking training on a treadmill with body weight support and walking training on the ground. Clinical Rehabilitation, 15, 515–527. Noble, J., & Prentice, S. (2006). Adaptation to unilateral change in lower limb mechanical properties during human walking. Experimental Brain Research, 169, 482–495. Noël, M., Cantin, B., Lambert, S., Gosselin, C. M., & Bouyer, L. J. (2008). An electrohydraulic actuated ankle foot orthosis to generate force fields and to test proprioceptive reflexes during human walking. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 16, 390–399.

179 Norman, K. E., Pepin, A., & Barbeau, H. (1998). Effects of drugs on walking after spinal cord injury. Spinal Cord, 36, 699–715. Olney, S. J. (2005). Training gait after stroke: A biomechnical perspective. In K. M. Refshauge, L. Ada & E. Ellis (Eds.), Science-based Rehabilitation: Theory into Practice (pp. 159–184). Sydney. Butterworth, Heinemann. Olney, S. J., Griffin, M. P., Monga, T. N., & McBride, I. D. (1991). Work and power in gait of stroke patients. Archives of Physical Medicine and Rehabilitation, 72, 309–314. Olney, S. J., Monga, T. N., & Costigan, P. A. (1986). Mechanical energy of walking of stroke patients. Archives of Physical Medicine and Rehabilitation, 67, 92–98. Olney, S. J., & Richards, C. L. (1996). Hemiparetic gait following stroke Part I: Characteristics. Gait & Posture, 4, 136–148. Pépin, A., Ladouceur, M., & Barbeau, H. (2003). Treadmill walking in incomplete spinal-cord-injured subjects: 2. Factors limiting the maximal speed. Spinal Cord, 41, 271–279. Perry, J., Garrett, M., Gronley, J. K., & Mulroy, S. J. (1995). Classification of walking handicap in the stroke population. Stroke, 26, 982–989. Regnaux, J. P., Pradon, D., Roche, N., Robertson, J., Bussel, B., & Dobkin, B. H. (2008). Effects of loading the unaffected limb for one session of locomotor training on laboratory measures of gait in stroke. Clinical Biomechanics, 23, 762–768. Reisman, D. S., Mclean, H., & Bastian, A. J. (2010). Split-belt treadmill training poststroke: A case study. Journal of Neurologic Physical Therapy, 34, 202–207. Richards, C., & Knutsson, E. (1974). Evaluation of abnormal gait patterns by intermittent-light photography and electromyography. Scandinavian Journal of Rehabilitation Medicine. Supplement, 3, 61–68. Richards, C. L., Malouin, F., Bravo, G., Dumas, F., & WoodDauphinee, S. (2004). The role of technology in task-oriented training in persons with subacute stroke: A randomized controlled trial. Neurorehabilitation and Neural Repair, 18, 199–211. Richards, C. L., Malouin, F., & Dean, C. (1999). Gait in stroke: Assessment and rehabilitation. Clinics in Geriatric Medicine, 15, 833–855. Richards, C. L., Malouin, F., & Dumas, F. (2009). Patterns of locomotor recovery after stroke. In J. E. A. Stein (Ed.), Stroke recovery and rehabilitation. New York, NY: Demos Medical Publishing, Chapter 16. Richards, C. L., Malouin, F., & Wood-Dauphinee, S. (2006). Outcomes measurement: Basic principles and applications in stroke rehabilitation. In M. E. E. A. Selzer (Ed.), Textbook of neural repair and neurorehabilitation. UK: Cambridge University Press. Richards, C. L., & Olney, S. J. (1996). Hemiparetic gait following stroke Part II: Recovery and physical therapy. Gait & Posture, 4, 149–162.

Robertson, G., Caldwell, G., Hamill, J., Kamen, G., & Whittlesey, S. (2004). Research methods in biomechanics. Human Kinetics, 320 pp. see: http://www.exrx.net/Store/ HK/ResearchMethodsBiomechanic.html. Salbach, N. M., Mayo, N. E., Wood-Dauphinee, S., Hanley, J. A., Richards, C. L., & Cote, R. (2004). A taskorientated intervention enhances walking distance and speed in the first year post stroke: A randomized controlled trial. Clinical Rehabilitation, 18, 509–519. Saunders, D. H., Greig, C. A., Mead, G. E., & Young, A. (2009). Physical fitness training for stroke patients. Cochrane Database of Systematic Reviews, 4. Art. No.: CD003316. DOI: 10.1002/14651858.CD003316. Shiavi, R., Bugle, H. J., & Limbird, T. (1987). Electromyographic gait assessment, Part 2: Preliminary assessment of hemiparetic synergy patterns. Journal of Rehabilitation Research and Development, 24, 24–30. States, R. A., Salem, Y., & Pappas, E. (2009). Overground gait training for individuals with chronic stroke: A cochrane systematic review. Journal of Neurologic Physical Therapy, 33, 179–186. Stewart, J. E., Barbeau, H., & Gauthier, S. (1991). Modulation of locomotor patterns and spasticity with clonidine in spinal cord injured patients. The Canadian Journal of Neurological Sciences, 18, 321–332. Sunnerhagen, K. S. (2010). Commentary on gait training in patients with spinal cord injury. Journal of Rehabilitation Medicine, 42, 527. Swinnen, E., Duerinck, S., Baeyens, J. P., Meeusen, R., & Kerckhofs, E. (2010). Effectiveness of robot-assisted gait training in persons with spinal cord injury: A systematic review. Journal of Rehabilitation Medicine, 42, 520–526. Teixeira-Salmela, L. F., Nadeau, S., McBride, I., & Olney, S. J. (2001). Effects of muscle strengthening and physical conditioning training on temporal, kinematic and kinetic variables during gait in chronic stroke survivors. Journal of Rehabilitation Medicine, 33, 53–60. Teixeira-Salmela, L. F., Nadeau, S., Milot, M. H., Gravel, D., & Requiao, L. F. (2008). Effects of cadence on energy generation and absorption at lower extremity joints during gait. Clinical Biomechanics (Bristol, Avon), 23, 769–778. Teixeira-Salmela, L. F., Olney, S. J., Nadeau, S., & Brouwer, B. (1999). Muscle strengthening and physical conditioning to reduce impairment and disability in chronic stroke survivors. Archives of Physical Medicine and Rehabilitation, 80, 1211–1218. Tilson, J. K., Sullivan, K. J., Cen, S. Y., Rose, D. K., Koradia, C. H., Azen, S. P., et al. (2010). Meaningful gait speed improvement during the first 60 days poststroke: Minimal clinically important difference. Physical Therapy, 90, 196–208. Van De Port, I. G., Wood-Dauphinee, S., Lindeman, E., & Kwakkel, G. (2007). Effects of exercise training programs

180 on walking competency after stroke: A systematic review. American Journal of Physical Medicine & Rehabilitation, 86, 935–951. Van Hedel, H. J., & Dietz, V. (2010). Rehabilitation of locomotion after spinal cord injury. Restorative Neurology and Neuroscience, 28, 123–134. Van Hedel, H. J., Wirth, B., & Dietz, V. (2005). Limits of locomotor ability in subjects with a spinal cord injury. Spinal Cord, 43, 593–603. Visintin, M., & Barbeau, H. (1994). The effects of parallel bars, body weight support and speed on the modulation of the locomotor pattern of spastic paretic gait. A preliminary communication. Paraplegia, 32, 540–553. Wevers, L., Van De Port, I., Vermue, M., Mead, G., & Kwakkel, G. (2009). Effects of task-oriented circuit class

training on walking competency after stroke: A systematic review. Stroke, 40, 2450–2459. Winter, D. A. (1991). The biomechanics and motor control of human gait: Normal, elderly and pathological. Waterloo, Canada: University of Waterloo Press. Wirz, M., Zemon, D. H., Rupp, R., Scheel, A., Colombo, G., Dietz, V., et al. (2005). Effectiveness of automated locomotor training in patients with chronic incomplete spinal cord injury: A multicenter trial. Archives of Physical Medicine and Rehabilitation, 86, 672–680. Yang, Y. R., Tsai, M. P., Chuang, T. Y., Sung, W. H., & Wang, R. Y. (2008). Virtual reality-based training improves community ambulation in individuals with stroke: A randomized controlled trial. Gait & Posture, 28, 201–206.

A. M. Green, C. E. Chapman, J. F. Kalaska and F. Lepore (Eds.) Progress in Brain Research, Vol. 192 ISSN: 0079-6123 Copyright Ó 2011 Elsevier B.V. All rights reserved.

CHAPTER 12

Involvement of the corticospinal tract in the control of human gait Dorothy Barthélemy{,*, Michael J. Grey{, Jens Bo Nielsen},} and Laurent Bouyerk,# {

School of Rehabilitation, Faculty of Medicine, Université de Montréal, Montréal, Québec, Canada { School of Sport and Exercise Sciences, University of Birmingham, Edgbaston, Birmingham, United Kingdom } Department of Exercise and Sport Sciences, Panum Institute University of Copenhagen, Copenhagen, Denmark } Department of Neuroscience and Pharmacology, Panum Institute University of Copenhagen, Copenhagen, Denmark k Department of Rehabilitation, Faculty of Medicine, Université Laval, Québec City, Canada # Center for Interdisciplinary Research inn Rehabilitation and Social Integration, Québec City, Québec, Canada

Abstract: Given the inherent mechanical complexity of human bipedal locomotion, and that complete spinal cord lesions in human leads to paralysis with no recovery of gait, it is often suggested that the corticospinal tract (CST) has a more predominant role in the control of walking in humans than in other animals. However, what do we actually know about the contribution of the CST to the control of gait? This chapter will provide an overview of this topic based on the premise that a better understanding of the role of the CST in gait will be essential for the design of evidence-based approaches to rehabilitation therapy, which will enhance gait ability and recovery in patients with lesions to the central nervous system (CNS). We review evidence for the involvement of the primary motor cortex and the CST during normal and perturbed walking and during gait adaptation. We will also discuss knowledge on the CST that has been gained from studies involving CNS lesions, with a particular focus on recent data acquired in people with spinal cord injury. Keywords: locomotion; spinal cord injury.

motor

cortex;

imaging;

*Corresponding author. Tel.: þ1-514-343-7712; Fax: þ1-514-343-6929 E-mail: [email protected] DOI: 10.1016/B978-0-444-53355-5.00012-9

181

transcranial

magnetic

stimulation;

coherence;

182

Assessment of the contribution of the corticospinal tract to normal walking It is often assumed that supraspinal centers play a larger role in the control of human gait than do similar centers in quadrupedal mammals (Eidelberg, 1981; Fedirchuk et al., 1998; Gorassini et al., 2004; Nielsen, 2003). However, how much do we actually know about this and what is the precise role of the corticospinal tract (CST) in the control of human gait? The involvement of cortical structures in the control of human gait can be assessed using several noninvasive methods, such as neuroimaging, transcranial magnetic and electrical stimulation (TMS and TES), and EMG–EMG and EEG–EMG coherence. Here, we will first review studies in which these techniques have been used to address the role of the CST in normal human gait. We will then discuss the role of the CST in modifications of gait and we will finally discuss the significance of corticospinal lesion for impairment of gait and to what an extent recovery of gait ability following brain and spinal cord injury (SCI) is contingent on corticospinal transmission.

Neuroimaging Recently, rapid progress in the neuroimaging field has made it possible to use functional imaging of the brain to investigate cortical involvement during human walking. Fukuyama et al. (1997) evaluated changes in brain activity during walking in healthy subjects using single-photon emission computed tomography (SPECT). This technique takes advantage of radioactive substances that are rapidly distributed in the brain in proportion to the regional blood flow and are then retained in the brain for hours. The distribution of the radioactive substance therefore represents the regional metabolic activity of the brain at the time of injection and an image of areas that are active during walking may consequently be obtained by injecting the

substance while subjects are walking. Fukuyama et al. (1997) found large activity in the primary sensorimotor areas, supplementary motor area (SMA), basal ganglia, and the cerebellum, suggesting that all of these supraspinal areas are involved in the control of gait (see also Hanakawa et al., 1999a,b). The SPECT technique has a limited spatial resolution, but later studies have confirmed these findings for glucose uptake measured by positron emission tomography (PET) during both walking (Mishina et al., 1999) and running (Tashiro et al., 2001). Both SPECT and PET require measurements over several minutes, whereas near infrared spectroscopy (NIRS) provides a possibility of obtaining a gross idea of regional blood flow in the brain in real time (Jobsis, 1977; Maki et al., 1995). Miyai et al. (2001) used this technique to demonstrate that the medial portion of the primary sensorimotor regions and SMAs were bilaterally activated during walking on a treadmill. However, neither of these studies were able to exclude the possibility that the cerebral activation was caused by sensory feedback from the moving limbs and that the cortical areas were therefore not actively involved in generating the locomotor activity. To address this problem, Christensen et al. (2000) compared active and passive bicycling movements, using oxygen-15-labeled H2O PET. Muscle activity during human bicycling is very similar to other forms of locomotion and is likely generated by a similar central network (Raasch and Zajac, 1999). Christensen et al. (2000) found that the major part of the activation in the sensorimotor areas during bicycling was in all likelihood generated by sensory feedback. However, when subtracting the activation during passive from that during active bicycling, significant activation was still observed in the leg area of the primary motor cortex (M1; Fig. 1a–d), suggesting that neurons in this area take an active part in generating the locomotor activity. This idea has been strengthened by noninvasive electrophysiological studies as will be outlined below.

183

(a)

10.3 t 4.3

z = 58

y = –34

(b) 8.7 t 4.3

z = 58

y = –34

(c) 3.5 t 2.5

z = 58

y = –34

(d) 4.7 t 4.3

z = 58

y = –2

Fig. 1. The t-statistics superimposed on the subject's average MR brain image measured by PET scan. (a) Active minus rest. (b) Passive minus rest. (c) Active minus passive. (d) Imagination minus rest. z and x indicate the Talairach coordinates for the slices displayed. The color bar indicates the scaling of the t-image. The color scale is linear. Bilateral activation is observed in the sensorimotor cortex during cycling. Active minus passive condition reveals that M1 is only activated during active cycling. This activation was correlated to cycling speed. Reproduced from Christensen et al. (2000).

Transcranial stimulation (TMS and TES) Several studies have applied TMS over the leg area of the motor cortex during walking. Most of these have evaluated the phase-dependent modulation of motor evoked potentials (MEPs) to obtain evidence of changes in cortical excitability (Barthelemy and Nielsen, 2010; Bonnard et al., 2002; Capaday et al., 1999; Schubert et al., 1997). A general finding in these studies has been that the MEPs are greatly modulated during the gait cycle, being largest when the muscle, in which

they are recorded, is active and smallest when the antagonist muscle is active. This is consistent with the idea that the excitability of the corticospinal neurons is modulated during the gait cycle with the highest activity when the target muscle is active. However, MEPs are mainly sensitive to changes in spinal motorneuronal (Matthews, 1999; Schneider et al., 2004) and interneuronal excitability (Geertsen et al., 2010; Nielsen et al., 1999) and cannot be used as a reliable measure of cortical excitability. Petersen et al. (1998) improved the technique in several ways: First,

184

they reduced the intensity of TMS so that it was just below the threshold for eliciting an MEP, which means that the CST volley did not drive the spinal motoneurones to discharge. At this stimulus intensity, TMS will still activate corticospinal pathways and influence the excitability of the spinal motoneurones, but without discharging them. They then used H-reflex testing to assess these excitability changes. This provided the possibility of dissociating early occurring (direct and/or fast conducting) from later occurring (indirect and/or slower conducting) corticospinal effects and to demonstrate the existence of inhibitory pathways activated by TMS. They observed that the earliest occurring facilitation of the soleus (SOL) H-reflex was very large in the stance phase of walking where the SOL muscle is active. In all likelihood, the earliest facilitation is mediated by the most direct, monosynaptic, and fastest conducting corticospinal pathway. To prove that this large stance-phase facilitation reflected the high excitability of the involved corticospinal neurons, they also tested the effect of TES on the SOL H-reflex. The idea behind this is that TES and TMS activate the same corticospinal pathways, but at different sites and are therefore differently sensitive to changes in the excitability of the corticospinal neurons (Day et al., 1987; Edgley et al., 1997; Nielsen et al., 1995). Whereas TES penetrates relatively deep into the brain and mainly activates the axons of the corticospinal neurons, TMS induces current in the superficial parts of the cortex where the cell bodies are located and activates the corticospinal neurons indirectly (i.e., transsynaptically) or directly close to the soma. In other words, an MEP evoked by TMS will reflect the presynaptic and postsynaptic activation state of the motor cortex, but an MEP evoked by TES will only reflect the activation state in subcortical structures and will not be modulated by the activation state of the motor cortex. Any state-dependent changes in TMS effects that are not seen with TES would therefore reflect the activation state of M1. A comparison of the

effects induced by the two modes of stimulation may therefore be used to obtain evidence of changes in corticospinal excitability. In contrast to what was observed for the early facilitation evoked by TMS, Petersen et al. (1998) observed that the early facilitation evoked by TES was quite small during walking. While this evidence is indirect, it strongly suggests that corticospinal neurons are active during walking and that they contribute to the muscle activity even during uncomplicated gait. In a subsequent study, Petersen et al. (2001) provided more direct evidence for a corticospinal contribution to gait. Based on an earlier study by Davey et al. (1994), they reasoned that if corticospinal neurons make a contribution to the activation of spinal motoneurones during walking, then activation of inhibitory mechanisms within the cortex by subthreshold TMS should reduce cortical output and cause a suppression of the ongoing EMG activity. Indeed, this is also what they observed (see Fig. 2a–c). Subthreshold TMS, which consists of a TMS pulse given at an intensity below motor threshold for corticospinal neurons, could activate inhibitory cortical interneurones and effectively suppress EMG activity in the tibialis anterior (TA) muscle during the swing phase of walking and in the SOL muscle during the stance phase. On the contrary, subthreshold TES, which does not activate the inhibitory cortical interneurones, showed no such effect, suggesting the involvement of motor cortex in the activation of leg muscles during gait. Similar findings have also been reported for the muscle activity in arm muscles during walking (Fig. 2d–f; Barthelemy and Nielsen, 2010).

EMG:EMG synchronization It is also possible to obtain evidence of the synaptic drive to spinal motoneurones by analyzing the coupling of motor unit activities in the active muscles in the time and frequency domains. In the time domain, one distinct feature of the

185 TA

(a)

PD

60 %

(d) 0.95 T Facilitation

(b)

55 %

(e)

(c)

50 %

(f)

10 ms

20 µV

0.72 T

Inhibition control With stim.

0.68 T

5 µV 20 ms

Fig. 2. Subthreshold inhibition in tibialis anterior (TA; a–c) and in posterior deltoid (PD; d–f). PD EMG during locomotion when subthreshold TMS was applied 700 ms (TA) and 600 ms (PD) after heel strike (dark trace) at different intensities. Background EMG with no stimulation is represented by the trace in light gray. The white arrows point to the facilitation and the black arrows point to the inhibited portion of the EMG. The vertical dashed line in (a) and (d) shows the onset of the facilitation. The vertical solid lines in (c) and (f) delineate the inhibition area. Reproduced in part from Petersen et al. (2001) and Barthelemy and Nielsen (2010).

coupling between motor unit activities both during voluntary static contraction and walking is shortterm synchrony, which is characterized by a very narrow 10–15 ms peak of synchronization between the motor units (Bremner et al., 1991; Halliday et al., 2003; Nielsen and Kagamihara, 1993). Shortterm synchrony is thought to indicate the presence of input to the motoneurons from branches of common last order neurones, but less direct coupling cannot be completely excluded and part of the common synaptic drive may arise from a less well-defined corticospinal activation (Datta et al., 1991; Farmer et al., 1993a; Kilner et al., 1999; see also Nielsen et al., 2008). There may be several sources of short-term synchrony, but since the synchronization is absent or greatly reduced in patients with SCI, cortical lesion, or internal capsule lesion (Datta et al., 1991; Farmer et al., 1993b; Hansen et al., 2005; Nielsen et al., 2008), motor cortex activity is likely to be involved in its occurrence. Hansen et al.

(2001) found that short-term synchrony was only observed between close synergists acting on the same joint, suggesting a very focused distribution of the terminals from the underlying network. They also observed that the amount of the synchronization was modulated greatly during the gait cycle. For the TA muscle, the synchronization that was largest corresponded to the two bursts of EMG activity in early and late swing, suggesting that the involved network mainly contributes to the TA EMG at these two times of the gait cycle. Although this may relate to corticospinal activity, it should be pointed out that short-term synchrony with a very similar distribution is also observed between motor units during fictive locomotion in cats with complete spinal transections (Nielsen et al., 2005). In the frequency domain, coupling between motor units is usually observed during static contraction in frequency bands around 8–12 and

186

15–35 Hz (Davey et al., 1993; Farmer et al., 1993a). There is good evidence to suggest that coherence in the latter frequency band reflects corticospinal activity. Coherence in a similar frequency band may be observed in paired EEG and EMG recordings. Indeed coupling at  20 Hz is observed in human and primates between EEG or MEG (magnetoencephalography) and EMG recorded simultaneously (Brown et al., 1998; Conway et al., 1995; Halliday et al., 1998; Mima and Hallett, 1999; Salenius et al., 1997). In addition, TMS selectively enhances coherence in that frequency band for both paired EEG:EMG and EMG:EMG recordings (Hansen and Nielsen, 2004). During walking, coherence is not seen in distinct frequency bands as during static contraction, but rather over a broad frequency range from 1 to 50 Hz with more or less easily identifiable peaks of coherence around 10 Hz and around 15–20/25 Hz (Halliday et al., 2003; Hansen et al., 2001, 2005; Nielsen et al., 2008). Some of the coherence observed during walking thus falls in the same frequency band as that observed during static contraction, but this does not necessarily indicate that similar mechanisms are involved. However, the observation that coherence in the 20–50 Hz frequency band is absent or greatly reduced in patients with stroke (Nielsen et al., 2008) or SCI (Barthelemy et al., 2010; Hansen et al., 2005) suggests that there is a possibility that the CST is also involved in the generation of the coherence in this frequency band during walking. Interestingly, coherence in the upper part of this frequency band has been shown to correlate with the maturation of gait during childhood (Petersen et al., 2010). This may potentially reflect maturation of the CST. The TMS and coherence observations support the idea that the motor cortex and the CST are an integrated part of the central network involved in the generation of EMG activity in leg and arm muscles during human walking, and provides evidence that although they seem automatic and reflexive in nature, rhythmical movements rely on a neuronal network that involves the motor cortex.

Corticospinal involvement in adaptation and perturbation of gait Modifications of gait in response to the requirements of the environment can be broadly separated into either proactive or reactive mechanisms, depending on the origin of the signal initiating the command. Proactive modifications originate in the central nervous system (CNS) (e.g., negotiating an obstacle), while reactive modifications result from the activation of the peripheral nervous system by the environment itself (e.g., stumbling reaction). These two types of gait modifications are first discussed separately below, followed by experiments requiring coordination between the two.

Proactive modifications of the gait pattern In the cat, experiments with more challenging locomotor tasks, such as ladder walking, obstacle navigation/avoidance have shown that corticospinal neurons increase their discharge (Amos et al., 1990; Armstrong, 1986; Beloozerova and Sirota, 1998; Drew, 1991). This may also be the case in human subjects, since Schubert et al. (1999) observed that TA and gastrocnemius medialis (GM) MEPs were larger during visually guided treadmill walking than during normal treadmill walking. Similar results were obtained by Bonnard et al. (2002) who showed that a facilitation of MEP is observed in rectus femoris and biceps femoris during a locomotor task that requires fine control of the proximal upper-leg muscles. As shown by the relatively small number of studies performed in humans on this topic, much remains to be learned regarding the neural mechanisms underlying proactive gait control.

Reactive modifications of the gait pattern It is essential that the muscle activity during gait is rapidly and efficiently adjusted according to the immediate requirements of the environment,

187

especially in relation to instability of the supporting ground. Some of these adjustments rely on spinal reflex mechanisms (af Klint et al., 2008, 2010; Grey et al., 2004; Sinkjaer et al., 2000), but longer-latency responses mediated by pathways traversing the brain stem, cerebellum, or cortex may also contribute (Christensen et al., 2001; Phillips, 1969). In humans, cutaneous stimulation during gait evokes multiple reflex responses in ankle and knee muscles similar to the stumbling corrective reactions described in the cat (Duysens et al., 1990; Yang and Stein, 1990). During static contraction, reflexes are observed with latencies of 40–50 ms, which is consistent with a spinal reflex pathway (Aniss et al., 1992; Nielsen et al., 1997). However, during gait, these short-latency responses are surprisingly small and only observed in very few subjects. The dominant response observed during gait has a latency around 70–80 ms (Baken et al., 2005), suggesting that reflexes with a definite spinal origin are suppressed during human gait. The latency of the reflexes observed during gait is sufficiently long that they can be mediated by a transcortical pathway (Christensen et al., 2000; Nielsen et al., 1997) and experiments using TMS have supported this possibility. Cutaneous stimulation of the foot (sural nerve and superficial peroneal nerve) thus facilitates muscular responses elicited by TMS at the latency of these reflexes, but has no effect on muscular responses evoked by TES (Christensen et al., 1999; Nielsen et al., 1997; Pijnappels et al., 1998). As already mentioned above, this is most easily explained by a different sensitivity of the two modes of stimulation to changes in cortical excitability. The observation thus suggests that the cutaneous stimulation activates a pathway that influences the excitability of the corticospinal neurons at the time of the reflex. This makes it likely that a transcortical reflex pathway is responsible for the cutaneous reflex. Further evidence in support of this notion comes from the observation that the cutaneous reflex is absent or small during walking in stroke patients (Zehr et al., 1998).

Similar observations have also been reported for stretch reflexes mediated by muscle spindle afferents (Christensen et al., 2001; Petersen et al., 1998). During static voluntary dorsiflexion stretch of the TA, muscle elicits a muscular response that may be subdivided in three main components, M1, M2, and M3. However, during the swing phase of the step cycle only the early (M1) and late (M3) components of the TA stretch reflex were evoked. In the stance phase, only one response could be induced, with a peak at a similar onset as M3. When TMS was applied at a time corresponding to the peak of the muscular response in stance phase and the late reflex response in the swing phase, MEPs elicited in the TA were strongly facilitated. Such facilitation was not observed during the earlier response or when MEPs were evoked by TES in the swing phase. To obtain further evidence of a cortical contribution to the M3 response in the stance phase, Zuur et al. (2009) used repetitive 1 Hz TMS to temporarily reduce corticospinal excitability (Chen et al., 1997; Muellbacher et al., 2000; see review Fitzgerald et al., 2006). Application of 1 Hz rTMS (repetitive transcranial magnetic stimulation) over the leg area of the motor cortex suppressed the late stretch reflex response during stance, further suggesting that the reflex is at least partly mediated by a transcortical pathway. Walking in a “force field” as a means of studying interactions between proactive and reactive mechanisms Force fields can be used to change the environmental demand into which movement is performed (Lackner and Dizio, 1994; Shadmehr and Mussa-Ivaldi, 1994). When subjects are walking in such a modified environment, locomotor output is gradually modified (Blanchette and Bouyer, 2009). This modification occurs partly through a change in central drive (Noel et al., 2009). Force field adaptation is therefore a model

188

of anticipatory gait modifications (proactive mechanism) triggered by a change in the environment (reactive mechanism). While a role for the cerebellum in adaptation of gait has been proposed in the literature (see Thach and Bastian, 2004 for review), what is the role of the motor cortex or of the CST in this adaptation? Recent experiments in our laboratory have addressed this issue (Alain et al., 2007). Changes in MEPs elicited by TMS were assessed in the TA muscle before, during, and after healthy subjects adapted to a force field applied to the ankle joint during treadmill walking. Preliminary data show that if dorsiflexion in the swing phase is assisted by the force field, TA EMG and MEP amplitude are decreased during gait adaptation. If dorsiflexion was resisted, both TA EMG and MEP were increased during gait adaptation. Changes in corticospinal excitability preceded changes in EMG and were task specific, that is, that they were not observed when subjects were tested during static dorsiflexion immediately after adaptation. Overall, these preliminary results suggest that the CST has an important role in adapting the locomotor output to the constraints of the environment. These adaptations can be proactive, reactive, or both. Supraspinal control may allow flexibility and proactive modulation of reflexes in response to a combination of motivational and environmental influences (Chan, 1983).

Walking after impairment to the CST or motor cortex: deficits and adaptive plasticity Level of CST excitability early after stroke or SCI is a predictor of locomotor recovery In the majority of incomplete SCI patients, MEPs can be recorded in TA, but the latency of the response is prolonged (slower spinal conduction velocity) and MEP amplitude is reduced (Chang and Lien, 1991). Curt et al. (1998) demonstrated that all patients who displayed normal MEP in

TA, that is, with a latency and amplitude similar to control, within the first 6 months posttrauma recovered full ambulatory capacity. For patients in whom MEPs could not be elicited in TA, only 11% recovered full gait ability whereas 78% showed little or no ambulatory activity. Similarly, stroke patients did not regain walking ability when MEPs could not be elicited in the first month poststroke, whereas patients who exhibited an MEP larger than 8% of normal amplitude in the first month regained independent gait (Piron et al., 2005).

Locomotor training induces cortical plasticity after stroke Using fMRI, Dobkin et al. (2004) assessed cortical activation during voluntary ankle movements performed by chronic stroke patients. In the first assessment, cortical activation was observed in the trunk and foot area of the sensorimotor cortex (SMC), but after 2 weeks of treadmill training, the quality of the voluntary ankle movement improved and activity was seen more focally in the foot representation of the SMC, resembling the activation pattern seen in control subjects. Thus, areas surrounding the lesion might be recruited in order to compensate for a deficit. Several studies have confirmed this reduction in cortical activation with functional improvement for both leg and arm movements (Enzinger et al., 2008; Lundell et al., 2010b; see also Ward, 2004 for hand function). The general idea is that patients need to recruit additional cortical areas in order to activate the muscles when function is strongly impaired. As function improves, less cortical resources appear to be required to generate the same functional output to the muscles. It is therefore surprising that Enzinger et al. (2009), using fMRI, found increased bilateral activation in SMC with functional recovery in patients, who underwent 4 weeks of treadmill training. The reason for this discrepancy has not been fully clarified. These studies all measured cortical

189

activation during voluntary ankle movements rather than during gait. Although the studies thus do provide evidence of cortical reorganization, they do not necessarily say anything about changes in the cortical activity during gait. It is therefore of interest that Miyai et al (2003) used NIRS to demonstrate reduced activation in the nonaffected hemisphere and enhanced activation in the affected hemisphere during gait in unilateral stroke patients following 2 months of inpatient multidisciplinary rehabilitation. This suggests that a functionally significant change induced by the rehabilitation in unilateral stroke patients may be to improve the balance in the contribution of the two hemispheres to the activation of the muscles during gait (Miyai et al., 2006).

Locomotor training induces cortical plasticity after a lesion to the spinal cord Cortical reorganization can also be assessed by electrophysiological techniques after training and recovery of function following incomplete SCI. After 3–5 months of intensive body-weight support treadmill training (BWSTT), Thomas and Gorassini (2005) showed that the amplitude of the MEP increased in leg muscles of SCI individuals. This increase was correlated to locomotor recovery assessed by the WISCI II scale and the 6-min walking test (distance that a SCI subject can walk in 6 min). As mentioned above, assessment of coherence during gait may also reflect corticospinal input. Norton and Gorassini (2006) showed an increase of coherence between knee muscles in the 24–40 Hz frequency band in SCI patients after they were trained with BWSTT. Since this coherence is not observed in able-bodied individuals, these results underline the plasticity of corticospinal inputs after an injury and also after training. Overall, these studies suggest that excitability of spared CST fibers is increased after training and that this corticospinal facilitation is “being used” during the improved gait (Yang and Gorassini, 2006).

Combining electrophysiological, imaging, and gait analysis to assess the role of the CST after injury Based on the above data, lesion to the motor cortex or CST will decrease corticospinal excitability and will induce plastic changes involving both ipsilateral and contralateral cortices. However, the vast majority of these studies did not investigate the actual locomotor pattern (kinematics and EMG) produced and it is not clear what are the gait impairments specifically due to lesion in the CST. The few studies that compared cortical excitability with gait performance often correlated it with gait scales (e.g., WISCI) or gait speed, which are not well correlated with the degree of CST damage (Ahn et al., 2006; Dawes et al., 2008; Jang et al., 2006) or sensitive enough to detect deficits in distal joints (e.g., ankle flexors). For example, a functional foot drop is often seen in patients after cortical or spinal cord damage (Burridge et al., 2001). This could be due to impairment of the corticospinal drive to the TA muscle. Seen this way, lesion to the CST might be better correlated to impairment in functional parameters at a specific phase of the gait cycle (e.g., abnormal muscle activation), rather than to actual walking speed. Recently, we addressed this issue and attempted to correlate parameters that reflect corticospinal function, such as intramuscular coherence of TA and corticospinal excitability measured by TMS, to an objective measure of foot drop in persons with incomplete SCI (ASIA D) in the chronic stage (more than 1 year; Barthelemy et al., 2010). Using kinematics, foot drop was determined by measuring toe elevation (Fig. 3a; Bensoussan et al., 2006; Chin et al., 2009; Weber et al., 2004). This measure was correlated with overground speed, the patients having the lowest toe elevation taking a longer time to walk 10 m (Fig. 3b). Latency and amplitude at rest of the MEP evoked in TA were significantly correlated to the degree of foot drop, the most impaired SCI participants having longer latencies and lower amplitudes at rest or no MEP at rest (Fig. 3c and d). We also assessed intramuscular

190 (a)

Toe elevation measurements

Correlation between speed and highest toe elevation

(b) 2 Speed (m/s)

(mm)

100 50 0 SWING

1,6 1,2 0,8 0,4 0

Amplitude

EMG

45 40 35 30 0

20 40 60 80 100 Highest toe elevation (mm)

20 60 100 Frequency (Hz) TA 1 TA 2

Coherence (area)

5,7

0.2

007CTRL

HC EMG recordings

120

Correlation between coherence and highest toe elevation

(f)

0.4

0

50

25

Coherence

(e)

120

55

Latency

Stimulation

80 100 20 40 60 Highest toe elevation (mm)

Correlation between MEP latency and highest toe elevation

(d)

MEP measurements

MEP latency (ms)

(c)

0

3

2 toe elev. 1 toe elev. Drag

2

Control

1

Short-term synchrony

4

0

0

20 40 60 80 100 120 Highest toe elevation (mm)

Fig. 3. (a) During swing, the toe elevates twice: at the beginning and at the end of swing (arrows). (b) Measurement of the highest elevation of the toe was plotted against overground speed for SCI participants. (c) TMS measurements consisted of latency and amplitude of MEP. (d) Both latency and amplitude (not shown) were correlated to highest toe elevation of the participants. (e) Coherence was measured from paired TA EMG recordings. (f) The area of coherence between 10 and 20 Hz was plotted against highest toe elevation of the participants. Grey boxes in (d) and (f) encompass the values of the control participants tested.

coherence of TA EMG during gait and measured coherence within the 10–20 Hz bracket but also in the 20–50 Hz bracket, such as reported in the control participants (Fig. 3e). This measure was also significantly correlated to the degree of foot drop, the most impaired SCI participants having little or no

coherence (Fig. 3f). Thus, these data suggest that impairment of CST function, determined by reduced MEP amplitude, increased MEP latency, and lack of common synaptic input to the TA motor units, is correlated to the degree of foot drop during gait.

191

We further investigated those deficits to determine anatomical correlates of CST lesion in SCI individuals with lesion to the cervical cord. We first estimated the reduction in spinal cord area, anteroposterior width, and left–right width as measures of atrophy using conventional structural MRI. Spinal atrophy had been observed in other populations with neurological deficits (Agosta et al., 2009; Krabbe et al., 1997; Sundblom et al., 2009), and good correlation had been reported between spinal cord area and several clinical tests (Bar-Zohar et al., 2008). In the present study, atrophy at the cervical level along the anteroposterior and left–right widths were clearly correlated to sensory and motor scores, respectively (Lundell et al., 2011). The atrophy in the left–right axis is thought to reflect a reduction in the size of the lateral columns, where the CST is located. Thus, atrophy in this axis could reflect impairment to the CST. Based on data shown above for stroke, a lesion to the CST would trigger plastic changes at the cortical level, both in the ipsilateral and contralateral hemisphere. This is indeed what was observed, patients showing a greater activation in ipsilateral SMC and premotor cortex (PMC) had more atrophy in the left–right axis of the spinal cord and had lowest scores in the clinical tests (Lundell et al., 2010b) These data suggest that lesion of descending tracts in lateral parts of the spinal cord, likely including CST, result in increased activation of SMC. Overall, these studies suggest that a lesion to the CST will bring a decrease in corticospinal excitability as tested by TMS, a reduction in the common synaptic drive to the TA muscle, and an impairment in foot placement during gait (i.e., foot drop). In parallel, the more damaged the CST, the more reorganization will take place in the ipsilateral and contralateral hemispheres, probably in an attempt to compensate for the deficits.

Enhancing recovery after a lesion to the CNS Recovery of function following lesions to the nervous system has been shown to involve

reorganization of networks both above and below the lesion (Dietz, 2010; Dobkin, 2000; Ward, 2005a,b). One frequently used marker of corticospinal reorganization is the excitability of the corticospinal pathway as investigated by using TMS. Hence, several rehabilitation therapies are now being developed to increase excitability in the affected motor cortex, including decreasing the dominant effect of the unaffected SMC in the rehabilitation process and thereby improving gait function (Kim et al., 2006; Miyai et al., 2003, 2006). Many approaches have already proven effective at increasing corticospinal excitability in patients, including gait training with partial body weight support (Dobkin et al., 2006; Harkema et al., 1997; Visintin et al., 1998) or with the use of a robot (Colombo et al., 2000, 2001), rTMS (Belci et al., 2004; Chen et al., 1997; Mansur et al., 2005), transcranial direct current stimulation (tDCS; Nitsche and Paulus, 2000) or paired associative stimulation (PAS; Jayaram and Stinear, 2008; Roy et al., 2007, 2010; Roy and Gorassini, 2008; Stinear and Hornby, 2005), and motor skill training (Carey et al., 2004; Perez et al., 2004). The latter study not only induced increased cortical excitability but also demonstrated the translation of increased corticospinal excitability into improvement of gait. Further, many of the techniques described in this review were recently included in a clinical initiative aiming at determining outcome measures that could reliably be used to measure impairment and effect of the treatment (Ellaway et al., 2007). Performing these measures at regular intervals is essential to decide if the therapeutic approaches activate the neuronal networks sufficiently to improve recovery.

Conclusion In this review, we have described a variety of techniques that provide a relatively detailed understanding of the contribution of the CST during walking. Although investigations using these techniques have added considerably to our

192

knowledge, many questions remain about the precise nature of the contribution of the CST to the adaptation of gait to environmental challenges and its role in the recovery of gait following lesions to the CNS. We believe the combination of gait analysis techniques with neurophysiological assessment techniques will allow us to more precisely determine the damage done to the CNS after an injury and allow us to design/ improve our therapeutic approaches that will enhance locomotor recovery.

Abbreviations ASIA BWSTT CNS CST EEG fMRI GM M1 MEG MEP NIRS PD PET PMC SCI SMA SMC SOL SPECT TA TES TMS WISCI

American spinal cord injury association body-weight support treadmill training central nervous system corticospinal tract electroencephalography functional magnetic resonance imaging gastrocnemius medialis primary motor cortex magnetoencephalography motor evoked potential near infrared spectroscopy posterior deltoid positron emission tomography premotor cortex spinal cord injury supplementary motor area sensorimotor cortex soleus single-photon emission computed tomography tibialis anterior transcranial electrical stimulation transcranial magnetic stimulation walking index for spinal cord injury

References af Klint, R., Mazzaro, N., Nielsen, J. B., Sinkjaer, T., & Grey, M. J. (2010). Load rather than length sensitive feedback contributes to soleus muscle activity during human treadmill walking. Journal of Neurophysiology, 103, 2747–2756. af Klint, R., Nielsen, J. B., Cole, J., Sinkjaer, T., & Grey, M. J. (2008). Within-step modulation of leg muscle activity by afferent feedback in human walking. Journal of Physiology, 586, 4643–4648. Agosta, F., Valsasina, P., Absinta, M., Sala, S., Caputo, D., & Filippi, M. (2009). Primary progressive multiple sclerosis: Tactile-associated functional MR activity in the cervical spinal cord. Radiology, 253, 209–215. Ahn, Y. H., Ahn, S. H., Kim, H., Hong, J. H., & Jang, S. H. (2006). Can stroke patients walk after complete lateral corticospinal tract injury of the affected hemisphere? Neuroreport, 17, 987–990. Alain, S., Barthelemy, D., Grey, M. J., Bouyer, L. J. G., Richards, C. L., & Nielsen, J. B. (2007). Rapid, task-specific modifications of cortico-spinal excitability during adaptation of human locomotion to elastic force fields applied to the ankle. Program No. 924.1 Neuroscience Meeting Planner. San Diego, CA: Society for Neuroscience 2007. Online. Amos, A., Armstrong, D. M., & Marple-Horvat, D. E. (1990). Changes in the discharge patterns of motor cortical neurones associated with volitional changes in stepping in the cat. Neuroscience Letters, 109, 107–112. Aniss, A. M., Gandevia, S. C., & Burke, D. (1992). Reflex responses in active muscles elicited by stimulation of lowthreshold afferents from the human foot. Journal of Neurophysiology, 67, 1375–1384. Armstrong, D. M. (1986). Supraspinal contributions to the initiation and control of locomotion in the cat. Progress in Neurobiology, 26, 273–361. Baken, B. C., Dietz, V., & Duysens, J. (2005). Phase-dependent modulation of short latency cutaneous reflexes during walking in man. Brain Research, 1031, 268–275. Barthelemy, D., & Nielsen, J. B. (2010). Corticospinal contribution to arm muscle activity during human walking. Journal of Physiology, 588, 967–979. Barthelemy, D., Willerslev-Olsen, M., Lundell, H., Conway, B. A., Knudsen, H., Biering-Sorensen, F., et al. (2010). Impaired transmission in the corticospinal tract and gait disability in spinal cord injured persons. Journal of Neurophysiology, 104, 1167–1176. Bar-Zohar, D., Agosta, F., Goldstaub, D., & Filippi, M. (2008). Magnetic resonance imaging metrics and their correlation with clinical outcomes in multiple sclerosis: A review of the literature and future perspectives. Multiple Sclerosis, 14, 719–727.

193 Belci, M., Catley, M., Husain, M., Frankel, H. L., & Davey, N. J. (2004). Magnetic brain stimulation can improve clinical outcome in incomplete spinal cord injured patients. Spinal Cord, 42, 417–419. Beloozerova, I. N., & Sirota, M. G. (1998). Cortically controlled gait adjustments in the cat. Annals of the New York Academy of Sciences, 860, 550–553. Bensoussan, L., Mesure, S., Viton, J. M., & Delarque, A. (2006). Kinematic and kinetic asymmetries in hemiplegic patients’ gait initiation patterns. Journal of Rehabilitation Medicine, 38, 287–294. Blanchette, A., & Bouyer, L. J. (2009). Timing-specific transfer of adapted muscle activity after walking in an elastic force field. Journal of Neurophysiology, 102, 568–577. Bonnard, M., Camus, M., Coyle, T., & Pailhous, J. (2002). Task-induced modulation of motor evoked potentials in upper-leg muscles during human gait: A TMS study. The European Journal of Neuroscience, 16, 2225–2230. Bremner, F. D., Baker, J. R., & Stephens, J. A. (1991). Correlation between the discharges of motor units recorded from the same and from different finger muscles in man. Journal fo Physiology, 432, 355–380. Brown, P., Salenius, S., Rothwell, J. C., & Hari, R. (1998). Cortical correlate of the piper rhythm in humans. Journal of Neurophysiology, 80, 2911–2917. Burridge, J. H., Wood, D. E., Taylor, P. N., & McLellan, D. L. (2001). Indices to describe different muscle activation patterns, identified during treadmill walking, in people with spastic dropfoot. Medical Engineering & Physics, 23, 427–434. Capaday, C., Lavoie, B. A., Barbeau, H., Schneider, C., & Bonnard, M. (1999). Studies on the corticospinal control of human walking. I. Responses to focal transcranial magnetic stimulation of the motor cortex. Journal of Neurophysiology, 81, 129–139. Carey, J. R., Anderson, K. M., Kimberley, T. J., Lewis, S. M., Auerbach, E. J., & Ugurbil, K. (2004). fMRI analysis of ankle movement tracking training in subject with stroke. Experimental Brain Research, 154, 281–290. Chan, C. W. (1983). Segmental versus suprasegmental contributions to long-latency stretch responses in man. Advances in Neurology, 39, 467–487. Chang, C. W., & Lien, I. N. (1991). Estimate of motor conduction in human spinal cord: Slowed conduction in spinal cord injury. Muscle & Nerve, 14, 990–996. Chen, R., Classen, J., Gerloff, C., Celnik, P., Wassermann, E. M., Hallett, M., et al. (1997). Depression of motor cortex excitability by low-frequency transcranial magnetic stimulation. Neurology, 48, 1398–1403. Chin, R., Hsiao-Wecksler, E. T., Loth, E., Kogler, G., Manwaring, S. D., Tyson, S. N., et al. (2009). A pneumatic power harvesting ankle-foot orthosis to prevent foot-drop. Journal of Neuroengineering and Rehabilitation, 6, 19. Christensen, L. O., Morita, H., Petersen, N., & Nielsen, J. (1999). Evidence suggesting that a transcortical reflex

pathway contributes to cutaneous reflexes in the tibialis anterior muscle during walking in man. Exp Brain Res, 124, 59–68. Christensen, L. O., Andersen, J. B., Sinkjaer, T., & Nielsen, J. (2001). Transcranial magnetic stimulation and stretch reflexes in the tibialis anterior muscle during human walking. Journal of Physiology, 531, 545–557. Christensen, L. O., Johannsen, P., Sinkjaer, T., Petersen, N., Pyndt, H. S., & Nielsen, J. B. (2000). Cerebral activation during bicycle movements in man. Experimental Brain Research, 135, 66–72. Colombo, G., Joerg, M., Schreier, R., & Dietz, V. (2000). Treadmill training of paraplegic patients using a robotic orthosis. Journal of Rehabilitation Research and Development, 37, 693–700. Colombo, G., Wirz, M., & Dietz, V. (2001). Driven gait orthosis for improvement of locomotor training in paraplegic patients. Spinal Cord, 39, 252–255. Conway, B. A., Halliday, D. M., Farmer, S. F., Shahani, U., Maas, P., Weir, A. I., et al. (1995). Synchronization between motor cortex and spinal motoneuronal pool during the performance of a maintained motor task in man. Journal of Physiology, 489(Pt 3), 917–924. Curt, A., Keck, M. E., & Dietz, V. (1998). Functional outcome following spinal cord injury: Significance of motor-evoked potentials and ASIA scores. Archives of Physical Medicine and Rehabilitation, 79, 81–86. Datta, A. K., Farmer, S. F., & Stephens, J. A. (1991). Central nervous pathways underlying synchronization of human motor unit firing studied during voluntary contractions. Journal of Physiology, 432, 401–425. Davey, N. J., Ellaway, P. H., Baker, J. R., & Friedland, C. L. (1993). Rhythmicity associated with a high degree of shortterm synchrony of motor unit discharge in man. Experimental Physiology, 78, 649–661. Davey, N. J., Romaiguere, P., Maskill, D. W., & Ellaway, P. H. (1994). Suppression of voluntary motor activity revealed using transcranial magnetic stimulation of the motor cortex in man. J Physiol, 477 (Pt 2), 223–235. Dawes, H., Enzinger, C., Johansen-Berg, H., Bogdanovic, M., Guy, C., Collett, J., et al. (2008). Walking performance and its recovery in chronic stroke in relation to extent of lesion overlap with the descending motor tract. Experimental Brain Research, 186, 325–333. Day, B. L., Thompson, P. D., Dick, J. P., Nakashima, K., & Marsden, C. D. (1987). Different sites of action of electrical and magnetic stimulation of the human brain. Neuroscience Letters, 75, 101–106. Dietz, V. (2010). Behavior of spinal neurons deprived of supraspinal input. Nature Reviews. Neurology, 6, 167–174. Dobkin, B. H. (2000). Spinal and supraspinal plasticity after incomplete spinal cord injury: Correlations between functional magnetic resonance imaging and engaged locomotor networks. Progress in Brain Research, 128, 99–111.

194 Dobkin, B., Apple, D., Barbeau, H., Basso, M., Behrman, A., Deforge, D., et al. (2006). Weight-supported treadmill vs over-ground training for walking after acute incomplete SCI. Neurology, 66, 484–493. Dobkin, B. H., Firestine, A., West, M., Saremi, K., & Woods, R. (2004). Ankle dorsiflexion as an fMRI paradigm to assay motor control for walking during rehabilitation. Neuroimage, 23, 370–381. Drew, T. (1991). Functional organization within the medullary reticular formation of the intact unanesthetized cat. III. Microstimulation during locomotion. Journal of Neurophysiology, 66, 919–938. Duysens, J., Trippel, M., Horstmann, G. A., & Dietz, V. (1990). Gating and reversal of reflexes in ankle muscles during human walking. Experimental Brain Research, 82, 351–358. Edgley, S. A., Eyre, J. A., Lemon, R. N., & Miller, S. (1997). Comparison of activation of corticospinal neurons and spinal motor neurons by magnetic and electrical transcranial stimulation in the lumbosacral cord of the anaesthetized monkey. Brain, 120(Pt 5), 839–853. Eidelberg, E. (1981). Consequences of spinal cord lesions upon motor function, with special reference to locomotor activity. Progress in Neurobiology, 17, 185–202. Ellaway, P. H., Catley, M., Davey, N. J., Kuppuswamy, A., Strutton, P., Frankel, H. L., et al. (2007). Review of physiological motor outcome measures in spinal cord injury using transcranial magnetic stimulation and spinal reflexes. Journal of Rehabilitation Research and Development, 44, 69–76. Enzinger, C., Dawes, H., Johansen-Berg, H., Wade, D., Bogdanovic, M., Collett, J., et al. (2009). Brain activity changes associated with treadmill training after stroke. Stroke, 40, 2460–2467. Enzinger, C., Johansen-Berg, H., Dawes, H., Bogdanovic, M., Collett, J., Guy, C., et al. (2008). Functional MRI correlates of lower limb function in stroke victims with gait impairment. Stroke, 39, 1507–1513. Farmer, S. F., Bremner, F. D., Halliday, D. M., Rosenberg, J. R., & Stephens, J. A. (1993a). The frequency content of common synaptic inputs to motoneurones studied during voluntary isometric contraction in man. Journal of Physiology, 470, 127–155. Farmer, S. F., Swash, M., Ingram, D. A., & Stephens, J. A. (1993b). Changes in motor unit synchronization following central nervous lesions in man. Journal of Physiology, 463, 83–105. Fedirchuk, B., Nielsen, J., Petersen, N., & Hultborn, H. (1998). Pharmacologically evoked fictive motor patterns in the acutely spinalized marmoset monkey (Callithrix jacchus). Experimental Brain Research, 122, 351–361. Fitzgerald, P. B., Fountain, S., & Daskalakis, Z. J. (2006). A comprehensive review of the effects of rTMS on motor cortical excitability and inhibition. Clinical Neurophysiology, 117, 2584–2596.

Fukuyama, H., Ouchi, Y., Matsuzaki, S., Nagahama, Y., Yamauchi, H., Ogawa, M., et al. (1997). Brain functional activity during gait in normal subjects: A SPECT study. Neuroscience Letters, 228, 183–186. Geertsen, S. S., Zuur, A. T., & Nielsen, J. B. (2010). Voluntary activation of ankle muscles is accompanied by subcortical facilitation of their antagonists. Journal of Physiology, 588, 2391–2402. Gorassini, M. A., Knash, M. E., Harvey, P. J., Bennett, D. J., & Yang, J. F. (2004). Role of motoneurons in the generation of muscle spasms after spinal cord injury. Brain, 127, 2247–2258. Grey, M. J., Mazzaro, N., Nielsen, J. B., & Sinkjaer, T. (2004). Ankle extensor proprioceptors contribute to the enhancement of the soleus EMG during the stance phase of human walking. Canadian Journal of Physiology and Pharmacology, 82, 610–616. Halliday, D. M., Conway, B. A., Christensen, L. O., Hansen, N. L., Petersen, N. P., & Nielsen, J. B. (2003). Functional coupling of motor units is modulated during walking in human subjects. Journal of Neurophysiology, 89, 960–968. Halliday, D. M., Conway, B. A., Farmer, S. F., & Rosenberg, J. R. (1998). Using electroencephalography to study functional coupling between cortical activity and electromyograms during voluntary contractions in humans. Neuroscience Letters, 241, 5–8. Hanakawa, T., Fukuyama, H., Katsumi, Y., Honda, M., & Shibasaki, H. (1999a). Enhanced lateral premotor activity during paradoxical gait in Parkinson's disease. Annals of Neurology, 45, 329–336. Hanakawa, T., Katsumi, Y., Fukuyama, H., Honda, M., Hayashi, T., Kimura, J., et al. (1999b). Mechanisms underlying gait disturbance in Parkinson's disease: A single photon emission computed tomography study. Brain, 122(Pt 7), 1271–1282. Hansen, N. L., Conway, B. A., Halliday, D. M., Hansen, S., Pyndt, H. S., Biering-Sorensen, F., et al. (2005). Reduction of common synaptic drive to ankle dorsiflexor motoneurons during walking in patients with spinal cord lesion. Journal of Neurophysiology, 94, 934–942. Hansen, N. L., Hansen, S., Christensen, L. O., Petersen, N. T., & Nielsen, J. B. (2001). Synchronization of lower limb motor unit activity during walking in human subjects. Journal of Neurophysiology, 86, 1266–1276. Hansen, N. L., & Nielsen, J. B. (2004). The effect of transcranial magnetic stimulation and peripheral nerve stimulation on corticomuscular coherence in humans. Journal of Physiology, 561, 295–306. Harkema, S. J., Hurley, S. L., Patel, U. K., Requejo, P. S., Dobkin, B. H., & Edgerton, V. R. (1997). Human lumbosacral spinal cord interprets loading during stepping. Journal of Neurophysiology, 77, 797–811.

195 Jang, S. H., Byun, W. M., Han, B. S., Park, H. J., Bai, D., Ahn, Y. H., et al. (2006). Recovery of a partially damaged corticospinal tract in a patient with intracerebral hemorrhage: A diffusion tensor image study. Restorative Neurology and Neuroscience, 24, 25–29. Jayaram, G., & Stinear, J. W. (2008). Contralesional paired associative stimulation increases paretic lower limb motor excitability post-stroke. Experimental Brain Research, 185, 563–570. Jobsis, F. F. (1977). Noninvasive, infrared monitoring of cerebral and myocardial oxygen sufficiency and circulatory parameters. Science, 198, 1264–1267. Kilner, J. M., Baker, S. N., Salenius, S., Jousmaki, V., Hari, R., & Lemon, R. N. (1999). Task-dependent modulation of 15–30 Hz coherence between rectified EMGs from human hand and forearm muscles. Journal of Physiology, 516 (Pt 2), 559–570. Kim, Y. H., You, S. H., Ko, M. H., Park, J. W., Lee, K. H., Jang, S. H., et al. (2006). Repetitive transcranial magnetic stimulation-induced corticomotor excitability and associated motor skill acquisition in chronic stroke. Stroke, 37, 1471–1476. Krabbe, K., Nielsen, J. E., Fallentin, E., Fenger, K., & Herning, M. (1997). MRI of autosomal dominant pure spastic paraplegia. Neuroradiology, 39, 724–727. Lackner, J. R., & Dizio, P. (1994). Rapid adaptation to Coriolis force perturbations of arm trajectory. Journal of Neurophysiology, 72, 299–313. Lundell, H., Barthelemy, D., Skimminge, A., Dyrby, T. B., Biering-Sorensen, F., & Nielsen, J. B. (2011). Independent spinal cord atrophy measures correlate to motor and sensory deficits in individuals with spinal cord injury. Spinal Cord, 49(1), 70–75. Lundell, H., Christensen, M. S., Barthelemy, D., WillerslevOlsen, M., Biering-Sorensen, F., & Nielsen, J. B. (2010b). Cerebral activation is correlated to regional atrophy of the spinal cord and functional motor disability in spinal cord injured individuals. Neuroimage, 54, 1254–1261. Maki, A., Yamashita, Y., Ito, Y., Watanabe, E., Mayanagi, Y., & Koizumi, H. (1995). Spatial and temporal analysis of human motor activity using noninvasive NIR topography. Medical Physics, 22, 1997–2005. Mansur, C. G., Fregni, F., Boggio, P. S., Riberto, M., GallucciNeto, J., Santos, C. M., et al. (2005). A sham stimulationcontrolled trial of rTMS of the unaffected hemisphere in stroke patients. Neurology, 64, 1802–1804. Matthews, P. B. (1999). The effect of firing on the excitability of a model motoneurone and its implications for cortical stimulation. Journal of Physiology, 518(Pt 3), 867–882. Mima, T., & Hallett, M. (1999). Electroencephalographic analysis of cortico-muscular coherence: Reference effect, volume conduction and generator mechanism. Clinical Neurophysiology, 110, 1892–1899.

Mishina, M., Senda, M., Ishii, K., Ohyama, M., Kitamura, S., & Katayama, Y. (1999). Cerebellar activation during ataxic gait in olivopontocerebellar atrophy: A PET study. Acta Neurologica Scandinavica, 100, 369–376. Miyai, I., Tanabe, H. C., Sase, I., Eda, H., Oda, I., Konishi, I., et al. (2001). Cortical mapping of gait in humans: A nearinfrared spectroscopic topography study. Neuroimage, 14, 1186–1192. Miyai, I., Yagura, H., Hatakenaka, M., Oda, I., Konishi, I., & Kubota, K. (2003). Longitudinal optical imaging study for locomotor recovery after stroke. Stroke, 34, 2866–2870. Miyai, I., Suzuki, M., Hatakenaka, M., & Kubota, K. (2006). Effect of body weight support on cortical activation during gait in patients with stroke. Exp Brain Res, 169, 85–91. Muellbacher, W., Ziemann, U., Boroojerdi, B., & Hallett, M. (2000). Effects of low-frequency transcranial magnetic stimulation on motor excitability and basic motor behavior. Clinical Neurophysiology, 111, 1002–1007. Nielsen, J. B. (2003). How we walk: Central control of muscle activity during human walking. The Neuroscientist, 9, 195–204. Nielsen, J. B., Brittain, J. S., Halliday, D. M., MarchandPauvert, V., Mazevet, D., & Conway, B. A. (2008). Reduction of common motoneuronal drive on the affected side during walking in hemiplegic stroke patients. Clinical Neurophysiology, 119, 2813–2818. Nielsen, J. B., Conway, B. A., Halliday, D. M., Perreault, M. C., & Hultborn, H. (2005). Organization of common synaptic drive to motoneurones during fictive locomotion in the spinal cat. Journal of Physiology, 569, 291–304. Nielsen, J., & Kagamihara, Y. (1993). Differential projection of the sural nerve to early and late recruited human tibialis anterior motor units: Change of recruitment gain. Acta Physiologica Scandinavica, 147, 385–401. Nielsen, J., Morita, H., Baumgarten, J., Petersen, N., & Christensen, L. O. (1999). On the comparability of Hreflexes and MEPs. Electroencephalography and Clinical Neurophysiology. Supplement, 51, 93–101. Nielsen, J., Petersen, N., & Ballegaard, M. (1995). Latency of effects evoked by electrical and magnetic brain stimulation in lower limb motoneurones in man. Journal of Physiology, 484(Pt 3), 791–802. Nielsen, J., Petersen, N., & Fedirchuk, B. (1997). Evidence suggesting a transcortical pathway from cutaneous foot afferents to tibialis anterior motoneurones in man. Journal of Physiology, 501(Pt 2), 473–484. Nitsche, M. A., & Paulus, W. (2000). Excitability changes induced in the human motor cortex by weak transcranial direct current stimulation. Journal of Physiology, 527(Pt 3), 633–639. Noel, M., Fortin, K., & Bouyer, L. J. (2009). Using an electrohydraulic ankle foot orthosis to study modifications in feedforward control during locomotor adaptation to force

196 fields applied in stance. Journal of Neuroengineering and Rehabilitation, 6, 16. Norton, J. A., & Gorassini, M. A. (2006). Changes in cortically related intermuscular coherence accompanying improvements in locomotor skills in incomplete spinal cord injury. Journal of Neurophysiology, 95, 2580–2589. Perez, M. A., Lungholt, B. K., Nyborg, K., & Nielsen, J. B. (2004). Motor skill training induces changes in the excitability of the leg cortical area in healthy humans. Experimental Brain Research, 159, 197–205. Petersen, N. T., Butler, J. E., Marchand-Pauvert, V., Fisher, R., Ledebt, A., Pyndt, H. S., et al. (2001). Suppression of EMG activity by transcranial magnetic stimulation in human subjects during walking. Journal of Physiology, 537, 651–656. Petersen, N., Christensen, L. O., & Nielsen, J. (1998). The effect of transcranial magnetic stimulation on the soleus H reflex during human walking. Journal of Physiology, 513 (Pt 2), 599–610. Petersen, T. H., Kliim-Due, M., Farmer, S. F., & Nielsen, J. B. (2010). Childhood development of common drive to a human leg muscle during ankle dorsiflexion and gait. Journal of Physiology, 588, 4387–4400. Phillips, C. G. (1969). The Ferrier lecture, 1968. Motor apparatus of the baboon's hand. Proceedings of the Royal Society of London. Series B: Biological Sciences, 173, 141–174. Pijnappels, M., van Wezel, B. M., Colombo, G., Dietz, V., & Duysens, J. (1998). Cortical facilitation of cutaneous reflexes in leg muscles during human gait. Brain Res, 787, 149–153. Piron, L., Piccione, F., Tonin, P., & Dam, M. (2005). Clinical correlation between motor evoked potentials and gait recovery in poststroke patients. Archives of Physical Medicine and Rehabilitation, 86, 1874–1878. Raasch, C. C., & Zajac, F. E. (1999). Locomotor strategy for pedaling: Muscle groups and biomechanical functions. Journal of Neurophysiology, 82, 515–525. Roy, F. D., & Gorassini, M. A. (2008). Peripheral sensory activation of cortical circuits in the leg motor cortex of man. Journal of Physiology, 586, 4091–4105. Roy, F. D., Norton, J. A., & Gorassini, M. A. (2007). Role of sustained excitability of the leg motor cortex after transcranial magnetic stimulation in associative plasticity. Journal of Neurophysiology, 98, 657–667. Roy, F. D., Yang, J. F., & Gorassini, M. A. (2010). Afferent regulation of leg motor cortex excitability after incomplete spinal cord injury. Journal of Neurophysiology, 103, 2222–2233. Salenius, S., Portin, K., Kajola, M., Salmelin, R., & Hari, R. (1997). Cortical control of human motoneuron firing during isometric contraction. Journal of Neurophysiology, 77, 3401–3405. Schneider, C., Lavoie, B. A., Barbeau, H., & Capaday, C. (2004). Timing of cortical excitability changes during the

reaction time of movements superimposed on tonic motor activity. Journal of Applied Physiology, 97, 2220–2227. Schubert, M., Curt, A., Colombo, G., Berger, W., & Dietz, V. (1999). Voluntary control of human gait: Conditioning of magnetically evoked motor responses in a precision stepping task. Experimental Brain Research, 126, 583–588. Schubert, M., Curt, A., Jensen, L., & Dietz, V. (1997). Corticospinal input in human gait: Modulation of magnetically evoked motor responses. Experimental Brain Research, 115, 234–246. Shadmehr, R., & Mussa-Ivaldi, F. A. (1994). Adaptive representation of dynamics during learning of a motor task. The Journal of Neuroscience, 14, 3208–3224. Sinkjaer, T., Andersen, J. B., Ladouceur, M., Christensen, L. O., & Nielsen, J. B. (2000). Major role for sensory feedback in soleus EMG activity in the stance phase of walking in man. Journal of Physiology, 523(Pt 3), 817–827. Stinear, J. W., & Hornby, T. G. (2005). Stimulation-induced changes in lower limb corticomotor excitability during treadmill walking in humans. Journal of Physiology, 567, 701–711. Sundblom, J., Melberg, A., Kalimo, H., Smits, A., & Raininko, R. (2009). MR imaging characteristics and neuropathology of the spinal cord in adult-onset autosomal dominant leukodystrophy with autonomic symptoms. AJNR. American Journal of Neuroradiology, 30, 328–335. Tashiro, M., Itoh, M., Fujimoto, T., Fujiwara, T., Ota, H., Kubota, K., et al. (2001). 18 F-FDG PET mapping of regional brain activity in runners. The Journal of Sports Medicine and Physical Fitness, 41, 11–17. Thach, W. T., & Bastian, A. J. (2004). Role of the cerebellum in the control and adaptation of gait in health and disease. Progress in Brain Research, 143, 353–366. Thomas, S. L., & Gorassini, M. A. (2005). Increases in corticospinal tract function by treadmill training after incomplete spinal cord injury. Journal of Neurophysiology, 94, 2844–2855. Visintin, M., Barbeau, H., Korner-Bitensky, N., & Mayo, N. E. (1998). A new approach to retrain gait in stroke patients through body weight support and treadmill stimulation. Stroke, 29, 1122–1128. Ward, N. S. (2004). Functional reorganization of the cerebral motor system after stroke. Current Opinion in Neurology, 17, 725–730. Ward, N. S. (2005a). Neural plasticity and recovery of function. Progress in Brain Research, 150, 527–535. Ward, N. S. (2005b). Plasticity and the functional reorganization of the human brain. International Journal of Psychophysiology, 58, 158–161. Weber, D. J., Stein, R. B., Chan, K. M., Loeb, G. E., Richmond, F. J., Rolf, R., et al. (2004). Functional electrical stimulation using microstimulators to correct foot drop: A case study. Canadian Journal of Physiology and Pharmacology, 82, 784–792.

197 Yang, J. F., & Gorassini, M. (2006). Spinal and brain control of human walking: Implications for retraining of walking. The Neuroscientist, 12, 379–389. Yang, J. F., & Stein, R. B. (1990). Phase-dependent reflex reversal in human leg muscles during walking. Journal of Neurophysiology, 63, 1109–1117.

Zehr, E. P., Fujita, K., & Stein, R. B. (1998). Reflexes from the superficial peroneal nerve during walking in stroke subjects. Journal of Neurophysiology, 79, 848–858. Zuur, A. T., Christensen, M. S., Sinkjaer, T., Grey, M. J., & Nielsen, J. B. (2009). Tibialis anterior stretch reflex in early stance is suppressed by repetitive transcranial magnetic stimulation. Journal of Physiology, 587, 1669–1676.

A. M. Green, C. E. Chapman, J. F. Kalaska and F. Lepore (Eds.) Progress in Brain Research, Vol. 192 ISSN: 0079-6123 Copyright Ó 2011 Elsevier B.V. All rights reserved.

CHAPTER 13

Vision restoration after brain and retina damage: The “residual vision activation theory” Bernhard A. Sabel*, Petra Henrich-Noack, Anton Fedorov and Carolin Gall Institute of Medical Psychology, Medical Faculty, Otto-von-Guericke-University Magdeburg, Magdeburg, Germany

Abstract: Vision loss after retinal or cerebral visual injury (CVI) was long considered to be irreversible. However, there is considerable potential for vision restoration and recovery even in adulthood. Here, we propose the “residual vision activation theory” of how visual functions can be reactivated and restored. CVI is usually not complete, but some structures are typically spared by the damage. They include (i) areas of partial damage at the visual field border, (ii) “islands” of surviving tissue inside the blind field, (iii) extrastriate pathways unaffected by the damage, and (iv) downstream, higher-level neuronal networks. However, residual structures have a triple handicap to be fully functional: (i) fewer neurons, (ii) lack of sufficient attentional resources because of the dominant intact hemisphere caused by excitation/inhibition dysbalance, and (iii) disturbance in their temporal processing. Because of this resulting activation loss, residual structures are unable to contribute much to everyday vision, and their “non-use” further impairs synaptic strength. However, residual structures can be reactivated by engaging them in repetitive stimulation by different means: (i) visual experience, (ii) visual training, or (iii) noninvasive electrical brain current stimulation. These methods lead to strengthening of synaptic transmission and synchronization of partially damaged structures (within-systems plasticity) and downstream neuronal networks (network plasticity). Just as in normal perceptual learning, synaptic plasticity can improve vision and lead to vision restoration. This can be induced at any time after the lesion, at all ages and in all types of visual field impairments after retinal or brain damage (stroke, neurotrauma, glaucoma, amblyopia, age-related macular degeneration). If and to what extent vision restoration can be achieved is a function of the amount of residual tissue and its activation state. However, sustained improvements require repetitive stimulation which, depending on the method, may take days (noninvasive brain stimulation) or months (behavioral training). By becoming again engaged in everyday vision, (re)activation of areas of residual vision outlasts the stimulation period, thus contributing to lasting vision restoration and improvements in quality of life.

*Corresponding author. Tel.: þ49-391-672-1800; Fax: þ49-391-672-1803 E-mail: [email protected] DOI: 10.1016/B978-0-444-53355-5.00013-0

199

200

Keywords: vision; restoration; rehabilitation; plasticity; current stimulation; training.

Introduction Humans rely on vision more than on any of the other senses, and more brain tissue is devoted to visual perception than to all other senses combined (Felleman and Van Essen, 1991). Thus, when the brain is damaged, the likelihood to suffer visual impairments is high and the consequences for quality of life are grave. Nineteen percent of persons > 70 years have visual impairments (Centers for Disease Control and Prevention, 2004), and visual loss is the most feared disease in the elderly (Aiello, 2008). There are many possible reasons for impairments or loss of vision after damage to the central nervous system. Functional deficits depend primarily on the location of the damage which may be in the retina, optic nerve, or higher-level visual structures of the brain. When the visual radiation or visual cortex is damaged, homonymous sectors of the visual field are lost, leading to scotomata or loss of the entire half of the visual field, a condition long known as hemianopia (Baumgarten, 1878; Poppelreuter, 1917). The etiology of visual field defects may be traumatic, inflammatory, or vascular, and the vision loss can proceed either acutely (as in stroke or brain trauma) or it can progress more slowly as in inflammatory degeneration of the optic nerve or retinal damage (e.g., glaucoma or age-related macular degeneration (AMD)). Because of its retinotopic organization and highly specific cortical organization, the visual system is generally believed not to recover very well after injury. The generally accepted notion is that patients are permanently left with irreparable blindness. However, there is some hope because vision loss is usually not complete but partial, having variable degrees of residual visual functions. A better understanding of how to stimulate the partially damaged visual system to improve its function is therefore not only scientifically interesting but also clinically relevant. Despite the lingering pessimism that vision loss

is permanent, searching new ways to help patients regain at least some of their lost vision is a scientific and clinical responsibility. It was long suspected that the brain had no capability of repair after an early spontaneous recovery phase which typically ends after the first few weeks of injury. But in recent years, we have witnessed scientific progress showing many examples where vision improvements were seen even well beyond this early recovery phase. Vision recovery as discussed in our review, also termed vision restoration, is limited to visual dysfunctions caused by damage of the central nervous system, that is, retina, optic nerve, and different brain regions. We do not discuss restoration of anterior eye problems (cornea or lense). Vision restoration also does not assume a “complete return” to normal function and it may manifest itself mainly in partial and sometime also in total recovery, depending on the individual case. The term “vision restoration” should not be misunderstood as implying “complete” restoration of function at all times because the extent of restoration is always rather variable and usually not complete. The term “restoration” should also not be misused to raise unfounded hope in patients with visual loss. It rather emphasizes the residual potential of the damage system to improve its function, in whatever extent, form or shape. Several issues cannot be discussed in detail in this review such as the role of plasticity in normal learning. Particularly, the study of normal “perceptual learning” was elegantly studied by other laboratories (see below) and their work should be consulted. The present review also does not focus on how patients who suffer from visual field defects may be able to “compensate” for their visual field loss by scanning the visual world more vigorously with eye movements, thus attempting to increase their “field of view.” It is possible that this compensation actually reduces the chance for restoration as the subjects learn to focus their attention more on the remaining, intact capacities.

201

The aim of the present review is to first summarize the current literature on vision restoration and on this basis formulate a new theory of the underlying mechanisms. Although visual system plasticity has been discussed at numerous occasions, we still lack a coherent theory of vision restoration after damage to the adult brain and this should be rectified. The residual vision activation theory was therefore conceived to provide a heuristic framework to sort and interpret the different observations and approaches, helping to guide us in a new and exciting research direction that provides hope for the many partially blind patients. “Neuroengineering”-like approaches that aim at restoring vision can be only mentioned here in passing: besides efforts to limit the lesion effects through neuroprotection, there are various attempts to replace the damaged tissue itself or to provide some alternative tissue that augments the damaged tissue or supports its biological regeneration. These include (i) artificial retinal or brain implants, (ii) retinal and cortical tissue transplants, (iii) nerve regeneration, and (iv) stem cell implantation. Because these approaches are mostly experimental at this time, they are not discussed in further detail here. Whereas the neuroengineering approach aims at replacing or augmenting the lost tissue itself— as if trying to fill the hole of a donut—the “neuroplasticity approach” of residual vision aims at altering the surviving brain tissue itself. It is by far the clinically more relevant topic and has received a lot of attention from different groups. Neuroplasticity studies focus on the residual (surviving) brain structures both at the site of the lesion (local) and in the brain network as a whole (global). While visual system plasticity is a well-described phenomenon in the developing, normal brain at an age well before the critical period, it is now consensus that the visual system plasticity is possible in older age; it is observed in perceptual learning in adults and elderly but also after different types of brain lesions, both in animals and in man. The present review summarizes the evidence of post-lesion plasticity of the partially damaged adult

visual system and its clinical impact and then formulates the residual vision activation theory. This theory was conceived to create a unified view of current empirical evidence of visual system repair and to explain mechanisms of vision restoration after lesions of the central visual pathway, including retina, optic nerve, postchiasmatic tracts, and radiations, striate (V1) but also extrastriate cortex.

Plasticity of the visual system Plasticity has been observed at many different levels of the visual system both in the normal and in the lesioned brain. In fact, plasticity is a rather normal, dynamic property that takes place in normal perceptual learning.

Perceptual learning Perceptual learning is a change in performance following training or practice which is typically investigated in visually healthy subjects (Fahle and Poggio, 2002; Li et al., 2004). Perceptual learning may improve different visual abilities such as detection of thresholds, gratings, hyperacuity, motion, or texture (Fahle, 2002, 2005; Fiorentini and Berardi, 1980; Gilbert et al., 2001; Polat and Sagi, 1994). The improvements are usually specific and they do not transfer easily between different stimuli or stimulus locations in the visual field. This specificity is attributed to response modifications of neuron assemblies at the earliest visual processing stages such as V1 (Fahle, 2005; Fahle and Skrandies, 1994; Hirsch and Gilbert, 1991). Perceptual learning may involve training attention to discriminate distinctive stimulus features (Gibson, 1969), increase alertness (Wolford et al., 1988), and establish stimulus–response associations (correlated activities) in sensory system of the brain. Practice is also able to increase the range of the lateral interactions sixfold in collinearity tasks (Polat et al., 2004; Polat and Sagi, 1994) which

202

appears to increase the efficacy of the collinear interactions between neighboring neurons. This, in turn, may improve the connectivity of remote neurons via local interactions which are also thought to be involved in receptive field (RF) plasticity after retinal lesions (for discussion see below).

Plasticity after retinal lesions Plasticity after acute retinal damage When the retina is damaged, visual impairments can recover spontaneously and there is considerable RF reorganization in upstream areas (Dreher et al., 2001; Eysel, 1997; Eysel and Grüsser, 1978; Eysel et al., 1999; Gilbert and Wiesel, 1992; Kaas et al., 1990). RF reorganization is a well-studied field showing how the brain reacts to injury by numerous neurophysiological changes on the molecular, cellular, and network level (see also Huxlin, 2008). Retinal lesions are often used in experimental animal models to study recovery and RF plasticity. Rather limited RF plasticity occurs in the lateral geniculate nucleus (LGN) of the thalamus after retina damage (Eysel and Grüsser, 1978), whereas in the visual cortex, up to 98% of the deafferented neurons developed new RFs within 3 months after retinal lesion in cats (Chino et al., 1995). Cortical reorganization is typically reflected in a displacement of the RF position and RF enlargement. The shift of RFs following retinal lesions has been reported both in cat's area 17 and 18 (Kaas et al., 1990; Young et al., 2002). The properties of these RFs are normal, except for elevation of contrast threshold (Chino et al., 1995) and changes in temporal characteristics of response (Darian-Smith and Gilbert, 1995; Heinen and Skavenski, 1991; Waleszczyk et al., 2003). Lesions of both retinal and cortical areas are typically accompanied by reduced GABAergic inhibition and increased glutamatergic excitation, leading to an increased spontaneous activity and excitability change of visual

activity in the region of cortical scotoma (cortical representation of retinal lesion; Giannikopoulos and Eysel, 2006) or in regions surrounding the cortical lesion (penumbra) (Dohle et al., 2009; Eysel et al., 1999; Imbrosci et al., 2010). Recovery of visual responses in the silenced area of the visual cortex is suggested to be mediated by anatomical (Darian-Smith and Gilbert, 1994) and functional changes of intrinsic cortical horizontal connection (Calford et al., 2003; Das and Gilbert, 1995; Palagina et al., 2009; Young et al., 2007). Keck et al. (2008) recently observed in adult mice with small retinal lesions a complete reorganization of dendritic spines in the deafferented cortex within 2 months. The rate at which postsynaptic connections perished and were reestablished was three times higher than in normal brain. Smirnakis et al., (2005) have questioned the existence of cortical reorganization as they could not see topographic changes in the BOLD response of adult macaques 7.5 months after retinal lesion, but the BOLD response was considered to be insensitive to changes in RFs by most investigators in the field (Calford et al., 2005). The body of literature on RF reorganization in the visual system is too large to be reviewed here and numerous publications by the groups of Eysel and Gilbert (see above), Chino et al. (1995), Calford et al. (2000), and others can be readily found (Huxlin, 2008).

Plasticity in AMD Age related macular degeneration (AMD) is a progressive, degenerative disorder that often causes a large scotoma in the central visual field which leads to fixation and reading problems. Such patients also show signs of perceptual plasticity as seen in the “filling-in” phenomenon (Cohen et al., 2003; McManus et al., 2008; Mendola et al., 2006; Zur and Ullman, 2003). Another sign of perceptual plasticity in AMD patients is that the patients spontaneously adopt

203

a new preferred retinal location (PRL) to achieve eccentric fixation (note: “preferred locus” does not imply that it is the most optimal locus). In such cases, patients compensate their foveal damage by using intact (or partly intact) regions at the edge of the damage to better be able to fixate objects (Greenstein et al., 2008; Schuchard and Fletcher, 1994). Eccentric fixation can also be learned which may improve reading (Nilsson et al., 1998, 2003; Watson et al., 2006), and, in cases where an unfavorable PRL has spontaneously developed, behavioral training can be used to shift the PRL to a more optimal location (Nilsson et al., 2003; Radvay et al., 2007). This is needed because an untrained PRL is sometimes located in an undesirable area such as on the left side of the scotoma, that is, in a position that is not optimal for reading. When patients are trained to relearn a new PRL above or below the scotoma, they may experience substantially increased reading speed (Nilsson et al., 1998, 2003), though it is still unclear, what the best position for such a PRL might be. Imaging studies showed that the visual cortex of AMD patients shows signs of cortical reorganization in areas of the cortex that topographically match the fovea (Baker et al., 2005, 2008). Cortical regions that previously processed only central (foveal) visual information could now be activated by peripheral stimuli and this reorganization is associated with development of eccentric vision (Schumacher et al., 2008). This type of reorganization apparently only occurs when the functional loss at the fovea was complete, that is, without tissue sparing. However, only small patient numbers were studied so far, requiring confirmation in larger clinical trials, which are currently underway in the UK (G. Rubin, personal communication). Yet, the spontaneous development of PRLs and the ability to retrain their location are signs of how the visual system uses plasticity mechanisms to adapt to the loss. Here, intact tissue takes over the role of the damaged regions.

Plasticity in glaucoma Glaucoma is the leading cause of visual field loss in all age groups (Ramrattan et al., 2001). It is a slowly developing retinal disease where elevated intraocular pressure leads to retinal ganglion cell (RGC) death. In contrast to AMD, field defects in glaucoma typically emerge from the periphery of the visual field. This may be one of the reasons why visual field impairments remain undetected by the patient for a long time until nerve cell loss has already progressed significantly with serious field impairments. Another reason for the late detection may be that the brain adapts to the slow loss by plastic changes: it compensates the retinal cell loss by some yet unknown mechanism (such as filling-in), keeping it subclinical or below conscious perception. While the progression of glaucoma-induced visual field loss can mostly be arrested by proper medication, the scotoma, once detected, is considered to be permanent, with no chance to improve. However, some RGCs survive within the damaged retinal regions (Pavlidis et al., 2003) and perhaps by the process of RF plasticity, the functionally deafferented visual cortex undergoes sensitivity changes. Gudlin et al. (2008) have carried out a pilot study with five patients that suffered stable primary open-angle glaucoma and trained them with near-threshold repetitive light stimuli (vision restoration training, VRT). They observed improvements in the perception of light stimuli in perimetries of the central visual field in four patients. This observation was later confirmed by a randomized, placebo-controlled clinical trial with 30 glaucoma patients with stable visual field loss at baseline who were randomized to one of two groups: either “VRT”, or “stimulus discrimination training” for 3 months (Gudlin, 2008). VRT significantly increased the detection performance in different perimetric tasks which confirms that even visual loss after retinal damage can be improved by training the visual field border. In summary, there are signs of considerable plasticity even in cases of retinal lesions which

204

can happen spontaneously (as shown in the PRLs) or are induced by perceptual training. While it is possible that there is plasticity on the retinal level as well or even in the damaged nerve itself, it appears that the functionally relevant change requires central visual pathway plasticity at the level of the lateral geniculate, the visual cortex or higher cortical networks.

Plasticity after optic nerve lesions Prechiasmatic (optic nerve) lesions typically have a traumatic or inflammatory origin with concentric narrowing of the visual field after compression of the outer portions of the optic nerve fibers.

Animal models Optic nerve damage has been a popular model to study neuroprotection, regeneration, and functional restoration after acute complete or incomplete injury (e.g., Benowitz and Yin, 2007, 2008; Heiduschka and Thanos, 2000; Lorber et al., 2008). Here, we will focus our discussion on restoration of function after partial optic nerve crush (ONC), which was studied for many years in our laboratory. ONC can be induced in adult rats by means of cross-action forceps which produce definable, reproducible lesions (Duvdevani et al., 1990). Though the rat has a relatively simple visual system compared to higher mammals, its contrast sensitivity function is roughly comparable to that of cats, monkeys, and humans (Keller et al., 2000). After ONC, only a definable, small number of cells survive after the injury, thus providing a small remnant of residual fibers (similar to Lashley's work, e.g., Lashley, 1939). If this spares just a small, minimum number of neurons and axons, spontaneous recovery of some visual functions can take place such as brightness or pattern discrimination, or orientation toward small moving targets (Duvdevani et al., 1990; Sautter

and Sabel, 1993; Sautter et al., 1991; Schmitt and Sabel, 1996a,b). Because the slope of recovery is typically about 2–3 weeks, ONC recovery can be correlated well with cellular and molecular changes that accompany recovery (Sabel et al., 1995; Sautter and Sabel, 1993; Sautter et al., 1991). Within the first 1–2 weeks, the number of RGCs is reduced by as much as 70–90% as a result of retrograde degeneration (Sabel et al., 1995, 1997; Sautter and Sabel, 1993). After that time, only about 10–30% of the RGCs survive and remain connected with their principle target, the superior colliculus (Prilloff et al., 2007; Rousseau and Sabel, 2001; Rousseau et al., 1999; Sabel et al., 1997; Sautter and Sabel, 1993). Although recovery is usually incomplete, the extent of recovery is remarkable: performance in visual tasks immediately after the damage is only 10–30% (which corresponds to the cell number at that time) but vision improves to about 80–90% within 2–3 weeks (Rousseau and Sabel, 2001; Sabel et al., 1997). The surviving, residual cells show morphological and functional signs of plasticity: their cell body size moderately increases (Rousseau and Sabel, 2001; Rousseau et al., 1999) and their calcium activity levels rise in a delayed and moderate manner (Fig. 1), unlike the fast calcium influx that precedes cell death (Prilloff et al., 2007). We believe that these cellular changes contribute to vision restoration because (i) the time course of the delayed cellular and metabolic changes is similar to the time course of functional recovery (Prilloff et al., 2007), and (ii) the hyperactivation of residual neurons leads to hyperresponsiveness to visual stimulation (Prilloff et al., 2007). These correlations in time of behavioral and neurobiological changes are an example of “within-systems” plasticity (see below).

Recovery of optic neuritis in humans Spontaneous restoration (recovery) of vision after optic nerve lesions is also seen in humans and

205 Recovery from optic nerve damage

(a)

Visual function

80 Percentage of control

70 60 Metabolic activity (2DG)

50 40 30

No. of neurons

20 10 0 0

2

4

6

8

10

12 14 16 18 Days after crush

20

22

26

28

30

Calcium activation in residual neurons

(b) 900

Bg

*

800

*

700

RGC type I RGC type II RGC type III Dying RGCs

*

600 FI (% Bg)

24

500

*

*

400

*

300

*

200 100 0

−2 Pre crush

2

4

6

10

15

Days after crush

Fig. 1. Recovery from partial optic nerve damage. The partial optic nerve crush in the rat serves as a model to study recovery from partial visual system damage. (a) Time course of behavioral and anatomical change after optic nerve crush. Despite an ongoing loss of retinal ganglion cells (RGCs), there is recovery of vision and metabolic (2DG) activity. The surviving cells seem to be able to compensate the loss rather well. (b) RGCs that manage to survive (RGC type II and III) show increased calcium activation and greater responsivity to visual stimulation starting at day 10-postlesion which is the time that significant recovery has taken place. These hyperfunctional cells may contribute to recovery of vision (see Prilloff et al., 2007); RGCs with massive calcium influx die within 6 days.

typically happens within the first weeks and months after damage. Recovery can happen even if conduction speed remains impaired as evident in longer latencies of the visual evoked potentials (VEPs; Levin et al., 2006; Russ et al., 2002; Werring et al., 2000). This suggests that mechanisms of recovery are probably not restricted to the optic nerve (within-systems

plasticity) but may also involve associated structures along the visual pathway (network plasticity). Korsholm et al. (2007) measured the effects of visual stimulation with functional imaging in 19 patients recovering from acute optic neuritis (ON) and found activations in the LGN of the thalamus and in the visual cortex in both the acute condition and after 3 and 6 months

206

post-lesion. In the acute phase, the LGN and visual cortex activation were significantly reduced. The difference in activation of the intact and the damaged eye, however, became smaller (recovered) over time and was no longer significant at 6 months. This could be explained by an increased activation of the retina of the damaged eye and also an activation reduction of the retina in the intact eye. Patients with ON undergo cortical and subcortical neuroplasticity as revealed by functional magnetic resonance imaging (fMRI; Korsholm et al., 2007, 2008). While adaptive cortical reorganization in higher visual areas was not directly observed in the Korsholm studies, extrastriate activations may happen, which is a sign of an adaptive reorganization of cerebral activity after acute ON (Toosy et al., 2002, 2005). Toosy et al. (2002) observed activations in the right insula/ claustrum, lateral parts of the temporal–parietal cortex and in thalamus. Thus, not only the primary structure of the visual system damage is involved in the post-lesion plasticity response but also secondary (and probably tertiary) structures. In summary, activity patterns along the entire axis of the visual system may change during spontaneous vision restoration (recovery), particularly in extrastriate areas, and these may very well be associated with performance improvements (Henriksson et al., 2007; Toosy et al., 2002, 2005). It has not been resolved to which extent these activation changes are necessary or sufficient for the recovery process, if they are adaptive or maladaptive, and which mechanisms and structures are involved. This needs further study.

Plasticity after post-chiasmatic lesions In contrast to lesions of the retina and optic nerve, damage in upstream brain regions (such as primary visual cortex) may leave different alternative pathways intact, depending on the lesion location. Especially in the early literature

the question of interest was this: Which structures are necessary for vision and how well can animals recover when visual structures are damaged?

Spontaneous recovery of vision in animals Lashley (1939) was, to the best of our knowledge, the first to report recovery of vision in rats. He found that with only small remnants of surviving tissue, amounting to as few as 700 cells in the LGN of the thalamus (which is about one-fiftieth of the normal number), visual discrimination ability was maintained. After cortical lesions, rats, just like other species (such as hamsters, hedgehogs, tree shrews, cats, and monkeys), were initially impaired in their ability to solve visual problems, but over time some visual functions recovered. To determine which brain areas are involved in recovery from visual cortex damage, Baumann and Spear (1977) first allowed the animals to recover from a visual cortex lesion and then removed additional areas of the brain. Loss of the lateral portions of the suprasylvian gyrus left the animals unable to recover, suggesting that this area played a special role in recovery. Also Fischman and Meikle (1965) suspected that other brain areas (pretectum or the suprasylvian gyrus) might be critical for recovery in such combined lesion cases. Recovery of vision has also been studied in cats. Wiesel and Hubel (1965), for example, noted some limited recovery in kittens with early visual deprivation induced by eye sutures, even if the deprived eye was reopened at a time well beyond the “critical period.” Also in adult cats, recovery of brightness discrimination was described after bilateral cortex or superior colliculus removal or simultaneous removal of both structures (in which case additional training was required, see below) (Urbaitis and Meikle, 1968). Only when all alternative pathways of the visual system were damaged simultaneously (total network lesion), recovery was no longer possible (Fischman and Meikle, 1965).

207

Since these early observations, many studies have been published on either electrophysiological measures of cortical reorganization or behavioral measures of vision restoration and reorganization following cortical deafferentation or silencing by retinal lesions. This includes studies of (i) RF reorganization after retinal (Chino et al., 1995; Gilbert and Wiesel, 1992) or cortical lesions (Eysel, 1997) which depends in its extent on visual experience (Milleret and Buser, 1984); (ii) recovery from monocular deprivation during or after the early critical period when the competing, intact eye is occluded or removed (He and Loop, 1991; Maire-Lepoivre et al., 1988; Mitchell et al., 1984; Smith and Holdefer, 1985; Spear and Ganz, 1975; van Hoff-van Duin, 1976); (iii) restoration of visual functions after additional brain lesions which lift inhibition by competing fibers to the deafferented zone (Di Stefano and Gargini, 1995; Wallace et al., 1989) or after loss of the intact, fixating eye in amblyopia (Tieman and Hirsch, 1983); and (iv) complete or incomplete spontaneous recovery of vision after lesions of the cortex (Baumann and Spear, 1977; De Weerd et al., 1993, 1994; Fabre-Thorpe et al., 1994; Wallace et al., 1989) or optic tract (Jacobson et al., 1977, 1979). In the macaque monkey, the primary visual cortex and visual association areas occupy about 50% of the total cortical mantle (Van Essen and Maunsell, 1980). Monkey studies on restoration of vision are more rare and they typically employ only very few animals. While specific lesions within visual pathways usually lead to stable deficits, there have been a few reports showing recovery of some visual functions in monkeys indicating that an initial loss of vision must not always be permanent. Zee et al. (1983), for example, created bilateral occipital lobectomies in monkeys, rendering the animals incapable of smooth pursuit eye movements 1 month postsurgery. In the subsequent months, however, the function recovered to normal levels. Also Mohler and Wurtz (1977) noted recovery in a visual detection paradigm within 3

weeks following either cortical or tectal injury, but no recovery was seen when both lesions were combined. Also, lesions in area MT produced pursuit eye-movement deficits from which the monkeys recovered within the relatively short period of about 1 week (Dürsteler et al., 1987; Newsome et al., 1985). This was attributed to the relatively small size lesion. Surprisingly, unilateral lesions produce sometimes more permanent deficits from which the animals do not recover. Segraves et al. (1987) offer the following explanation for this apparent paradox: “First, the monkey with unilateral striate lesions presumably relies upon the intact striate cortex for input to the smooth pursuit system. However, a monkey with a bilateral striate lesion is left with only subcortical and residual extrastriate visual mechanisms, and so may use them more fully. The effect is analogous to the tendency of monkeys with unilateral rhizotomies to avoid use of the deafferented limb until the intact one is mechanically restrained” (p. 3056). This is related to the “Sprague effect” as discussed below.

Spontaneous recovery of vision in humans Traditionally, geniculostriate lesions were considered to result in complete and permanent visual loss in the topographically related area of the visual field (Holmes, 1918; Poppelreuter, 1917) though the maintenance of movement perception was noticed by some clinical investigators (Riddoch, 1917). Lesions of the occipital cortex typically cause contralateral visual field defects termed hemianopia or quadrantanopia, depending on their size. Teuber et al. (1960, 1975) was among the first to systematically observe recovery of vision in patients. He extensively studied soldiers with gunshot wounds of the brain acquired during the Korea war and found better recovery in younger patients. Further, the spontaneous shrinkage of the resultant scotoma depended on the age at lesion (Teuber, 1975).

208

In humans with brain injury, recovery of visual functions is the rule and not the exception. But patients with visual field defects typically have a poor prognosis if they do not spontaneously recover early on. The speed of recovery depends on the lesion characteristics: whereas in cases with partial defects maximal recovery is achieved within the first 48 h (Pambakian and Kennard, 1997; Pambakian et al., 2005), recovery from complete hemianopia occurs usually within the first few weeks. About half of the patients show partial recovery and less than 10% of patients recover their full field of vision back (Zhang et al., 2006). There are only very rare cases of spontaneous vision recovery beyond this time point (Poggel et al., 2001). Nelles et al. (2002, 2007) studied patients with ischemic lesions of the visual cortex using functional imaging. While in a control group, they observed the maximum activity in hemifield stimulation in the contralateral visual cortex and bilaterally in the extrastriate cortex, the patients showed increased ipsilateral activation in the extrastriate cortex when stimulating the hemianopic hemifield. Although many studies are available on spontaneous and traininginduced visual field recovery, cortical reorganization processes after acquired visual cortex lesions are rarely examined (Dilks et al., 2007). Schoenfeld et al. (2002) described a young hemianopic patient (age 22 years), who spontaneously recovered some motion and color perception at 1 month post-lesion. Functional neuroimaging showed activation of areas V4/8, V5, and V2/3 with no activation in his damaged V1. Magnetoencephalographic recordings revealed more posterior activation areas V2/3 followed by activation of the MTþ and V4/8 complex. Other functional imaging studies have also shown that stimulating fields of residual (or recovered) vision leads to activation of extrastriate cortical regions which was interpreted as a sign of reorganization (Rausch et al., 2000). But other studies could not confirm this (Barbur et al., 1993; Zeki and Ffytche, 1998). Clearly, this is an area requiring further experimentation.

In this context, it is noteworthy that patients, especially those with blindness early in life, show a massive reorganization of the brain involving multimodal activation of nonvisual senses. Here, the brain recruits visual cortex for other functions, for example, processing of tactile input in reading Braille (Sadato, 2005). This type of “transmodal plasticity” is important for the patient to compensate their vision loss and remain able to orient and navigate in space.

Reorganization in congenital and early blindness Congenital and early blindness are different in that they originate early in life, when the brain has a considerable developmental plasticity potential. Here, the “visual” cortex of the blind processes somatosensory and auditory information, which suggests a rewiring of neural associations sending nonvisual sensory information to the visual cortex (Noppeney, 2007; Ptito and Kupers, 2005; Ptito et al., 2008). Consequently, “intermodal plasticity” appears to outpace within-systems plasticity in the early blind. Park et al. (2007) studied the neural reorganization in the visual cortex in early blind patients with diffusion tensor functional imaging. Mainly in the primary visual pathway, reduced anisotropy and increased diffusion were found compared to emmetropic subjects. Changes in regional diffusion were observed not only in the visual pathway but also in nonvisual areas such as the U-fibers of the parietal lobe, the striatum, the pulvinar, and the inferior and superior longitudinal fasciculus. These changes are adjustments to the early loss of visual system structures. These adjustments, in turn, may represent hyperfunctions of other sensory systems (especially hearing and somatosensory functions) which the blind need for orientation in space (Ptito et al., 2008). Another study that highlights the potential of residual capacities of congenitally blind people was presented by Gothe et al. (2002) who applied transcranial magnetic stimulation (TMS) to excite

209

the visual cortex of patients with congenital blindness. Patients reported phosphenes in different retinotopic positions even inside areas of perimetric blindness, and these phosphenes could be provoked by TMS more easily in patients with some residual vision. Recently, a remarkable case of plasticity in congenital blindness was reported by Ostrovsky et al. (2006). An adult subject from India, S.R.D., who was blind from birth on until age 12 at which time she underwent surgery for removal of dense congenital cataracts on both eyes, still had acuity impairments at age 32, but, surprisingly, S.R.D.'s acuity was proficient on mid- and high-level visual tasks. The authors concluded that the human brain retains an impressive capacity for visual learning well into late childhood which was observed also in three other subjects in a later study (Ostrovsky et al., 2009).

Residual vision Clinically relevant plasticity occurs not primarily in regions of “absolute blindness” but in “areas of residual vision” (ARVs). They are located at the visual field borders and in islands of residual vision in regions of presumed “total” blindness.

Residual vision at the visual field border The difficulty to appreciate the existence of residual vision at all has, besides conceptual issues, a technical origin. Current perimetric methods were not designed to measure ARVs, visual cognition, or subjective vision. They were designed to measure vision loss that results from eye diseases (such as glaucoma). Thus, standard perimetry methods are not very detailed (low resolution) and have other limitations when applied to the studies of the more subtle phenomenon of visual system plasticity: they simply ignore the weaker residual visual functions. This may be one source of some controversy over the

interpretation of visual field expansion results (Sabel and Trauzettel-Klosinski, 2005). Just paying attention to intact regions and damaged areas (absolute defects) is insufficient. Rather, ARVs that are known also as “relative defects” are key in our understanding of visual system recovery (Kasten et al., 1998a; Sabel, 1999; Sabel and Kasten, 2000; Sabel et al., 2004; Widdig et al., 2003; Zihl and von Cramon, 1979). Thus, a more precise diagnostic with higher resolution needs to complement the existing static supraliminal- and threshold-perimetry. Also, visual information in daily life is normally processed binocularly. For these reasons in our laboratory, we usually evaluate visual fields with both eyes open using a specifically developed computer-based method termed “high-resolution perimetry” (HRP; Kasten et al., 1998a,b). This method presents suprathreshold light stimuli repeatedly in random sequences (Kasten et al., 1998a). As Fig. 2 shows, binocular HRP reveals areas of inconsistent detections (gray areas), intact areas (white), with reliable stimulus detection, and areas of absolute blindness (black areas). The brightness of the light stimuli is of decisive relevance for the characterization of the intact, residual (partially damaged), and absolute damaged areas. When using brighter (high-contrast) light stimuli, the intact visual field area appears larger than when darker (low-contrast) light stimuli are used. Typically, there is not a sharp border between the damaged and intact visual field but rather a more smooth “transition zone” (relative defects) which varies considerably between patients in size or shape (Kasten et al., 1998a). These “fuzzy” transition zones are particular prominent in patients with prechiasmatic (optic nerve) lesions as they have a rather inhomogeneous, scattered visual field defect (Fig. 2). We have proposed that these transition zones are the functional representation of partially damaged brain areas and termed them “ARVs” (Fig. 3). They are the regions where plasticity mostly occurs. Here, neurons survived the damage similar to what we found in our rat

210 Areas of residual vision

(a)

20

20

15 10 5

15

0 −5 −10

10

−15 −20 −25 −20 −15 −10 −5

0

5

10 15 20 25

5

20 15 10

0

5 0 −5

−5

−10 −15 −20 −25 −20 −15 −10 −5

0

5

10 15

20

−10

25 20 15

−15

10 5 0

−20

−5 −10 −15 −20 −25 −20 −15 −10 −5

0

5

10 15 20

100 % 80 % 60 % 40 % 20 % 0% Fixpoint

−25 −20 −15 −10 −5

0

5

10

15

20

25

25

(b)

Intact

ARV

blind

(c)

Full input

(d) Reduced input

Complete damage

Fig. 2. Areas of residual vision (ARVs). (a) To assess the visual fields with high-resolution computer-based perimetry (HRP), suprathreshold stimuli are presented at random from which simple detection charts (here: 3) can be created. Intact visual field sector is shown in white; black represents regions of absolute blindness. When superimposed, the new chart (right) reveals gray areas where response accuracy is inconsistent. They are known as ARVs or relative defects. Gray regions are interpreted by us as representing partial damage where only some cells remain connected with their target structure. Thus, partial structure leads to partial function (b, c). The disconnected cells will degenerate retrogradely due to lack of trophic support. (b) Different brain regions (square) can suffer different severities of deafferentation, shown in different shades of gray in the visual field map. The extent of deafferentation has a functional correlate: the greater the loss, the lower the functional accuracy, ranging from intact (white) through shades of gray (i.e., ARV) to black (blind). (c) The concept of stimulation-induced synchronization after partial nervous system damage. While neurons of the intact regions fire in a synchronized manner to drive normal vision (here they jointly fire action potentials in perfect temporal coordination), areas of partial damage are nonsynchronized, with poor firing synchrony. In blind (black) regions, no neuronal firing is elicited due to complete loss of neurons. After external stimulation which is induced by training or during electric current stimulation, the partially damaged regions are forced to fire jointly in temporal coordination. It is hypothesized that repeated stimulation induces a synaptic plasticity of the partially damaged structures (shown here) and downstream areas (not shown here), stabilizing their synchronous firing beyond the treatment period (aftereffects). This improved or stabilized synchronization is one of the proposed neurophysiological mechanism of vision restoration.

211 (a)

Plasticity in striate and extrastriate regions

P4

V2–V5

LGN

P1 Striate

P2 90% Retina

V1

10% P3 Extra striate

Tectum/pulvinar

Restored visual fields

(b) Before

After 20

Training

20 15

15

10

10

5

5

0

0

–5

–5

–10

–10

–15

–15

–20 0

5

10

15

20

–20 30

–25 –20 –15 –10 –5

0

5

10

15

20

ACS

–25 –20 –15 –10 –5

100 % 80 % 60 % 40 % 20 % 0% Fixpoint

Fig. 3. Vision restoration pathways: striate and extrastriate. (a) This highly simplified diagram shows the primary and secondary visual structures in the human brain. In the normal brain, the main retinofugal pathway is the retino-geniculo-striate pathway (striate route) which comprises the majority (>90%) of retinofugal fibers. It supplies striate cortex and higher cortical regions with neuronal information for normal perception. Only a small proportion of retinofugal fibers (probably 40 Kasten et al. 2001 16 Yes Yes VFI of about 2.7 in the experimental group versus 0.53 in the control group Julkunen et al.

2003

5

Yes

No

VFI between 5 and 10 in three patients; partly validated by visual evoked potentials; subjective improvements in four patients

Sabel et al.

2004

16

Yes

No

Reinhard et al.

2005

17

Yes

No

Werth and Seelos

2005

17

Yes

Yes

Kasten et al.

2006

15

Yes

No

Julkunen et al.

2006

1

Yes

No

No VFI when checked with Scanning Laser Ophthalmoscope, shift of the visual field border about 5–7 in perimetry and computer campimetry No VFI in SLO; increased reading speed in 6% of the patients; satisfaction with training in two-third of the patients (Note: same patient sample than Sabel et al. (2004) study Study of children; VFI improvements only in experimental group; control of eye movements; validation by fMRI; 11 of 17 patients recovered vision 1 year after the training; none in the control group Improvements of stimulus detection in campimetry about 3.8%; decrease errors in perimetry OD (2.2%) and OS (3.5%); improvements of visual field independent from eye movements Normalization of P100 latencies after visual training of the border region, increased cerebral blood flow restricted to the occipital lobe in a follow-up study at 3 three months

Follow-up

No No

No

Yes

On average no significant decline at 23 months follow-up At 3 months followup increased or stable area of normal vision in three patients No

No

Yes

No

Yes

(Continued)

218 Table 1. Studies reporting visual improvements after intervention (Continued) Study

Year

N

Preand postdesign

Control group

Results

Follow-up

Schmielau and Wong Kasten et al.

2007

20

Yes

No

No

2007

23

Yes

No

Marshall et al.

2008

6

Yes

No

Gall et al.

2008

85

Yes

No

Gudlin et al.

2008

5

Yes

No

Poggel et al.

2008

19

Yes

No

Jung et al.

2008

10

Yes

Yes

Romano et al.

2008

161

Yes

No

Mueller et al.

2008

17

Yes

No

Jobke et al.

2009

18

Yes

No

Poggel et al.

2010

19

Yes

No

Marshall et al.

2010

7

Yes

No

Raemaekers et al.

2011

8

Yes

No

VFI of 11.3 on average in 17 patients; subject. improvements in daily life VFI in HRP (4.2%), fewer misses within the central 30 perimetrically (3.7% OD, 4.4% OS), VFI did not benefit from doublestimulation Significant increase in BOLD activity in border zone detections after VRT, relative improvement in response times in the border zone, brain activation changes with a shift of attention from the nontrained seeing field to the trained border zone VFI < 5% detected stimuli in 42% of the patients, 5–10% in 24% and >10% in 28% of patients VFI in HRP and 30 white/white perimetry after the first treatment, stable effects after training-free interval of 3 months Significantly improved detection and reaction times in perimetric and HRP-tests along the visual field border; no improvement in visual acuity VRT compared to intact visual field VRT Improved binocular reading speed, foveal sensitivity (trend), HRP detection by 16–17%, but in both groups Mean absolute VFI of 12.8% after VRT, improvements of 3% in 76% of patients Training effects of 3.5% (OD) and 1.5% (OS) after 6 months of daily VR training, minor training effects of long-term training VFI were twice as good as after extrastriate VRT (4.2%) than after standard VRT (2.4%) VFI in HRP from 53.6% to 57.6%, increase of intact field size of more than 16% or border shifts of more than 18 Average improvement in stimulus detection rate by microperimetry of 12.5% (range–1.4% to 38.9%). Six of 7 patients had  3% improvement in stimulus detection by home-based perimetry VFI with shifts of the central visual field border ranging between 1 and 7

No

Yes

No

No

No

No

No No

Yes

Yes (at 6 months)

No

No

219 Table 1. Studies reporting visual improvements after intervention (Continued) Study

Year

N

Preand postdesign

Control group

Results

(b) Perceptual training with different stimuli inside the blind field or in amblyopia Hyvärinen et al. 2002 5 Yes No Improvement of flicker sensitivity in the blind hemifield equal to that in the normal hemifield in two patients, increased recognition of (non-) flickering letters at 20 eccentricity in one patient Polat et al. 2004 77 Yes Yes Training with spatial frequency tasks; improvement in contrast sensitivity, visual acuity improved by 78% above baseline with the greater improvement in amblyopics with lower initial acuity Sahraie et al. 2006 12 Yes No Repeated stimulation inside the blind visual field resulted in improvements deep in the field defect, discrimination performance increased monotonically with increasing contrast Raninen et al. 2007 2 Yes No Improvement of flicker sensitivity in the blind hemifield within 20 respectively 30 eccentricity, recognition of flickering letters at 10 eccentricity Henriksson et al. 2007 1 Yes No Visual information of flicker training was mainly processed in the intact hemisphere, representation of both the intact and the blind hemifield takes place in the same set of cortical areas in the intact hemisphere Chokron et al. 2008 9 Yes No Objective improvement of behavioral tasks in nine patients and objective enlargement of the visual field in 8/9 patients Roth et al. 2009 28 Yes No No improvement with flicker-stimulation training deep in the blind field Jobke et al. 2009 18 Yes No Detection performance increased twice as much after extrastriate VRT (4.2%) than after standard VRT (2.4%) Polat et al. 2009 5 Yes No Training with Gabor patterns; Visual acuity improvement of 1.5 Snellen lines, improvement of contrast sensitivity in children Sahraie et al. 2010 4 Yes No Improved detection ability in 3/4 patients after visual detection training of spatial grating patches within the field defect (C) Noninvasive alternating current stimulation of the brain Gall et al. 2010b 1 Yes No Detection ability increased from 3.44% to 17.75%, mean perimetric threshold from 0 to 2.21 dB Fedorov et al. 2010 446 Yes No VFI in 40.4% (OD) and 49.5% (OS) of the patients after rtACS, significant increase of visual acuity (OD: 0.02; OS: 0.015), further improvement after a second treatment course

Follow-up

Yes

Yes

No

Yes

No

No

Yes Yes

No

Yes

No

Yes

(Continued)

220 Table 1. Studies reporting visual improvements after intervention (Continued) Study

Year

N

Preand postdesign

Control group

Results

Follow-up

Sabel et al.

2010

22

Yes

Yes

Yes

Gall et al.

2011

42

Yes

Yes

Significantly greater visual field defect reductions in the rtACS group (69.25%) than in the placebo group (16.93%), decrease of reaction times in rtACS- but not in placebo patients Detection ability increase in the defective visual field was significantly larger after rtACS (41.1%) than after sham-stimulation (13.6%)

increasing the function of the border areas have been conducted with stroke and trauma patients who suffered hemianopia or scotomata. Zihl and von Cramon (1979, 1985) have trained the border itself using repetitive visual field testing. Others have stimulated the border region after first identifying ARVs and focusing the training area on them (see Kasten and Sabel, 1995; Kasten et al., 1998a,b,c). The training is aimed at the border region especially at regions of partial damage, that is, ARVs (gray in Fig. 2). This approach is termed VRT. Extensive training over the course of up to 6 months leads to a reduction of the scotoma size (Julkunen et al., 2003, 2006; Kasten et al., 1998a,b; Marshall et al., 2008; Mueller et al., 2008; Poggel et al., 2008; Romano et al., 2008; Sabel and Kasten, 2000; Sabel et al., 2004; Werth, 2008; Widdig et al., 2003), effectively enlarging the visual field by primarily transforming the ARVs into intact areas (Fig. 3). About half to two-thirds of the patients achieve visual field expansions (at an average of 5 degrees of visual angle), but training success varies considerably between patients; some patients (1/3) do not respond to the therapy, others show moderate improvements (1/3), and yet others (1/3) have larger types of field expansions (Mueller et al., 2008; Romano et al., 2008; Sabel et al., 2004; Sabel and Kasten, 2000).

No

The effects of VRT were confirmed by others as well. Romano et al. (2008), for example, carried out a clinical observational study and found improvements which were even superior to our previous studies. However, while in our trials, hemianopics were included irrespective of their visual defect characteristics, Romano recruited patients that had clearly identifiable ARV before therapy, which increased the likelihood of improvement. This points toward a special role of ARVs in the recovery process. While patients with ARVs respond well to therapy, those devoid of ARVs, that is, with areas of absolute blindness only (“sharp” visual field borders), do not benefit as much. Therefore, when the criterion of training success is “improvements in the field of absolute blindness only” (ignoring improvements in ARVs) training has no effects (Reinhard et al., 2005; Roth et al., 2009). Yet others used different forms of visual field border training (Schmielau and Wong, 2007) and found reliable improvements. Such improvements are, however, limited: when a second 6-months training period is given to the patients, there are no significant, additional effects detectable, at least not when using simple stimulus detections (Mueller et al., 2008). Also Bergsma and van der Wildt (2010) trained 11 subjects with cerebral blindness with VRT, confirming a “gradual

221

shift of the visual field border” which was independent of the type of stimulus-set used during training while eye fixation was monitored. Another interesting observation was reported by Jung et al. (2008) who stimulated the border region in one group of patients with anterior ischemic optic neuropathy and the intact region in another group. Here, detection improvements were seen in both. As one would expect, children of different ages also benefit from vision training. Werth and Moehrenschlager (1999) and Werth (2008) looked at very young children at preschool ages, and Mueller et al. (2008) studied older children of school age. In these studies, significant visual field improvements were noted as well. They benefited as much, if not more from the training than adults, but it is unclear if the training effects at young or adolescent age are more pronounced than in adulthood or older age. This is an open issue. Not all experiments found evidence for training-induced visual field improvements in patients with brain damage (Balliett et al., 1985; Reinhard et al., 2005), but these studies suffer some methodological and interpretative flaws. Balliett et al. (1985), for example, used very small stimuli for training, and training time was much shorter (a few weeks only) than those of all other studies (several months). In the study by Reinhard et al. (2005), a visual field expansion could not be confirmed with a laser scanning ophthalmoscope (SLO), but when the more sensitive high-resolution and standard perimetry methods were used, visual field enlargement could be found in the same patients (Sabel et al., 2004). A closer analysis of the exact visual field topographies indicated that SLO measurements were in fact very difficult for patients to perform, and the detection task of the SLO was also not the one which was trained (Sabel et al., 2004). In fact, the SLO chart displayed ARVs as areas of absolute blindness, suggesting that the SLO is not sufficiently sensitive to detect areas of relative loss (for details, see Kasten et al., 2008; Sabel et al., 2004). Yet,

despite their technical and interpretative limitations, the only two negative studies (Balliett et al., 1985; Reinhard et al., 2005) point us to important methodological issues, which have been the source of some controversial discussions (Sabel et al., 2004). In summary, the vast majority of studies found consistently a rather positive outcome of visual training, and they outnumber experimental studies with null findings by far. There is also physiological and brain-imaging evidence for training effect. Physiological observations have the advantage of providing a more “objective” means to document vision restoration and plasticity after visual field training. Julkunen et al. (2003, 2006), for example, measured VEPs before and after visual training of the border region and showed a normalization of P100 latencies. After therapy, the same patient showed an increased blood flow in cortical and subcortical structures as measured by PET. In a follow-up study 3 months after the end of training, the increased cerebral blood flow was restricted to the occipital lobes (Julkunen et al., 2006). Raninen et al. (2007) trained with flicker light or flickering letters twice a week for the period of 1 year which improved flicker sensitivity in the blind hemifield although no evidence of visual field changes were observed in perimetry. Henriksson et al. (2007) found evidence for reorganized visual cortices using both magnetoencephalography and fMRI recordings after training. The pattern of change suggests that structures surviving the injury had now participated in the processing of visual information, that is, the training affected not small (residual) areas alone but had an influence on the brain network as a whole, that is, other brain regions. This is in line with Marshall et al. (2008) who treated six chronic, right hemianopic patients with VRT and applied fMRI while patients were responding to stimuli in the trained visual border zone. The results of the trained region were compared with those of the nontrained seeing field before and after 1 month of VRT. The authors found a significant increase in BOLD activity in

222

border zone detections, and this correlated with a relative improvement in response times in the border zone. An analysis of the BOLD patterns revealed brain activation changes that were consistent with a shift of attention from the nontrained seeing field to the trained border zone. The effect appeared to have been mediated by the anterior cingulate and dorsolateral frontal cortex in conjunction with other higher-order visual areas in the occipitotemporal and middle temporal regions. In summary, training of the visual field border region does not only result in improved parameters of vision (such as light detection in perimetry), but it also leads to increased neuronal activation in wider regions of neuronal networks. Because the intact hemisphere also seems to contribute to recovery, vision restoration appears to be the result of both local and global influences (see discussion below). Training alternative pathways (blindsight) Some investigators have repeatedly trained deep in the blind field with the goal to enhance “blindsightlike” responses. Similar to visual field border training, they found improvements in visual detection performance (Chokron et al., 2008; Sahraie et al., 2006). The most famous blindsight case is patient GY (Weiskrantz, 1996, 2009; Weiskrantz et al., 1974) who was trained (by virtue of repeated testing) over many years and showed some remarkable improvements throughout this time (see also Chokron et al., 2008; Stoerig, 2008). Sahraie et al. (2006) asked a group of 12 cortically blind patients to discriminate simple grating stimuli for a 3-month training period. Repeated stimulation inside the blind visual field (and not only at the border zone, as in VRT) resulted in improvements deep in the field defect. But Sahraie noted (personal communication) that it is important to start first with the stimulation of the border region. In this kind of blindsight training it is apparently necessary to costimulate the border region in a way that patients are able to

see some portion of the visual stimulus at the beginning of the training, similar to “prompting” during behavioral shaping. This may also explain why others failed to improve visual fields: they used a simple flickering stimulus which was presented deep in the blind field, ignoring or avoiding ARVs (Roth et al., 2009). The need to stimulate the border region in early training phases makes it difficult to clearly separate an “alternative pathway” (blindsight)-training from the classic visionrestoration training. Perhaps ARVs are initially needed for some prompting, an issue requiring further study. Other investigators have studied the effects of vision training using slightly different training paradigms. Chokron et al. (2008) treated nine patients with unilateral occipital damage for 22 weeks using several blindsight-like forced-choice visual tasks: pointing to visual targets, letter recognition, visual comparison between the two hemifields, target localization, and letter identification. An improvement was found in all behavioral tasks for all patients and visual field enlargements of the contralesional visual field for all except one patient. Huxlin et al. trained both animals (Huxlin and Pasternak, 2004) and patients with brain lesions (Huxlin et al., 2009) to perform a movement detection task. This behavioral paradigm stimulated preferentially extrastriate pathways that directly innervate V5 via the LGN of the thalamus or through the tectal/pulvinar route with an array of moving dots. They found that in animals or patients with V1 lesions, movement perception could be improved. Das and Huxlin (2010) report brain activation changes after such a visual training task in a single subject with cortical blindness. Before training, they found widespread hyperactivation in V1/V2, V3, and hMTþ of the intact hemisphere, with no measureable activity on the damaged side. However, after intensive global direction discrimination training of the blind field (involving as many as 30.000 trials), the hyperactivation of the intact hemisphere was reduced toward

223

control levels while a recovered activation pattern was seen in regions of on the lesioned side, including perilesional tissue (V1/V2) and V3a and hMTþ confirmed the behavioral studies (Huxlin et al., 2009). Jobke et al. (2009) used a combined striate/ extrastriate training approach. Here, visual stimuli were presented to the border region using classic VRT, while simultaneously stimulating deep in the blind field with a moving spiral. The aim of this “extrastriate-VRT” (eVRT) was to create a maximal behavioral stimulation of both the residual structures (ARVs) in the border region plus a stimulation of the entire blind field sector to activate extrastriate (blindsight) pathways. Whereas standard VRT significantly improved stimulus detection by 2.9%, eVRT patients improved by 5.8%, doubling the extent of vision restoration. This confirms the hypothesis that extrastriate pathways, bypassing the damaged visual cortex, can be recruited to contribute to vision restoration. While visual border stimulation and blindsighttraining are different in principle, in practice residual regions (“islands of residual vision” within the blind field or border zones between intact and blind field) are probably trained together with the alternative pathways in most studies. Just as in saccadic eye-movement training, it is difficult to fully avoid ARV stimulation when presenting visual stimuli to patients with visual field defects in blindsight paradigms. Likewise, when attempting to train only the border regions alone, a certain proportion of stimuli (about 20%) are located in the blind field, leading to a “mini-blindsight”-training, that is, unintentionally also stimulating the extrastriate pathways in the blind field. Likewise, when aiming at the blind field only, it can hardly be avoided to also excite residual tissue. Training amblyopia Amblyopia is another visual disorder where research of perceptual learning (training) has contributed to our understanding of residual vision. Amblyopia refers to a unilateral

or bilateral decrease of vision caused by abnormal binocular visual experience during the “critical period” early in life (Levi and Carkeet, 1993). It leads to serious deficits in parameters of spatial vision such as impairments in visual acuity (VA), contrast sensitivity, vernier acuity, spatial distortion, spatial interactions, or contour detection (for reviews, see Hess et al., 1990; Levi and Carkeet, 1993). On a physiological level, amblyopia is thought to be caused by alterations in orientationselective neurons and their interactions in the primary visual cortex (Polat, 1999). The standard amblyopia therapy in children is to use an eye patch of the normal eye which forces the brain to use the visual input from the weaker (amblyopic) eye (Li et al., 2005). It has been a long-held notion that this approach is effective only when applied up to the critical age of 8–9 years. Therefore, any recovery was seen just as an “extension of normal development,” not an instance of vision restoration. But even in adults with amblyopia, recovery of visual functions can be achieved with occlusion therapy (Wick et al., 1992) and it also has been noted spontaneously after the loss of vision in the good eye (El Mallah et al., 2000). In fact, stimulating the amblyopic eye by repetitive practice can induce plasticity in adults, effectively improving visual functions (Fronius et al., 2005, 2006; Levi and Polat, 1996; Levi et al., 1997; Polat et al., 2004). For example, Polat et al. (2004) described a visual-training procedure specifically designed to train the abnormal lateral interactions by probing spatial interactions with flanker tasks. Training consisted of a Gabor patch detection task. VA improved by 78% above baseline with the improvement being greater in patients with lower initial acuity and improvements in contrast sensitivity at all spatial frequencies and in binocular functions. The authors pointed out that long-lasting effects are typical for perceptual learning and that it is a sign that “the high spatial frequencies are used after the treatment in daily tasks and thus are naturally practiced” (Polat, 2008).

224

An interesting observation was the transfer of these improvements to other tasks. Though treatment was monocular, targeting the lateral interactions of the amblyopic eye, led to a transfer of improvement to other, unrelated functions such as VA and binocular functions. In contrast to perceptual learning, where improvement is usually specific to the trained task (Fahle, 2005), the transfer of functions in amblyopic patients shows that there are nonspecific elements involved in the vision restoration process. It was proposed that practice restored normal balance between excitation and inhibition (Mizobe et al., 2001; Polat, 1999; Polat and Sagi, 2006; Polat et al., 1997). In summary, training (practice) of visual functions is currently the most widely used method to alter visual system plasticity and induce vision restoration. Compensatory (eye movement) training Another training method for hemianopia is saccadic exploration training. As Das and Huxlin (2010) recently summarized, there is some evidence on how patients with cortical blindness attempt spontaneously to compensate their deficit by eye movements toward the hemianopic field. Based on this observation, some authors argue that training such eye movements would actually help patients increase visual orientation (though it usually seems not to enlarge visual fields). We do not discuss this approach in any detail here, as training of eye-movement behavior does not aim at vision restoration but at field of view enlargement so that patients utilize the intact visual field sectors more (for review, see Kerkhoff and Schindler, 2000; Kerkhoff et al., 1992a,b, 1994). Also, we do not believe that training patients to move their eyes around more vigorously has a long-lasting benefit because (i) whenever the patient looks to the right he misses the left, and (ii) more eye movements means greater effort for integration of moving images. Still, it is interesting to note that patients performing saccadic training also experience “unintended” visual field

enlargements (Kerkhoff et al., 1992a,b, 1994) which is not really surprising because these types of trainings never specifically avoided the simultaneous stimulation of the border regions where residual structures are present. Actually, a recent study found that eye movement training had no greater effects than attention training alone (Lane et al. 2010). This suggest that eye movement training may actually enhance restoration and not only compensation.

Activating residual vision by electrical current stimulation Invasive current stimulation methods The attempt to restore visual circuitry by artificial means with invasive electrical stimulation has been around for almost 100 years. Here, the goal was to stimulate optic nerve or visual cortex by invasive current stimulation methods to replace or augment lost visual input by artificial electrical signals. Historically, the first experiment of electrical stimulation to excite the visual system was reported by Foerster (1929). He stimulated visual cortex to produce phosphene perceptions and found that their appearance depended on where the cortex was stimulated. These findings formed the basis of the concept of the visual prosthesis, where local electrical stimulation in human visual cortex was used to excite phosphenes to help facilitate visual perception. Chronic stimulation was later achieved by electrodes which were implanted directly into cortex (Brindley and Lewin, 1968; Brindley and Rushton, 1977; Brindley et al., 1972; Dobelle et al., 1974, 1976; Pollen, 1977). But it turned out that such cortical stimulation would be only of limited clinical use: the resolution was not only too low, but it also carried a high risk of inducing seizures in patients. Later attempts of applying low current microstimulation of visual cortex achieved a better resolution and improved safety, although this

225

approach has never gone beyond the experimental stage. Here, it was of interest if visually perceived phosphenes are useful to create spatial patterns of sufficiently high resolution such that subjects would recognize objects in the environment, to check if perception was stable for months or years, and to determine how a blind person with very old visual cortex lesions, who had become accustomed to the blindness, would respond to electrical stimulation (Bak et al., 1990; Schmidt et al., 1996). In animal experiments, electrically evoked responses of visual cortex were recorded during electrical stimulation of the optic nerves (Bartley and Ball, 1969; Malis and Kruger, 1956). Intact rabbit optic nerves were stimulated by needle electrodes implanted in the optic disc and electrically evoked potentials (EEPs) could be recorded in the primary visual cortex (Sakaguchi et al., 2004). More recently, optic nerves were stimulated by an invasive method in the clinical setting by Sakaguchi et al. (2009) using a chronically (6 months) implanted, direct optic nerve electrode in a single blind patient with retinitis pigmentosa. Visual sensations were elicited by electrical stimulation through each electrode. This type of study provided the basis for the most recent work on retinal implants which is discussed in Chapter 1. There is also an early Russian tradition of using invasive electrical stimulation approaches to treat vision loss. Here, multiple deep brain microelectrodes were used for subcorticography and diagnostic stimulation for the investigation of extrapyramidal movement disorders, central pain, epilepsy, and obsessive–compulsive disorders (Bechtereva et al., 1972). This initial work in the field of stereotaxic neurology was later extended to the stimulation of damaged optic nerves using implanted electrodes with the goal to induce recovery of vision (Bechtereva et al., 1985). They found significant vision recovery after 3–4 weeks and this recovery remained stable for over 2 years.

From our own studies, we know that visual cortex remains responsive to create visual percepts even in cases of congenital blindness: when V1 is stimulated by TMS in congenitally blind patients, phosphenes are still generated in retinotopic order (Gothe et al., 2002). It is this residual processing capacity that provides an anchor for enhancing visual functions in the blind.

Noninvasive current stimulation methods In contrast to invasive approaches, noninvasive approaches are aimed at influencing brain physiology on a network level and this, in turn, might affect sensitization of deafferented regions or synchronization (entrainment) of neuronal network firing with long-lasting (plasticity) changes (so-called aftereffects, see Zaehle et al., 2010). These methods do not aim at “replacing” the lost retinal cells or neuronal circuitry or stimulating brain nuclei locally which is what retina or brain implants try to achieve. The work on noninvasive electrical current stimulation in the visual system was first developed in Russia, where Bechtereva's started off with invasive methods using specific stimulation protocols (see above). These protocols were subsequently applied also noninvasively in patients with visual system damage (Chibisova et al., 2001; Fedorov et al., 2005). Here, electrodes were attached to the eye orbit and repetitive, transorbital, alternating current stimulation (rtACS) was applied. In a large clinical observational study of 446 patients with optic nerve damage, they measured visual fields before and after 10 days of rtACS treatment (Fedorov et al., 2010). rtACS led to significant VA improvements and visual field enlargements. On average, visual field sizes improved by up to 9% over baseline. Also, VA significantly increased in both eyes. In a subsequent double-blind and placebo-controlled clinical trial, optic nerve patients were randomly assigned to an rtACS or a sham group

226

(Sabel et al., 2010). The treatment was given daily for 20–40 min for 10 days (EBS Technologies GmbH, Kleinmachnow, Germany). In the rtACS group, significant vision improvements were seen in detection accuracy evident as a shrinkage of the scotoma by > 40% change from baseline, reaction time ( 19.63 ms), static near-threshold perimetry, and VA (Fig. 5). The improvements of visual functioning in the rtACS group were stable at a 2-month treatment-free follow-up, and they were associated with improvement in the patient's quality of life as assessed by standard questionnaires. Thus, noninvasive current stimulation using rtACS can be used to reduce visual

field defects in patients with long-term optic nerve lesions. Electroencephalogram (EEG) power-spectra analysis also showed significantly increased alpha-activity, especially in occipital sites following rtACS (data not published, see Fig. 6). In view of these findings, we proposed that rtACS leads to increased neuronal network synchronization which is substantiated by lasting bilateral synchronous waves of alpha- and theta-ranges in central and occipital brain areas. This “synchronization hypothesis” assumes that by firing “artificial” electrical trains of impulses at predetermined frequencies to the brain, neuronal

Visual field improvement after repetitive transorbital alternating current stimulation (b) (a) 1 3

3 1

4

Baseline

2

2 2

Reduction of visual field defect (%)

(c)

1

4

3

100

rtACS-group (n = 12)

80

Sham-group (n = 10)

60

4

(d) After rtACS

69.25**

40 20

16.93* 0

Fig. 5. Visual field dynamics after alternating current stimulation. Patients with optic nerve damage were treated with repetitive, transorbital, noninvasive brain stimulation (rtACS, EBS Technologies GmbH, Kleinmachnow, Germany). (a) The montages of the electrodes which were placed on the skin around the eye ball. (b and d) The visual field charts before and after 10 days of rtACS in a single case with traumatic optic neuropathy (Gall et al., 2010b, for explanation of charts, see Fig. 2). As demonstrated by the brightening of the chart, the patient improved from 3% detection performance to 23%. The area of improvement was located in the lower left quadrant which already had some minimal residual vision even before therapy. (c) The group results of a double blind, randomized, placebo-controlled study (unpublished).

227

Power spectra changes after alternating current stimulation (a)

(b)

10 5 0 –5 –10

Pre

*

-15

F7

*

*

-11,7

Fp2

Fp1

Fz

F3

-8,1

9,5

FC5

FC1

*

*

10,5 -15

-10,3

FC6

*

-10,9

C3

Cz

*

-8,6

* T8

-5,8

-6,1

*

-10,9

CP1

CP2

*

11,5

-7,4

*

P3

*

CP6

Post

P7

-10

C4

*

13

F8

*

FC2

* T7

CP5

*

F4

Pz

P4 30,4

P8

*

11,5

O1

0

Alpha power spectra, mV2

*

O2

95 Slow waves activity Alpha activity

Fig. 6. rtACS and EEG power spectra changes. When patients with partial optic nerve lesions are stimulated by noninvasive, alternating currents (rtACS), this leads to lasting EEG power spectra changes. (a) Alpha activation in the brain of a single patient before and 24 h after a 10-day rtACS treatment. Alpha activity was highest in posterior brain region before treatment, where the visual cortex is located. After 10 days, alpha power increased across the brain, extending more anteriorally. (b) Results of a clinical trial shows average EEG changes in a group of optic nerve patients. The bars show the power of alpha and slow wave activity in different brain regions. Stars indicate significant changes. The percentage change were 11% and 30% of alpha power at occipital sites (O1 and O2), after rtACS while alpha activity increased slow waves decreased in different brain regions, where primary visual cortex is located. This was not seen in a sham group. These EEG power changes are indicative of an increased synchronization state of the brain which outlasts the stimulation period.

networks are forced to propagate synchronous firing which, when repeated many times, induces a “learned synchronization response (LSR)” in the damaged pathways. This idea of LSR is compatible with the observation that synchronization can be entrained by external, transcranial pulsed stimulations and such alpha entrainment has already been observed in normal subjects (Zaehle

et al., 2010). As a consequence of this increased synchronization, the injured visual system reacts to the reduced and unchanged input in a more sensitive manner (supersensitivity), similar to what the brain does on its own during the natural or training-induced recovery phase where spontaneous visual phosphenes are seen (Poggel et al., 2007; Tan and Sabel, 2006; Tan et al., 2006).

228

Here, one can think of cortical plasticity as serving the role of an “amplifier” that increases the signal above noise in an area with reduced visual input. There are other studies using noninvasive visual system stimulation, particular with transcranial direct current stimulation (tDCS) protocols: in normal subjects, tDCS induces changes in phosphene thresholds and excitability (plasticity) of the human primary visual cortex (Antal et al., 2003), affecting different visual perception phenomena (Antal et al., 2006, 2008; Chaieb et al., 2008). The only other alternating current stimulation study is one from a Japanese group (Fujikado et al., 2006) who applied transcorneal electrical stimulation (TES) in patients with ischemic optic neuropathy. TES was applied only once for 30 min at 600–800 mA with a frequency of 20 Hz and this led to improvements in VA in six of eight treated patients. Due to the small sample, a definitive conclusion about this approach is still pending. Also, Inomata et al. (2007) studied TES of the retina to treat longstanding retinal artery occlusion. Here, TES (20 Hz biphasic pulses, 30 min, up to 1100 mA) was delivered by a bipolar contact lens electrode once a month for 3 months. VA was found to have improved in two cases, and the visual fields were improved in all three cases. Improvements in the electroretinogram indicate some recovery of function distal to RGCs which may explain the visual field improvements. When viewed together, noninvasive current applications can (i) provoke visual percepts (phosphenes) in visual cortex, (ii) lead to excitability changes in visual cortex and other brain structures, and (iii) improve visual functions after damage to the optic nerve showing some therapeutic efficacy. This is quite similar to what is seen after visual training (Fig. 7). Vision restoration, subjective vision, and activities of daily life Improvement of psychophysical parameters or plasticity of RF changes may be of great interest

to scientists, but unless vision restoration is shown to be clinically relevant, contributing to a higher quality of life, clinicians will not pay attention and patients will not become aware of this new vision restoration option nor use it. Obviously, vision loss and blindness have a much feared negative impact on functional abilities and quality of life. In patients where the visual field loss is caused by cerebral damage, the reduction of quality-of-life domains is mainly due to problems in reading, driving, visual clarity, and peripheral vision (Gall et al., 2009, 2010a,c). The status of vision-related quality of life is somewhat dependent on the size of visual field loss after damage to the post-chiasmatic (Gall et al., 2008; Papageorgiou et al., 2007) or prechiasmatic pathway (Cole et al., 2000). These impairments are typically assessed by perimetry and VA tests, but this type of evaluation may fail to assess certain aspects of visual disability that are identified by visually impaired persons as being important for their daily well-being (see below). The optimal approach to measure quality of life in vision research is therefore to measure both visionrelated and health-related quality of life (Franke and Gall, 2008).

Subjective improvements after visual border training As discussed above, different types of VRTs can improve stimulus detection in patients with postchiasmatic and optic nerve lesions (Julkunen et al., 2003; Kasten et al., 1998a,b; Sabel et al., 2004). Here, about two-third of the patients reported subjective improvements as measured in post-training interviews (Mueller et al., 2003) or by analysis of pre- and post-training drawings of subjective visual field sizes (Poggel et al., 2008). Other studies have developed their own methods and confirmed subjective improvements (Chokron et al., 2008; Julkunen et al., 2003; Sabel et al., 2004). Everyday life activities were also recorded in hemianopic patients by structured

229 Visual field size improvements after optic nerve damage

30

No. of patients

25

Mean change after rtACS (10 days) 39.6%

Mean change after behavioral training (VRT, 6 months) 34.7%

20

15

10

5

0 0 15 30 45 60 75 90

120 150 0 10 20 30 40 50 60 70 80 90 100 Relative improvement of detection accuracy (%)

120

140

Fig. 7. Vision restoration after optic nerve damage: rtACS treatment versus behavioral training. To compare the extent of recovery from optic nerve damage after rtACS with vision restoration training (VRT), we plotted the number of patients versus the different improvement levels (in percent change over baseline). As the figure demonstrates, 10 days of rtACS resulted in similar activation of residual vision (39.6% detection improvement over baseline) as 6 months of visual training (34.7% improvement). The data were taken from different studies (rtACS: Fedorov et al., 2010; VRT: Kasten et al., 1998b) but plotted on the same scale to allow comparison.

post-training interviews in a larger sample (n ¼ 69) (Mueller et al., 2003). Here, the percentage of patients reporting training-induced subjective improvements were as follows: reading (43.5%), ability to avoid collisions (31.9%), general vision improvement (47.8%), ability to perform hobby activities (29%), and confidence in mobility (75.4%). Objective improvements of visual field parameters correlated significantly with the number of named activities of daily living categories, but not all patients who reported subjective improvements also showed objective improvements in perimetry results, that is, there was a certain number of cases with a “mismatch” (see below).

To try getting a better handle on subjective vision, we recently adopted the National Eye Institute-Visual Functioning Questionnaire (NEIVFQ) as a standardized instrument to assess vision-related quality of life and found significant improvements after visual field training in hemianopic patients (Gall et al., 2008) which were also correlated with objective perimetry results. Given that questionnaires are sufficiently sensitive to detect VFI, standardized questionnaires of health- and especially vision-related quality of life should be used on a regular basis in future rehabilitation studies (Bouwmeester et al., 2007). This will enhance our understanding of the clinical relevance of functional improvements

230

and standardized methods can also be more easily compared between laboratories.

Subjective improvements after noninvasive current stimulation One may argue that patients having undergone a long and laborious training for many months may be biased to report subjective visual improvements after such a substantial effort and time commitment. We therefore used a nontraining type therapy, rtACS, which might be less prone to such artifacts. We have measured subjective visual functioning and vision-related quality of life before and after rtACS and assessed self-estimated visual and health-related quality of life (Gall et al., 2011). rtACS led to partial restoration of visual fields which was accompanied by improvements of vision-related quality of life (NEI-VFQ) and health-related quality of life (Short Form Health Survey, SF-36). Some, but not all, NEI-VFQ scales were sensitive to improvements in visual field size after rtACS, and particularly, the subscale “general vision” improved to a clinically relevant extent in the rtACS group. The improvements were dependent on the magnitude of the visual field expansion: rtACS-treated patients with detection improvements > 20% had a significantly greater increase in NEI-VFQ scores than patients with smaller detection improvements (< 20%). Thus, rtACS treatment is capable of modifying the adult visual system in a noninvasive manner and this is of subjective, functional relevance to the patients’ everyday life. It is interesting to note that only some NEIVFQ scales were sensitive to visual field expansions after visual training or rtACS. In any event, vision restoration studies ought to include assessments of vision-related quality of life, a meaningful and valuable complement to objective visual field data that better reflects on the patient's individual self-perceived situation. Because the correlation of both is modest at best,

these assessments represent different aspects of vision. One reason why this relationship between subjective and objective visual measures is only small to moderate is the mismatch problem, an issue that adds complexity to the discussion of vision restoration (see section “The mismatch problem”). In any event, a definitive advantage of using questionnaires such as NEI-VFQ is that they help to weigh the risk (ratio of effort/cost) and benefits of interventions.

The mismatch problem Visual field impairments are typically assessed by perimetry. However, perimetry was not designed to assess vision in everyday life, and the detection of small dots presented on ambient background is not a typical real life event. The visual world is much more complex, comprising different shapes, colors, contours, cluttered scenes, moving objects, etc. In reference to vision restoration, the question is frequently asked how perimetric improvements and everyday vision relate. Also, critics claim that self-perceived training effects may be “only psychological” or “subjective” and therefore “not real.” We have found that there is only a small to moderate overlap of subjective vision and perimetric measures. In many patients, perimetric improvements are associated with subjective improvements, but in other patients, there is seemingly a mismatch: subjective improvements can be reported without visual field expansions and, vice versa, visual field expansions may happen without being subjectively noticed (Mueller et al., 2003; Fig. 4). Also Chokron et al. (2008) described a patient who experienced a progression of subjective improvement after vision training despite lack of improvement in perimetry. Of course, the location and size of the scotoma has a large influence on individual subjective vision and this alone can account for some of the unexplained variance. For example, a gain of visual field size at or near fixation has a much greater subjective impact than peripheral visual field gains (Poggel et al., 2008).

231

But both the low correlations and the mismatch problem raise another possibility: other factors of vision might account for this mismatch: (i) the “intact field” also has subtle deficits in visual cognition (contour integration deficits), (ii) temporal processing (reaction time) is impaired, (iii) spatial resolution (VA) is reduced, and (iv) steady fixation of the eyes and eye-movement control may be impaired, making the perception of stationary or moving objects more demanding (Mueller et al., 2003; Paramei and Sabel, 2008; Schadow et al., 2009). In the context of a discussion on residual vision, the issue of subjective visual improvements is complex because everyday life vision is dependent on different factors: (i) visual field size, (ii) exact location of the field defect (foveal vs. peripheral), (iii) deficits in the “intact” field sector, (iv) temporal processing deficits, (v) decline in spatial resolution, and (vi) variable degrees of residual vision at the border zone or deep in the blind field (with unconscious elements of vision (blindsight). Further, (vii) fixation accuracy and (viii) eye movements are part of the subjective vision equation. Thus, mismatch cases, where subjective vision improves while the visual field size remains unchanged, cannot disprove vision restoration as being “purely psychological.” We rather propose that functions other than those tested with perimetry have improved as well. Indeed, VRT speeds up reaction time (Kasten and Sabel, 1995; Kasten et al., 1998b; Mueller et al., 2003), increases VA (e.g., Kasten et al., 1998b), and improves fixation accuracy (Kasten et al., 1998b). Just as subjective visual impairments are a multifactorial and rather complex affair, so is the subjective improvement associated with vision restoration (Poggel et al., 2008). Alternative explanations of vision restoration The claim that vision restoration is possible at all after lesions in the adult brain is shared by many scientists (see below), but it has also attracted some opposition. Although most critics have not

actually studied vision restoration experimentally, their theoretical arguments are based on circumstantial evidences but nevertheless raised considerable debate. Yet, this discussion is valuable as it directs our attention to possible alternative explanations which need to be carefully considered, particularly related to the following issues.

Vision restoration is just normal learning Some sceptics have argued that vision is just an effect of perceptual learning and that there may be no true restoration of vision. We concur with the argument and believe that perceptual learning actually is an important element when the brain tries to repair the damage. To the best of our knowledge, no author studying vision restoration has claimed that restoration requires pathologyinduced repair mechanism(s). In fact, just as in normal perceptual learning—which requires many repetitions (Fahle and Poggio, 2002) in massed practice sessions (see above)—vision restoration is not easily accomplished either. It also requires many stimulus presentations, in the order of 50,000–100,000, which usually takes months of laborious work. Here, it is not only the intact structure that is involved, but rather perceptual learning takes place within the partially damaged structures (within-systems plasticity) or the remaining (even intact) neuronal networks (network plasticity).

Is restoration just the result of spontaneous recovery? This argument is frequently raised but ignores that all vision stimulation procedures such as training or current stimulation were given to patients with lesions that were many years old (e.g., 6.8 years in Kasten et al., 1998a,b). Because spontaneous recovery is only rarely seen beyond 6 months postlesion (see discussion above), these

232

clinical improvements many years after damage cannot be explained by spontaneous recovery. Functional improvements are caused by attention changes Similar to the “its-just-learning” argument, we agree that there is a special role of attention in vision restoration. Attention is, in fact, a major contributing factor to vision restoration. Both behavioral and brain-imaging evidence exist supporting the special role of attention in restoration. Just as in normal perceptual learning, attention is a necessary requirement for improving visual functions and also for long-term and stable vision restoration after visual system damage. Is vision restoration an artifact of eye movements? This is perhaps the most serious concern when interpreting vision restoration studies. The issue was raised that vision restoration is not due to increased visual detection (perceptual improvement), but that the visual border shift can rather be explained by increased eye movements toward the scotoma after training, leading to an apparent, but not real, shift of the visual field border. In principle, there are different ways how eye movements could influence diagnostic testing and only “mimic” a visual field expansion: (i) the eyes could scan more in both directions, to the right or left side; (ii) the eyes could intermittently and preferentially saccade toward the visual field border, resulting in an artificial shift of the scotoma away from fixation; and (iii) the eye position could permanently shift toward the hemianopic side, which would require the establishment of eccentric fixation which patients with central vision loss, such as AMD patients, regularly do. First of all, there are some logical problems with these possibilities. Let us consider the three cases during the post-therapy perimetric assessment: (i) firstly, if the eyes would scan more in

both directions, the patient would have as many detection gains by looking toward the blind side as detection losses by looking to the other side; (ii) if the patient would intermittently scan toward the hemianopic side only, then the patient would not only have to move the eyes just prior to the short stimulus presentation (which cannot be anticipated). Doing this while having to pay attention to the fixation point would be an extremely hard task. (iii) Stable, eccentric fixation does not occur in hemianopic patients who are able to fixate well (an inclusion criterion in restoration studies); also the blind spot remains in its expected place. Besides these arguments of logic, there are many experimental indications why the “eyemovement artifact hypothesis” is unreasonable. 1. The nature of the visual field border: If the “eye-movement artifact hypothesis” was correct, one would expect that the training-induced visual field border shifts as a whole to one side. However, the border shift dynamics are rather variable: some patients show a shift of the entire border to the hemianopic side, while others show a shift only in one sector of the border (Fig. 8a). Also, in patients with glaucoma, the visual field borders move in a ring-like centrifugal direction toward the periphery (Gudlin et al., 2008) which is incompatible with moving eyes preferentially toward one side. Thus, the local border shift and the centropedal border shift dynamics are incompatible with the eyemovement theory. 2. Blind spot position: If patients would intermittently or continuously move their eyes toward the scotoma, the position of the blind spot would shift, which is not what is seen (Kasten et al., 2006). eye movements: Actual 3. Measuring measurements of the eye positions with an eye tracker before versus after training are the most direct way to clarify the role of eye movements in restoration. We found no increase but rather a decrease of eye movements after VRT (Fig. 8b) (Kasten et al.,

233 (a)

Vision restoration and eye movements

−25 −20 −15 −10 −5

(b)

0

5

10 15 20

20

20

15

15

10

10

5

5

0

0

−5

−5

−10

−10

−15

−15

−20

−20 −5 0

25

5

10 15 20

25

Average visual field border shift (5°)

60 50

%

40

Left

Right

30 Pre Post

20 10 0

>−5° −5° −4° −3°

−2° −1°

1°

2°

3°

4°

5°

>5°

Fig. 8. Vision restoration and eye movements. Eye movements need to be observed in studies of vision restoration. (a) In a patient with a complete upper right and incomplete lower right damage before and after vision restoration training (VRT). Visual field improvement occurred mainly in the lower right quadrant shifting the relative border; but the upper visual field border did not change (the arrows indicate the border position parallel to the vertical midline). If eye movements were responsible for the border shift, then the entire border would be expected to shift position, not just the lower half. (b) Eye movements can also be directly measured before and after VRT. This graph shows the time spent at fixation or to the right or left. A pre- versus posttraining comparison did not find any differences. If anything, fixation improved (see Kasten et al., 2006). Thus, eye movements cannot explain visual field border shifts and therefore vision restoration is real.

2006). There was also no evidence for preferred directions of the eye movements before or after restoration. Moreover, the eye movements were rather small: 95% of the times the eyes were positioned  2 around fixation before training and 99% after training, that is, fixation quality actually increased. 4. Eye movement-adjusted retinal charts: Another approach to determine the role of eye movements is to adjust visual field charts as a function of eye movements. When this is done (Fig. 9), stimulus detections (hits) after training

are observed in areas of the visual field sector that previously had been blind. 5. Fixation performance: If eye movements were more frequent after VRT, then one would expect fixation performance to worsen. Actually, there is no reason to assume that training would induce patients to start moving their eyes around more because the training task requires stable fixation performance. In fact, none of the patients carrying out training developed eccentric fixation (Reinhard et al., 2005) and fixation performance actually improved in all of our prior studies.

234 Eye tracker adjusted visual charts 15

(a)

15

(b)

10

10 5

5 0

80 %

Y

100 % 0

–5

–5 –10

–10

Before

60 %

–15 –20

–15

–10

–5

0

5

10

15

–15

20

40 %

–20 15

15

(c) 10

20 %

–10

0

10

20

0

10

20

(d)

10

5

5

0

After –20

–15

–10

–5

0

5

10

15

Y

0% 0

–5

–5

–10

–10

–15

–15

20

–20

–10

X

160

**

No. of stimuli

140 120

n = 16

100 80

**

60 40 20

Pre Post Missed stimuli

Pre Post Hits

Fig. 9. Eye-tracker adjusted visual charts. Eye movements always occur during perimetric assessment, even when patients are instructed to fixate well. To evaluate the role of eye movements in restoration, an eye-tracker adjusted visual field chart was calculated in which the position of the recording was adjusted to accommodate the position of the eye at the time of stimulus presentation. Upper graph: The left(a, c) shows HRP charts before and after VRT. On the right, the adjusted charts are displayed. Clearly, the improved areas as shown in (c) can also be found when the stimulus positions are adjusted as a function of the eye position at the time of presentation. Numerous stimulus detections (hits) are now seen on the far right after VRT (d) in the previous area of absolute blindness (shown in (b)). Lower panel shows the average results of a group of patients (n ¼ 16). Detection performance was expressed as number of hits (left) and number of misses (right) inside the previously blind field only. Whereas the number of misses significantly decreased, the number of hits significantly increased.

6. Locally induced visual field border shifts: Training residual vision with an attention cue amplifies restoration precisely in the region where the cue was positioned (Fig. 10).

This leads to a selective and regionally restricted visual field border shift which can also not be achieved by eye movements.

235 Vision restoration and attention Acute attention

Chronic attention (training)

20⬚

15 10 5

0⬚

0 –5 -10

-20⬚

-15 -20 -15 -10 -5

0

5

10

15

20 15

100 % 80 % 60 % 40 % 20 % 0% Fixation point

10 5 0 -5 -10 -15 -20 -15 -10 -5

0

5

10 15 20

Fig. 10. Vision restoration and attention. To evaluate the role of attention in vision restoration, hemianopic patients were asked to focus their attention to a cue in the shape of a square (attentional spotlight) that was positioned on the border region. The left panel shows the immediate effects of such local attention on the visual field. When the cue (square) is placed on the upper visual field border, the number of stimulus detections is increased compared to a comparable square shaped region without attention (lower visual field). Thus, attention led to residual functions in the previously blind regions (see Poggel et al., 2006). The panels on the right show the effects of daily attention training over the course of 6 months with a cue positioned also on the visual field border. Vision restoration developed precisely inside the region (square) which was activated by the attentional spot light (Poggel et al., 2004). This shows that attention is a key factor in vision restoration.

7. Visual field improvements were recently confirmed with microperimetry, which allows the exclusion of eye movement artifacts (Marshall et al., 2010). While we acknowledge that the majority of visual stimulus presentations are given when the eye is not exactly at fixation, there is no evidence of a change or induction of preferred saccades toward the scotoma after restoration training. Actually, eye movements are a physiological necessity and they do always occur. Though eye movements cannot explain restoration, they are always a possible source of error (variability) in visual field diagnosis. Monitoring

their influence therefore helps to control and reduce the variability which increases the validity of vision restoration measurements. In summary, eye-movement artifact cannot rule out the available experimental evidence in favor of vision restoration.

Factors influencing vision restoration To understand mechanisms of restoration and to optimize outcome, several potential factors have to be considered: patient demographics, the nature of the disease, and the topography of the specific visual field defects and transfer effects.

236

Patient parameters Age of the patient (at least after early adulthood) has no major influence on vision restoration outcome (Mueller et al., 2007; Zihl and von Cramon, 1979, 1985). The activation of residual vision and plasticity is also not gender dependent (Mueller et al., 2007). It is possible that restoration is greater in children before or at school age (Werth, 2008; Werth and Seelos, 2005), but outcome of children versus adults has never been directly compared.

Lesion parameters Lesion age Our experience is that lesion age (at least at times beyond 6 months) has little, if any, influence on training-induced vision restoration. Although most of the spontaneous visual field recovery occurs early after the lesion, all visual field restoration studies (using training or noninvasive brain stimulation) have treated patients with lesions older than 6 months. Having very old lesions was of no apparent disadvantage for prognosis. Little is known if vision restoration is more effective when applied during the very early spontaneous recovery phase. Our preliminary studies did not find any evidence for this and patients tended to actually do worse if behavioral training started earlier (Mueller et al., 2006), though this needs further study before a conclusion can be reached. Lesion type Vision restoration occurs no matter where the lesion is located along the visual system pathway. Contrary to our intuition, more peripheral (retinal and optic nerve) lesions tend to have greater restoration potential than central lesions of the visual radiation or visual cortex (Kasten et al., 1998a,b). This is surprising but highlights the

special role of visual cortex in post-lesion plasticity. Perhaps “cortical amplification” simply works better when the cortex is not deafferented or damaged directly.

Visual field defect type It does not matter whether the visual field defect is a smaller type of scotoma, a quadrantanopia, a hemianopia, or a peripheral, concentric visual field loss (as in glaucoma). All lesion types may respond to treatment, with no major difference between any of them. The only known parameter that matters for restoration, though, is the size and topography of ARVs (Guenther et al., 2009). This is in line with the argument that vision plasticity is mediated by residual structures. Visual field topography Visual field defects may have areas of absolute blindness or ARVs (relative defects), where patients respond unreliably to visual detection tasks (Fig. 2). The size of these ARVs, that is, the degree of residual vision, is currently the only factor that has a notable influence on outcome. Though large ARVs are no guarantee that functions will improve after therapy, the size of the ARVs is positively correlated with outcome. A detailed analysis of the visual field topography also attests to the special role of this factor (see below). Here, self-organizing map (SOM) chart analyses revealed that the vision restoration hot spots are not randomly distributed but they are a function of the amount of residual activity in the immediate surround (Guenther et al., 2009). For more detail, see legend to Fig. 11. In our experience, 80% of the visual field locations that improve (vision restoration hot spots) are located in the ARVs; only 20% are found deep in the blind field (unpublished observation). Though stimulation of only the field of absolute blindness has been tried by several authors,

237 (a) Dynamic chart construction Vertical visual angle [°]

Baseline chart

Dynamic chart

Posttraining chart

20

20

20

10

10

10

0

0

0

−10

−10

−10

−20 −20

−10

0

10

−20

20

Horiz. visual angle [°]

−20 −20

−10

0

10

20

Horiz. visual angle [°]

−20

−10

0

10

20

Horiz. visual angle [°]

(b) Feature calculation: neighborhood activity

(c) Self-organizing maps (SOM): restoration hot spots 15.2 mm

0 mm

0.6

0

+

+ 0

0.33

0

+ 0

0.8

0

0

+

+ 0

0.8

0

0

+ Hot spots 0 Cold spots

Distance to scotoma

Neighborhood activity

Defectdepth

Quadrantanopia

Hemianopia

Fig. 11. Factors influencing vision restoration hot spots. To study factors (features) of the visual field defect that influence whether a given spot of the visual field can recover, “restoration hot spots” (improved vision) and “cold spots” (no change) were determined as shown in (a): Visual field charts before (left panel) and after vision restoration training (middle panel) are used to calculate the dynamic chart (right panel) which shows the change pre- versus post-VRT (for explanation of charts see Fig. 2). “Hot spots” are indicated by dark square, cold spots by gray squares. (b) One sample feature which was of special interest: “neighborhood activity.” A computer-simulation used data mining method of SOMs to examine for each spot of the baseline visual chart ((a), left panel) a value that represents this feature (in this example of a feature, low values represents little activity and high values much activity in each spots immediate surround/neighborhood). The SOM then calculated to what extent this feature at baseline is able to predict a “hot spot.” This feature represents residual visual activity (indicated by levels of gray) in the immediate neighborhood of a given spot. (c) The results of the SOM-analysis for different such features in 2D SOM charts. For each feature, a separate chart was created which was subdivided in a hot spot (þ) and cold spot region (0), separated from each other by a border line. The gray levels represent how well a given feature (e.g., neighborhood activity) is associated with cold or hot spots. In these SOM charts, white represent tight associations while gray and black represent no association. As the graph shows, the features “neighborhood activity” and “residual activity of the spot itself” (defect depth) are closely associated with the hot spot (þ) region. Other features, such as type of visual field defect (quadrantanopia/hemianopia) or distance to the scotoma are not associated with the occurrence of hot spots. Thus, an SOM analysis revealed that residual activity within a limited surround has a great influence on vision restoration and, in fact, predicted the restoration potential rather well (see Guenther et al., 2009).

238

we question if they tested sufficiently for possible residual vision inside the blind field. Namely, an area that appears completely blind may actually show residual vision when tested with brighter stimuli (Kasten et al., 2008).

Specificity and transfer of training effects When vision restoration is accomplished by training the question arises if training effects are specific or if they transfer to other functions as well. In the normal brain, improvements in perceptual learning tasks are rather specific to the features that were trained (spatial frequency, orientation), to the retinal position, or to the eye which was trained (Gilbert et al., 2001; Karni and Sagi, 1991). However, there are also examples of transfer (Beard et al., 1995), though only “easy-tolearn” tasks seem to be transferable (Gilbert et al., 2001). The information on transfer in clinical cases is still rather ambiguous. In patients with hemianopia, for example, vision training of the border region, that is, in ARVs, a task involving the detection of small dots presented at different brightness levels, also improved color detection (Kasten et al., 2001) and VA (Kasten et al., 1998b; Mueller et al., 2007), both of which were not trained. Also in regained regions of the visual field, there is not only improved detection of simple light stimuli (which were used for training) but also improvement of VA, critical flicker fusion frequency, and color vision (which was not trained) (Bergsma and Van der Wildt, 2008). Thus, there is no clear conclusion as to the transferability of training effects. This may depend on the task and the size and localization of the lesion or other nonspecific factors (such as attention and temporal processing). We need to keep in mind, though, that in contrast to normal subjects, patients with brain damage may have problems with more general cognitive functions such as attention, temporal processing, contour integration, brain synchronization, etc. It is therefore reasonable to assume that a specific

training task may lead to gains in these general factors which, in turn, would benefit other visual tasks as well (such as color recognition). In fact, it is not conceivable that any particular, specific training task is “specific” in the true sense since carrying out a visual task (even if simple) always engages other functions (e.g., visual attention) as well. It is practically impossible to train single features alone (such as a contour without a shape, a shape without attention, etc.). As a consequence, stimulation of more general functions (such as attention or brain electrophysiological synchronization) might be beneficial to a specific task or to a variety of tasks. In this context, the observation is of interest that visual improvements occur also in regions well outside of the trained region itself (unpublished observations). Thus, while the “specificity” issue is difficult to answer at this point, it seems clear that some generalization always occurs because any specific training task has also more generalized, global effects on visual cognition.

Neurobiological mechanisms of vision restoration The “minimal residual structure” hypothesis The minimal residual structure hypothesis (Sabel, 1997) states the following: as long as a small, minimum number of cells survive within the damaged structure, recovery of function is possible. This hypothesis needs to be expanded to include the downstream neuronal networks: it is the total number of fibers of the different pathways surviving the lesion that determines how much information reaches higher cortical regions. For example, after lesions in the retina or optic nerve, no alternative routes exist whereby visual information can travel to the brain. In such peripheral lesions, the functional loss (and the restoration potential) is at least initially a function of the number of surviving cells. After post-chiasmatic lesions, even a complete lesion, for example, of V1, does not lead to a complete functional impairment: information can still travel alternative extrastriate routes to

239

reach higher cortical regions (e.g., V2–V5) via the tectal–pulvinar pathways or direct geniculate–V5 fibers. The term minimal residual structure should therefore include both the local structure and all other alternative or upstream structures that contribute to visual information processing and restoration. Although a certain minimal number of residual cells is critical for vision restoration to occur at all, because of the “network” plasticity the precise number of surviving cells in the lesion site is a rather poor predictor of restoration (Sautter and Sabel, 1993). As Fig. 1 shows, with only about 20% of the RGCs, rats reach up to 80% performance in visual tasks. For a given visual performance, the questions are as follows: (i) How much primary tissue (number of cells and their connections) is left? (ii) Are there other (rerouting) pathways for visual information to reach higher-up brain regions? The smaller the residual capacities of both, the lower the chances for vision restoration. The Pasik and Pasik study (1973) may serve to illustrate this point: 14 macaque monkeys were trained in a two-choice task (light vs. no light) following bilateral occipital cortex removal. Immediately after the injury, the animals were blind, bumping into objects and falling from platforms, though pupillary reactions and eye movements appeared intact. After 3 months of recovery time, the monkeys were again able to carry out brightness discrimination tasks and reach for visual stimuli. Concomitant removal of ventrolateral portions of the temporal lobe, posterior portions of the parahippocampal gyrus, the pulvinar, the superior colliculus, or the medial tectum resulted in only a temporary visual defect from which the animals recovered. But when the lateral pretectal region was injured as well, which caused a bilateral, severe degeneration of the nucleus of the accessory optic tract, the monkeys were no longer able to relearn the brightness discrimination task. Thus, not only does the visual system have capacities to recover from damage, but more generally speaking, just as Lashley (1939) stated, the

extent of visual dysfunction depends on how much of the visual system as a whole is injured (which Lashley termed the “principle of mass action”). While this somewhat holistic interpretation of visual system function seems perhaps a bit simplistic, yet from today's point of view it points to the great potential that just a small amount of residual tissue may have. Apparently, only a surprisingly small number of neurons is required for functional restoration to take place and this is also known in other brain systems. In studies of patients with Parkinson's disease, for example, a loss of dopamine cells greater than 70% has to occur before patients start even noticing symptoms, that is, the brain is able to compensate up to 70% loss rather well. Also, recovery of spinal cord lesions is a rule when lesions are not complete, that is, leaving behind small remnants of residual tissue, in the order of 5–20%, which is similar to the 80% vision recovery in rats with only 20% RGCs (Fig. 1). Given this tremendous dynamics of the brain, it is perhaps not surprising that the number of surviving neurons correlates rather poorly with functional outcome. First, the plasticity of the areas of primary damage introduces variability to behavioral performance, and second, both downstream neuronal nuclei and alternative pathways contribute in a significant way to recovery and reorganization (amplification) of vision. This markedly dilutes the structural–functional correlation and is a source of variability. The good news is that it gives more therapeutic wiggle room.

Within-systems plasticity In order to get a better understanding of the mechanisms of vision restoration, we need to differentiate between plasticity of residual tissue in the damaged structure itself—within-systems plasticity - and the responses of all other brain regions located downstream, the “network plasticity”. Within-systems plasticity relates to changes in the remnants of the damaged structure itself

240

(see Sabel, 1997). This kind of plasticity involves local cellular changes—such as activation (or reactivation) of surviving cells (Prilloff et al., 2007) or enhancement of their synaptic transmission (synaptic plasticity). Within-systems plasticity can best be studied in animal experiments where an incomplete (partial) lesion can be studied at the cellular level, such as in optic nerve preparations. It can also be investigated in areas surrounding the injured visual cortex (Imbrosci et al., 2010; Wood et al., 1974) or spared tissue remnants of injured superior colliculus (Stein and Weinberg, 1978). The key question here is this: “How many neurons need to survive and what changes do they undergo so that restoration of vision becomes possible?” We studied this issue by creating partial crush lesions of the ONC in adult rats. Here, only about 10–20% of RGCs are sufficient for vision recovery to occur (Sautter and Sabel, 1993). That this rather small number of cells sustain function confirms observations by Lashley (1939) who estimated that as little as one-sixtieth of the neocortex is sufficient for visual discrimination. Chow (1968) and Galambos et al. (1967) found 2–3% of the optic tract fibers to the LGN to be sufficient for “normal vision” (which appears to be an overstatement). Chow and Stewart's figure is about 28% (Chow and Stewart, 1972) much larger than the figure given by Hubel and Wiesel (1970) who observed that with as little as 1% of the cells responsive, limited recovery of form deprivation is possible. The finding of RGC hyperactivation after partial optic nerve damage supports the concept of within-systems plasticity: a delayed, moderate calcium hyperactivation of surviving RGCs was associated with greater responsiveness of the cells to visual stimuli (Prilloff et al., 2007).

deafferentation. It includes also other, nondeafferented regions involved in the postlesion response. For example, partial retinal or optic nerve damage will lead to primary deafferentation both in the superior colliculus, the main retinofugal target in the rat and in the LGN of the thalamus, the main retinofugal target in humans. Visual cortex would then be the region of secondary deafferentation. If network plasticity exists, one would predict that damaging all nuclei of the network should reduce the chance of recovery toward zero if no other pathways can drive the function. Indeed, combined, simultaneous lesions of all alternative pathways result in more severe deficits and recovery is precluded. For example, there is less recovery in cats with combined visual cortex and suprasylvian gyrus lesions (Wood et al., 1974), when creating combined lesions of different visual areas simultaneously, there may still be visual sparing (in luminous flux). But when the suprachiasmatic nucleus was also damaged, the last visual structure still available, there was no restoration whatsoever (Pasik and Pasik, 1973). Thus, a loss of all visual structures clearly precludes restoration (which is not at all surprising). Fortunately, in the clinical world, complete visual system lesions are extremely rare (other than complete eye or optic nerve damage). As a consequence, even in patients considered to be “legally blind,” there is almost always some degree of residual vision and therefore some restoration potential. In lesions acquired later in life, complete (and not only apparent) blindness is an extremely rare exception and at least some residual vision is usually present.

Receptive field plasticity in deafferented brain structures Network plasticity Network plasticity refers to all changes in areas not directly affected by the injury but suffering from “primary” and/or “secondary” (functional)

Cell loss in one structure of the brain destroys their projection fibers to remote areas. If these fibers are excitatory, deafferentation depression in remote regions takes place, a phenomenon

241

also known as “diaschisis” (Von Monakow, 1914). If the projection fibers are inhibitory, deafferentation excitation results. In cases of deafferentation depression, spontaneous recovery may be achieved by reactivation of metabolic activity which happens spontaneously during the early recovery period, for example, after optic nerve damage in the LGN and visual cortex (Schmitt and Sabel, 1996a,b, 1998). This reactivation may be mediated either by surviving neurons increasing their strength to above-normal levels (Prilloff et al., 2007) or by excitability changes in the deafferentation zone itself (Giannikopoulos and Eysel, 2006). In the computation model of cortical plasticity, increased neuronal gain in the deafferented zone has been shown to be crucial for consistent experimental RF shifts (Young et al., 2007). Whatever the mechanism, the region of primary deafferentation is key in the restoration equation and reorganization of its neuronal network. The classic example of network plasticity in the visual system is RF reorganization, a field pioneered by Eysel (e.g., Eysel and Grüsser, 1978). He and others showed RF reorganization after retinal lesions with RF shifts up to 5–9 of visual angle and up to 10-fold initial increase of RF size followed by shrinkage to nearly normal levels (Darian-Smith and Gilbert, 1995; Giannikopoulos and Eysel, 2006; Gilbert and Wiesel, 1992; Heinen and Skavenski, 1991; Kaas et al., 1990; Waleszczyk et al., 2003). Also, direct injury to visual cortex produces a local RF reorganization seen as both increases in RF size but also in shift of RF location (Eysel, 1997). RF reorganization also occurs in the surround of the lesion, the “penumbra,” leading to physiological hypoexcitability and more distally, hyperexcitability (Dohle et al., 2009; Eysel, 1997). This RF reorganization is mediated by long-range intracortical horizontal connections which are either activated after deafferentation (Darian-Smith and Gilbert, 1995) and which show axonal sprouting (Darian-Smith and Gilbert, 1994).

Lateral influences of cortical interneurons have also been proposed to underlie the “filling-in phenomenon,” that is, the curious observation that a retinal scotoma is subjectively perceived to be much smaller than expected (Murakami et al., 1997). In fact, the compensation potential of even the normal brain is so great that by filling-in the blind spot escapes conscious detection. In addition, the size of the RF varies considerably in the normal brain, depending on the brain's synchronization state (Wörgötter et al., 1998). Thus, RFs plasticity is dependent on lateral influences from neighboring regions, which can exert either inhibitory or excitatory influences. It is likely that RF plasticity is the mechanism of both normal learning and adaptation of the visual system to damage. Because lateral interactions in visual cortex are involved in perceptual learning (Gilbert, 1998; Gilbert et al., 2001) and RF plasticity (Gilbert and Wiesel, 1992), the possibility exists that lateral influences are also involved in vision restoration following behavioral training or electrical stimulation. Recent findings by Raemaekers et al. (2011) are consistent with this possibility (further discussed below). If the assumption is true that lateral interaction contributes to vision restoration, one would expect that vision restoration does not exceed the boundaries imposed by the lateral extent of these interactions. We have studied this question in visual field charts of hemianopic patients after a repetitive perceptual learning task (training). We reasoned that if RF plasticity is involved in vision restoration similar to that found in cats (Giannikopoulos and Eysel, 2006) or monkeys (Gilbert and Wiesel, 1992), improvements should not be distributed randomly in the visual field. Rather, they should be a function of the distance of the immediate surround and span a finite distance, that is, their influence is spatially limited. Visually driven spike activity recovers within a deafferented region up to 3.5 mm from the scotoma border (Das and Gilbert, 1995;

242

Giannikopoulos and Eysel, 2006). Others report even larger range of RF shifts (5–6 mm; Waleszczyk et al., 2003). Computer simulations and physiological recordings from macaque area MT suggest that dynamic alterations in neural activation alone are sufficient to allow large RF changes (Sober et al., 1997) and perilesion cortical activity is a critical factor in reorganization (Eysel et al., 1999). Thus, restoration of vision in patients (i) may be mediated by areas which are not completely but only partially damaged, (ii) it may be influenced by perilesion activity of the cortical surround, and (iii) if reorganization of RFs is the underlying neuronal substrate, visual field expansions should be spatially limited. Based on these considerations, we have hypothesized that vision restoration is governed by the same rules and principles imposed by the spatial limits of lateral interaction. To test this, we have measured in visual field charts of hemianopic patients the precise topography of changes after stimulation by first creating dynamic visual field charts and then calculating the differences before versus after training. The obtained “dynamic charts” then permitted the identification of areas of the visual field where vision restoration occurred (hot spots) and those where it did not (cold spots). Using “SOM”-data mining tools, we then related the location of the restoration hot and cold spots to certain features in the baseline visual field topography (Guenther et al., 2009). The goal of this approach was to uncover possible rules of RF plasticity and to check the influence of lateral interactions (Fig. 11). Indeed, when the location of the restoration hot spots was compared to the precise topography of baseline charts, we found that vision restoration follows indeed rules of RF plasticity: restoration hot spots were primarily located in areas of the visual field that had either a high level of local residual activity and greater amounts of residual activity in the immediate spatial 5 surround. The level of global activity (lesion size) or other parameters (such a type of visual field defect) were of no influence (Guenther et al., 2009).

This observation is confirmed by recent imaging studies by Raemaekers et al. (2011). They found direct evidence for RF plasticity: in hemianopic patients participating in VRT RF changes in visual cortex could be imaged. The findings are thus compatible with our own observations of a special role of lateral interactions in vision restoration. The authors concluded that small visual field enlargements (such as those at the border region of the visual field) could be explained by this more “local” RF plasticity, but that massive visual field expansions, which are sometimes observed in patients, cannot be explained by this mechanism. Yet, one important question remains: Is RF plasticity good or a bad? Whether RF reorganization (RF location shift or enlargement) is functionally adaptive or maladaptive is not yet clear. Enlarged RFs might facilitate detection, but at the same time, they might reduce the ability to see in ambient light or detect objects at higher resolution or more complex objects. Likewise, a shift of the RFs location might be helpful to engage deafferented regions of the brain to participate in visual processing, but if this is helpful at all or instead leads to scrambling or noise in regions adjacent to the lesion remains to be determined. There is one phenomenon that nicely illustrates the ambiguous role of RF plasticity. Dilks et al. (2007) described a patient with a left upper quadrantanopia carrying out detection tasks of different shapes (squares, circles, triangles). When presented near the lesion in the lower left quadrant, the subject perceived objects as vertically elongated, extending toward and into the damaged area. The lesion affected “visionfor-perception” tasks as well as visually guided motor response (vision-for-action). fMRI measurements confirmed the hypothesis that the deprived cortex became responsive to nearby (intact) regions in a retinocentric manner, an issue also related to the filling-in effect. One may argue that visual distortions might be maladaptive from the point of view of “what is it?,”

243

but they might be adaptive from the point of view “is there something?” Currently, we cannot tell if cortical reorganization after lesions is a good or bad thing (enhancing perception or distorting is). Clearly, this issue is a critical one in need of further study.

The role of downstream networks After considering plasticity of the damaged structure itself and the primary deafferented structure, let us now turn to neuronal networks beyond the deafferented region. It is reasonable to assume that reorganization does not stop at the area of primary deafferentation. It also leads to changes in secondary brain structures. In addition, as we know from post-chiasmatic damage, information flow can bypass the lesion site, using a detour of alternative routes to higher cortical regions (see the discussion on blindsight above), leading to a kind of remote neuronal network response.

The excitation–inhibition balance From the network point of view, reestablishing homeostasis, an evolutionary principle, is the key goal, that is, the proper balancing of excitation and inhibition. This issue receives sparse attention in the vision restoration literature. Let us consider, for example, the case of an incomplete hemianopia caused by an incomplete V1 lesion. Here, we have a loss of lateral interactions (presumably horizontal cells) which impairs local information. Second, there is the loss of longrange interhemispheric fibers that terminate in the mirror-symmetric position of the opposite, intact hemisphere, particularly in the region that corresponds to the vertical midline. Because interhemispheric fibers are believed to be inhibitory (Sprague, 1966), their loss would result in a hyperexcitation of the intact hemisphere with a secondary inhibitory ripple effect by the

reciprocal interhemispheric, inhibitory fibers originating from the intact side and terminating on the damaged side. The final outcome would be a disaster for the damaged hemisphere: additional inhibition of all those regions that were partially damaged (ARVs). ARVs are probably the greatest victims of excitation–inhibition dysbalance. In the RF microenvironment, the consequences are several-fold. We expect local inhibitory and over-excitatory changes in the immediate surround of the lesion plus a functionally hyperactive state of the intact hemisphere with a secondary overinhibition of the reciprocal interhemispheric inhibitory back-projections. In this scenario, any cells that managed to survive inside the blind or partially blind field would be inhibited from the opposite (intact) hemisphere (ARV suppression) and this happens irrespective of whether they are located in the ARVs near the visual border or in any islands of residual vision. The net outcome of all of this would be as follows: the intact hemisphere ends up with subtle deficits in vision, possibly by being hyperexcitable, and additionally, residual tissue on the lesion side is suppressed by overinhibition. If this theory of imbalance is correct, restoration would be a rebalancing act: (i) inhibiting the overexcited intact tissue and/or (ii) increasing excitability of the residual tissue in the region of the lesion. There are several lines of evidence that network balance is critical for proper vision. There are (i) subtle deficits in the intact hemisphere and (ii) visual hallucinations found during spontaneous and training-induced visual recovery which have been interpreted as signs of interhemispheric dysbalance. In addition, rebalancing can restore vision: (iii) restoration of vision can be achieved by additional lesions in the hemisphere contralateral to the lesion (also called the “Sprague effect”), (iv) cortical reorganization as demonstrated by imaging studies, and (v) the restoration effects of functional silencing of the intact side while stimulating ARVs (as in VRT).

244

Subtle deficits in the intact hemisphere The presumably intact visual field in hemianopes is actually impaired. It has difficulties to detect incomplete figures embedded in a noisy background (Paramei and Sabel, 2008; Schadow et al., 2009). For example, three hemianopia patients had to detect with their intact side of the visual field a figure (square) composed of interrupted contours created by Gabor patches embedded in a random patch array (Paramei and Sabel, 2008). Two of the patients had marked deficits in the response accuracy and reaction times and also showed “figure confabulations.” This can be explained by impaired top-down influences from higher visual centers and/or loss of proper interhemispheric balance, both of which impair the function of the intact hemisphere. This interpretation was confirmed by Gamma response analyses of EEG recordings (Schadow et al., 2009). Interestingly, Corbetta et al. (2005) found in patients with attentional dysfunctions after parietal lesions fMRI evidence of hyperactivation in contralateral, intact cortical regions. As hyperactivation declined, attentional dysfunction recovered. Thus, reduced transcallosal inhibitory interaction (directly or indirectly) may reinstate interhemispheric balance after lesions and interhemispheric modulation may improve perceptual functioning and recovery. Hyperactivations as a result of the loss of inhibitory fibers in deafferented brain regions may also explain the figure confabulation in parietal patients (a kind of “reverse diaschisis”) (Paramei and Sabel, 2008): In a way, the patient's “expectation” of a visual stimulus (such as a square) “outcompeted” the evidence from the sensory input, or, in Corbetta's terms, there was a top-down bias along with a decreased stimulus-driven capture. Visual hallucinations during recovery of vision The notion of top-down hyperactivation is also in agreement with observations in hemianopic patients that report simple phosphene perceptions (hallucinations) during the time of spontaneous recovery and during

training-induced visual field expansions. Kölmel (1985, 1993) was the first to propose that visual hallucinations in partially blind patients are a positive sign of neural plasticity and recovery of function. But direct proof of a link between hallucinations and recovery of visual functions was first shown by Poggel et al. (2007). They observed hallucinations in hemianopic patients during the days and weeks of early spontaneous recovery and also again when visual field recovery was induced by training. Here, simple and complex hallucinations were associated in time and space with increased visual field size and recovery: (i) hallucinations were more frequently in patients who benefited from training, (ii) they were typically located in ARVs, and (iii) hallucinations coincided in time with the period of greatest visual field expansion. It should be mentioned in passing that patients are usually aware that these “hallucinations” are not real. But the patients usually do not talk about it because they are worried that it is seen by others as signs of a psychiatric disorder (which it is not). Restoration of vision by additional lesions Because the intact hemisphere has an inhibitory effect on contralateral, cortical, and subcortical areas (such as the superior colliculus), cross-hemispheric inhibition may contribute to dysfunction. Consequently, lifting this inhibition may restore functions. This was first demonstrated by Sprague (1966) in cats where additional lesions in the intact hemisphere restored some of the lost functions induced by a tectal lesion (sometimes referred to as the Sprague effect). Here, a unilateral lesion of the superior colliculus led to orienting deficits which were counteracted by contralateral visual cortex damage. Perhaps the most impressive proof of interhemispheric interactions after brain lesions in humans was published by Pöppel and Richards (1974). They described a hemianopic patient who also had a small lesion in the contralateral intact field. Here, an “island of vision” of vision was seen in the otherwise absolute blind

245

hemifield. Because this island of vision was mirror symmetric to the island of blindness, this effect was interpreted as an example of a lifting deafferentation effect. Cortical reorganization as demonstrated by imaging studies fMRI studies have revealed the first evidence of visual system reorganization in humans. Whereas in healthy subjects, brain activation is found particularly in contralateral V1 (area 17), in patients with post-geniculate lesions, in contrast, activation changes are found bilaterally in the extrastriate areas with a stronger activation on the intact (contralesional) hemisphere (areas 18 and 19) (Nelles et al., 2002, 2007). Brodtmann et al. (2009) showed that bilateral striate and ventral extrastriate activation was reduced in stroke patients, while activation increased in dorsal sites, indicating a greater utilization of the dorsal visual system. These findings are in agreement with the interhemispheric imbalance hypothesis. Cortical reorganization was also reported in patients suffering from macular degeneration who develop a preferred retinal locus (PRL) (see discussion above). Imaging studies showed that the PRL has a larger cortical representation than other retinal regions of the same eccentricity (Liu et al., 2010) and isoeccentric peripheral locations are represented in the formerly foveal cortex (Dilks et al., 2009). Thus, there are both “active” mechanisms of reorganization which are use dependent and passive ones which are use independent. When stimulating the brain by behavioral training, activation changes in fMRI are observed. After eye-movement training, for example, there are changes in the unaffected extrastriate cortex (Nelles et al., 2010). Also when patients carry out VRT, activations are found in the anterior cingulate and dorsolateral frontal cortex together with other higher order visual areas in the occipitotemporal and middle temporal regions (Marshall et al., 2008). Along similar lines, Henriksson et al. (2007) trained hemianopic

patients using flicker stimulation which caused an ipsilateral representation of the trained visual hemifield in different cortical areas, including the primary visual cortex. Similar findings were reported by Raninen et al. (2007). Retinotopic mapping was used by Ho et al. (2009) to show residual visual function in a patient with complete homonymous hemianopia. Retinotopic representation was found in the surviving visual cortex around the infarcted area and stimulating the blind field led to a response in extrastriate areas above the calcarine sulcus. So far, fMRI imaging results are compatible with the concept of large scale visual cortex reorganization (network plasticity). Unfortunately, in many studies, the patient numbers have been too small (often single cases) to reach final conclusions on the generality and reliability of cortical reorganization by brain imaging (however, see Raemaekers et al., 2011). More information is now required using larger patient samples in combination with sophisticated behavioral paradigms. Functional silencing of the intact hemisphere The easiest way to achieve functional balance is to silence the undamaged hemisphere by simply exposing the subject to darkness. In this manner, both hemispheres are functionally inactivated, processing less visual information. In this situation, a functional “advantage” can be created for ARVs by stimulating them selectively while the intact field remains in the dark. This is what the classic VRT does: here patients train in the dark which has two simultaneous effects: functional silencing of intact regions while being able to activate ARVs (see discussion of the restoration potential of VRT above). In a way, VRT creates a double-punch situation: on one hand, a reduced activation by darkness of the intact hemisphere with the consequence of reduced interhemispheric inhibition in the lesioned hemisphere, and on the other hand, the simultaneous activation by visual stimulation of the previously inhibited ARVs inside or near the lesion.

246

The net result is a functional rebalancing which, if practiced regularly, is stabilized. Thus, the imbalance between excitatory and inhibitory neural influences aggravates the visual loss after brain lesions. As the brain can be induced to reach a more homeostatic, balanced state (by training or brain current stimulation), partial vision restoration is achieved. If homeostatic balance is the key in vision restoration, then it should not matter if inhibition is reinstated in regions that are overexcited or excitation is reinstated in areas suffering deafferentation depression (diaschisis). Preferably both can be used to reestablish the balance at different levels of the nervous system: at local, lateral interactions or at long-range intra- and interhemispheric (transcallosal and subcortical) projections. Both “within-systems plasticity” and “network reorganization” are part of the post-lesion response in the brain. They act in concert to optimize residual vision and restoration, but there may also be some maladaptive elements to reorganization which need to be explored. Because lesions vary greatly from minor loss to complete blindness, the extent of restoration is variable as well. But it is the sum of local (within-systems plasticity) and global (network plasticity) influences that will determine the final extent of recovery of perception.

Cellular mechanisms of vision restoration and plasticity How can such within systems and network plasticity be explained on a cellular level? We believe that the cellular mechanism of vision restoration involves the strengthening of surviving neurons in the damaged system itself and/or reorganization of higher-up (intact) neuronal networks. This issue was already discussed above. Building on the assumption that a stable within-systems and network plasticity change requires stable changes at the synaptic level, the questions arise how synaptic plasticity can be achieved by surviving cells.

We would like to propose that vision restoration rests upon cellular and molecular mechanisms of normal learning. We believe that just as in the normal brain, repetitively activating surviving (residual) cells lead to synaptic plasticity, and this is relevant for both for surviving cells of the damaged structure itself and for cells in upstream networks (network plasticity). Interestingly, it seems less critical as to which precise method of stimulation is used to achieve reactivation of residual structures: (1) regular training where patients (or the animals) have to respond to many thousands of visual stimuli or (2) by noninvasive brain current stimulation protocols which are fairly nonspecific (see above). At a cellular level, learning was studied in rodents where the concept of LTP was established (Bliss and Lomo, 1973). LTP is defined as a longlasting enhancement in the cell response to highfrequency stimulation (Fig. 12). LTP maintenance is mediated by both an increased transmitter release per presynaptic impulse and an increased postsynaptic responsiveness to a fixed amount of transmitter (Voronin et al., 1995). LTP has been induced also in the human visual system by noninvasive “photic tetanus” (Sale et al., 2010). Interestingly, LTP as a model of learning and memory has been used already to investigate post-lesional plasticity and, of most relevance here, in residual structures in the vicinity of a lesion (Dohle et al., 2009; Huemmeke et al., 2004). Electrophysiological recordings of ex vivo/in vitro preparations of the post-lesional visual cortex revealed that LTP is enhanced while LTD (long-term depression) is impaired (Imbrosci et al., 2010). This “metaplasticity” may provide the physiological/ molecular basis of the rewiring of synaptic connections and restoration of visual function. Postsynaptic NMDA receptors play a special role in LTP in the lesion surround, and this is compatible with the hypothesis that LTP in horizontal connections in visual cortex might comprise the cellular mechanism of vision restoration (Imbrosci et al., 2010). However, although these results suggest that post-lesion neuronal plasticity is possible,

247 Synaptic plasticity after partial brain injury Residual neurons

Presynaptic

Postsynaptic

Resting state

Stimulated Fig. 12. Synaptic plasticity after partial brain injury. The hypothesis of within-systems plasticity proposes that synaptic plasticity contributes to restoration of vision. It assumes that in a partially injured area of the brain, the physiological activity (sum of all action potentials) produced by surviving neurons is below normal values, insufficient to drive the postsynaptic neuron (partially deafferented structure) at full throttle (upper panel). By stimulating the presynaptic neurons of the partially damaged region by training or electrical stimulation, the silenced activation state (middle panel) changes to greater activation. Repeated activation then elevates cell activity (number of action potentials) above-normal levels, strengthening synaptic efficacy (lower panel). This, in turn, leads to induction of long-term synaptic plasticity which outlasts the stimulation period. On a molecular level of analysis, the process of synaptic plasticity is achieved by the release of trophic factors from postsynaptic cells (adapted from Kolarow et al., 2007)

one has to keep in mind that there is still a significant loss of function in neuronal populations in the vicinity of cell death/damage (Aoyagi et al., 1998; Henrich-Noack et al., 2005). Also, as following traumatic optic nerve damage, there are molecular changes in the surviving cells such as alterations in the splicing variance of different NMDA receptors (Kreutz et al., 1998). It is not clear if these alterations are adaptive or maladaptive. In any event, considering the evidence for both, metaplasticity and silencing of neurons,

post-lesion plasticity may be induced by overcoming injury-related blockades. Interestingly, at a cellular level, this is possible by mechanisms of learning: high-frequency stimulation which under physiological conditions induces LTP is also able to restore lost functions in silent neuronal populations after brain injury (Henrich-Noack et al., 2005). However, restoration of function by learning is not possible at very early post-lesion times. It can therefore be hypothesized that neurons need some time after an impact to

248

recover and adapt their molecular/morphological integrity. After this delay, changes in the micromilieu or inhibitory feedback loops prevent the post-lesion neuronal plasticity as ex vivo investigation of silenced neuronal population shows normal function and plasticity. This restoration of plasticity and function also depends on changes in postsynaptic sensitivity (HenrichNoack et al., 2005), and this is compatible with the hypothesis that hallucinations reported by visually impaired patients are a sign of post-lesion denervation supersensitivity. LTP or LTD, supported by the release of trophic factors, may explain the strengthening of synaptic transmission (plasticity), possibly also involving axon terminal sprouting, but single cells alone do not explain the reaction of the entire residual network. Rather, network reactions as a whole generate function and alter RF plasticity. LTP/LTD may thus provide the cellular condition for an overall change at the network level.

regions fire in a synchronized manner to drive normal vision (jointly firing action potentials and oscillating network in perfect temporal coordination), areas of partial damage are initially nonsynchronized, with poor firing synchrony. After external stimulation (induced by training or during electric current stimulation), the partially damaged regions are forced to fire jointly in temporal coordination. We hypothesize that such a repeated stimulation induces a “forced synchronized firing” which then leads to synaptic plasticity of the partially damaged structures and downstream areas. By doing this repeatedly, LTP-like mechanisms lead to stabilized synchronous firing in the network which lasts beyond the treatment period (aftereffects). This improved or stabilized synchronization is a key mechanism of the proposed neurophysiological mechanism of vision restoration.

Vision restoration and attention Vision restoration and neuronal synchronization When a visual stimulus hits the retina, retinal cells fire together in a timely synchronized fashion and information travels to higher brain centers. Under normal conditions, this synchronization works perfectly, evoking many secondary ripple effects in the brain (such as oscillations) which jointly create the percept. However, when cells are lost and primary and secondary disorganization of neuronal networks happens, one would expect a loss of synchrony, that is, a worst coordination of timed events. A slowing of mental processing would be expected which is what we see in patients who show reduced reaction times and feel uncomfortable observing the fast moving world (navigating in a busy crowd or driving a car). Thus, to restore vision requires a better neuronal synchronization. Figure 2 shows the concept of “stimulationinduced synchronization” after partial nervous system damage. While neurons in intact brain

It is well known that neural activation enhances visuospatial attention. Behavioral, neurophysiological, and imaging experiments show that focusing attention to a specific part of the visual field benefits visual processing in that area, for example, reaction times are reduced, and stimuli are detected or discriminated more easily then when attention is distributed more diffusely across the visual field or focused elsewhere (Eriksen and Rohrbaugh, 1970; Nakayama and Mackeben, 1989; Posner, 1980; Treisman and Gelade, 1980). This can be explained by increased neuronal activation (synchronization) in circumscribed regions of the visual cortex as shown in single-cell recordings in animals (Gilbert, 1998; Ito and Gilbert, 1999), electrophysiological experiments in humans (Mangun and Hillyard, 1987) and in brain-imaging studies (Martinez et al., 1999; Somers et al., 1999). In the attention spotlight, the signal-to-noise ratio increases, resulting in improved performance of the normal brain. The benefit of attention is particularly obvious under difficult perceptual conditions with low

249

signal-to-noise ratio. Attentional load modulates not only responses of invisible stimuli in human primary visual cortex, but it also improves normal vision at low contrast viewing conditions (Bahrami et al., 2007). Such an attentional advantage is also found in patients with visual field defects. Here, visuospatial cues can acutely improve detection performance: when patients are asked to focus their attentional spotlight at the visual field border, this immediately enhances perceptual performance within a few hundred milliseconds after stimulus presentation precisely in the region of the cue (Poggel et al., 2006; see Fig. 10). This demonstrates that residual vision can be immediately accessed by activating attentional resources. In this context, a rather curious observation by Schendel and Robertson (2004) might be of interest. They reported that visual detection can be (acutely) increased in hemianopic patients by placing their arm near the visual stimuli when this was located in the blind hemifield. This arm placement might have simply increased the attention to the stimulus location, elevating its excitability. Another example is a patient with near-blindness that one of us (B. A. S.) studied in Wisconsin/USA in 2000. When asked to describe what he sees he said: “just darkness, nothing else.” When being confronted with a sudden noise (B. A. S. clapped his hands unexpectedly near the patients ears, a rather startling sound), the patient suddenly said he could see a person (the attending physician) standing in front of him, stating with joy: “I can see the doc, he is wearing a red tie” (which he actually did). This is a dramatic example of how temporary vision improvement that can be achieved by raising the patients level of alterness or attention. Directing the attentional spotlight repetitively onto the ARVs in a repetitive practice tasks (training) leads to permanent improvement of vision in patients suffering from visual field defects. Poggel et al. (2004) combined standard VRT in hemianopics with an attentional cueing

task focusing attention to ARVs and found this to enhance the restoration level of VRT (Fig. 10). Also, the Jung et al. (2008) study, where training of the intact region of the visual field in patients with anterior ischemic optic neuropathy improved function, was taken to conclude that this “may reflect diffusely increased visual attention (neuronal activation), or improvement of an underlying subclinical abnormality in the seeing visual field” (p. 145). The Poggel study supports the hypothesis that both ARVs normally receive insufficient attentional resources; the intact visual field sector simply captures all of the attention in everyday life, at the expense of the partially damaged areas by inhibiting them (excitatory/inhibitor dysbalance). In summary, attention plays a key role in the plasticity of partially damaged areas of V1 at a network level, even in the presumably intact visual field sector.

The residual vision activation theory Only partially damaged brain systems have a potential for restoration of vision. Clearly, if there is no structure (e.g., complete eye damage), there is no chance for recovery. Rather, recovery or restoration of function requires some minimal amount of tissue that is (or becomes) dedicated to this task. One of us has earlier postulated the “hypothesis of minimal residual structures” (Sabel, 1997). It is remarkable how much a relatively small number of cells can accomplish. Rats with mild optic nerve injury with only 10–20% of the RGCs survival (Sautter and Sabel, 1993) recovered their ability to perform visual tasks again in about 2–3 weeks. The recovery was not complete and also not all animals recovered to the same extent. Yet, visual performance improved from complete visual dysfunction to about 70–80% performance. This rather remarkable and unexpected observation suggests that a small percentage of neurons and their intact axons are sufficient to allow considerable recovery.

250

The behavioral tasks we used to test our rats employed rather simple brightness discrimination or pattern discrimination tasks. We do not know if perhaps some more complex functions such as ambient stimulus perception, fast or complex tasks might remain deficient. Nevertheless, these findings clearly attest to a considerable postlesion plasticity potential after partial visual system damage which opens new possibilities to take advantage of this restoration potential by developing new therapeuties. Our own research and that of others have confirmed the restoration potential of residual vision. There are many publications on the subject of vision restoration and plasticity by now from different fields of study (Table 1). With this review, we have attempted to arrange these many puzzle pieces to a coherent picture. Based on several decades of research by us and others since the 1970s, we now propose the residual vision activation theory as follows: Damage to visual structures is usually not complete but some structures survive the damage. Together with structures of the intact hemisphere, they provide residual capacities to support vision restoration. Residual structures include (i) partially damaged tissue that sustains “areas of residual vision” (ARV) at the visual field border, (ii) “islands of residual vision” inside the blind field, (iii) alternate visual pathways unaffected by the damage (sustaining “blindsight”), and (iv) down-stream, higher-level neuronal networks. Because patients with retina or brain damage tend to focus their attention on the “intact” visual field sectors in everyday life, a result of a hyperactivation of the intact hemisphere, residual structures lack sufficient attentional resources, reducing their activation state and impairing physiological activation and synchronization. Residual structures thus suffer a triple handicap: (i) they have fewer neurons, (ii) they are disturbed in their excitation/inhibition balance and temporal processing, and (iii) they lack sufficient attentional activation. ARVs

are therefore down-regulated, unable to contribute much to every-day vision. “Non-use” then impairs their synaptic strength even further. Residual structures can be (re-)activated/restored by engaging them in repetitive activation and stimulation. This repetitive activation of residual vision can be achieved by different means such as (i) visual experience, (ii) visual training, or (iii) noninvasive electrical current brain stimulation. This may lead to reorganization by the strengthening of synaptic transmission of the partially damaged structures themselves (“withinsystems plasticity”) and of downstream neuronal networks (“network plasticity”) in cortical or subcortical areas of the damaged and the intact hemisphere. This leads to improved synchronization of neuronal firing in the brain network. Cellular mechanisms of vision restoration are similar to, if not identical with those involved in normal perceptual learning (such as long-term potentiation) which is why long-lasting reorganization and re-synchronization of synaptic plasticity can sustain long term activation of residual structures. Vision restoration should therefore not be regarded as a pathology-specific phenomenon but an expression of normal learning which explains that it can be induced at any time after the lesion, at all ages and in most, if not all, visual field impairments (scotoma, tunnel vision, hemianopia, acuity loss), irrespective of their etiology (e.g. stroke, neurotrauma, glaucoma, amblyopia, AMD). However, vision restoration is rarely complete and does not take place in all patients. If and to what extent restoration can be achieved is a function of the precise nature and extent of residual structures and their activation state. In addition, the extent of restoration depends on the proper activation methodology and appropriate parameters, and it requires the allocation of sufficient attentional resources directed toward the residual structures. Thus, the more ARV are available, the greater is the restoration potential. Whereas the acute activation of residual vision leads to only temporary functional

251 improvements, permanent improvements require repetitive stimulation for many days, weeks or months (depending on the stimulation method). By becoming again engaged in every day vision, (re-) activation and synchronization of ARV outlasts the stimulation period, leading to long-term improvements in vision and quality of life.

Considering the large body of evidence, we now have many reasons to be more optimistic about the fate of partial blindness. The visual system has an excellent potential for plasticity and self-repair, much more than previously thought which is a paradigm shift. If in doubt, consider the following quote by perhaps the most prominent visual system scientist, Torsten N. Wiesel, who received the Nobel Prize for his work on visual system specificity and RF organization. In a lecture held at the symposium “Restoration of vision after brain damage” during the “VISION 2005” meeting (organized by B. A. S. and T. Wiesel; Royal National Institute of the Blind, London) he emphasized the value of vision restoration research: Restoration of vision after damage is an issue I am very interested in and I think that there is progress; to find different means of restoring visual functions is very interesting and encouraging. . . (My experiments on receptive field enlargements) are hard evidence that it is possible to restore (visual) function through time. In this case we did not make any special effort by stimulating the eyes,. . . trying to restore visual functions. . . but this kind of experiment gives you hope that there is more to learn from this kind of experiments and also from the clinical work that it should be possible to have patients restore vision in spite of initially apparent lack of vision. . .

Time will tell if the proposed residual vision activation theory is a paradigm shift (Kuhn, 1962) in the fields of low vision, neuro-ophthalmology, and restorative neurology. In any event,

we hope the theory will stimulate others to get engaged in further discussion and experimental verification. We do not expect that each individual aspect of the proposed theory will hold forever. But it is a start to better understand the complex mechanisms of how the brain may overcome visual impairments. For sure, new aspects will have to be added to the proposed theory. But nevertheless, it provides a heuristic basis for further studies in the field of vision restoration, a rather complex issue in restorative neuroscience. Hopefully, the theory will inspire others to carry out new experiments and develop new treatment options. Perhaps the current theory will be modified or extended at some point. In this manner, vision restoration may mature to become a more widely accepted subject. The theory should lead our way to go beyond the widely accepted notion that (partial) blindness after retinal and cerebral damage is forever and unchangeable. Rather than turning a “blind eye” on vision restoration as a real possibility, we shall recognize that the theory is a basis for a more hopeful attitude: that vision restoration is possible and that new and innovative solutions may be found that reduce the impact of visual impairments. Future research and development will help improve visual impairments, extending far beyond the conceptual borders that currently limit our view. We are at the dawn of a better medical care for patients that greatly suffer from partial blindness which is inflicted by retinal and cerebral visual injury. Acknowledgments We thank Steffi Matzke and Sylvia Prilloff for their excellent help preparing the chapter, and special thanks to W. Waleszczyk (Nencki Institute of Experimental Biology, Warsaw, Poland) for insightful comments on a previous version of the chapter.

252

References Aiello, L. P. (2008). Targeting intraocular neovascularization and edema—One drop at a time. The New England Journal of Medicine, 359, 967–969. Antal, A., Artl, S., Nitsche, M. A., Chadaide, Z., & Paulus, W. (2006). Higher variability of phosphene thresholds in migraineurs than in controls: A consecutive transcranial magnetic stimulation study. Cephalalgia, 26, 865–870. Antal, A., Boros, K., Poreisz, C., Chaieb, L., Terney, D., & Paulus, W. (2008). Comparatively weak after-effects of transcranial alternating current stimulation (tACS) on cortical excitability in humans. Brain Stimulation, 1, 97–105. Antal, A., Kincses, T. Z., Nitsche, M. A., & Paulus, W. (2003). Modulation of moving phosphene thresholds by transcranial direct current stimulation of V1 in human. Neuropsychologia, 41, 1802–1807. Aoyagi, A., Saito, H., Abe, K., & Nishiyama, N. (1998). Early impairment and late recovery of synaptic transmission in the rat dentate gyrus following transient forebrain ischemia in vivo. Brain Research, 799, 130–137. Azzopardi, P., & Cowey, A. (2001). Motion discrimination in cortically blind patients. Brain, 24, 30–46. Bahrami, B., Lavie, N., & Rees, G. (2007). Attentional load modulates responses of human primary visual cortex to invisible stimuli. Current Biology, 17, 509–513. Bak, M., Girvin, J. P., Hambrecht, F. T., Kufta, C. V., Loeb, G. E., & Schmidt, E. M. (1990). Visual sensations produced by intracortical microstimulation of the human occipital cortex. Medical & Biological Engineering & Computing, 28, 257–259. Baker, C. I., Dilks, D. D., Peli, E., & Kanwisher, N. G. (2008). Reorganization of visual processing in macular degeneration: Replication and clues about the role of foveal loss. Vision Research, 48, 1910–1919. Baker, C. I., Peli, E., Knouf, N., & Kanwisher, N. G. (2005). Reorganization of visual processing in macular degeneration. The Journal of Neuroscience, 25, 614–618. Balliett, R., Blood, K. M., & Bach-y-Rita, P. (1985). Visual field rehabilitation in the cortically blind? Journal of Neurology, Neurosurgery and Psychiatry, 48, 1113–1124. Barbur, J. L., Watson, J. D. G., Frackowiak, R. S. J., & Zeki, S. (1993). Conscious visual perception without V1. Brain, 116, 1293–1302. Bartley, S. H., & Ball, R. J. (1969). Effects of intermittent photic input on brightness, hue, saturation, visual acuity, and stereopsis. American Journal of Optometry and Archives of American Academy of Optometry, 46, 315–318. Baumann, T. P., & Spear, P. D. (1977). Role of supersylvian visual area in behavioral recovery from effects of visual cortex damage in cats. Brain Research, 138, 445–468. Baumgarten, P. (1878). Hemiopie nach Erkrankung der occipitalen Hirnrinde. Centralblatt für die Medicinischen Wissenschaften, 16, 369–371.

Beard, B. L., Levi, D. M., & Reich, L. N. (1995). Perceptual learning in parafoveal vision. Vision Research, 27, 953–965. Bechtereva, N. P., Bondartchuk, A. N., Gretchin, V. B., Iliukhina, V. A., Kambarova, D. K., Matveev, Y. K., et al. (1972). Structural–functional organization of the human brain and the pathophysiology of the Parkinsonian type hyperkineses. Confinia Neurologica, 34, 14–17. Bechtereva, N. P., Shandurina, A. N., Khilko, V. A., Lyskov, E. B., Matveev, Y. K., Panin, A. V., et al. (1985). Clinical and physiological basis for a new method underlying rehabilitation of the damaged visual nerve function by direct electric stimulation. International Journal of Psychophysiology, 2, 257–272. Benowitz, L. I., & Yin, Y. (2007). Combinatorial treatments for promoting axon regeneration in the CNS: Strategies for overcoming inhibitory signals and activating neurons’ intrinsic growth state. Developmental Neurobiology, 67, 1148–1165. Benowitz, L., & Yin, Y. (2008). Rewiring the injured CNS: Lessons from the optic nerve. Experimental Neurology, 209, 389–398. Benson, P. J., Guo, K., & Blakemore, C. (1998). Direction discrimination of moving gratings and plaids and coherence in dot displays without primary visual cortex (V1). The European Journal of Neuroscience, 10, 3767–3772. Bergsma, D. P., & Van der Wildt, G. J. (2008). Properties of the regained visual field after visual detection training of hemianopsia patients. Restorative Neurology and Neuroscience, 26, 365–375. Bergsma, D. P., & van der Wildt, G. (2010). Visual training of cerebral blindness patients gradually enlarges the visual field. The British Journal of Ophthalmology, 94, 88–96. Bliss, T. V., & Lomo, T. (1973). Long-lasting potentiation of synaptic transmission in the dentate area of the anaesthetized rabbit following stimulation of the perforant path. The Journal of Physiology, 232, 331–356. Bouwmeester, L., Heutnik, J., & Lucas, C. (2007). The effect of visual training for patients with visual field defects due to brain damage: A systematic review. Journal of Neurology, Neurosurgery and Psychiatry, 78, 555–564. Brindley, G. S., Donaldson, P. E. K., Falconer, M. A., & Rushton, D. N. (1972). The extent of the region of occipital cortex that when stimulated gives phosphenes fixed in the visual field. Journal of Physiology, London, 225, 57P–58P. Brindley, G. S., & Lewin, W. S. (1968). The sensations produced by electrical stimulation of the visual cortex. The Journal of Physiology, 96, 479–493. Brindley, G. S., & Rushton, D. N. (1977). Observations on the representation of the visual field on the human occipital cortex. In F. T. Hambrecht & J. B. Reswick (Eds.), Functional electrical stimulation: Applications in neural prostheses (pp. 261–276). New York: Marcel Dekker. Brodtmann, A., Puce, A., Darby, D., & Donnan, G. (2009). Serial functional imaging poststroke reveals visual cortex reorganization. Neurorehabilitation and Neural Repair, 23, 150–159.

253 Calford, M. B., Chino, Y. M., Das, A., Eysel, U. T., Gilbert, C. D., Heinen, S. J., et al. (2005). Neuroscience: Rewiring the adult brain. Nature, 438, E3–E4. Calford, M. B., Wang, C., Taglianetti, V., Waleszczyk, W. J., Burke, W., & Dreher, B. (2000). Plasticity in adult cat visual cortex (area 17) following circumscribed monocular lesions of all retinal layers. Journal of Physiology, London, 524, 587–602. Calford, M. B., Wright, L. L., Metha, A. B., & Taglianetti, V. (2003). Topographic plasticity in primary visual cortex is mediated by local corticocortical connections. The Journal of Neuroscience, 23, 6434–6442. Chaieb, L., Antal, A., & Paulus, W. (2008). Gender-specific modulation of short-term neuroplasticity in the visual cortex induced by transcranial direct current stimulation. Visual Neuroscience, 25, 77–81. Center for Disease Control and Prevention, Aging Trends, No. 2, March 2004. Chibisova, A. N., Fedorov, A. B., & Fedorov, N. A. (2001). Neurophysiological characteristics of compensation-recovery processes in the brain during rehabilitation of the neurosensory impairment of the visual and acoustic systems. Fiziologiia Cheloveka, 27, 14–21. Chino, Y. M., Smith, E. G., Kaas, J. H., Sasaki, Y., & Cheng, H. (1995). Receptive field properties of deafferented visual cortical neurons after topographic map reorganization in adult cats. The Journal of Neuroscience, 15, 2417–2433. Chokron, S., Perez, C., Obadia, M., Gaudry, I., Laloum, L., & Gout, O. (2008). From blindsight to sight: Cognitive rehabilitation of visual field defects. Restorative Neurology and Neuroscience, 26, 305–320. Chow, K. L. (1968). Visual discrimination after extensive ablation of optic tract and visual cortex in cats. Brain Research, 9, 363–366. Chow, K. L., & Stewart, D. L. (1972). Reversal of structural and functional effects of long-term visual deprivation in cats. Experimental Neurology, 34, 409–433. Cohen, S. Y., Lamarque, F., Saucet, J. C., Provent, P., Langram, C., & LeGargasson, J. F. (2003). Filling-in phenomenon in patients with age-related macular degeneration: Differences regarding uni- or bilaterality of central scotoma. Graefe's Archive for Clinical and Experimental Ophthalmology, 241, 785–791. Cole, S. R., Beck, R. W., Moke, P. S., Gal, R. L., & Long, D. T. (2000). The National Eye Institute Visual Function Questionnaire: Experience of the ONTT. Investigative Ophthalmology & Visual Science, 41, 1017–1021. Cooke, S. F., & Bear, M. F. (2010). Visual experience induces long-term potentiation in the primary visual cortex. The Journal of Neuroscience, 30, 16304–16313. Corbetta, M., Kincade, M. J., Lewis, C., Snyder, A. Z., & Sapir, A. (2005). Neural basis and recovery of spatial attention deficits in spatial neglect. Nature Neuroscience, 8, 1603–1610.

Cowey, A. (1967). Perimetric study of field defects in monkeys after cortical and retinal ablations. The Quarterly Journal of Experimental Psychology, 19, 232–245. Cowey, A., & Stoerig, P. (1991). The neurobiology of blindsight. Trends in Neurosciences, 14, 140–145. Cowey, A., & Stoerig, P. (1995). Blindsight in monkeys. Nature, 373, 247–249. Darian-Smith, C., & Gilbert, C. D. (1994). Axonal sprouting accompanies functional reorganization in adult cat striate cortex. Nature, 368, 737–740. Darian-Smith, C., & Gilbert, C. D. (1995). Topographic reorganization in the striate cortex of the adult cat and monkey is cortically mediated. The Journal of Neuroscience, 15, 1631–1647. Das, A., & Gilbert, C. D. (1995). Long-range horizontal connections and their role in cortical reorganization revealed by optical recording of cat primary visual cortex. Nature, 375, 780–784. Das, A., & Huxlin, K. R. (2010). New approaches to visual rehabilitation for cortical blindness: Outcomes and putative mechanisms. The Neuroscientist, 16, 374–387. De Weerd, P., Sprague, J. M., Raiguel, S., Vandenbussche, E., & Orban, G. A. (1993). Effects of visual cortex lesions on orientation discrimination of illusory contours in the cat. The European Journal of Neuroscience, 5, 1695–1710. De Weerd, P., Sprague, J. M., Vandenbussche, E., & Orban, G. A. (1994). Two stages in visual texture segregation: A lesion study in the cat. The Journal of Neuroscience, 14, 929–948. Di Stefano, M., & Gargini, C. (1995). Recovery of the nasal field loss in esotropic cats after section of the optic chiasm. Neuroreport, 6, 1341–1346. Dilks, D. D., Baker, C. I., Peli, E., & Kanwisher, N. (2009). Reorganization of visual processing in macular degeneration is not specific to the “preferred retinal locus” The Journal of Neuroscience, 29, 2768–2773. Dilks, D. D., Serences, J. T., Rosenau, B. J., Yantis, S., & McCloskey, M. (2007). Human adult cortical reorganization and consequent visual distortion. The Journal of Neuroscience, 27, 9585–9594. Dobelle, W. H., Mladejovsky, M. G., Evans, J. R., Roberts, T. S., & Girvin, J. P. (1976). 'Braille' reading by a blind volunteer by visual cortex stimulation. Nature, 259, 111–112. Dobelle, W. H., Mladejovsky, M. G., & Girvin, J. P. (1974). Artificial vision for the blind: Electrical stimulation of visual cortex offers hope for a functional prosthesis. Science, 183, 440–444. Dohle, C. I., Eysel, U. T., & Mittmann, T. (2009). Spatial distribution of long-term potentiation in the surround of visual cortex lesions in vitro. Experimental Brain Research, 199, 423–433. Dreher, B., Burke, W., & Calford, M. B. (2001). Cortical plasticity revealed by circumscribed retinal lesions or artificial scotomas. Progress in Brain Research, 134, 217–246. Dürsteler, M. R., Wurtz, R. H., & Newsome, W. T. (1987). Directional pursuit deficits following lesions of the foveal representations within the superior temporal sulcus of the

254 macaque monkey. Journal of Neurophysiology, 57, 1262–1287. Duvdevani, R., Rosner, M., Belkin, M., Sautter, J., Sabel, B. A., & Schwartz, M. (1990). Graded crush of the tat optic nerve as a brain injury model: Combining electrophysiological and behavioral outcome. Restorative Neurology and Neuroscience, 2, 31–38. El Mallah, M. K., Chakravarthy, U., & Hart, P. M. (2000). Amblyopia: Is visual loss permanent? The British Journal of Ophthalmology, 84, 952–956. Eriksen, C. W., & Rohrbaugh, J. W. (1970). Some factors determining efficiency of selective attention. The American Journal of Psychology, 83, 330–342. Eysel, U. T. (1997). Perilesional cortical dysfunction and reorganization. In H.-J. Freund, B. A. Sabel & O. W. Witte (Eds.), Brain plasticity. Advances in Neurology, 73, pp. (195–206). Eysel, U. T., & Grüsser, O.-J. (1978). Increased transneuronal excitation of the cat lateral geniculate nucleus after acute deafferentation. Brain Research, 158, 107–128. Eysel, U. T., Schweigart, G., Mittmann, T., Eyding, D., Qu, Y., Vandesande, F., et al. (1999). Reorganization in the visual cortex after retinal and cortical damage. Restorative Neurology and Neuroscience, 15, 153–164. Fabre-Thorpe, M., Levesque, F., & Buser, P. (1994). Preservation of pointing accuracy toward moving targets after extensive visual cortical ablations in cats. Cortex, 30, 585–601. Fahle, M. (2002). Perceptual learning: Gain without pain? Nature Neuroscience, 5, 923–924. Fahle, M. (2005). Perceptual learning: Specificity versus generalization. Current Opinion in Neurobiology, 15, 154–160. Fahle, M., & Poggio, T. (2002). Perceptual learning. Cambridge, MA: MIT Press. Fahle, M., & Skrandies, W. (1994). An electrophysiological correlate of learning in motion perception. German Journal of Ophthalmology, 3, 427–432. Fedorov, A. B., Chibisova, A. M., & Tchibissova, J. M. (2005). Impulse modulating therapeutic electrical stimulation (IMTES) increases visual field size in patients with optic nerve lesions. International Congress Series, 1282, 525–529. Fedorov, A. B., Jobke, S., Chibisova, A. N., Chibisova, J. M., Gall, C., & Sabel, B. A. (2010). Restoration of vision: noninvasive current stimulation enhances brain synchronization after optic nerve damage - a retrospective observational study with 446 patients. World Ophthalmology Congress, Berlin, Germany, June, 5–9. Felleman, D. J., & Van Essen, D. C. (1991). Distributed hierarchical processing in the primate cerebral cortex. Cerebral Cortex, 1, 1–47. Fendrich, R., Wessinger, C. M., & Gazzaniga, M. S. (1992). Residual vision in a scotoma: Implications for blindsight. Science, 258, 27. Fendrich, R., Wessinger, C. M., & Gazzaniga, M. S. (1993). Sources of blindsight (technical comment). Science, 261, 494–495.

Fiorentini, A., & Berardi, N. (1980). Perceptual learning specific for orientation and spatial frequency. Nature, 287, 43–44. Fischman, M. W., & Meikle, T. H. Jr. (1965). Visual intensity discrimination in cats after serial tectal and cortical lesions. Journal of Comparative and Physiological Psychology, 59, 193–201. Foerster, O. (1929). Beitrage zur Pathophysiologie der Sehbahn und der Sehsphare. Journal für Psychologie und Neurologie, 39, 435–463. Franke, G. H., & Gall, C. (2008). Quality of life—Methodology and clinical practice aspects with a focus on ocular medicine. Ophthalmologe, 105, 727–734. Fronius, M., Cirina, L., Cordey, A., & Ohrloff, C. (2005). Visual improvement during psychophysical training in an adult amblyopic eye following visual loss in the contralateral eye. Graefe's Archive for Clinical and Experimental Ophthalmology, 243, 278–280. Fronius, M., Cirina, L., Kuhli, C., Cordey, A., & Ohrloff, C. (2006). Training the adult amblyopic eye with ‘perceptual learning’ after vision loss in the non-amblyopic eye. Strabismus, 14, 75–79. Fujikado, T., Morimoto, T., Matsushita, K., Shimojo, H., Okawa, Y., & Tano, Y. (2006). Effect of transcorneal electrical stimulation in patients with nonarteritic ischemic optic neuropathy or traumatic optic neuropathy. Japanese Journal of Ophthalmology, 50, 266–273. Galambos, R., Norton, T. T., & Frommer, G. P. (1967). Optic tract lesions sparing pattern vision in cats. Experimental Neurology, 18, 8–25. Gall, C., Franke, G. H., & Sabel, B. A. (2010a). Vision-related quality of life in first stroke patients with homonymous visual field defects. Health and Quality of Life Outcomes, 88, 33. Gall, C., Fedorov, A. B., Ernst, L., Borrmann, A., & Sabel, B. A. (2010b). Repetitive transorbital alternating current stimulation (rtACS) in optic neuropathy—Single case study. NeuroRehabilitation, 27, 335–341. Gall, C., Wagenbreth, C., Sgorzaly, S., & Sabel, B. A. (2010c). Parafoveal vision impairments and their influence on reading performance and self-evaluated reading abilities. Graefe's Archive for Clinical and Experimental Ophthalmology, 248, 863–875. Gall, C., Lucklum, J., Sabel, B. A., & Franke, G. H. (2009). Vision- and health-related quality of life in patients with visual field loss after postchiasmatic lesions. Investigative Ophthalmology & Visual Science, 50, 2765–2776. Gall, C., Mueller, I., Kaufmann, C., Franke, G. H., & Sabel, B. A. (2008). Visual field defects after cerebral lesions from the subject's perspective: Health- and visionrelated quality of life assessed by SF-36 and NEI-VFQ. Der Nervenarzt, 79, 185–194. Gall, C., Schmidt, S., Brandt, S., Fedorov, A. B., & Sabel, B. A. (2011). Non-invasive alternating current stimulation impoves subjective visual functioning in optic neuropathy. Neuroscience, Washington, 12.11.–16.11.2011. Ganz, L., & Fitch, M. (1968). The effect of visual deprivation on perceptual behavior. Experimental Neurology, 22, 638–660.

255 Giannikopoulos, D. V., & Eysel, U. T. (2006). Dynamics and specificity of cortical map reorganization after retinal lesions. Proceedings of the National Academy of Sciences of the United States of America, 103, 10805–10810. Gibson, E. (1969). Principles of perceptual learning and development. New York: Appleton-Century-Crofts. Gilbert, C. D. (1998). Adult cortical dynamics. Physiological Reviews, 78, 467–485. Gilbert, C. D., Sigman, M., & Crist, R. E. (2001). The neural basis of perceptual learning. Neuron, 31, 681–697. Gilbert, C. D., & Wiesel, T. N. (1992). Receptive field dynamics in adult primary visual cortex. Nature, 356, 150–152. Gothe, J., Brandt, S. A., Irlbacher, K., Röricht, S., Sabel, B. A., & Meyer, B. U. (2002). Changes in visual cortex excitability in blind subjects as demonstrated by transcranial magnetic stimulation. Brain, 125, 479–490. Greenstein, V. C., Santos, R. A., Tsang, S. H., Smith, R. T., Barile, G. R., & Seiple, W. (2008a). Preferred retinal locus in macular disease: Characteristics and clinical implications. Retina, 28, 1234–1240. Greenwald, M. J., & Parks, M. M. (1999). Treatment of amblyopia. In: T. Duane (Ed.), Clinical ophthalmology, (Vol. 1. Hagerstown: Harper and Row. Gudlin, J. (2008) Objective and subjective effects of systematic visual stimulation in glaucoma patients. Zugl.: Magdeburg Univ., Fak. für Naturwiss., Dissertation, 2008. Gudlin, J., Mueller, I., Thanos, S., & Sabel, B. A. (2008). Computer based vision restoration therapy in glaucoma patients: A small open pilot study. Restorative Neurology and Neuroscience, 26, 403–412. Guenther, T., Mueller, I., Preuss, M., Kruse, R., & Sabel, B. A. (2009). Treatment outcome prediction model of visual field recovery using Self-Organizing-Maps. IEEE Transactions on Biomedical Engineering, 56, 572–581. He, Z. J., & Loop, M. S. (1991). Luminance sensitivity recovers quickly in monocularly deprived cats. Vision Research, 31, 1633–1637. Heiduschka, P., & Thanos, S. (2000). Restoration of the retinofugal pathway. Progress in Retinal and Eye Research, 19, 577–606. Heinen, S. J., & Skavenski, A. A. (1991). Recovery of visual responses in foveal V1 neurons following bilateral foveal lesions in adult monkey. Experimental Brain Research, 83, 670–674. Held, R., & Hein, A. (1963). Movement-produced stimulation in the development of visually guided behaviour. Journal of Comparative and Physiological Psychology, 56, 872–876. Henrich-Noack, P., Gorkin, A. G., Krautwald, K., Pforte, C., Schröder, U. H., & Reymann, K. G. (2005). Tetanusinduced re-activation of evoked spiking in the post-ischemic dentate gyrus. Neuroscience, 133, 571–581. Henriksson, L., Raninen, A., Näsänen, R., Hyvärinen, L., & Vanni, S. (2007). Training-induced cortical representation

of a hemianopic hemifield. Journal of Neurology, Neurosurgery and Psychiatry, 78, 74–81. Hess, R. F., Field, D., & Watt, R. J. (1990). The puzzle of amblyopia. In C. Blakemore (Ed.), Vision: Coding and efficiency. Cambridge: Cambridge University Press. Hirsch, J. A., & Gilbert, C. D. (1991). Synaptic physiology of horizontal connections in the cat's visual cortex. The Journal of Neuroscience, 11, 1800–1809. Ho, Y. C., Cheze, A., Sitoh, Y. Y., Petersen, E. T., Goh, K. Y., Gjedde, A., et al. (2009). Residual neurovascular function and retinotopy in a case of hemianopia. Annals of the Academy of Medicine, Singapore, 38, 827–831. Holmes, G. (1918). Disturbances of visual orientation. The British Journal of Ophthalmology, 2, 449–506. Hubel, D. H., & Wiesel, T. N. (1970). The period of susceptibility to the physiological effects of unilateral eye closure in kittens. The Journal of Physiology, 206, 419–436. Huemmeke, M., Eysel, U. T., & Mittmann, T. (2004). Lesioninduced enhancement of LTP in rat visual cortex is mediated by NMDA receptors containing the NR2B subunit. The Journal of Physiology, 559, 875–882. Humphrey, N. K. (1970). What the frog's eye tells the monkey's brain. Brain, Behavior and Evolution, 3, 324–337. Humphrey, N. K. (1974). Vision in a monkey without striate cortex: A case study. Perception, 3, 241–255. Huxlin, K. R. (2008). Perceptual plasticity in damaged adult visual systems. Vision Research, 48, 2154–2166. Huxlin, K. R., Martin, T., Kelly, K., Riley, M., Friedman, D. I., Burgin, W. S., et al. (2009). Perceptual relearning of complex visual motion after V1 damage in humans. The Journal of Neuroscience, 29, 3981–3991. Huxlin, K. R., & Pasternak, T. (2004). Training-induced recovery of visual motion perception after extrastriate cortical damage in the adult cat. Cerebral Cortex, 14, 81–90. Hyvärinen, L., Raninen, A. N., & Näsänen, R. E. (2002). Vision rehabilitation in homonymous hemianopia. NeuroOphthalmology, 27, 97–102. Imbrosci, B., Eysel, U. T., & Mittmann, T. (2010). Metaplasticity of horizontal connections in the vicinity of laser lesions in rat visual cortex. The Journal of Physiology, 588, 4695–4703. Inomata, K., Shinoda, K., Ohde, H., Tsunoda, K., Hanazono, G., Kimura, I., et al. (2007). Transcorneal electrical stimulation of retina to treat longstanding retinal artery occlusion. Graefe's Archive for Clinical and Experimental Ophthalmology, 245, 1773–1780. Ito, M., & Gilbert, C. D. (1999). Attention modulates contextual influences in the primary visual cortex of alert monkeys. Neuron, 22, 593–604. Jacobson, S. G., Eames, R. A., & McDonald, W. I. (1979). Optic nerve fibre lesions in adult cats: Pattern of recovery of spatial vision. Experimental Brain Research, 36, 491–508.

256 Jacobson, S. G., McDonald, W. I., & Perry, P. A. (1977). Pattern of recovery of spatial vision after pre-geniculate nerve fibre lesions in adults cats. The Journal of Physiology, 269, 25P–26P. Jobke, S., Kasten, E., & Sabel, B. A. (2009). Vision restoration through extrastriate stimulation in patients with visual field defects—A double-blind and randomized experimental study. Neurorehabilitation and Neural Repair, 23, 246–555. Julkunen, L., Tenovuo, O., Jääskeläinen, S. K., & Hämäläinen, H. (2003). Rehabilitation of chronic post-stroke visual field defect with computer-assisted training: A clinical and neurophysiological study. Restorative Neurology and Neuroscience, 21, 19–28. Julkunen, L., Tenovuo, O., Vorobyev, V., Hiltunen, J., Teräs, M., Jääskeläinen, S. K., et al. (2006). Functional brain imaging, clinical and neurophysiological outcome of visual rehabilitation in a chronic stroke patient. Restorative Neurology and Neuroscience, 24, 123–132. Jung, C. S., Bruce, B., Newman, N. J., & Biousse, V. (2008). Visual function in anterior ischemic optic neuropathy: Effect of Vision Restoration Therapy—A pilot study. Journal of the Neurological Sciences, 268, 145–149. Kaas, J. H., Krubitzer, L. A., Chino, Y. M., Langston, A. L., Polley, E. H., & Blair, N. (1990). Reorganization of retinotopic cortical maps in adult mammals after lesions of the retina. Science, 248, 229–231. Karmarker, U. R., & Dan, Y. (2006). Experience-dependent plasticity in adult visual cortex. Neuron, 52, 577–585. Karni, A., & Sagi, D. (1991). Where practice makes perfect in texture discrimination: Evidence for primary visual cortex plasticity. Proceedings of the National Academy of Sciences of the United States of America, 88, 4966–4970. Kasten, E., Bunzenthal, U., Müller-Oehring, E. M., Mueller, I., & Sabel, B. A. (2007). Vision restoration does not benefit from costimulation: A pilot study. Journal of Clinical and Experimental Neuropsychology, 29, 569–584. Kasten, E., Bunzenthal, U., & Sabel, B. A. (2006). Visual field recovery after vision restoration therapy (VRT) is independent of eye movements: An eye tracker study. Behavioural Brain Research, 175, 18–26. Kasten, E., Guenther, T., & Sabel, B. A. (2008). Inverse stimuli in perimetric performance reveal larger visual field defects: Implications for vision restoration. Restorative Neurology and Neuroscience, 26, 355–364. Kasten, E., Müller-Oehring, E., & Sabel, B. A. (2001). Stability of visual field enlargements following computer-based restitution training—Results of a follow-up. Journal of Clinical and Experimental Neuropsychology, 23, 297–305. Kasten, E., & Sabel, B. A. (1995). Visual field enlargement after computer training in brain-damaged patients with homonymous deficits: An open pilot trial. Restorative Neurology and Neuroscience, 8, 113–127. Kasten, E., Wuest, S., & Sabel, B. A. (1998a). Variability of stimulus detection, form discrimination, and color

recognition with suprathreshold campimetry in brain damaged patients. Neuroophthalmology, 20, 161–176. Kasten, E., Wüst, S., Behrens-Baumann, W., & Sabel, B. A. (1998b). Computer-based training for the treatment of partial blindness. Natural Medicines, 4, 1083–1087. Kasten, E., Wüst, S., & Sabel, B. A. (1998c). Residual vision in transition zones in patients with cerebral blindness. Journal of Clinical and Experimental Neuropsychology, 20, 581–598. Keck, T., Mrsic-Flogel, T. D., Vaz Afonso, M., Eysel, U. T., Bonhoeffer, T., & Hübener, M. (2008). Massive restructuring of neuronal circuits during functional reorganization of adult visual cortex. Nature Neuroscience, 11, 1162–1167. Keller, J., Strasburger, H., Cerutti, D. T., & Sabel, B. A. (2000). Automated assessment of the visual contrast sensitivity in the hooded rat. Journal of Neuroscience Methods, 97, 103–110. Kerkhoff, G., Münßinger, U., Eberle-Strauss, G., & Stögerer, E. (1992a). Rehabilitation of hemianopic alexia in patients with postgeniculate visual field disorders. Neuropsychological Rehabilitation, 2, 21–42. Kerkhoff, G., Münßinger, U., Haaf, E., Eberle-Strauss, G., & Stögerer, E. (1992b). Rehabilitation of homonymous scotomata in patients with postgeniculate damage of the visual system: Saccadic compensation training. Restorative Neurology and Neuroscience, 4, 245–254. Kerkhoff, G., Münßinger, U., & Meier, E. K. (1994). Neurovisual rehabilitation in cerebral blindness. Archives of Neurology, 51, 474–481. Kerkhoff, G., & Schindler, I. (2000). Möglichkeiten und Grenzen restorativer Trainingsmethoden bei prä- und postchiasmatischen Skotomen. Zeitschrift für Neuropsychologie, 11, 82–85. Kolarow, R., Brigadski, T., & Lessmann, V. (2007). Postsynaptic secretion of BDNF and NT-3 from hippocampal neurons depends on calcium calmodulin kinase II signaling and proceeds via delayed fusion pore opening. The Journal of Neuroscience, 27, 10350–10364. Kölmel, H. W. (1985). Complex visual hallucinations in the hemianopic field. Journal of Neurology, Neurosurgery and Psychiatry, 48, 29–38. Kölmel, H. W. (1993). Visual illusions and hallucinations. Baillere‘s Clinical Neurology, 2, 243–264. Korsholm, K., Madsen, K. H., Frederiksen, J. L., Rowe, J. B., & Lund, T. E. (2008). Cortical neuroplasticity in patients recovering from acute optic neuritis. Neuroimage, 42, 836–844. Korsholm, K., Madsen, K. H., Frederiksen, J. L., Skimminge, A., & Lund, T. E. (2007). Recovery from optic neuritis: An ROI-based analysis of LGN and visual cortical areas. Brain, 130, 1244–1253. Kreutz, M. R., Böckers, T. M., Bockmann, J., Seidenbecher, C., Kracht, B., Vorwerk, C. K., et al. (1998). Splicing of the retinal NR1 receptor is altered after optic nerve injury: The preferential expression of the NR1-b

257 isoforms is crucial for retinal ganglion cell survival. The Journal of Neuroscience, 18, 8278–8291. Kuhn, R. (1962). The speed of cerebral circulation. The New England Journal of Medicine, 267, 689–695. Lane, A. R., Smith, D. T., Ellison, A., & Schenk, T. (2010). Visual exploration training is no better than attention training for treating hemianopia. Brain, 133, 1717–1728. Lashley, K. (1939). The mechanisms of vision XVI. The function of small remnants of the visual cortex. The Journal of Comparative Neurology, 70, 45–67. Levi, D. M., & Carkeet, A. (1993). Amblyopia: A consequence of abnormal visual development. In K. Simons (Ed.), Early visual development, normal and abnormal (pp. 391–407). Oxford, NY: Oxford University Press. Levi, D. M., & Polat, U. (1996). Neural plasticity in adults with amblyopia. Proceedings of the National Academy of Sciences of the United States of America, 93, 6830–6834. Levi, D. M., Polat, U., & Hu, Y. S. (1997). Improvement in Vernier acuity in adults with amblyopia. Practice makes better. Investigative Ophthalmology & Visual Science, 38, 1493–1510. Levin, N., Orloy, T., Dotan, S., & Zohary, E. (2006). Normal and abnormal fMRI activation patterns in the visual cortex after recovery from optic neuritis. Neuroimage, 33, 1161–1168. Li, W., Piech, V., & Gilbert, C. D. (2004). Perceptual learning and top-down influences in primary visual cortex. Nature Neuroscience, 7, 651–657. Li, R. W., Young, K. G., Hoenig, P., & Levi, D. M. (2005). Perceptual learning improves visual performance in juvenile amblyopia. Investigative Ophthalmology & Visual Science, 46, 3161–3168. Liu, T., Cheung, S. H., Schuchard, R., Glielmi, C., Hu, X., He, S., et al. (2010). Incomplete cortical reorganization in macular degeneration. Investigative Ophthalmology & Visual Science, 51, 6826–6834. Lorber, B., Berry, M., & Logan, A. (2008). Different factors promote axonal regeneration of adult rat retinal ganglion cells after lens injury and intravitreal peripheral nerve grafting. Journal of Neuroscience Research, 86, 894–903. Maire-Lepoivre, E., Przybyskawski, J., & Gary-Bobo, E. (1988). Visual field in dark-reared cats after an extended period of recovery. Behavioural Brain Research, 28, 245–251. Malis, L. I., & Kruger, L. (1956). Multiple response and excitability of cat's visual cortex. Journal of Neurophysiology, 19, 172–186. Mangun, G. R., & Hillyard, S. A. (1987). The spatial allocation of visual attention as indexed by event-related brain potentials. Human Factors, 29, 195–211. Marshall, R. S., Ferrera, J. J., Barnes, A., Xian, Z., O'Brian, K. A., Chmayssani, M., et al. (2008). Brain activity associated with stimulation therapy of the visual borderzone in hemianopic stroke patients. Neurorehabilitation and Neural Repair, 22, 136–144.

Marshall, R. S., Chmayssani, M., O’Brian, K. A., Handy, C. & Greenstein, V. C. (2010) Visual field expansion after visual restoration therapy. Clin Rehabil, 24, 1027–1035. Martinez, A., Anllo-Vento, L., Sereno, M. I., Frank, L. R., Buxton, R. B., Dubowitz, D. J., et al. (1999). Involvement of striate and extrastriate visual cortical areas in spatial attention. Nature Neuroscience, 2, 364–369. McManus, J. N., Ullman, S., & Gilbert, C. D. (2008). A computational model of perceptual fill-in following retinal degeneration. Journal of Neurophysiology, 99, 2086–2100. Mendola, J. D., Conner, I. P., Sharma, S., Bahekar, A., & Lemieux, S. (2006). fMRI measures of perceptual filling-in in the human visual cortex. Journal of Cognitive Neuroscience, 18, 363–375. Milleret, C., & Buser, P. (1984). Receptive field sizes and responsiveness to light in area 18 of the adult cat after chiasmotomy. Postoperative evolution; role of visual experience. Experimental Brain Research, 57, 73–81. Mitchell, D. E., Murphy, K. M., & Kaye, M. G. (1984). The permanence of the visual recovery that follows reverse occlusion of monocularly deprived kittens. Investigative Ophthalmology & Visual Science, 25, 908–917. Mizobe, K., Polat, U., Pettet, M. W., & Kasamatsu, T. (2001). Facilitation and suppression of single striate-cell activity by spatially discrete pattern stimuli presented beyond the receptive field. Visual Neuroscience, 18, 377–391. Mohler, C. W., & Wurtz, R. H. (1977). Role of striate cortex and superior colliculus in visual guidance of saccadic eye movements in monkeys. Journal of Neurophysiology, 40, 74–94. Moore, T., Rodman, H. R., Repp, A. B., Gross, C. G., & Mezrich, R. S. (1996). Greater residual vision in monkeys after striate cortex damage in infancy. Journal of Neurophysiology, 76, 3928–3933. Mueller, I., Gall, C., Kasten, E., & Sabel, B. A. (2008). Longterm learning of visual functions in patients after brain damage. Behavioural Brain Research, 191, 32–42. Mueller, I., Mast, H., & Sabel, B. A. (2007). Recovery of visual function after brain injury: A large-sample study using Vision Restoration Therapy. Restorative Neurology and Neuroscience, 25, 563–572. Mueller, I., Poggel, D. A., Kenkel, S., Kasten, E., & Sabel, B. A. (2003). Vision Restoration Therapy (VRT) after brain damage: Subjective improvements of activities of daily life and their relationship to visual field enlargements. Visual Impairment Research, 5, 157–178. Mueller, I., Sabel, B. A., & Kasten, E. (2006). Comparison of early and late rehabilitation of stroke and cerebral trauma patients with visual field defects. Der Nervenarzt, 77, 694–701. Murakami, I., Komatsu, H., & Kinoshita, M. (1997). Perceptual filling-in at the scotoma following a monocular retinal lesion in the monkey. Visual Neuroscience, 14, 89–101.

258 Nakayama, K., & Mackeben, M. (1989). Sustained and transient components of focal visual attention. Vision Research, 29, 1631–1647. Nelles, G., de Greiff, A., Pscherer, A., Forsting, M., Gerhard, H., Esser, J., et al. (2007). Cortical activation in hemianopia after stroke. Neuroscience Letters, 426, 34–38. Nelles, G., Pscherer, A., de Greiff, A., Gerhard, H., Forsting, M., Esser, J., et al. (2010). Eye-movement training-induced changes of visual field representation in patients with post-stroke hemianopia. Journal of Neurology, 257, 1832–1840. Nelles, G., Widman, G., de Greiff, A., Meistrowitz, A., Dimitrova, A., Weber, J., et al. (2002). Brain representation of hemifield stimulation in poststroke visual field defects. Stroke, 33, 1286–1293. Newsome, W. T., Wurtz, R. H., Dürsteler, M. R., & Mikami, A. (1985). Punctate chemical lesions of striate cortex in macaque monkeys: Effects on visually guided saccades. Experimental Brain Research, 58, 392–399. Nilsson, U. L., Frennesson, C., & Nilsson, S. E. (1998). Location and stability of a newly established eccentric retinal locus suitable for reading, achieved through training of patients with a dense central scotoma. Optometry and Vision Science, 75, 873–878. Nilsson, U. L., Frennesson, C., & Nilsson, S. E. (2003). Patients with AMD and a large absolute central scotoma can be trained successfully to use eccentric viewing, as demonstrated in a scanning laser ophthalmoscope. Vision Research, 43, 1777–1787. Noppeney, U. (2007). The effects of visual deprivation on functional and structural organization of the human brain. Neuroscience and Biobehavioral Reviews, 31, 1169–1180. Ostrovsky, Y., Andalman, A., & Sinha, P. (2006). Vision following extended congenital blindness. Psychological Science, 17, 1009–1014. Ostrovsky, Y., Meyers, E., Ganesh, S., Mathur, U., & Sinha, P. (2009). Visual parsing after recovery from blindness. Psychological Science, 20, 1484–1491. Palagina, G., Eysel, U. T., & Jancke, D. (2009). Strengthening of lateral activation in adult rat visual cortex after retinal lesions captured with voltage-sensitive dye imaging in vivo. Proceedings of the National Academy of Sciences of the United States of America, 106, 8743–8747. Pambakian, A. L., Currie, J., & Kennard, C. (2005). Rehabilitation strategies for patients with homonymous visual field defects. Journal of Neuroophthalmology, 25, 136–142. Pambakian, A. L., & Kennard, C. (1997). Can visual function be restored in patients with homonymous hemianopia? The British Journal of Ophthalmology, 81, 324–328. Papageorgiou, E., Hardiess, G., Schaeffel, F., Wiethoelter, H., Karnath, H. O., Mallot, H., et al. (2007). Assessment of vision-related quality of life in subjects with homonymous

visual field defects. Graefe's Archive for Clinical and Experimental Ophthalmology, 245, 1749–1758. Paramei, G. V., & Sabel, B. A. (2008). Contour integration deficits on the intact side of the visual field in hemianopic patients. Behavioural Brain Research, 188, 109–124. Park, H. J., Jeong, S. O., Kim, E. Y., Kim, J. I., Park, H., Oh, M. K., et al. (2007). Reorganization of neural circuits in the blind on diffusion direction analysis. Neuroreport, 18, 1757–1760. Pasik, T., & Pasik, P. (1973). Extrageniculostriate vision in the monkey. IV. Critical structures for light vs. no-light discrimination. Brain Research, 56, 165–182. Pavlidis, M., Stupp, T., Naskar, R., Cengiz, C., & Thanos, S. (2003). Retinal ganglion cells resistant to advanced glaucoma: A post mortem study of human retinas with the carbocyanine dye DiI. Investigative Ophthalmology & Visual Science, 44, 5196–5205. Payne, B. R., Lomber, S. G., MacNeil, M. A., & Cornwell, P. (1996). Evidence for greater sight in blindsight following damage of primary visual cortex early in life. Neuropsychologia, 34, 741–774. Poggel, D. A., Kasten, E., Müller-Oehring, E. M., Bunzenthal, U., & Sabel, B. A. (2006). Improving residual vision by attentional cueing in patients with brain lesions. Brain Research, 1097, 142–148. Poggel, D. A., Kasten, E., Müller-Oehring, E. M., Sabel, B. A., & Brandt, S. A. (2001). Unusual spontaneous and training induced visual field recovery in a patient with a gunshot lesion. Journal of Neurology, Neurosurgery and Psychiatry, 70, 236–239. Poggel, D. A., Kasten, E., & Sabel, B. A. (2004). Attentional cueing improves vision restoration therapy in patients with visual field loss. Neurology, 63, 2069–2076. Poggel, D. A., Mueller, I., Kasten, E., Bunzenthal, U., & Sabel, B. A. (2010). Subjective and objective outcome measures of computer-based vision restoration training. NeuroRehabilitation, 27, 173–187. Poggel, D. A., Mueller, I., Kasten, E., & Sabel, B. A. (2008). Multifactorial predictors and outcome variables of vision restoration training in patients with post-geniculate visual field loss. Restorative Neurology and Neuroscience, 26, 321–339. Poggel, D. A., Müller-Oehring, E. M., Gothe, J., Kenkel, S., Kasten, E., & Sabel, B. A. (2007). Visual hallucinations during spontaneous and training-induced visual field recovery. Neuropsychologia, 45, 2598–2607. Polat, U. (1999). Functional architecture of long-range perceptual interactions. Spatial Vision, 12, 143–162. Polat, U. (2008). Restoration of underdeveloped cortical functions: Evidence from treatment of adult amblyopia. Restorative Neurology and Neuroscience, 26, 413–424. Polat, U., Ma-Naim, T., Belkin, M., & Sagi, D. (2004). Improving vision in adult amblyopia by perceptual learning.

259 Proceedings of the National Academy of Sciences of the United States of America, 101, 6692–6697. Polat, U., Ma-Naim, T., & Spierer, A. (2009). Treatment of children with amblyopia by perceptual learning. Vision Research, 49, 2599–2603. Polat, U., & Sagi, D. (1994). Spatial interactions in human vision: From near to far via experience-dependent cascades of connections. Proceedings of the National Academy of Sciences of the United States of America, 91, 1206–1209. Polat, U., & Sagi, D. (2006). Temporal asymmetry of collinear lateral interactions. Vision Research, 46, 953–960. Polat, U., Sagi, D., & Norcia, A. M. (1997). Abnormal longrange spatial interactions in amblyopia. Vision Research, 37, 737–744. Pollen, D. A. (1977). Responses of single neurons to electrical stimulation of the surface of the visual cortex. Brain, Behavior and Evolution, 14, 67–86. Pöppel, E., Held, R., & Frost, D. (1973). Residual visual function after brain wounds involving the central visual pathways in man. Nature, 243, 295–296. Pöppel, E., & Richards, W. (1974). Light sensitivity in cortical scotomata contralateral to small islands of blindness. Experimental Brain Research, 21, 125–130. Poppelreuter, W. (1917). Die psychischen Schädigungen durch Kopfschuss im Kriege 1914–1916. Bd. I. Die Störungen der niederen und höheren Sehleistungen durch Verletzungen des Okzipitalhirns. Leipzig: L. Voss. Posner, M. I. (1980). Orienting of attention. The Quarterly Journal of Experimental Psychology, 32, 3–25. Prieto-Diaz, J. S.-D. C. (2000). Strabismus (4th ed.). Boston: Butterworth-Heinemann. Prilloff, S., Henrich-Noack, P., Kropf, S., & Sabel, B. A. (2010). Experience-dependent plasticity and vision restoration in rats after optic nerve crush. Journal of Neurotrauma, 27, 2295–2307. Prilloff, S., Noblejas, M. I., Chedhomme, V., & Sabel, B. A. (2007). Two faces of calcium activation after optic nerve trauma: Life or death of retinal ganglion cells in vivo depends on calcium dynamics. The European Journal of Neuroscience, 25, 3339–3346. Ptito, M., Fumal, A., de Noordhout, A. M., Schoenen, J., Gjedde, A., & Kupers, R. (2008). TMS of the occipital cortex induces tactile sensations in the fingers of blind Braille readers. Experimental Brain Research, 184, 193–200. Ptito, M., & Kupers, R. (2005). Cross-modal plasticity in early blindness. Journal of Integrative Neuroscience, 4, 479–488. Radvay, X., Duhoux, S., Koenig-Supiot, F., & Vital-Durand, F. (2007). Balance training and visual rehabilitation of agerelated macular degeneration patients. Journal of Vestibular Research, 17, 183–193. Raemaekers, M., Bergsma, D. P., van Wezel, R. J., van der Wildt, G. J., & van den Berg, A. V. (2011). Effects of vision restoration training on early visual cortex in patients with

cerebral blindness investigated with functional magnetic resonance imaging. Journal of Neurophysiology, 105, 872–882. Ramrattan, R. S., Wolfs, R. C. W., Panda-Jonas, S., Jonas, J. B., Bakker, D., Pols, H. A., et al. (2001). Prevalence and causes of visual field loss in the elderly and associations with impairment in daily functioning. Archives of Ophthalmology, 119, 1788–1794. Raninen, A., Vanni, S., Hyvärinen, L., & Näsänen, R. (2007). Temporal sensitivity in a hemianopic visual field can be improved by long-term training using flicker stimulation. Journal of Neurology, Neurosurgery and Psychiatry, 78, 66–73. Rausch, M., Widdig, W., Eysel, U. T., Penner, I. K., & Tegenthoff, M. (2000). Enhanced responsiveness of human extra visual areas to photic stimulation in patients with severely reduced vision. Experimental Brain Research, 135, 34–40. Reinhard, J., Schreiber, A., Schiefer, U., Kasten, E., Sabel, B. A., Kenkel, S., et al. (2005). Does visual restitution training change absolute homonymous visual field defects? A fundus controlled study. The British Journal of Ophthalmology, 89, 30–35. Riddoch, G. (1917). Dissociation of visual perceptions due to occipital injuries with special reference to appreciation of movement. Brain, 40, 15–57. Riesen, A. H. (1961). Excessive arousal effects of stimulation after early sensory deprivation. In P. Solomon, P. E. Kubzansky, P. H. Leiderman, J. Mendalson & D. Wexler (Eds.), Sensory deprivation (pp. 34–40). Cambridge, MA: Harvard University Press. Romano, J. G., Schulz, P., Kenkel, S., & Todd, D. P. (2008). Visual field changes after a rehabilitation intervention: Vision restoration therapy. Journal of the Neurological Sciences, 273, 70–74. Romei, V., Brodbeck, V., Michel, C., Amedi, A., PascualLeone, A., & Thut, G. (2008a). Spontaneous fluctuations in posterior alpha-band EEG activity reflect variability in excitability of human visual areas. Cerebral Cortex, 18, 2010–2018. Romei, V., Rihs, T., Brodbeck, V., & Thut, G. (2008b). Resting electroencephalogram alpha-power over posterior sites indexes baseline visual cortex excitability. Neuroreport, 19, 203–208. Roth, T., Sokolov, A. N., Messias, A., Roth, P., Weller, M., & Trauzettel-Klosinski, S. (2009). Comparing explorative saccade and flicker training in hemianopia: A randomized controlled study. Neurology, 72, 324–331. Rousseau, V., Engelmann, R., & Sabel, B. A. (1999). Restoration of vision III: Soma swelling dynamics predicts neuronal death or survival after optic nerve crush in vivo. Neuroreport, 10, 3387–3391. Rousseau, V., & Sabel, B. A. (2001). Restoration of vision IV: Compensatory soma size increase of retinal ganglion cells predicts recovery of vision after partial optic nerve crush. Restorative Neurology and Neuroscience, 18, 177–189.

260 Russ, M. O., Cleff, U., Lanfermann, H., Schalnus, R., Enzensberger, W., & Kleinschmidt, A. (2002). Functional magnetic resonance imaging in acute unilateral optic neuritis. Journal of Neuroimaging, 12, 339–350. Sabel, B. A. (1997). Unrecognized potential of surviving neurons: Within-systems plasticity, recovery of function and the hypothesis of the minimal residual structure. The Neuroscientist, 3, 366–370. Sabel, B. A. (1999). Restoration of vision I: Neurobiological mechanisms of restoration and plasticity after brain damage— A review. Restorative Neurology and Neuroscience, 15, 177–200. Sabel, B. A., Fedorov, A. B., Hermann, C., & Gall, C. (2010). Restoration of vision. Non-invasive current stimulation for the treatment of visual field defects in optic neuropathy a single–center prospective trial, World Ophthalmology Congress, Berlin, Germany. June 5–9. Sabel, B. A., & Kasten, E. (2000). Restoration of vision by training of residual functions. Current Opinion in Ophthalmology, 11, 430–436. Sabel, B. A., Kasten, E., & Kreutz, M. R. (1997). Recovery of vision after partial visual system injury as a model of postlesion neuroplasticity. Advances in Neurology, 73, 251–276. Sabel, B. A., Kenkel, S., & Kasten, E. (2004). Vision restoration therapy (VRT) efficacy as assessed by comparative perimetric analysis and subjective questionnaires. Restorative Neurology and Neuroscience, 22, 399–420. Sabel, B. A., Sautter, J., Stoehr, T., & Siliprandi, R. A. (1995). A behavioural model of excitotoxicity: Retinal degeneration, loss of vision, and subsequent recovery after intraocular NMDA administration in adult rats. Experimental Brain Research, 106, 93–105. Sabel, B. A., & Trauzettel-Klosinski, S. (2005). Improving vision in a patient with homonymous hemianopia. Journal of Neuroophthalmology, 25, 143–149. Sadato, N. (2005). How the blind “see” Braille: Lessons from functional magnetic resonance imaging. The Neuroscientist, 11, 577–582. Sahraie, A., MacLeod, M. J., Trevethan, C. T., Robson, S. E., Olson, J. A., Callaghan, P., et al. (2010). Improved detection following Neuro-Eye Therapy in patients with post-geniculate brain damage. Experimental Brain Research, 206, 25–34. Sahraie, A., Trevethan, C. T., MacLeod, M. J., Murray, A. D., Olson, J. A., & Weiskrantz, L. (2006). Increased sensitivity after repeated stimulation of residual spatial channels in blindsight. Proceedings of the National Academy of Sciences of the United States of America, 103, 14971–14976. Sakaguchi, H., Fujikado, T., Kanda, H., Osanai, M., Fang, X. Y., Nakauchi, K., et al. (2004). Electrical stimulation with a needle type electrode placed into the optic nerve in rabbit eyes. Japanese Journal of Ophthalmology, 48, 552–557. Sakaguchi, H., Kamei, M., Fujikado, T., Yonezawa, E., Ozawa, M., Cecilia-Gonzalez, C., et al. (2009). Artificial vision by direct optic nerve electrode (AV-DONE)

implantation in a blind patient with retinitis pigmentosa. Journal of Artificial Organs, 12, 206–209. Sale, A., De Pasquale, R., Bonaccorsi, J., Pietra, G., Oliveri, D., Berardi, N., et al. (2010). Visual perceptual learning induces LTP in the visual cortex. Neuroscience, 172, 219–225. Sale, A., Maya Vetencourt, J. F., Medini, P., Cenni, M. C., Baroncelli, L., De Pasquale, R., et al. (2007). Environmental enrichment in adulthood promotes amblyopia recovery through a reduction of intracortical inhibition. Nature Neuroscience, 10, 679–681. Sanders, M. D., Warrington, E. K., Marshall, J., & Weiskrantz, L. (1974). ‘Blindsight’: Vision in a field defect. Lancet, 1, 707–708. Sautter, J., & Sabel, B. A. (1993). Recovery of brightness discrimination in adult rats despite progressive loss of retrogradely labelled retinal ganglion cells after controlled optic nerve crush. The European Journal of Neuroscience, 5, 680–690. Sautter, J., Schwartz, M., Duvdevani, R., & Sabel, B. A. (1991). GM1 ganglioside treatment reduces visual deficits after graded crush of the rat optic nerve. Brain Research, 565, 23–33. Schadow, J., Dettler, N., Paramei, G. V., Lenz, D., Fründ, I., Sabel, B. A., et al. (2009). Impairments of Gestalt perception in the intact hemifield of hemianopic patients are reflected in gamma-band EEG activity. Neuropsychologia, 47, 556–568. Schendel, K., & Robertson, L. C. (2004). Reaching out to see: Arm position can attenuate human visual loss. Journal of Cognitive Neuroscience, 16, 935–943. Schmidt, E. M., Bak, M. J., Hambrecht, F. T., Kufta, C. V., Rourke, D. K. O., & Vallabhanath, P. (1996). Feasibility of a visual prosthesis for the blind based on intracortical microstimulation of the visual cortex. Brain, 119, 507–522. Schmielau, F., & Wong, E. K. Jr. (2007). Recovery of visual fields in brain-lesioned patients by reaction perimetry treatment. Journal of Neuroengineering and Rehabilitation, 4, 31. Schmitt, U., & Sabel, B. A. (1996a). Basic fibroblast growth factor treatment protects from visual deficits but does not increase retinal ganglion cell survival following controlled optic nerve crush. Restorative Neurology and Neuroscience, 10, 61–68. Schmitt, U., & Sabel, B. A. (1996b). MK-801 increases retinal ganglion cell death but reduces visual deficits after controlled optic nerve crush. Journal of Neurotrauma, 13, 791–800. Schmitt, U., Sabel, B. A., Cross, R., Samson, F. E., & Pazdernik, T. L. (1998). Recovery of metabolic activity in retinofugal targets after traumatic optic nerve injury is independent of retinofugal input. Restorative Neurology and Neuroscience, 13, 153–161. Schneider, G. E. (1969). Two visual systems. Brain mechanisms for localization and discrimination are dissociated by tectal and cortical lesions. Science, 163, 895–902. Schoenfeld, M. A., Noesselt, T., Poggel, D., Tempelmann, C., Hopf, J.-M., Woldorff, M. G., et al. (2002). Analysis of pathways mediating preserved vision after striate cortex lesions. Annals of Neurology, 52, 814–824.

261 Schuchard, R. A., & Fletcher, D. C. (1994). Preferred retinal locus. A review with applications in low vision rehabilitation. Ophthalmology Clinics of North America, 7, 243–256. Schumacher, E. H., Jacko, J. A., Primo, S. A., Main, K. L., Moloney, K. P., Kinzel, E. N., et al. (2008). Reorganization of visual processing is related to eccentric viewing in patients with macular degeneration. Restorative Neurology and Neuroscience, 26, 391–402. Segraves, M. A., Goldberg, M. E., Deng, S., Bruce, C. J., Ungerleider, L. G., & Mishkin, M. (1987). The role of striate cortex in the guidance of eye movement in the monkey. The Journal of Neuroscience, 7, 3040–3058. Smirnakis, S. M., Brewer, A. A., Schmid, M. C., Tolias, A. S., Schüz, A., Augath, M., et al. (2005). Lack of long-term cortical reorganization after macaque retinal lesions. Nature, 435, 300–307. Smith, D. C., & Holdefer, R. N. (1985). Binocular competitive interactions and recovery of visual acuity in long-term monocularly deprived cats. Vision Research, 25, 1783–1794. Sober, A. J., Stark, J. M., Yamasaki, D. S., & Lytton, W. W. (1997). Receptive field changes after strokelike cortical ablation: A role for activation dynamics. Journal of Neurophysiology, 78, 3438–3443. Somers, D. C., Dale, A. M., Seiffert, A. E., & Tootell, R. B. H. (1999). Functional MRI reveals spatially specific attentional modulation in human primary visual cortex. Proceedings of the National Academy of Sciences of the United States of America, 96, 1663–1668. Spear, P. D., & Barbas, H. (1975). Recovery of pattern discrimination ability in rats receiving serial or one-stage visual cortex lesions. Brain Research, 94, 337–346. Spear, P. D., & Ganz, L. (1975). Effects of visual cortex lesions following recovery from monocular deprivation in the cat. Experimental Brain Research, 23, 181–201. Sprague, J. M. (1966). Interaction of cortex and superior colliculus in mediation of visually guided behavior in the cat. Science, 153, 1544–1547. Stein, D. G., & Weinberg, D. (1978). Impairment and recovery of visual functions after bilateral lesions of superior colliculus. Physiology & Behavior, 20, 323–329. Stoerig, P. (1993). Sources of blindsight. Science, 26, 493–494. Stoerig, P. (2006). Blindsight, conscious vision, and the role of primary visual cortex. Progress in Brain Research, 155, 217–234. Stoerig, P. (2008). Functional rehabilitation of partial cortical blindness? Restorative Neurology and Neuroscience, 26, 291–303. Stoerig, P., & Cowey, A. (1997). Blindsight in man and monkey. Brain, 120, 535–559. Tan, C. S., & Sabel, B. A. (2006). Dynamic changes in visual acuity as the pathophysiologic mechanism in Charles Bonnet syndrome (visual hallucinations). European Archives of Psychiatry and Clinical Neuroscience, 256, 62–63.

Tan, C. S., Sabel, B. A., & Goh, K. Y. (2006). Visual hallucinations during visual recovery after central retinal artery occlusion. Archives of Neurology, 63, 598–600. Teuber, H.-L. (1975). Effects of focal brain injury on human behavior. Nervous System, 2, 457–480. Teuber, H.-L., Battersby, W. S., & Bender, M. B. (1960). Visual field defects after penetrating missile wounds of the brain. Cambridge, MA: Harvard University Press. Tieman, S. B., & Hirsch, H. V. (1983). Removal of the moreexperienced eye decreases visual field deficits in cats reared with unequal alternative monocular exposure. Brain Research, 271, 170–173. Toosy, A. T., Hickman, S. J., Miszkiel, K. A., Jones, S. J., Plant, G. T., Altmann, D. R., et al. (2005). Adaptive cortical plasticity in higher visual areas after acute optic neuritis. Annals of Neurology, 57, 622–633. Toosy, A. T., Werring, D. J., Bullmore, E. T., Plant, G. T., Barker, G. J., Miller, D. H., et al. (2002). Functional magnetic resonance imaging of the cortical response to photic stimulation in humans following optic neuritis recovery. Neuroscience Letters, 330, 255–259. Treisman, A., & Gelade, G. (1980). A feature-integration theory of attention. Cognitive Psychology, 12, 97–136. Urbaitis, J. C., & Meikle, T. H., Jr., (1968). Relearning a darklight discrimination by cats after cortical and collicular lesions. Experimental Neurology, 20, 295–311. Van Essen, D. C., & Maunsell, J. H. R. (1980). Two-dimensional maps of the cerebral cortex. The Journal of Comparative Neurology, 191, 255–281. Van Hoff-van Duin, J. (1976). Development of visuomotor behavior in normal and dark-reared cats. Brain Research, 104, 233–241. Von Monakow, C. V. (1914). Die Lokalisation in der Großhirnrinde und der Abbau der Funktion durch kortikale Herde. Wiesbaden: J.F. Bergmann. Von Noorden, G. K. (1981). New clinical aspects of stimulus deprivation amblyopia. American Journal of Ophthalmology, 92, 416–421. Voronin, L., Byzov, A., Kleschevnikov, A., Kozhemyakin, M., Kuhnt, U., & Volgushev, M. (1995). Neurophysiological analysis of long-term potentiation in mammalian brain. Behavioural Brain Research, 66, 45–52. Waleszczyk, W. J., Wang, C., Young, J. M., Burke, W., Calford, M. B., & Dreher, B. (2003). Laminar differences in plasticity in area 17 following retinal lesions in kittens or adult cats. The European Journal of Neuroscience, 17, 2351–2368. Wallace, S. F., Rosenquist, A. C., & Sprague, J. M. (1989). Recovery from cortical blindness mediated by destruction of nontectal fibers in the commissure of the superior colliculus in the cat. The Journal of Comparative Neurology, 284, 429–450.

262 Watson, G. R., Schuchard, R. A., De l`Aune, W. R., & Watkins, E. (2006). Effects of preferred retinal locus placement on text navigation and development of advantageous trained retinal locus. Journal of Rehabilitation Research and Development, 43, 761–770. Weiskrantz, L. (1996). Blindsight revisited. Current Opinion in Neurobiology, 6, 215–220. Weiskrantz, L. (2009). Is blindsight just degraded normal vision? Experimental Brain Research, 192, 413–416. Weiskrantz, L., Warrington, E. K., Sanders, M. D., & Marshall, J. (1974). Visual capacity in the hemianopic field following a restricted occipital ablation. Brain, 97, 709–728. Werring, D. J., Bullmore, E. T., Toosy, A. T., Miller, D. H., Barker, G. J., MacManus, D. G., et al. (2000). Recovery from optic neuritis is associated with a change in the distribution of cerebral response to visual stimulation: A functional magnetic resonance imaging study. Journal of Neurology, Neurosurgery and Psychiatry, 68, 441–449. Werth, R. (2008). Cerebral blindness and plasticity of the visual system in children. A review of visual capacities in patients with occipital lesions, hemispherectomy or hydranencephaly. Restorative Neurology and Neuroscience, 26, 377–389. Werth, R., & Moehrenschlager, M. (1999). The development of visual functions in cerebrally blind children during a systematic visual field training. Restorative Neurology and Neuroscience, 15, 229–241. Werth, R., & Seelos, K. (2005). Restitution of visual functions in cerebrally blind children. Neuropsychologia, 43, 2011–2023. Wessinger, C. M., Fendrich, R., & Gazzaniga, M. S. (1997). Islands of residual vision in hemianopic patients. Journal of Cognitive Neuroscience, 9, 203–221. Wick, B., Wingard, M., Cotter, S., & Scheiman, M. (1992). Anisometropic amblyopia: Is the patient ever too old to treat? Optometry and Vision Science, 69, 866–878. Widdig, W., Pleger, B., Rommel, O., Malin, J. P., & Tegenthoff, M. (2003). Repetitive visual stimulation: A neuropsychological approach to the treatment of cortical blindness. NeuroRehabilitation, 18, 227–237. Wiesel, T. N., & Hubel, D. H. (1965). Extent of recovery from the effects of visual deprivation in kittens. Journal of Neurophysiology, 28, 1060–1072. Wolford, G., Marchak, F., & Hughes, H. (1988). Practice effects in backward masking. Journal of Experimental Psychology. Human Perception and Performance, 14, 101–112.

Wood, C. C., Spear, P. D., & Braun, J. J. (1974). Effects of sequential lesions of suprasylvian gyri and visual cortex on pattern discrimination in the cat. Brain Research, 66, 443–466. Wörgötter, F., Suder, K., Zhao, Y., Kerscher, N., Eysel, U. T., & Funke, K. (1998). State-dependent receptive-field restructuring in the visual cortex. Nature, 396, 165–168. Wüst, S., Kasten, E., & Sabel, B. A. (2002). Blindsight after optic nerve injury indicates functionality of spared fibers. Journal of Cognitive Neuroscience, 14, 243–253. Young, J. M., Waleszczyk, W. J., Burke, W., Calford, M. B., & Dreher, B. (2002). Topographic reorganization in area 18 of adult cats following circumscribed monocular retinal lesions in adolescence. Journal of Physiology, London, 541, 601–612. Young, J. M., Waleszczyk, W. J., Wang, C., Calford, M. B., Dreher, B., & Obermayer, K. (2007). Cortical reorganization consistent with spike timing-but not correlation-dependent plasticity. Nature Neuroscience, 10, 887–895. Zaehle, T., Rach, S., & Herrmann, C. S. (2010). Transcranial alternating current stimulation enhances individual alpha activity in human EEG. PLoS ONE, 5, e13766. Zee, D. S., Butler, P. H., Optican, L. M., Tusa, R. J., & Gücer, G. (1983). Effects of bilateral occipital lobectomies on eye movements in monkeys: Preliminary observations. In A. Roucoux & M. Crommelink (Eds.), Physiological and pathological aspects of eye movements (pp. 225–232). Boston: Junk. Zeki, S., & Ffytche, D. H. (1998). The Riddoch syndrome: Insights into the neurobiology of conscious vision. Brain, 121, 24–45. Zhang, X., Kedar, S., Lynn, M. J., Newman, N. J., & Biousse, V. (2006). Natural history of homonymous hemianopia. Neurology, 66, 901–905. Zihl, J., Pöppel, E., & von Cramon, D. (1977). Diurnal variation of visual field size in patients with postretinal lesions. Experimental Brain Research, 27, 245–249. Zihl, J., & von Cramon, D. (1979). Restitution of visual function in patients with cerebral blindness. Journal of Neurology, Neurosurgery and Psychiatry, 42, 312–322. Zihl, J., & von Cramon, D. (1985). Visual field recovery from scotoma in patients with postgeniculate damage. A review of 55 cases. Brain, 108, 335–365. Zur, D., & Ullman, S. (2003). Filling-in of retinal scotomas. Vision Research, 43, 971–982.

A. M. Green, C. E. Chapman, J. F. Kalaska and F. Lepore (Eds.) Progress in Brain Research, Vol. 192 ISSN: 0079-6123 Copyright Ó 2011 Elsevier B.V. All rights reserved.

CHAPTER 14

Real-time functional magnetic imaging—brain–computer interface and virtual reality: promising tools for the treatment of pedophilia Patrice Renaud{,*, Christian Joyal{, Serge Stoleru}, Mathieu Goyette}, Nikolaus Weiskopfk and Niels Birbaumer# { {

Université du Québec en Outaouais, Institut Philippe-Pinel de Montréal, Montréal, Québec, Canada Université du Québec à Trois-Rivières, Institut Philippe-Pinel de Montréal, Montréal, Québec, Canada } INSERM 669, Paris, France } Institut Philippe-Pinel de Montréal, Université de Montréal, Montréal, Québec, Canada k University College London, London, United Kingdom # University of Tübingen, Tübingen, Germany

Abstract: This chapter proposes a prospective view on using a real-time functional magnetic imaging (rt-fMRI) brain–computer interface (BCI) application as a new treatment for pedophilia. Neurofeedback mediated by interactive virtual stimuli is presented as the key process in this new BCI application. Results on the diagnostic discriminant power of virtual characters depicting sexual stimuli relevant to pedophilia are given. Finally, practical and ethical implications are briefly addressed. Keywords: fmri bci; virtual reality; pedophilia; eye-tracking; penile plethysmography; interactivity. Pedophilia is a psychiatric disorder of acting-out. Sex offenders in general show deeply entrenched cognitive and affective deficits specifically related to their sexual disorders. Sexually deviant behaviors of the latter furthermore appear to be strongly driven by critical proximate environments. These clinical features of pedophilia require the

development of therapeutics addressing the issues of impulsivity, self-regulation, and voluntary control. Recent developments in behavioral neurobiology, psychology, and human–computer engineering are indeed leading to potential new treatments of pedophilia that would be based on BCI. Pedophilia is a criminal sexual deviance commonly found among the paraphilias examined in forensic psychiatry. Although its prevalence is difficult to quantify, victims of pedophilia are numerous. Trocmé et al. (1994), for example, have

*Corresponding author. Tel.: 819 595 3900, poste 4412 E-mail: [email protected] DOI: 10.1016/B978-0-444-53355-5.00014-2

263

264

reported a sexual abuse incidence rate of 1.57 cases per 1000 children in 1993 in Ontario (Canada). Psychological and financial costs are also important. For instance, a recent economic study on the cost of crime and delinquency in France, estimated that the total cost of rape and other sexual offenses amounts to 4.1 billion Euros per year. As more than half of sexual offenses are committed on minors, the cost of pedophilia is colossal, not only financially but also in terms of public health. Still, therapeutic approaches to this problem are notoriously far from yielding satisfactory results, pedophilia is among the most difficult mental disorder to treat, and recidivism rates are high (Hanson and Morton-Bourgon, 2009). In the only psychological study that used randomized assignment of patients to a treated or to a nontreated control group (Marques et al., 2005), no significant differences in the rate of sexual recidivism were found between the two groups at various time points over a 12-year long follow-up. The goal of this chapter is to present a promising avenue to evaluate and treat pedophilia: a functional magnetic resonance imaging (fMRI) BCI neurofeedback approach using virtual dynamic environments. The diagnosis of pedophilia is based on three main criteria: (a) the recurrence over a period of at least 6 months of intense sexual fantasies, sexual urges, or behaviors involving sexual activity toward prepubescent children (generally under 13 years of age); (b) the presence of fantasies, sexual urges, or behaviors that cause significant psychological distress or personal difficulties to the patient in the accomplishment of his activities; (c) the requirement that the aggressor be at least 16 years old and at least 5 years older than the victim (APA, 2000; 302.2). The etiology of sexual deviance is complex and multifactorial, including such factors as emotional regulation problems, cognitive distortions, and social difficulties, which interact synergistically to trigger the acting-out (Hanson and Harris, 2000; Marshall, 1989; Proulx et al., 1999; Thornton, 2002; Ward and Beech, 2004, 2006). However, deviant sexual interest represents the strongest determining

factor in recidivism among sexual offenders in general and among pedophiles in particular (Hanson and Bussière, 1998; Hanson and Morton-Bourgon, 2005). According to the “integrated theory of sexual offending” (ITSO) proposed by Ward and Beech (2006), neuroanatomical and neuropsychological predispositions should also be considered in the understanding of self-regulation of offending (Joyal et al., 2007; Pennington, 2002; Redouté et al., 2000; Stoléru et al., 1999). Selfregulation of offending, which is at the core of the problem, is based on a combination of internal (e.g., neurological dysfunctions, anxiety, and cognitive distortions) and external (environmental) processes that drive the person to manifest (or not) the goal-oriented behavior (i.e., to act out or not; Ward and Beech, 2006; Ward and Hudson, 1998). Voluntarily or automatically, this behavioral regulation process, which is conducted overtime and in different contexts, manifests itself through the modulation of perceptual-motor processes and attention (Baumeister and Heatherton, 1996; Karoly, 1993; Renaud et al., 2010a,b; Ward and Hudson, 1998). The concept of self-regulation applied to sexual delinquents is used to explain the dynamic variations in intentionality of these individuals, which in turn reflects the existing hierarchical relations between sexual motivation, inhibiting processes, deviant sexual arousal, and the acting out itself. If self-regulation deficits and neurobiological anomalies are associated with pedophilia, neurofeedback might serve as a treatment. The goal of the approach would be to modulate through conditioning abnormal activation of specific brain regions associated with pedophilia. With the emergence of neuroimaging studies characterizing pedophilia, this avenue is now achievable. Preliminary neurofeedback data are promising among sexual offenders, although they are limited at this stage to electroencephalography (EEG) used with rapists and not applied to pedophilia as such (Giovannoni, 2010). Because pedophilia, or more precisely sexual arousal toward children, is based on complex circuits involving specific cortical

265

regions and subcortical structures, fMRI BCI is needed. To achieve this goal, the first step is to describe the neurological determinants of sexual arousal toward children.

Functional neuroanatomy of deviant sexual arousal and pedophilia Because pedophilia is among the most difficult mental disorder to treat (e.g., Grossman et al., 1999; Hanson and Morton-Bourgon, 2009), recent evidence-based interventions abandoned the idea of curing pedophilia and put emphasis on increasing voluntary control over sexual arousal and enhancing self-management skills (e.g., Seto, 2009). In line with self-regulation deficits, anomalies of the so-called action selection and control system are at the center of neuropsychological theories of pedophilia (e.g., Joyal et al., 2007; Ward and Beech, 2006). Worth of noting is the fact that cognitive inhibition capacities depend heavily on frontostriatal neural loops, which are defective both in pedophilia and obsessive–compulsive disorders (e.g., Breiter et al., 1996; Schiffer et al., 2009). More specifically, cortical and subcortical regions associated with sexual arousal and inhibition (e.g., the frontal cortex, including the anterior cingulate cortex (ACC), the caudate nucleus, and the amygdala) are abnormally activated in pedophiles (Sartorius et al., 2008; Schiffer et al., 2008; Schiltz et al., 2007). Among these regions, the ACC might be targeted as a region of interest (ROI) for modulation with neurofeedback because it is consistently involved not only in obsessive–compulsive disorders but also in sexual arousal, both among the general population (Arnow et al., 2002; Beauregard et al., 2001; Ferretti et al., 2005; Joyal et al., 2007; Karama et al., 2002; Mouras et al., 2003; Redouté et al., 2000; Stoléru et al., 1999) and pedophiles (Schiffer et al., 2008). In pedophiles, the ACC activation is comparable to that of nonpedophile males in response to sexually arousing stimuli when the stimuli are equally

arousing and adapted for each group (sexually stimulating photographs of children and adults, respectively; Schiffer et al., 2008; see also Stoléru and Mouras, 2007). The ACC has a central role in the widely accepted neurobehavioral model of sexual arousal proposed by Serge Stoleru, one of us (Redouté et al., 2000; Stoléru et al., 1999). It is viewed as a motivational component related to motor preparation, autonomic processes, and the sexual response. The ACC is indeed known to play a role in autonomic functions (e.g., blood pressure and heart rate regulation), and cognitive functions such as reward anticipation, decision making, empathy, and emotion (e.g., Allman et al., 2001). Thus, using fMRI, neurofeedback could be used to modulate (diminish) the ACC activation associated with sexual arousal among pedophiles.

Brain–computer interfaces BCI allow two-way communication between the brain and a computer (or an external device such as an artificial limb). EEG-based BCI is known for decades, being first used to reduce impulsivity among persons with attentional deficit and hyperactivity disorder (ADHD; Lubar and Shouse, 1976). Niels Birbaumer, one of us, also used EEG neurofeedback to allow completely paralyzed persons to drive an electronic spelling device through voluntary modulation of their slow cortical potentials (Birbaumer et al., 1999). Since then, EEG neurofeedback has been used to treat various psychiatric symptoms, including social cognition in persons with autism (Jarusiewicz, 2002), obsessions and compulsions (Hammond, 2002, 2003), and, above all, impulsivity and ADHD (see Arns et al., 2009 for a metaanalysis). EEG-based BCI, however, only allow the modulation of brain waves and large cortical areas. In order to modulate specific and/or subcortical structures, fMRI BCI is needed. fMRI BCI can be defined as a set of closedloop system technologies with signal acquisition, signal analysis, and signal feedback as its major

266

Signal acquisition: fMRI, PPG, eye-tracking

Virtual Interactive Sexual Stimulus Pedophile patient

Signal processing and analysis

BCI–GUI Signal feedback

Fig. 1. Subsystems of an fMRI BCI applied to pedophilia: signal acquisition, signal preprocessing, signal analysis, and signal feedback (adapted from Sitaram et al., 2008); fMRI, functional magnetic imaging; PPG, penile plethysmography; BCI–GUI, brain–computer interface–graphic user interface.

components (Fig. 1; Sitaram et al., 2008, 2009). fMRI BCI is a real-time application of fMRI (rtfMRI) using neurofeedback, that is, a feedback of neural metabolic activity brought to the attention of the human recipient through the use of an interactive graphic display, to condition area specific brain processes. Fluctuations in blood oxygen level dependent (BOLD) responses are taken as measures of such neural metabolic activity. A growing number of clinical applications are associated with fMRI BCI, although it is still mainly limited to motor or basic learning functions (see deCharms, 2008; Sitaram et al., 2007 for reviews). In fMRI BCI applications, brain ROI may be selected beforehand from theoretical considerations or following a functional localizer strategy (Weiskopf et al., 2007). Functional localizers are functional acquisition scans that are performed in order to determine the precise location, in each patient considered individually,

of the target brain area that will be trained subsequently. Thus, sexual arousal can either be treated by modulating the activation of brain structures known to be associated (e.g., the ACC) or by modulating brain structures that were activated for this individual in particular when sexually aroused. fMRI BCI as a conditioning device relies on a set of operant or instrumental conditioning principles that were first developed by Skinner (1953; Skinner and Campbell, 1947) and that hark back to Thorndike's Law of Effect (Birbaumer, 2006). These principles essentially revolve around the possibility of establishing contingencies between the active behaviors of a living organism and sequences of stimuli-events consequently delivered from the environment to the latter. In the guise of reinforcements or punishments, either positive or negative, these contingencies act to increase or diminish the probability of the occurrence of specific behaviors. As applied to fMRI BCI, these

267

principles may be used in conjunction with mental strategies (covertly repeated cognitive contents) to condition the metabolic activity of a single ROI or networks of ROIs (deCharms, 2008; Johnston et al., 2010; Sitaram et al., 2009). In turn, these changes of brain activity brought about by operant conditioning translate into changes in motor, emotional, and cognitive responses. In recent years, fMRI BCI has emerged as a promising tool to address mental health problems as well as psychiatric disorders. Self-regulation of emotion and language processing through volitional control of the metabolic activity of the ACC (deCharms et al., 2004, 2005; Weiskopf et al., 2003), anterior insula (Caria et al., 2007; Sitaram et al., 2007), and right inferior frontal gyrus (Rota et al., 2009) opens the door to numerous potential therapeutic applications, and notably to new criminal rehabilitation methods in forensic psychiatry (Karim et al., 2010; Sitaram et al., 2009). Let's now turn toward the question of how fMRI BCI might be used to address the topics of pedophilia.

fMRI BCI applied to pedophilia: a prospective view This section presents a prospective view on how fMRI BCI might be used to assist therapeutic processes aiming at modifying deviant sexual

interests in pedophile patients. This view is based on the neurophysiology of deviant sexuality, the cognitive behavioral therapy (CBT) approach to treat pedophilia, and virtual 3D technology to assess pedophilia. The main purpose of such an fMRI BCI system is to help pedophile patients develop voluntary control of the activity of brain structures involved in deviant sexual arousal as well as in cognitive and motor intentionality leading to sexual acting-out. More specifically, the procedure should aim at training patients in lowering their ACC activation while being exposed to the presence of sexual stimuli morphologically and behaviorally simulating prepubescent children. This ACC BOLD signal learned modulation should furthermore be achieved from a device whose feedback delivery would be mediated by a virtual interactive sexual stimulus (VISS), the behavior of which would have to be made contingent upon the patients’ ACC activation (see Figs. 1 and 2). VISS are synthetic and interactive 3D characters reproducing the physical appearance and behaviors of naked individuals. In an fMRI BCI system developed to train pedophiles, the VISS are graphic simulations of prepubescent individuals. VISS are designed in this fMRI BCI application to serve simultaneously as a means to induce sexual arousal and as a medium to convey brain conditioning. Furthermore, because they are plastic and malleable,

Fig. 2. Caucasian avatars used in Renaud et al. (2009, 2010b) and Goyette et al. (2010); 6- to 7-year-old characters; 10- to 12-year-old characters; adolescents and adults (courtesy of BehaVR Inc.).

268

VISS could be made to fit idiosyncrasies and specific sexual interests (Renaud et al., 2010a). In addition to MRI technologies, the proposed fMRI BCI application to treat pedophilia should comprise two other psychophysiological inputs, that is, measures of penile tumescence and gaze behavior relative to the VISS (see signal acquisition in Fig. 1). These additional measurements would, respectively, allow for the validating of sexual arousal induction and the controlling of overt attention during the procedure. Volumetric penile plethysmography (PPG) measures variations of blood volume in the penis and is used particularly to assess sexual arousal (Freund, 1965; Laws and Marshall, 2003). Volumetric PPG components consist of a glass cylinder that fits over the penis and an inflatable cuff installed at the base of the penis to isolate the air inside the cylinder from the outside atmosphere (Stoléru et al., 1999). As mentioned earlier, deviant sexual interest as measured by PPG represents the strongest predictive factor for recidivism among sexual offenders (Hanson and Bussière, 1998; Hanson and Morton-Bourgon, 2005). Standardized sexual stimuli, either audio tapes or slides, are usually used to prompt sexual arousal during PPG-based clinical assessments of sex offenders. Ocular saccades and fixations can be picked up using eye-tracking (ET) techniques to monitor the perceptual content actively entertained by the patient put in presence of the VISS. This is an important precaution to insure the validity of this particular type of fMRI BCI procedure because visual avoidance of sexual stimuli represents a typical strategy used by pedophile patients to falsify their PPG results during clinical assessment of sexual preferences (Kalmus and Beech, 2005; Renaud, 2006; Renaud et al., 2009, 2010a,b). This control strategy based on visual avoidance might easily distort the results expected from the application described in Fig. 1, simply by giving way to inhibitions of ACC activation and erectile response based on not taking into account critical aspects of the VISS, that is,

without any learning effects. This combining of fMRI BCI with ET may furthermore gives access to a sharper analysis of the perceptual-motor trajectories meshed with the targeted ROI's activation (Righi et al., 2010). As depicted in Fig. 1, fMRI, PPG, and ET signals will be combined together via signal processing and analysis and BCI–GUI (graphic user interface) subsystems to deliver psychophysiologically driven feedback as enacted by the VISS interactive behavior. For instance, a patient showing a systematic visual avoidance strategy when his PPG signal is on the rise could be presented with a modified VISS animation; the interactive character could move toward the visual avoidance or display a novel more attention attracting sequence of behaviors. The VISS needed by the proposed fMRI BCI application have to display the required anthropometric properties, that is, the right body proportions and behaviors to accurately simulate juvenile attributes. Figure 2 shows virtual characters that were used in virtual immersion using PPG and ET measurements to assess sexual preferences among pedophile patients (Goyette et al., 2010; Renaud et al., 2009, 2010b). Figure 3 displays results of a receiver operating characteristic (ROC)1 analysis showing the discriminating value of these sexual stimuli for diagnostic

1 Receiver ROC analysis is a nonparametric test based on Z distributions that is in increasing use in the field of sexual aggression for determining the accuracy of classification given by instruments measuring sexual arousal and risks of recidivism (Marshall and Fernandez, 2003). The analysis is based on specificity (Sp) and sensitivity (Se); two complementary dimensions of a good diagnostic instrument (Allaire and Cismaru, 2007; Streiner and Carney, 2007). In the data shown in Fig. 3, Sp corresponds to the probability that VISS show a nondeviant index in a group of sexually nondeviant individuals while Se is the probability that VISS show a deviant index in a group of sexual aggressors toward children. The area under the curve (AUC) generated by the combination of Se and 1  Sp (or false positive proportion) is to be compared to the diagonal of no (or chance) information (AUC ¼ 0.5) to give us the general accuracy of the instrument.

269 1.0

Sensitivity

0.8

0.6

0.4

0.2 VISS pubescent child included VISS pubescent child excluded 0.0 0.0

0.2

0.4 0.6 1-Spécificity

0.8

1.0

Fig. 3. Receiver operating characteristic (ROC) analyses of the value of virtual interactive sexual stimulus (VISS) and circumferential penile plethysmography (PPG) to discriminate between groups of nondeviants (n ¼ 36) and pedophiles (n ¼ 22); the accuracy of the deviance indices obtained with the VISS are significantly greater than chance: area under the curve (AUC) using prepubescent and pubescent characters (AUC ¼ 0.86, Z ¼ 4.58, p < 0.001; 95% CI 0.76–0.96); using only prepubescent characters (AUC ¼ 0.85, Z ¼ 4.39, p < 0.001; 95% CI 0.74–0.96); these data show an excellent classification accuracy for VISS (Hosmer and Lemeshow, 2000).

sensitivity (Se) and specificity (Sp). These results are similar to those generally obtained with standard audio sexual stimuli (Goyette et al., 2010; Marshall and Fernandez, 2003) and show that virtual characters have the potential to generate a specific sexual arousal corresponding to the presence of past sexual behaviors toward children, an essential premise for fMRI BCI applied to pedophilia. To simulate the presence of a prepubescent child next to a pedophile patient during an fMRI BCI procedure requires a scripting of the patient–VISS interaction. As explained above, the VISS behavior will have to embody the neurofeedback aiming at reinforcing the lowering of the ACC activation. Hence, the possibility to develop an interactive scripting in which a continuous and reversible VISS behavior (e.g., to raise the arm) could be used as neurofeedback of the ACC BOLD level (e.g., the higher the arm in

the air, the lower the ACC activation level). In this specific interactive context, the patient would receive the instruction as to focus his attention and cognitive activity in order to bring about and maintain the targeted VISS behavior (e.g., to maintain the arm at its apex). Finally, as advocated by leading authors in the field, in order to favor the inhibition of the ACC activity, a strategy of covert mental rehearsal would have to be used by the patient while in fMRI BCI training (deCharms, 2008; Johnston et al., 2010; Sitaram et al., 2009). This kind of covert mental rehearsal is akin to covert sensitization, that is, a CBT currently used in the treatment of sex offenders (Cautela and Kearney, 1990; Cautela and Wisocki, 1971). The cognitive content of this mental rehearsal could, for instance, be related to the aversive repercussions of the deviant sexual behavior of the patient for

270

himself. The patient would thus have to mentally rehearse such content while monitoring his ACC neurofeedback as enacted by the VISS. Conclusion Although this prospective view of an fMRI BCI application to treat pedophilia still has to be developed and tested, the elements required for its carrying out are at hand. Because pedophilia destroys innocent victims’ life and costs tremendously to society, it is important to achieve this project and assess the potential of fMRI BCI to face efficiently this critical issue. Beyond technological and scientific considerations, the materialization of this project and its possible use in forensic clinical practice demand careful ethical guidelines. While the purpose of this chapter was not to examine these aspects, it is nevertheless important to mention that such an fMRI BCI based treatment should be used with the patient's informed consent and be part of a larger therapeutic process involving CBT techniques and a strong pledge to the empowerment of patients and the protection of young victims. Acknowledgments This chapter was made possible thanks to funding from the Institute of Neurosciences, Mental Health and Addiction (Canadian Institute of Health Research), and the Ministère des relations internationales du Québec (Programme Samuel-deChamplain).

Abbreviation ACC ADHD AUC BCI BOLD

anterior cingulate cortex attention-deficit hyperactivity disorder area under the curve brain–computer interface blood oxygen level dependent

CBT EEG ET ITSO PPG ROC ROI rt-fMRI Se Sp VISS

cognitive behavioral therapy electroencephalography eye-tracking integrated theory of sexual offending penile plethysmography receiver operating characteristic region of interest real-time functional magnetic resonance imaging sensitivity specificity virtual interactive sexual stimulus

References Allaire, J. F., & Cismaru, D. (2007). ROCTools: A statistical software for the analysis of ROC curves. Montréal, Canada: Statistics Consulting Group of Centre de recherche de l'Institut Philippe-Pinel de Montréal. Allman, J. M., Hakeem, A., Erwin, J. M., Nimchinsky, E., & Hof, P. (2001). The anterior cingulate cortex. The evolution of an interface between emotion and cognition. Annals of the New York Academy of Sciences, 935, 107–117. American Psychiatric Association. (2000). Diagnostic and Statistical Manual of Mental Disorders (4th ed.). text revision, Washington, DC. Arnow, B. A., Desmond, J. E., Banner, L. L., Glover, G. H., Solomon, A., Polan, M. L., et al. (2002). Brain activation and sexual arousal in healthy, heterosexual males. Brain, 125, 1014–1023. Arns, M., de Ridder, S., Strehl, U., Breteler, M., & Coenen, A. (2009). Efficacy of neurofeedback treatment in ADHD: The effects on inattention, impulsivity and hyperactivity: A meta-analysis. Clinical EEG and Neuroscience, 40, 179–189. Baumeister, R. F., & Heatherton, T. F. (1996). Self-regulatory failure: An overview. Psychological Inquiry, 7(1), 1–15. Beauregard, M., Lévesque, J., & Bourguoin, P. (2001). Neural correlates of the conscious regulation of emotions. Journal of Neuroscience, 21, RC165. Birbaumer, N. (2006). Breaking the silence: Brain–computer interfaces (BCI) for communication and motor control. Psychophysiology, 43, 517–532. Birbaumer, N., Ghanayim, N., Hinterberger, T., Iversen, I., Kotchoubey, B., Kübler, A., et al. (1999). A spelling device for the paralysed. Nature, 398(6725), 297–298. Breiter, H. C., Rauch, S. L., Kwong, K. K., Baker, J. R., Weisskoff, R. M., Kennedy, D. N., et al. (1996). Functional magnetic resonance imaging of symptom provocation in obsessive-compulsive disorder. Archives of General Psychiatry, 53, 595–606.

271 Caria, A., Veit, R., Sitaram, R., Lotze, M., Weiskopf, N., Grodd, W., et al. (2007). Regulation of anterior insular cortex activity using real-time fMRI. Neuroimage, 35(3), 1238–1246. Cautela, J. R., & Kearney, A. J. (1990). Behaviour analysis, cognitive therapy, and covert conditioning. Journal of Behavior Therapy and Experimental Psychiatry, 21(2), 83–90. Cautela, J. R., & Wisocki, P. (1971). Covert sensitization for the treatment of sexual deviation. Psychological Record, 21, 37–48. deCharms, R. C. (2008). Applications of real-time fMRI. Nature Reviews. Neuroscience, 9, 720–729. deCharms, R. C., Christoff, K., Glover, G. H., Pauly, J. M., Whitfield, S., & Gabrielli, D. E. (2004). Learned regulation of spatially localized brain activation using real-time fMRI. Neuroimage, 21(1), 436–443. deCharms, R. C., Maeda, F., Glover, G. H., Ludlow, D., Pauly, J. M., Soneji, D., et al. (2005). Control over brain activation and pain learned by using real-time functional MRI. Proceedings of the National Academy of Sciences of the United States of America, 102(51), 18626–18631. Ferretti, A., Caulo, M., Del Gratta, C., Di Matteo, R., Merla, A., Montorsi, F., et al. (2005). Dynamics of male sexual arousal: Distinct components of brain activation revealed by fMRI. Neuroimage, 26, 1086–1096. Freund, K. (1965). Diagnosing heterosexual paedophilia by means of a test for sexual interest. Behaviour Research and Therapy, 3(4), 229–234. Giovannoni, J. (2010). The efficacy of neurofeedback with resistant sex abusers. 29th conference of the Association for the Treatment of Sexual Abusers (ATSA), Phoenix, Arizona. Goyette, M., Rouleau, J. L., Renaud, P., Trottier, D., Longpré, N., & Barsetti, I. (2010). Animated synthetic stimuli: Profiles analysis and comparison of the classification accuracy based on penile plethysmography responses. (in preparation). Grossman, L. S., Martis, B., & Fichtner, C. G. (1999). Are sex offenders treatable? A research overview. Psychiatric Services, 50, 349–361. Hammond, D. C. (2002). Treatment of the obsessional subtype of obsessive–compulsive disorder with neurofeedback. Biofeedback, 32, 9–12. Hammond, D. C. (2003). QEEG-guided neurofeedback in the treatment of obsessive compulsive disorder. Journal of Neurotherapy, 7, 25–52. Hanson, K. R., & Bussière, M. T. (1998). Predicting relapse: A meta-analysis of sexual offender recidivism studies. Journal of Consulting and Clinical Psychology, 66(2), 348–362. Hanson, K. R., & Harris, A. J. R. (2000). Where should we intervene? Dynamic predictors of sexual offence recidivism. Criminal Justice and Behavior, 27(1), 6–35. Hanson, K. R., & Morton-Bourgon, K. (2005). The characteristics of persistent sexual offenders: A meta-analysis of recidivism studies. Journal of Consulting and Clinical Psychology, 73(6), 1154–1163.

Hanson, K. R., & Morton-Bourgon, K. E. (2009). The accuracy of recidivism risk assessments for sexual offenders: A meta-analysis of 118 prediction studies. Psychological Assessment, 21, 1–21. Hosmer, D. W., & Lemeshow, S. (2000). Applied logistic regression (pp. 1–351) (2nd ed.). New York: John Wiley and Sons. Jarusiewicz, B. (2002). Efficacy of neurofeedback for children in the autistic spectrum: A pilot study. Journal of Neurotherapy, 6, 39–49. Johnston, S. J., Boehm, D., Goebel, R., & Liden, D. E. J. (2010). Neurofeedback: A promising tool for the self-regulation of emotion networks. Neuroimage, 49, 1066–1072. Joyal, C. C., Black, D. N., & Dassilva, B. (2007). The neuropsychology and neurology of sexual deviance: A review and pilot study. Journal of Sex Abuse, 19(2), 155–173. Kalmus, E., & Beech, A. R. (2005). Forensic assessment of sexual interest: A review. Agression and Violent Behavior, 10(2), 193–217. Karama, S., Lecours, A. R., Leroux, J. M., Bourgouin, P., Beaudoin, G., Joubert, S., et al. (2002). Areas of brain activation in males and females during viewing of erotic film excerpts. Human Brain Mapping, 16, 1–13. Karim, A. A., Schneider, M., Lotze, M., Veit, R., Sauseng, P., Braun, C., et al. (2010). The truth about lying: Inhibition of the anterior prefrontal cortex improves deceptive behaviour. Cerebral Cortex, 20, 205–213. Karoly, P. (1993). Mechanisms of self-regulation: A systemps view. Annual Review Psychology, 44, 23–52. Laws, D. R., & Marshall, W. L. (2003). A brief history of behavioural and cognitive-behavioural approaches to sexual offenders, part 1, early developments. Sexual Abuse: A Journal of Research and Treatment, 15, 75–92. Lubar, J. F., & Shouse, M. N. (1976). EEG and behavioural changes in a hyperkinetic child concurrent with training of the sensorimotor rhythm (SMR): A preliminary report. Biofeedback and Self Regulation, 1, 293–306. Marques, J. K., Wiederanders, M., Day, D. M., Nelson, C., & van Ommeren, A. (2005). Effects of a relapse prevention program on sexual recidivism: Final results from California's sex offender treatment and evaluation project (SOTEP). Sexual Abuse: A Journal of Research and Treatment, 17, 79–107. Marshall, W. L. (1989). Intimacy, loneliness and sexual offenders. Behaviour Research and Therapy, 27(5), 491–503. Marshall, W. L., & Fernandez, Y. M. (2003). Sexual preferences: Are they useful in the assessment and treatment of sexual offenders? Aggression and Violent Behaviour, 8, 131–143. Mouras, H., Stoleru, S., Bittoun, J., Glutron, D., PelegriniIssac, M., Paradis, A.-L., et al. (2003). Brain processing of visual sexual stimuli in healthy men: A functional magnetic resonance imaging study. Neuroimage, 20, 855–869. Pennington, B. F. (2002). The development of psychopathology: Nature and nurture. New York: Guilford Press.

272 Proulx, J., Perreault, C., & Ouimet, M. (1999). Pathways in the offending process of extra-familial sexual child molesters. Sexual Abuse: A Journal of Research and Treatment, 11(2), 117–129. Redouté, J., Stoléru, S., Grégoire, M.-C., Costes, N., Cinotti, L., Lavenne, F., et al. (2000). Brain processing of visual sexual stimuli in human males. Human Brain Mapping, 11(3), 162–177. Renaud, P. (2006). Method for providing data to be used by a therapist for analyzing a patient behaviour in a virtual environment. U.S. Patent No. 7128577, Washington DC: U.S. Patent and Trademark Office. Renaud, P., Chartier, S., Rouleau, J.-L., Proulx, J., Trottier, D., Bradford, J. P., et al. (2009). Gaze behaviour nonlinear dynamics assessed in virtual immersion as a diagnostic index of sexual deviancy: Preliminary results. Journal of Virtual Reality and Broadcasting, 6, no. 3, urn:nbn: de:0009-6-17538, ISSN 1860–2037. Renaud, P., Goyette, M., Chartier, S., Zhornicki, S., Trottier, D., Rouleau, J.-L., et al. (2010a). Sexual affordances, perceptual-motor invariance extraction and intentional nonlinear dynamics: Sexually deviant and nondeviant patterns in male subjects. Nonlinear Dynamics, Psychology, and Life Sciences, 14(5), 463–489. Renaud, P., Rouleau, J. L., Proulx, J., Trottier, D., Goyette, M., Bradford, J. P., et al. (2010b). Virtual characters designed for forensic assessment and rehabilitation of sex offenders: Standardized and made-to-measure. Journal of Virtual Reality and Broadcasting, 7, no. 5, urn: nbn:de:0009-6-26466, issn1860-2037. Righi, G., Blumstein, S., Mertus, G., & Worden, M. S. (2010). Neural systemps underlying lexical competition: An eye tracking and fMRI study. Journal of Cognitive Neuroscience, 22(2), 213–224. Rota, G., Sitaram, R., Veit, R., Erb, M., Weiskopf, N., Dogil, G., et al. (2009). Self-regulation of regional cortical activity using real-time fMRI: The right inferior frontal gyrus and linguistic processing. Human Brain Mapping, 30, 1605–1614. Sartorius, A., Ruf, M., Kief, C., Demirakca, T., Bailer, J., Ende, G., et al. (2008). Abnormal amygdala activation profile in paedophilia. European Archives of Psychiatry and Clinical Neuroscience, 258, 271–277. Schiffer, B., Paul, T., Gizewski, E., Forsting, M., Leygraf, N., Schedlowski, M., et al. (2008). Functional brain correlates of heterosexual paedophilia. Neuroimage, 41, 80–91. Schiffer, B., Peschel, T., Paul, T., Gizewski, E., Forsting, M., Leygraf, N., et al. (2009). Structural brain abnormalities in the frontostriatal system and cerebellum in paedophilia. Journal of Psychiatric Research, 41, 753–762. Schiltz, K., Witzel, J., Northoff, G., Zierhut, K., Gubka, U., Fellmann, H., et al. (2007). Brain pathology in pedophilic offenders: Evidence of volume reduction in the right amygdala and related diencephalic structures. Archives of General Psychiatry, 64, 737–746.

Seto, M. C. (2009). Paedophilia. Annual Review of Clinical Psychology, 5, 391–407. Sitaram, R., Caria, A., & Birbaumer, N. (2009). Hemodynamic brain–computer interfaces for communication and rehabilitation. Neural Networks, 22, 1320–1328. Sitaram, R., Caria, A., Veit, R., Gaber, T., Rota, G., Kuebler, A., et al. (2007). fMRI brain–computer interface: A tool for neuroscientific research and treatment. Computer Intelligence and Neuroscience, doi:10.1155/2007/25487 Article ID 25487. Sitaram, N., Weiskopf, N., Caria, A., Veit, R., Erb, N., & Birbaumer, N. (2008). fMRI brain computer interfaces: A tutorial on methods and applications. IEEE Signal Processing Magazine, 95–106. Skinner, F. (1953). Science and human behaviour. New York: Macmillan. Skinner, B. F., & Campbell, S. L. (1947). An automatic shocking-grid apparatus for continuous use. Journal of Comparative and Physiological Psychology, 40, 305–307. Stoléru, S., Grégoire, M. C., Gérard, D., Decety, J., Lafarge, E., Cinotti, L., et al. (1999). Neuroanatomical correlates of visually evoked sexual arousal in human males. Archives of Sexual Behavior, 28(1), 1–21. Stoléru, S., & Mouras, H. (2007). Brain functional imaging studies of sexual desire and arousal in human males. In E. Janssen (Ed.), The psychophysiology of sex (pp. 3–34). Bloomington: Indiana University Press. Streiner, D. L., & Carney, J. (2007). What's under the ROC? An introduction to receiver operating characteristics curves. Canadian Journal of Psychiatry, 52(2), 121–128. Thornton, D. (2002). Constructing and testing a framework for dynamic risk assessment. Sexual Abuse: Journal of Research and Treatment, 14(2), 139–154. Trocmé, N., McPhee, D., Kwan Tam, K., & Hay, T. (1994). Ontario incidence study of reported child abuse and neglect. Toronto, Ontario: The Institute for the Prevention of Child Abuse. Ward, T., & Beech, A. R. (2004). The etiology of risk: A preliminary model. Sexual Abuse: Journal of Research and Treatment, 16(4), 271–284. Ward, T., & Beech, A. R. (2006). An integrated theory of sexual offending. Aggression and Violent Behaviour, 11(1), 44–63. Ward, T., & Hudson, S. M. (1998). A model of the relapse process in sexual offenders. Journal of Interpersonal Violence, 13(6), 700–725. Weiskopf, N., Sitaram, R., Josephs, O., Veit, R., Scharnoewski, F., Goebel, R., et al. (2007). Real-time functional magnetic resonance imaging: Methods and applications. Magnetic Resonance Imaging, 25, 989–1003. Weiskopf, N., Veit, R., Erb, M., Mathiak, K., Grodd, W., Goebel, R., et al. (2003). Psysiological self-regulation of regional brain activity using real-time functional magnetic resonance imaging (fMRI): Methodelogy and exemplary data. Neuroimage, 19(3), 577–586.

A. M. Green, C. E. Chapman, J. F. Kalaska and F. Lepore (Eds.) Progress in Brain Research, Vol. 192 ISSN: 0079-6123 Copyright Ó 2011 Elsevier B.V. All rights reserved.

CHAPTER 15

Shaping plasticity to enhance recovery after injury Numa Dancause{,* and Randolph J. Nudo{ {

{

Département de Physiologie, Université de Montréal, Montréal, Québec, Canada Landon Center on Aging, Kansas University Medical Center, Kansas City, Kansas, USA

Abstract: The past decade of neuroscience research has provided considerable evidence that the adult brain can undergo substantial reorganization following injury. For example, following an ischemic lesion, such as occurs following a stroke, there is a cascade of molecular, genetic, physiological and anatomical events that allows the remaining structures in the brain to reorganize. Often, these events are associated with recovery, suggesting that they contribute to it. Indeed, the term plasticity in stroke research has had a positive connotation historically. But more recently, efforts have been made to differentiate beneficial from detrimental changes. These notions are timely now that neurorehabilitative research is developing novel treatments to modulate, increase, or inhibit plasticity in targeted brain regions. We will review basic principles of plasticity and some of the new and exciting approaches that are currently being investigated to shape plasticity following injury in the central nervous system. Keywords: Cortex; Stimulation; Plasticity; Recovery; Rehabilitation; Stroke. Concept of a plastic brain

injuries require a longer time (Jang, 2009). The neural bases for such recovery, especially in the absence of specific rehabilitative interventions, have intrigued scientists and clinicians for centuries. It has only been in the past 25 years that modern neurophysiological, neuroanatomical, and neuroimaging tools have been brought to bear on this question, resulting in startling findings regarding the degree of structural and functional plasticity of the central nervous system. To explain recovery in the absence of interventions—a phenomenon known as spontaneous recovery—three basic theories have been proposed. First, since remote structures connected to the site of injury often go through a temporary

The adult nervous system is organized in a way that allows for substantial recovery of lost functions after acquired brain injuries. For example, after stroke, the most dramatic recovery in motor function occurs within the first 30 days, though moderate and severe stroke survivors continue to improve for at least 90 days (Duncan et al., 1992). Recovery profiles after focal traumatic brain injury are similar, though diffuse *Corresponding author. Tel.: þ(514) 343-6317; Fax: þ(514) 343-6113 E-mail: [email protected] DOI: 10.1016/B978-0-444-53355-5.00015-4

273

274

period of depressed metabolism and blood flow (diaschisis), it is widely held that at least part of the recovery must be due to the resolution of this process. Second, changes in joint and muscle kinematic patterns are common after cortical injury, and compensatory patterns are often used to accomplish tasks in either subtle or fundamentally different ways. Third, the nervous system undergoes a process of local, and sometimes distant, rewiring. While it is assumed that this is an adaptive process, it is possible that maladaptive plasticity occurs as well. Studies investigating postinjury adaptive plasticity, in the form of modulation of long-term potentiation, long-term depression, unmasking, synaptogenesis, dendritogenesis, and functional map plasticity, have exploded over the past decade and are arguably the most exciting areas in the field of neuroscience due to their implications for understanding and treating injury-related functional deficits. Various plasticity mechanisms underlying functional recovery are embodied in the theory of vicariation—the ability of one part of the brain to substitute for the function of another (Slavin et al., 1988). Since modern views of brain organization recognize that the cerebral cortex is arranged in a distributed, hierarchical fashion, vicariation does not necessarily require that a function lost after damage is taken over by a totally unrelated structure, as suggested by some early interpretations but that other related components of the distributed network reorganize to support the recovered function. A number of studies supportive of this theory have demonstrated that the motor cortex of adult mammals changes its activation patterns in response to cortical injuries. Rat and nonhuman primate studies using intracortical microstimulation (ICMS) to derive detailed maps of the functional representations in the motor cortex have suggested that the neural substrates mediating recovery reside within the peri-infarct cortex (Castro-Alamancos and Borrel, 1995; Glees and Cole, 1949; Nudo et al., 1996b), spared motor areas in the injured hemisphere, such as the

premotor cortex (Dancause et al., 2006; Frost et al., 2003; Liu and Rouiller, 1999) and the supplementary motor area (Eisner-Janowicz et al., 2008). Physiological reorganization of the neural activity in the sensory cortex of the uninjured hemisphere has also been shown (Rema and Ebner, 2003; Reinecke et al., 2003). Neural reorganization within these spared motor regions of the injured and uninjured hemisphere is thought to be necessary for postinjury recovery of motor function (Biernaskie et al., 2005; Castro-Alamancos et al., 1992; Conner et al., 2005; Kleim et al., 2003a; Liu and Rouiller, 1999; Rouiller et al., 1998). Plasticity in topographic maps as a functional index In the 1980s, a fundamental change in our thinking about cortical plasticity occurred, spurred by innovative neurophysiological studies in the somatosensory cortex (Buonomano and Merzenich, 1998). While it had been known for many years that functional plasticity occurs in the cerebral cortex of developing animals, these studies demonstrated that the topographic organization of the representation of skin surfaces in the somatosensory cortex of adult monkeys is modifiable as a result of peripheral nerve injury, disuse or behavioral training. These studies provided credence to the vicariation hypothesis, and impetus for increasing investigation of neurophysiological and neuroanatomical plasticity in the normal and injured cerebral cortex. Subsequently, parallel studies were conducted in other sensory areas of the cerebral cortex, as well as in the motor cortex of experimental animals and in humans. All of these studies have provided strong support for the notion that plasticity of cortical maps is a general trait of cerebral cortex even in mature animals and that rules of temporal coincidence and behavioral context drive emergent properties of cortical modules regardless of their specific cortical location. Assuming that map plasticity and behavioral abilities are interrelated, as they appear to be,

275

these studies have enormous importance for our understanding of the process of recovery after central and peripheral nervous system injury. Topographic maps can be tracked overtime in individual animals, and thus can be used as biological markers of recovery. Further, by examining cellular and molecular correlates of map change, it may be possible to more fully understand the neural mechanisms underlying neuroplasticity, and ultimately control these mechanisms for rehabilitative purposes. Two-stage process of map plasticity The notion that input and output properties in cortical sensory and motor areas are plastic throughout life is now widely accepted and is generalizable across all cortical regions. With respect to relevance for rehabilitation, most studies have focused on the primary motor cortex (M1), primarily because (a) it is often affected by clinical stroke due to its blood supply by the middle cerebral artery, (b) the clinical effects of stroke in M1 are often devastating (hemiparesis), and (c) the close link between neurons in M1 and motoneurons in the spinal cord via the corticospinal (CS) tract allows the relationship between cortical physiology and motor behavior to be examined at various levels of analysis. It is likely that at least two processes are involved in the alteration of topographic maps in both somatosensory and motor cortex. First, there is an immediate unmasking phenomenon that cannot be explained on the basis of neuroanatomical sprouting. Instead, there is likely a change in the efficacy of existing synapses allowing subthreshold inputs to be expressed. The expression of inputs from ascending fibers arriving in the cerebral cortex is largely controlled by inhibitory interneurons that utilize g-amino butyric acid, subtype a (GABAa) as a neurotransmitter. GABAa receptor binding in layer IV of adult monkey area 3b is reduced in the deprived cortex within hours of peripheral nerve injury and this

reduction persists for at least several weeks, if not permanently (Garraghty et al., 2006; Wellman et al., 2002). This result is consistent with the long-held belief that peripheral nerve transection leads to a disinhibition of tonically suppressed inputs. In the motor cortex of rats, similar mechanisms supporting rapid reorganization of representation borders in the motor cortex have been reported. For example, cortico-cortical connections exist between the vibrissae and the forelimb representations in the rat cortex. Under normal conditions, the projections from the vibrassae to the forelimb representation are inhibited by local GABAergic control. Because of this inhibition, the cortical stimulation results in vibrissae and not forelimb movements. Thus, the local inhibition contributes to the definition of the physiological border between these two representations. If bicuculine, a competitive antagonist of GABAa receptors, is injected into the forelimb representation, it blocks local inhibition of cortico-cortical projections from the vibrissae. Consequently, some sites where vibrissae movement could be evoked prior to the injection now evoke forelimb movements. Thus, the removal of the local GABAergic inhibition in the forelimb representation results in a rapid expension of the forelimb area into neighboring vibrassae areas (Jacobs and Donoghue, 1991). The second phase, lasting at least several weeks, results in the remainder of the deprived cortex gradually becoming responsive to other inputs. In somatosensory cortex, this translates into input from adjacent skin surfaces. It is likely that dendritic sprouting plays a role in this longer-term alteration in reorganizational maps (Hickmott and Steen, 2005). Systematic changes occur in dendritic arborization of layer II/III pyramidal and layer IV spiny stellate cells. There appears to be a progressive expansion of distal but not proximal regions of the dendritic trees of both basal and apical dendrites (Churchill et al., 2004). Blockade of the NMDA receptor at the time of peripheral nerve injury has no effect on the immediate stage of unmasking but prevents long-term reorganization from occurring.

276

Interestingly, NMDA receptor blockade has no apparent effect on the organization of normal area 3b cortex, or on the topography of injury-induced reorganized 3b (Myers et al., 2000). However, by 1 month after injury, GABAb receptor binding is reduced, and AMPA receptor binding is increased. Comparisons have been made between topographic changes in somatosensory cortex and changes in the hippocampus during LTP (Garraghty et al., 2006). Map plasticity in motor cortex Neurophysiological maps can be derived in motor cortex using various stimulation techniques in experimental animals, even under anesthesia. Somatotopic organization of the motor cortex was elegantly shown in the early 1950s in both humans (Penfield and Rasmussen, 1952) and monkeys (Woolsey et al., 1952) using epidural cortical stimulation. In the late 1960s, the development of invasive cortical stimulation techniques, or ICMS, resulted in the capacity to stimulate many more focal sites (Stoney et al., 1968) and thus to create much higher resolution motor maps and precise identification of borders between the representations of different body parts. Using these techniques, it was possible to evaluate the effect of learning on the organization of the M1. The first studies explored cortical plasticity within M1 of normal (uninjured) monkeys and provided a basic description of fundamental properties of cortical plasticity relative to the distal and proximal forelimb representation in M1. In these studies, in order to investigate reorganization associated with motor learning, the M1 representational map of the distal forelimb (digits, wrist, forearm movements) is documented before the animals are trained. After training and improvement of the function on the task, the mapping is redone and compared to the data collected prior to training. Functional reorganization of motor maps was found to be dependent on prior behavioral experience. Reach and retrieval training on the Klüver board results in an increase in the representation of the digits in M1 and a

relative decrease in the representation of the wrist and forearm (Fig. 1). If the task is changed to require the monkeys to turn a handle to receive a food pellet, shifting the behavioral demand from digit action to wrist action, wrist representations expand at the expense of digit representations (Nudo et al., 1996a). Further, multijoint movement representations appear after training, reflecting joint combinations and sequences used in the actual task. Multijoint representations thus are driven by temporally correlated activation of two or more movements during training and may reflect the development of muscle and joint synergies in motor cortex. Experiments in rodents using a skilled forelimb-reaching task have provided similar results. Two weeks of training on a single pellet retrieval task induces an expansion of distal forelimb (wrist/digit) movement representations within the sensorimotor cortex and increases the number of synapses per neuron in motor cortex (Kleim et al., 1998, 2004). Reorganization is not simply due to increased use. Rats trained on tasks that do not specifically necessitate skilled use of the forelimb do not exhibit an expansion of the distal forelimb representation (Kleim et al., 1998, 2002; Remple et al., 2001). These behaviorally driven changes appear to be skill- or learning-dependent, as training animals on a task requiring no additional learning or skill acquisition (e.g., monkeys retrieving pellets from a large well), results in no changes in either motor maps or synaptic density. Similarly in monkeys, cortical maps remain stable from one mapping session to the next unless the monkeys are required to learn a new motor skill (Plautz et al., 2000). Early studies using cortical surface stimulation techniques suggested that the hand representation in M1 of adult primates undergoes substantial remodeling following small lesions, and the cortical remodeling is correlated with functional recovery (Glees and Cole, 1950). Using more modern ICMS techniques, Nudo and Milliken (1996) found that movements represented in the infarcted zone did not reappear in the cortical sector surrounding the infarct. Instead, relatively

277

Fig. 1. Representation of finger extension, finger flexion, and wrist abduction movements in M1 derived from pre- and posttraining mapping procedures in squirrel monkeys. Finger extension movements are shown in black, finger flexion movements in dark gray, wrist abduction movements in light gray, and other distal forelimb movements in stippling. Relative area devoted to both finger extension and finger flexion movements increased after pellet retrieval training. Figures are reproduced from Nudo et al. (1996a) with permissions.

small, subtotal lesions in representations of hand movements resulted in widespread reduction in the spared hand representations adjacent to the lesion, and apparent increases in adjacent proximal representations (see Fig. 2c). It was reasonable to hypothesize that training techniques used in uninjured monkeys to demonstrate skill-dependent changes could have an adaptive influence on motor representations after cortical injury. In another experiment, small ischemic lesions were made in the M1 hand area, sparing a large portion of this area. Deficits in motor skill were apparent in the pellet retrieval task. Within about 5 days, the monkeys were able to participate in the task again. Here, instead of letting the animal spontaneously recover, repetitive training was introduced using a protocol similar to the one used to induce motor learning in control animals (Nudo et al., 1996a). As monkeys regained proficiency at retrieval from large food wells, they advanced to progressively smaller wells. For most monkeys, pellet retrieval proficiency returned within about

2 weeks. At that point, the M1 hand area was explored with ICMS techniques once again. In monkeys receiving training, the spared M1 hand area was not statistically different from their baseline maps. That is, instead of a reduction in M1 hand area as seen in spontaneously recovered monkeys, the hand area was retained. In some cases, the hand area clearly expanded into former proximal representations (Nudo et al., 1996b). Due to the similarities of the training regimen to constraintinduced movement therapy (CIMT), this study has been cited as one of the first detailed demonstrations of the neurophysiological basis for poststroke physiotherapy. Widespread effects in motor cortex networks after focal lesions Focal lesions in a small portion of M1 have very different effects on remote hand representations in premotor cortex. Because of reciprocal connectivity of these areas with M1, if M1 is injured, there

278 (a)

Prelesion

(b)

Lesion

(d)

Therapy

M R 1 mm (c)

Spontaneous

Nonresponsive Elbow/shoulder Face Digit Wrist/forearm Fig. 2. Cartoon depicting the reorganization of the distal forelimb representation in M1 following an ischemic lesion. In these studies, the motor map in M1 is derived prior to the lesion (a). Then, a focal ischemic lesion is made in the distal forelimb area ((b); lesion location superimposed on prelesion map). If no intervention is performed and the animal spontaneously recovers, when the motor map is derived again there is a further loss of the distal forelimb representation that occurred after the initial lesion ((c); map status after spontaneous recovery). In a separate group of animals, if therapy based on movement repetition, motor learning, and restraint of the less-impaired forelimb is performed, then the loss of distal forelimb representation is prevented ((d); map status after recovery with therapy). In some cases, there is an increase of distal forelimb representation in the perilesional tissue.

are inevitable effects in secondary motor areas. Using the ventral premotor cortex (PMv) as a model for these remote effects, as early as a few days after M1 injury, neurons in PMv undergo substantial changes in expression of proteins thought to be involved in neuroprotection and angiogenesis (Stowe et al., 2008). In the chronic stage after M1 injury, there is a linear relationship between the size of the M1 infarct and enlargement of the hand representation in PMv. After lesions in M1 that destroy less than 30% of the hand area, the PMv hand representation actually shrinks slightly. However, after

progressively larger M1 injuries, the PMv representation expands in proportion to the M1 loss (Dancause et al., 2006; Frost et al., 2003). These remote effects of M1 lesions have now been extended to the hand representation in the supplementary motor area (SMA), again relating remote map expansion with M1 lesion size (Eisner-Janowicz et al., 2008). Due to a rich network of reciprocal intracortical connections, after focal injury to M1, remote areas are triggered to reorganize their axonal projection pathways. In rats, after cortical injury, the cerebral cortex in the intact

279

hemisphere sprouts crossed axonal projections to the striatum of the injured side of the brain (Napieralski et al., 1996). There is also evidence that plasticity of intrinsic intracortical pathways can occur. Recently, using a squirrel monkey model of M1 infarct, we discovered that several months after an M1 infarct, PMv intracortical axons developed an aberrant trajectory (Dancause et al., 2005). Axons projecting toward the site of the lesion made sharp turns and avoided the lesion zone. A substantial number of axons then turned more caudally, heading lateral to circumvent the central sulcus, and finally terminated in a parietal area within the somatosensory cortex, area 1 (possibly both areas 1 and 2). This de novo cortical connection represented axonal growth of more than 1 cm, a very long distance in the small squirrel monkey brain (Fig. 3). Moreover, it was also shown that CS projections can reorganize. Following a lesion in M1 and Brodmann area 6, the ipsilesional SMA increases its projections to the contraleral spinal cord in laminae VII and IX (McNeal et al., 2010). The increase of projections is correlated with the recovery and a secondary lesion of SMA reinstated the motor deficits. Thus,

(a)

the intriguing possibility exists that areas remote from cortical injury adaptively reorganize to compensate for the loss of M1 CS output by sending larger numbers of CS axons to terminate on the dennervated motor neurons. All of these studies provide strong support for the presence of major anatomical rewiring following injury in the mature brain, such as occurs as a consequence of stroke. This impressive anatomical reorganization could be supported by sequential waves of neuronal growth-promoting genes following the injury (Carmichael et al., 2005). Understanding the functional significance of injury-induced sprouting is an important topic for future research. Postlesion map changes: adaptive, maladaptive or epiphenomenal? Though remote areas expand following an M1 lesion, the functional significance of this change is still unclear. SMA may be an excellent model for understanding these effects, since it receives its blood supply from the anterior cerebral artery and is often spared after MCA strokes. The SMA

(b) M

CS

CS R

1/2

1/2

3b 3a

3a

1 mm

Prelesion PO/IP

3b

Postlesion PO/IP

Fig. 3. Cartoon summary of the reorganization that occurs in PMv following M1 lesion. (a) Prior to the lesion, PMv distal forelimb representation is interconnected with M1 and with secondary somatosensory areas in the posterior operculum. (b) Following a large M1 infarct that destroys most of the distal forelimb representation in M1, the cortical area devoted to this representation in PMv expands (red arrows). These physiological changes are associated with abrupt changes of axon orientation and the border of M1 and novel projections to the primary somatosensory cortex. CS, central sulcus; PO/IP, posterior operculum/inferior parietal cortex. Modified with permission from Dancause, Reviews in the Neurosciences 2006.

280

proximal forelimb and postural compensation (Whishaw et al., 2004). Compensatory use of proximal musculature is also commonly observed in humans after stroke (Cirstea and Levin, 2000). Functional outcomes improve overtime, but true recovery may be masked (Whishaw, 2000; Whishaw et al., 1991) or even hindered (Alaverdashvili et al., 2007, 2008) by use of alternative movement strategies (Levin et al., 2009). Thus, if plasticity in remaining motor areas forms the basis for motor recovery (vicariation), and if motor skill acquisition drives the topography of motor maps, plasticity in spared structures likely supports compensatory motor strategies, rather than recovery of the original movement patterns (Fig. 4). Structural and functional reorganization is not limited to spared regions of the injured hemisphere but may occur in homotopic regions of the intact hemisphere as well (Jones and Schallert, 1992). However, Jones and colleagues have provided substantial evidence that structural 1.2

Area (mm2)

of the two hemispheres are heavily interconnected and share dense reciprocal projections with M1 (Rouiller et al., 1994). It has also been estimated that 23% of SMA CS neurons project to the ipsilateral cord (Dum and Strick, 1996). But a recent study in SMA after extensive ischemic lesions that extended across the M1, the dorsal premotor cortex (PMd), and PMv hand areas questions a direct functional relationship of map reorganization to recovery (Eisner-Janowicz et al., 2008). In this study, behavioral recovery was limited to the first 3 weeks postinjury. Behavioral performance remained relatively constant and suboptimal throughout the next 10 weeks. However, maps of the hand area in SMA actually contracted in the first 3 postinfarct weeks. They subsequently expanded over the next 10 weeks. This temporal mismatch is not easily explained by a simple relationship of remote reorganization in a single area to behavioral recovery. Also, SMA has greater influence on motor neuron pools controlling proximal, rather than distal muscles (Boudrias et al., 2006). In the EisnerJanowicz study, the changes that were seen in the SMA hand representation after injury were attributed to wrist and forearm movement representations, not more distal, finger and thumb representations. In these chronic stages, monkeys were able to reach out and touch the pellet board, but were not able to retrieve pellets from the wells or even insert their fingers into the wells. This is not unlike human stroke survivors who can use proximal musculature to propel the limb forward, but do not have distal control over hand movements. Thus, it is possible that SMA may contribute to the development of compensatory movement patterns that rely in more proximal musculature. Similar results were recently seen in a focal traumatic brain injury model targeting the motor cortex in rats (Nishibe et al., 2010). While changes were seen in the rostral forelimb area after injury to the caudal forelimb area, there was a redistribution of movement representations from distal to proximal. Following a cortical injury, animals adjust the kinematics of forelimb movements to compensate for deficits in the affected musculature, often resulting in both

0.8 Combined Distal Proximal

0.4

0

Control

Injury

Fig. 4. Changes in motor representations in the rostral forelimb area of rats after a controlled cortical impact in the caudal forelimb area. In this study, while the total (combined) rostral forelimb area did not change, distal representations were reduced in area, while proximal representations were enlarged. Several examples using microstimulation techniques in animals suggest that remote reorganization supports learning of compensatory movement patterns rather than recovery of the lost movement patterns. Modified from Nishibe et al. (2010) and reproduced with permission.

281

changes in both homotopic and heterotopic areas of the intact, contralateral cortex are related to hyperreliance on the intact limb, rather than recovery of the impaired limb (Allred et al., 2008; Bury and Jones, 2002; Chu and Jones, 2000). Human neuroimaging studies have also repeatedly shown bihemispheric changes in activation patterns after stroke. However, the functional significance of increased activity in the intact hemisphere is still subject to intense debate (Schallert et al., 2003). It is not yet clear whether changes in fMRI patterns represent an adaptive, maladaptive, or ephiphenomenal effect (Dancause, 2006; Nowak et al., 2009). Treatment strategies currently being developed for stroke patients There are currently multiple strategies being developed to increase the recovery of patients following a stroke. Approaches used in the hours following the lesion generally try to limit the extent of damage and prevent further cell death. For example, intervention targeting the vascular system, such as tissue plasminogen activator (t-PA) administration, applied in the first few hours following the lesion can decrease lesion size and disability (Lansberg et al., 2009a,b). Similarly, approaches to decrease hyperthermia (Colbourne et al., 2000; Corbett et al., 2000) or the inflammatory response (Patel et al., 1993; Yrjanheikki et al., 1998) initiated within the first few hours following the lesion have shown to increase the neural survival and to decrease the motor deficits in rodent models of stroke. In the subsequent days, patients go through the acute and subacute phases of recovery. Most of the behavioral improvements occur in this period that is considered to last about 3 months (Duncan, 1997). Rehabilitation, traditionally based on neurofacilitation or functional retraining, usually takes place within these 3 months and aims at increasing adaptive plasticity

in the tissue that survived the lesion (Nudo and Dancause, 2007; Shumway-Cook and Woollacoot, 2001). The demonstration in animals and humans that cortical maps are malleable as a function of experience in both normal and brain-injured individuals has contributed to the rapid development of new rehabilitative approaches based on experience-dependent plasticity mechanisms. More recently, the use of CIMT was shown to increase motor function in stroke patients in the chronic phase of recovery (Taub et al., 1999, 2002). CIMT was developed on the basis of pioneering animal studies by Taub and colleagues (Taub, 1980; Taub and Morris, 2001). Due to the extensive amount of research that was conducted to test the efficacy of CIMT, it has arguably become the most mature approach among rehabilitative treatments. CIMT consists of (a) constraint of the less-affected upper extremity, typically with a sling or glove and (b) either shaping or task practice with the impaired upper limb. Shaping includes immediate feedback concerning movements, individualized tasks, prompting and cueing, and progressive increase in the difficulty of the tasks. Task practice consists of repetitive practice of a single individualized task in specified blocks without feedback prompting or cueing. While it is generally thought that the sensorymotor experience with the impaired limb is most important, the differential contributions of constraint and the type of practice (shaping or task practice) are confounded (Uswatte et al., 2006). Interestingly, in the nonhuman primate studies that examined map plasticity in the peri-infarct motor cortex and demonstrated a positive effect of rehabilitative training, the behavioral paradigm was a combination of shaping and task practice principles, since the monkeys repeated a single task (pellet retrieval from small wells) in blocks of trials, but the task was made progressively harder by decreasing well diameter (Nudo et al., 1996b). A series of experiments to evaluate the presence of cortical representation changes, paralleling the

282

behavioral changes resulting from CIMT therapy, has also been performed in humans (Liepert et al., 1998, 2000a; Taub et al., 2003, Wittenberg et al., 2003). These studies reported increased cortical representations of the affected arm following treatment, an upper limb representational map size that was similar in both affected and less-affected hemispheres at a 6 months follow-up and shifts of the center of the output map, suggesting recruitment of adjacent brain areas. Recently, efficacy of CIMT for stroke recovery was tested in a multisite randomized controlled trial in 222 stroke survivors, called EXCITE (extremity constraint-induced therapy evaluation; Wolf et al., 2006). This trial demonstrated improvements in upper extremity functional endpoints compared with control groups up to 2 years after treatment. This is despite the fact that individuals were enrolled in the chronic period after stroke. Several details regarding the optimum protocol still remain. The two most critical factors: duration and intensity of treatment (dosage), and the time of onset for the treatment after stroke are still unresolved. A recent trial in 52 stroke survivors suggested that early treatment (enrollment within 9 days after stroke) with CIMT and at higher doses resulted in less improvement (Dromerick et al., 2009). Whether this clinical trial result is due to early excitotoxic effects of intense use remains to be established (Kozlowski et al., 1996). In the past few years, several other novel approaches have been explored to increase the adaptive plasticity in the subacute stage of recovery, where much of the neural reorganization supporting the recovery is expected to occur. Often, these approaches are used as adjuncts to conventional rehabilitation. For example, the use of pharmacological manipulation to increase arousal and learning during training (Barbay and Nudo, 2009; Feeney et al., 1982; Gladstone and Black, 2000; Papadopoulos et al., 2009), and the use of pharmacological agents to increase sprouting and anatomical plasticity (Fang et al.,

2010; Tsai et al., 2007) are currently being investigated by several groups. Restoration of function in the peri-infarct area can be aided by pharmacologic treatment. It has long been known that amphetamine paired with training can enhance recovery after lesions (Feeney et al., 1982). In addition, the pairing of amphetamine with training enhances expression of GAP-43 and synaptophysin in both the intact and damaged hemispheres, presumably indicative of synaptogenesis and axonal sprouting (Stroemer et al., 1998). A significant new finding by Carmichael and colleagues (Clarkson et al., 2010) sheds more light on the early events after focal stroke and suggests potential new targets for therapy. These investigators found excessive tonic inhibition in the peri-infarct zone after a stroke-like injury in the cortex of mice. The inhibition is mediated by extrasynaptic GABAa receptors. The novel approach in this study was to administer a benzodiazepine inverse agonist specific for a subset of the GABAa receptors at various times after stroke. This treatment resulted in sustained, improved motor function. Further, genetically altering the same subset of GABAa receptors also improved poststroke recovery. Thus, it may be possible to substantially improve the effect of poststroke rehabilitative interventions by pharmacologically manipulating tonic inhibition in very specific subsets of receptor types. Another strategy under intensive investigation is the use of cortical stimulation to increase or decrease the activity of targeted brain areas. Recently, the use of both invasive and noninvasive stimulation techniques to favor recovery from stroke has been the focus of extensive research. The use of stimulation has the potential advantages of manipulating the function of specific targeted areas to favor recovery with few, if any, side effects. The following sections will focus on the development of this approach in stroke and review both the literature from animal models as well as the current state of our efforts in humans.

283

Fundamental rational for the use of stimulation to favor recovery Electrical stimulation has been attempted or is currently used to treat many neurological conditions. In the early 1970s, there were reports of chronic implants to stimulate the cerebellum and thalamic nuclei for cerebral palsy, spasticity, and epilepsy, and chronic pain (Hosobuchi et al., 1973; Rosenow et al., 2002). However, it is only in the 1990s that stimulation was shown to be significantly effective to improve tremor, in particular in Parkinson's disease (PD) patients (Benabid et al., 1991; Blond and Siegfried, 1991; Eskandar et al., 2003). Obviously, the body of evidence for the use of stimulation to shape cortical plasticity and favor recovery after stroke is not as extensive as for other conditions such as PD. There are, however, convincing fundamental data providing a strong rationale for its use. ICMS at high intensity can inhibit neural activity in the vicinity of the stimulating electrode (Asanuma and Ward, 1971). Inversely, it can excite distant neurons via polysynaptic activation (Stoney et al., 1968) likely through cortico-cortical networks. Deoxyglucose, a metabolic marker, is increased in other cortical areas interconnected with the site of stimulation (Sharp and Ryan, 1984) but not in subcortical structures (Sharp et al., 1982). Most importantly, ICMS can be used to generate rapid cortical reorganization of motor representation in rats (Nudo et al., 1990). In these experiments, the cortical representation of the caudal and rostral forelimb areas (CFA and RFA, respectively) and the adjacent representations such as the neck and vibrissa were defined. The border between two specific representations (e.g., forelimb and vibrissa) was precisely identified with interstimulation distances of about 200 mm. Then, a stimulating electrode was placed in one representation and repetitive stimulation was applied for 1–3 h. Following the stimulation, the motor representations were mapped again and the borders between representations redefined. These borders dramatically changed

location following the stimulations. The stimulated representation expanded substantially into the unstimulated, neighboring representation, a phenomenon quite similar to the one we previously described in association with motor learning (Kleim et al., 1998, 2002; Plautz et al., 2000; Remple et al., 2001). Brain stimulation to favor recovery after stroke It took several years for the use of cortical stimulation to be applied to increase recovery from stroke. In early animal models, electrodes were first placed under the dura and later on, epidurally, as in eventual clinical trials. So far, the general approach has been to combine stimulation of the perilesional cortex with rehabilitative approaches based on movement repetition and the great majority of the studies have been conducted in rodents. Following ischemic lesions, cortical stimulation can increase excitability and the motor responses evoked from the stimulated cortex (Teskey et al., 2003). In addition, stimulation was shown to increase dentritic density in the stimulated cortex (Adkins-Muir and Jones, 2003). Stimulation can also favor the reorganization of representational maps in the stimulated cortex (Kleim et al., 2003b) a result that was also shown in nonhuman primates (Plautz et al., 2003). In the later study, the behavioral training, which consisted of repetitive dextrous finger movements to retrieve food pellets, was combined with stimulation for about an hour each day for several days. The treatment resulted in representational map expansion that was substantially greater than that following spontaneous recovery (Frost et al., 2003). In all of these previous studies, the increased plasticity was associated with an increase of motor recovery. Whereas very few data are currently available on the parameters of treatment, it appears that both anodal and cathodal stimulations can increase recovery (Adkins et al., 2006; Kleim et al., 2003b) but that cathodal stimulation is

284

more efficient in the early phase following the lesion (Kleim et al., 2003b). In comparison to anodal stimulation, cathodal stimulation may increase survival of vulnerable neurons (Adkins et al., 2006). Another limitation is our understanding of the effects of stimulation frequency. In studies of recovery in rats after stroke, frequencies in the range of 50–100 Hz appear to be effective (Adkins-Muir and Jones, 2003; Adkins et al., 2006, 2008). What is needed to reconcile these issues is a more clear understanding of the mechanisms underlying stimulationinduced plasticity. Stimulation to favor rewiring after injury We have previously discussed support for the rewiring capacity of the adult brain following injury. Of course, key questions are whether these anatomical changes are optimal and if they could be molded or directed to increase recovery. If so, can this be done using stimulation? Following a lesion, EEG studies have shown that there is an appearance of delta waves on the ipsilesional side (Gloor et al., 1977; Hirose et al., 1981). In rats, a comparable low-frequency synchronized neuronal activity is observed in perilesional cortex after an ischemic lesion of the cortex (Carmichael and Chesselet, 2002). Synchronous activity is initially observed in the perilesional cortex and on subsequent days appears to spread to other areas of the ipsilesional and contralesional hemispheres. If the perilesional activity is blocked with tetrodotoxine (TTX), an inhibitor of Na channels, distant cortical synchronous neuronal activity is not observed. Furthermore, in the rats with synchronous neural activity, the contralesional cortex formed atypical connections with the ipsilesional striatum. This novel pattern of connections was not present in rats treated with TTX in the perilesional cortex and thus they did not develop the synchronous neural activity. These results suggest that the synchronous neuronal activity initiated in the perilesional cortex and spreading to other

distant cortical areas supports major rewiring of the connections of these distant cortical areas. In a recent set of experiments (Brus-Ramer et al., 2007; Carmel et al., 2010), stimulation has been shown to increase sprouting and recovery following lesion of the CS tract. Epidural stimulation of the CFA of the “normal” hemisphere following a lesion was used to promote recovery following lesions of the CS at the level of the rostral medulla. The stimulation of the contralesional cortex was started on the day after the lesion and the rats received trains of stimulation for 6 h daily, for 10 days. With time after injury, the group with cortical stimulation recovered better than the control group without stimulation. Furthermore, BDA was injected in the stimulated cortex after recovery and anatomical reconstruction showed that there was an increase of ipsilateral projections from the contralesional cortex to the lower cervical segments. These results suggest that the contralesional hemisphere could, if stimulated, take over some of the lost function from the ipsilesional hemisphere by increasing its ipsilateral control. Does this exploitation of the ipsilateral CS pathway by the contralesional hemisphere require stimulation? Recent studies in humans after stroke have suggested that increased ipsilateral control may not be the pathway through which the contralesional hemisphere acts vicariously for the control of the impaired limb. Indeed, singlepulse transcranial magnetic stimulation (TMS) to stimulate the contralesional cortex does not result in an increase of fast CS output to the recovered arm (Alagona et al., 2001), even if the stimulation is specifically applied to the area showing an increased activation in relation to movements of the impaired limb (Gerloff et al., 2006). Moreover, using stimulation of the contralesional hemisphere to increase recovery is diametrically opposite to the ideas of interhemispheric competition and imbalance that are currently dominating the landscape of stimulation protocol development in humans (Nowak et al., 2009; see also section “The role of the contralesional

285

hemisphere in recovery after a stroke”). According to these ideas, the contralesional hemisphere would be detrimental to the impaired limb. It is difficult to reconcile the results from Brus-Ramer et al. (2007) with these hypotheses and, as we will see in the next sections, many studies designed based on the concepts of interhemispheric imbalance have shown to successfully improve performance in stroke patients. These contradictory results may emphasize that our current understanding of interhemispheric interactions and their role in the recovery from stroke are at best incomplete. The use of brain stimulation to favor recovery from stroke in humans In humans, the use of stimulation with epidural electrodes comparable to those used previously in animal studies initially gave promising results (Brown and Burns, 2001; Brown et al., 2006; Huang et al., 2008; Levy et al., 2008). However, testing of the efficacy of the stimulation on a larger stroke population in the phase III trial gave less conclusive results (Plow et al., 2009). These data stress that further understanding of the mechanisms supporting the effect of stimulation on recovery after stroke are required for the optimization of these new treatment strategies. Several limitations of our current knowledge of the use of stimulation in stroke have been outlined above and animal models can surely be useful to troubleshoot some of them. But perhaps, most importantly, animal models testing the effects of stimulation have to date used stereotypical lesion locations and sizes. One of the major challenges for the development of treatment strategies for the stroke population is the heterogeneity of lesions and consequently of the source of deficits. It is quite possible that patients with different lesions would benefit from different stimulation treatments, including either different stimulation parameters or alternative locations of stimulation. Supporting these assumptions, in contrast to rats

with milder initial impairments following ischemic lesions, animals with severe initial impairments fail to benefit from an identical cortical stimulation treatment (Adkins et al., 2008). Whereas the results from the latest trial for the use of invasive stimulation in humans after stroke were disappointing, they certainly should not be seen as a sign that invasive stimulation as a treatment should be abandoned. Instead, it may be viewed as a reminder of the complexity of stroke and that more work needs to done to better define our parameters of treatment. Perhaps in the stroke population, the choices of stimulation type and location need to be adapted to each patient. Whereas the development of patient specific protocols would require considerable work, the diagnostic capacity that could support the development of stimulation treatments in this direction is available. Invasive stimulation does have many advantages such as the degree of precision of the stimulation site, capacity to simultaneously and differentially stimulate at many locations and the almost limitless duration of stimulation treatments. It may also be recalled that the successful use and implantation of invasive stimulation as a viable treatment for PD also required many years of research. Noninvasive brain stimulation techniques in humans after stroke In humans, transcranial direct current stimulation (tDCS; Fregni et al., 2005; Lang et al., 2005; Nitsche et al., 2005) or TMS (Hummel and Cohen, 2006; Raux et al., 2010) have been used to promote recovery. Whereas the precision of the stimulation site is lower and duration of stimulation treatment is more limited, these approaches have the great advantage of being noninvasive, lowering the potential for complications. The use of these methods has yielded very exciting results and currently, much effort is being invested to develop protocols that use noninvasive stimulation techniques to favor recovery from stroke.

286

The use of tDCS in stroke patients is quite new and, while not unanimous (Hesse et al., 2007), many preliminary studies are showing encouraging results. Many more studies are likely to use this approach that has many advantages in comparison to TMS, such as its low cost, the ease of use and the fact that it can be combined with rehabilitative treatments. tDCS delivers weak direct currents to the cortex via two electrodes that polarize the neural tissue. The active electrode, either the anode to increase or the cathode to decrease excitability, is placed on the scalp over the brain area to modulate (Nitsche et al., 2003, 2007). In chronic stroke patients, a single treatment including the activation of the lesioned hemisphere with anodal tDCS can increase the excitability of the ipsilesional M1 and produce transient improvements in motor performance (Hummel et al., 2005). The application of tDCS over the course of several days prolonged this effect (Boggio et al., 2007). In addition, cathodal tDCS delivered to the contralesional hemisphere can also result in an improvement in motor functions (Fregni et al., 2005; Hummel and Cohen, 2005; see also contralesional hemisphere treatment below). tDCS can also be paired with other approaches. In a recent study, tDCS was combined with peripheral nerve stimulation of the radial and ulnar nerve at the wrist (Celnik et al., 2009). Peripheral nerve stimulation can increase corticomotor excitability (Kaelin-Lang et al., 2002), force (Conforto et al., 2002), and function (Conforto et al., 2007). This pairing of the two stimulation approaches increased the effects of rehabilitative training in stroke patients and was more effective than either type of stimulation alone used with training. To date, the large majority of human studies using noninvasive stimulation in stroke patients have used TMS. As for tDCS, TMS can increase or decrease activity. Repetitive high-frequency TMS applied directly over the ipsilesional hemisphere increases its excitability and the amplitude of electromyographic (EMG) it can evoke in the

impaired limb. These increases are associated with an enhancement of the impaired limb movement accuracy and speed in chronic stroke patients (Kim et al., 2006). Another pattern of stimulation, excitatory theta burst stimulation (Huang et al., 2005), can decrease reaction times of the paretic hand when applied to the ipsilesional M1 (Talelli et al., 2007). However, for both tDCS and TMS, most studies have used a single treatment session and evaluated the short-term effect on motor performance (Ameli et al., 2009; Kim et al., 2006; Lomarev et al., 2007; Yozbatiran et al., 2009). To our knowledge, only one study investigated the effect of treatment using excitatory TMS stimulation protocol over the ipsilesional M1 in multiple consecutive days. This protocol was applied in patients that sustained the stroke within less than 15 days. The stimulations resulted in an increase of motor performance, even 1 year after the treatment (Khedr et al., 2010). These results suggest that stimulation of the ipsilesional hemisphere may be a viable treatment option to increase recovery. However, they also raise several questions. Why are these results different than the ones obtained with the invasive stimulation? Are both approaches acting through different mechanisms? Are the effects obtained in the study using TMS more beneficial because the stimulation was done early after the lesion instead of in chronic patients? Once again, there is an important need for further investigation to establish parameters of treatments and understand their mechanisms of action. Other alternative approaches, also using noninvasive stimulation protocols, have been developed to favor recovery after stroke. In these approaches, noninvasive stimulation is used to inhibit the contralesional cortex. The next sections will review our current understanding of the changes in the contralesional hemisphere after stroke and their role in recovery. Then, we will look at what we know about the effects of contralesional inhibition on recovery from stroke.

287

The role of the contralesional hemisphere in recovery after a stroke Human fMRI and PET studies have shown that there is an early increase of activity in the contralesional hemisphere following a stroke. This increased contralesional activity is associated with a decrease of excitability and of CS output intensity from the ipsilesional cortex to the impaired hand (Alagona et al., 2001; Carey et al., 2006; Catano et al., 1995, 1996; Di Lazzaro et al., 2008; Heald et al., 1993; Jaillard et al., 2005; Liepert et al., 2000b; Manganotti et al., 2002; Marshall et al., 2000; Rapisarda et al., 1996). In animal studies, an acute increase of somatosensory evoked potentials in the contralesional cortex has also been reported in the hours following lesion (Hossmann et al., 1985; Meyer et al., 1985; Sakatani et al., 1990). The early increase of activity in the contralesional hemisphere and its negative impact on the impaired limb are generally explained with the concept of interhemispheric competition. In this hypothesis, the loss of neural tissue in the ipsilesional hemisphere results in a decrease of interhemispheric inhibition from the ipsilesional hemisphere (Liepert et al., 2000b), creating an interhemispheric imbalance. The resulting increase of contralesional activity would in turn contribute to the ipsilesional diaschisis through an increase of its callosal inhibition to the ipsilesional cortex. By doing so, the changes of activity in the contralesional hemisphere would be detrimental to the impaired limb. Longitudinal imaging studies in humans generally support this view (Carey et al., 2006; Jaillard et al., 2005; Marshall et al., 2000; Nhan et al., 2004; Ward et al., 2004). Early after the lesion, these studies report atypically high levels of contralesional activity and low ipsilesional activity. With recovery, there is an increase of ipsilesional and decrease of contralesional activity presumably due to a reduction of the interhemispheric imbalance (Cicinelli et al., 1997, 2003; Heald et al., 1993; Liepert et al., 2000b; Turton et al., 1996).

However, the negative role of the contralesional cortex in the recovery is far from being unanimously accepted (Schallert et al., 2003). In the chronic phase of recovery, plasticity in the contralesional hemisphere has been shown to support compensatory behavior and learning with the less-impaired limb in rats (Bury and Jones, 2002; Jones and Schallert, 1992, 1994). Although not related to the recovery of the impaired limb, the plasticity in the contralesional hemisphere could thus make a significant contribution to the overall recovery and maximize the autonomy of stroke survivors. Furthermore, in the chronic phase after stroke the contralesional hemisphere may undergo adaptive plasticity to be more involved in the control of the impaired limb. Some support for this hypothesis is provided in many human functional imaging studies reporting that the increased activation in the contralesional cortex is associated with recovery (Cramer et al., 1997; Riecker et al., 2010; Schaechter and Perdue, 2008; Seitz et al., 1998). It was also shown that inhibition of contralesional areas with atypically high activity in chronic stroke patients can interfere with performance of the impaired hand, whereas inhibition of comparable locations in control subject did not (Lotze et al., 2006). Similarly, the inhibition of the contralesional hemisphere in rats that recovered from large ischemic infarcts generates more behavioral deficits of the impaired forelimb in comparison to control animals (Biernaskie et al., 2005). Inhibition of the contralesional cortex to improve recovery from stroke The use of contralesional inhibition to increase recovery is based on the demonstration that suppression of one motor cortex leads to an increased excitability of the contralateral motor cortex (Gilio et al., 2003; Heide et al., 2006; Schambra et al., 2003). In most studies, slow rTMS is used to suppress the contralesional hemisphere excitability (Maeda et al., 2002). In chronic

288

stroke patients, the application of inhibitory stimulation protocols over the contralesional hemisphere increases motor output from the ipsilesional to the paretic limb (Takeuchi et al., 2005). Furthermore, in chronic stroke patients, stimulation protocols to inhibit the contralesional activity have also been shown to produce improvement in motor performance of the hand function (Mansur et al., 2005; Nowak et al., 2008; Takeuchi et al., 2005). In acute patients, one study showed that repetitive theta burst stimulation over the intact hemisphere increase the excitability of the ipsilesional motor cortex (Di Lazzaro et al., 2008). As for the use of ipsilesional stimulation, to date the great majority of studies using contralesional inhibition have looked at the effect of a single session of stimulation on stroke patients and the reported effects were short-lasting. Only two studies have investigated the effect of multiple sessions of contralesional inhibition to favor recovery of the impaired limb. The first one, in chronic stroke patients, showed that slow rTMS for 5 days in the contralesional hemisphere can significantly increase CS excitability in the ipsilesional hemisphere and improve motor performance of the impaired limb (Fregni et al., 2006). These effects were still identifiable after 2 weeks. The other study, in acute stroke patients (7–20 days after stroke), showed that 5 consecutive days of treatment to inhibit the contralesional hemisphere increases the ipsilesional output to the paretic upper limb and enhanced recovery (Khedr et al., 2009). The effects were still present 3 months after the treatment. Therefore, it appears that contralesional inhibition could be favorable to recovery. General conclusions Relying on a much clearer understanding of plasticity principles, novel treatment protocols following brain lesion are rapidly emerging and evolving. Several research groups are investigating mechanisms

and testing the effects of these novel approaches in animal models and in humans. One of these approaches, which is particularly promising, is the use of stimulation. To date, most of the protocols are designed with the assumption that following a lesion, the ipsilesional cortex is hypoactive and should be activated and that the contralesional cortex is hyperactive and should be inhibited. Accordingly, stimulation in humans has been used to either increase the activity in the ipsilesional hemisphere or to inhibit the contralesional cortex. However, some studies have provided results that are contradictory to these hypotheses (Biernaskie et al., 2005; Brus-Ramer et al., 2009; Carmel et al., 2010; Pomeroy et al., 2007), and stress that our current hypotheses may not be entirely accurate. These dominating views were largely extrapolated from evidence collected in control subject. It is quite possible that ipsilesional function and rules of interhemispheric interactions are changed following stroke. Nonetheless, several of the novels studies on the use of stimulation after stroke have provided encouraging results and support the notion that stimulation could become a valuable addition to traditional rehabilitative approaches. While much work still needs to be done, it is possible that one day we will be able to use these techniques to increase favorable and decrease detrimental plasticity to maximize recovery. It is likely that to maximize benefit this shaping of plasticity will have to be adapted to the patient's remaining intact structures. Acknowledgment Numa Dancause is currently holding a Chercheur Boursier Junior 1 salary award from the Fonds de la Recherche en Santé du Québec and a New Investigator salary award from the Canadian Institutes of Health Research. Randolph J. Nudo is supported by NIH Grant NS030853 and a United States Department of Defense Investigator-Initiated Award.

289

References Adkins, D. L., Campos, P., Quach, D., Borromeo, M., Schallert, K., & Jones, T. A. (2006). Epidural cortical stimulation enhances motor function after sensorimotor cortical infarcts in rats. Experimental Neurology, 200, 356–370. Adkins, D. L., Hsu, J. E., & Jones, T. A. (2008). Motor cortical stimulation promotes synaptic plasticity and behavioral improvements following sensorimotor cortex lesions. Experimental Neurology, 212, 14–28. Adkins-Muir, D. L., & Jones, T. A. (2003). Cortical electrical stimulation combined with rehabilitative training: Enhanced functional recovery and dendritic plasticity following focal cortical ischemia in rats. Neurological Research, 25, 780–788. Alagona, G., Delvaux, V., Gerard, P., De Pasqua, V., Pennisi, G., Delwaide, P. J., et al. (2001). Ipsilateral motor responses to focal transcranial magnetic stimulation in healthy subjects and acute-stroke patients. Stroke, 32, 1304–1309. Alaverdashvili, M., Foroud, A., Lim, D. H., & Whishaw, I. Q. (2008). “Learned baduse” limits recovery of skilled reaching for food after forelimb motor cortex stroke in rats: A new analysis of the effect of gestures on success. Behavioural Brain Research, 188, 281–290. Alaverdashvili, M., Lim, D. H., & Whishaw, I. Q. (2007). No improvement by amphetamine on learned non-use, attempts, success or movement in skilled reaching by the rat after motor cortex stroke. The European Journal of Neuroscience, 25, 3442–3452. Allred, R. P., Adkins, D. L., Woodlee, M. T., Husbands, L. C., Maldonado, M. A., Kane, J. R., et al. (2008). The vermicelli handling test: A simple quantitative measure of dexterous forepaw function in rats. Journal of Neuroscience Methods, 170, 229–244. Ameli, M., Grefkes, C., Kemper, F., Riegg, F. P., Rehme, A. K., Karbe, H., et al. (2009). Differential effects of high-frequency repetitive transcranial magnetic stimulation over ipsilesional primary motor cortex in cortical and subcortical middle cerebral artery stroke. Annals of Neurology, 66, 298–309. Asanuma, H., & Ward, J. E. (1971). Patterns of contraction of distal forelimb muscles produced by intracortical stimulation in cats. Brain Research, 27, 97–109. Barbay, S., & Nudo, R. J. (2009). The effects of amphetamine on recovery of function in animal models of cerebral injury: A critical appraisal. NeuroRehabilitation, 25, 5–17. Benabid, A. L., Pollak, P., Gervason, C., Hoffmann, D., Gao, D. M., Hommel, M., et al. (1991). Long-term suppression of tremor by chronic stimulation of the ventral intermediate thalamic nucleus. Lancet, 337, 403–406. Biernaskie, J., Szymanska, A., Windle, V., & Corbett, D. (2005). Bi-hemispheric contribution to functional motor

recovery of the affected forelimb following focal ischemic brain injury in rats. The European Journal of Neuroscience, 21, 989–999. Blond, S., & Siegfried, J. (1991). Thalamic stimulation for the treatment of tremor and other movement disorders. Acta Neurochirurgica. Supplementum, 52, 109–111. Boggio, P. S., Nunes, A., Rigonatti, S. P., Nitsche, M. A., Pascual-Leone, A., & Fregni, F. (2007). Repeated sessions of noninvasive brain DC stimulation is associated with motor function improvement in stroke patients. Restorative Neurology and Neuroscience, 25, 123–129. Boudrias, M. H., Belhaj-Saif, A., Park, M. C., & Cheney, P. D. (2006). Contrasting properties of motor output from the supplementary motor area and primary motor cortex in rhesus macaques. Cerebral Cortex, 16, 632–638. Brown, G. T., & Burns, S. A. (2001). The efficacy of neurodevelopmental treatment in paediatrics. The British Journal of Occupational Therapy, 64, 235–244. Brown, J. A., Lutsep, H. L., Weinand, M., & Cramer, S. C. (2006). Motor cortex stimulation for the enhancement of recovery from stroke: A prospective, multicenter safety study. Neurosurgery, 58, 464–473. Brus-Ramer, M., Carmel, J. B., Chakrabarty, S., & Martin, J. H. (2007). Electrical stimulation of spared corticospinal axons augments connections with ipsilateral spinal motor circuits after injury. The Journal of Neuroscience, 27, 13793–13801. Brus-Ramer, M., Carmel, J. B., & Martin, J. H. (2009). Motor cortex bilateral motor representation depends on subcortical and interhemispheric interactions. The Journal of Neuroscience, 29, 6196–6206. Buonomano, D. V., & Merzenich, M. M. (1998). Cortical plasticity: From synapses to maps. Annual Review of Neuroscience, 21, 149–186. Bury, S. D., & Jones, T. A. (2002). Unilateral sensorimotor cortex lesions in adult rats facilitate motor skill learning with the “unaffected” forelimb and training-induced dendritic structural plasticity in the motor cortex. The Journal of Neuroscience, 22, 8597–8606. Carey, L. M., Abbott, D. F., Egan, G. F., O'Keefe, G. J., Jackson, G. D., Bernhardt, J., et al. (2006). Evolution of brain activation with good and poor motor recovery after stroke. Neurorehabilitation and Neural Repair, 20, 24–41. Carmel, J. B., Berrol, L. J., Brus-Ramer, M., & Martin, J. H. (2010). Chronic electrical stimulation of the intact corticospinal system after unilateral injury restores skilled locomotor control and promotes spinal axon outgrowth. The Journal of Neuroscience, 30, 10918–10926. Carmichael, S. T., Archibeque, I., Luke, L., Nolan, T., Momiy, J., & Li, S. (2005). Growth-associated gene expression after stroke: Evidence for a growth-promoting region in peri-infarct cortex. Experimental Neurology, 193, 291–311.

290 Carmichael, S. T., & Chesselet, M. F. (2002). Synchronous neuronal activity is a signal for axonal sprouting after cortical lesions in the adult. The Journal of Neuroscience, 22, 6062–6070. Castro-Alamancos, M. A., & Borrel, J. (1995). Functional recovery of forelimb response capacity after forelimb primary motor cortex damage in the rat is due to the reorganization of adjacent areas of cortex. Neuroscience, 68, 793–805. Castro-Alamancos, M. A., Garcia-Segura, L. M., & Borrell, J. (1992). Transfer of function to a specific area of the cortex after induced recovery from brain damage. The European Journal of Neuroscience, 4, 853–863. Catano, A., Houa, M., Caroyer, J. M., Ducarne, H., & Noel, P. (1995). Magnetic transcranial stimulation in nonhaemorrhagic sylvian strokes: Interest of facilitation for early functional prognosis. Electroencephalography and Clinical Neurophysiology, 97, 349–354. Catano, A., Houa, M., Caroyer, J. M., Ducarne, H., & Noel, P. (1996). Magnetic transcranial stimulation in acute stroke: Early excitation threshold and functional prognosis. Electroencephalography and Clinical Neurophysiology, 101, 233–239. Celnik, P., Paik, N. J., Vandermeeren, Y., Dimyan, M., & Cohen, L. G. (2009). Effects of combined peripheral nerve stimulation and brain polarization on performance of a motor sequence task after chronic stroke. Stroke, 40, 1764–1771. Chu, C. J., & Jones, T. A. (2000). Experience-dependent structural plasticity in cortex heterotopic to focal sensorimotor cortical damage. Experimental Neurology, 166, 403–414. Churchill, J. D., Tharp, J. A., Wellman, C. L., Sengelaub, D. R., & Garraghty, P. E. (2004). Morphological correlates of injury-induced reorganization in primate somatosensory cortex. BMC Neuroscience, 5, 43. Cicinelli, P., Pasqualetti, P., Zaccagnini, M., Traversa, R., Oliveri, M., & Rossini, P. M. (2003). Interhemispheric asymmetries of motor cortex excitability in the postacute stroke stage: A paired-pulse transcranial magnetic stimulation study. Stroke, 34, 2653–2658. Cicinelli, P., Traversa, R., & Rossini, P. M. (1997). Post-stroke reorganization of brain motor output to the hand: A 2–4 month follow-up with focal magnetic transcranial stimulation. Electroencephalography and Clinical Neurophysiology, 105, 438–450. Cirstea, M. C., & Levin, M. F. (2000). Compensatory strategies for reaching in stroke. Brain, 123(Pt 5), 940–953. Clarkson, A. N., Huang, B. S., Macisaac, S. E., Mody, I., Carmichael, S. T. (2010). Reducing excessive GABA-mediated tonic inhibition promotes functional recovery after stroke. Nature, 468, 305–309. Colbourne, F., Corbett, D., Zhao, Z., Yang, J., & Buchan, A. M. (2000). Prolonged but delayed postischemic hypothermia: A long-term outcome study in the rat middle

cerebral artery occlusion model. Journal of Cerebral Blood Flow and Metabolism, 20, 1702–1708. Conforto, A. B., Cohen, L. G., Dos Santos, R. L., Scaff, M., & Marie, S. K. (2007). Effects of somatosensory stimulation on motor function in chronic cortico-subcortical strokes. Journal of Neurology, 254, 333–339. Conforto, A. B., Kaelin-Lang, A., & Cohen, L. G. (2002). Increase in hand muscle strength of stroke patients after somatosensory stimulation. Annals of Neurology, 51, 122–125. Conner, J. M., Chiba, A. A., & Tuszynski, M. H. (2005). The basal forebrain cholinergic system is essential for cortical plasticity and functional recovery following brain injury. Neuron, 46, 173–179. Corbett, D., Hamilton, M., & Colbourne, F. (2000). Persistent neuroprotection with prolonged postischemic hypothermia in adult rats subjected to transient middle cerebral artery occlusion. Experimental Neurology, 163, 200–206. Cramer, S. C., Nelles, G., Benson, R. R., Kaplan, J. D., Parker, R. A., Kwong, K. K., et al. (1997). A functional MRI study of subjects recovered from hemiparetic stroke. Stroke, 28, 2518–2527. Dancause, N. (2006). Vicarious function of remote cortex following stroke: Recent evidence from human and animal studies. The Neuroscientist, 12, 489–499. Dancause, N., Barbay, S., Frost, S. B., Plautz, E. J., Chen, D., Zoubina, E. V., et al. (2005). Extensive cortical rewiring after brain injury. The Journal of Neuroscience, 25, 10167–10179. Dancause, N., Barbay, S., Frost, S. B., Zoubina, E. V., Plautz, E. J., Mahnken, J. D., et al. (2006). Effects of small ischemic lesions in the primary motor cortex on neurophysiological organization in ventral premotor cortex. Journal of Neurophysiology, 96, 3506–3511. Di Lazzaro, V., Pilato, F., Dileone, M., Profice, P., Capone, F., Ranieri, F., et al. (2008). Modulating cortical excitability in acute stroke: A repetitive TMS study. Clinical Neurophysiology, 119, 715–723. Dromerick, A. W., Lang, C. E., Birkenmeier, R. L., Wagner, J. M., Miller, J. P., Videen, T. O., et al. (2009). Very early constraint-induced movement during stroke rehabilitation (VECTORS): A single-center RCT. Neurology, 73, 195–201. Dum, R. P., & Strick, P. L. (1996). Spinal cord terminations of the medial wall motor areas in macaque monkeys. The Journal of Neuroscience, 16, 6513–6525. Duncan, P. W., & Lai, S. M. (1997). Stroke recovery. Top Stroke Rehabil, 4(3), 51–58. Duncan, P. W., Goldstein, L. B., Matchar, D., Divine, G. W., & Feussner, J. (1992). Measurement of motor recovery after stroke. Outcome assessment and sample size requirements. Stroke, 23, 1084–1089.

291 Eisner-Janowicz, I., Barbay, S., Hoover, E., Stowe, A. M., Frost, S. B., Plautz, E. J., et al. (2008). Early and late changes in the distal forelimb representation of the supplementary motor area after injury to frontal motor areas in the squirrel monkey. Journal of Neurophysiology, 100, 1498–1512. Eskandar, E. N., Flaherty, A., Cosgrove, G. R., Shinobu, L. A., & Barker, F. G. 2nd (2003). Surgery for Parkinson disease in the United States, 1996 to 2000: Practice patterns, short-term outcomes, and hospital charges in a nationwide sample. Journal of Neurosurgery, 99, 863–871. Fang, P. C., Barbay, S., Plautz, E. J., Hoover, E., Strittmatter, S. M., & Nudo, R. J. (2010). Combination of NEP 1–40 treatment and motor training enhances behavioral recovery after a focal cortical infarct in rats. Stroke, 41, 544–549. Feeney, D. M., Gonzalez, A., & Law, W. A. (1982). Amphetamine, haloperidol, and experience interact to affect rate of recovery after motor cortex injury. Science, 217, 855–857. Fregni, F., Boggio, P. S., Mansur, C. G., Wagner, T., Ferreira, M. J., Lima, M. C., et al. (2005). Transcranial direct current stimulation of the unaffected hemisphere in stroke patients. Neuroreport, 16, 1551–1555. Fregni, F., Boggio, P. S., Valle, A. C., Rocha, R. R., Duarte, J., Ferreira, M. J., et al. (2006). A sham-controlled trial of a 5day course of repetitive transcranial magnetic stimulation of the unaffected hemisphere in stroke patients. Stroke, 37, 2115–2122. Frost, S. B., Barbay, S., Friel, K. M., Plautz, E. J., & Nudo, R. J. (2003). Reorganization of remote cortical regions after ischemic brain injury: A potential substrate for stroke recovery. Journal of Neurophysiology, 89, 3205–3214. Garraghty, P. E., Arnold, L. L., Wellman, C. L., & Mowery, T. M. (2006). Receptor autoradiographic correlates of deafferentation-induced reorganization in adult primate somatosensory cortex. The Journal of Comparative Neurology, 497, 636–645. Gerloff, C., Bushara, K., Sailer, A., Wassermann, E. M., Chen, R., Matsuoka, T., et al. (2006). Multimodal imaging of brain reorganization in motor areas of the contralesional hemisphere of well recovered patients after capsular stroke. Brain, 129, 791–808. Gilio, F., Rizzo, V., Siebner, H. R., & Rothwell, J. C. (2003). Effects on the right motor hand-area excitability produced by low-frequency rTMS over human contralateral homologous cortex. The Journal of Physiology, 551, 563–573. Gladstone, D. J., & Black, S. E. (2000). Enhancing recovery after stroke with noradrenergic pharmacotherapy: A new frontier? The Canadian Journal of Neurological Sciences, 27, 97–105. Glees, P., & Cole, J. (1949). The reappearance of coordinated movements of the hand after lesions in the hand area of the

motor cortex of the rhesus monkey. Journal of Physiology, 108, 33 Proc. Glees, P., & Cole, J. (1950). Recovery of skilled motor functions after small repeated lesions in motor cortex in macaque. Journal of Neurophysiology, 13, 137–148. Gloor, P., Ball, G., & Schaul, N. (1977). Brain lesions that produce delta waves in the EEG. Neurology, 27, 326–333. Heald, A., Bates, D., Cartlidge, N. E., French, J. M., & Miller, S. (1993). Longitudinal study of central motor conduction time following stroke. 2. Central motor conduction measured within 72 h after stroke as a predictor of functional outcome at 12 months. Brain, 116(Pt 6), 1371–1385. Heide, G., Witte, O. W., & Ziemann, U. (2006). Physiology of modulation of motor cortex excitability by low-frequency suprathreshold repetitive transcranial magnetic stimulation. Experimental Brain Research, 171, 26–34. Hesse, S., Werner, C., Schonhardt, E. M., Bardeleben, A., Jenrich, W., & Kirker, S. G. (2007). Combined transcranial direct current stimulation and robot-assisted arm training in subacute stroke patients: A pilot study. Restorative Neurology and Neuroscience, 25, 9–15. Hickmott, P. W., & Steen, P. A. (2005). Large-scale changes in dendritic structure during reorganization of adult somatosensory cortex. Nature Neuroscience, 8, 140–142. Hirose, G., Saeki, M., Kosoegawa, H., Takado, M., Yamamoto, T., & Tada, A. (1981). Delta waves in the EEGs of patients with intracerebral hemorrhage. Archives of Neurology, 38, 170–175. Hosobuchi, Y., Adams, J. E., & Rutkin, B. (1973). Chronic thalamic stimulation for the control of facial anesthesia dolorosa. Archives of Neurology, 29, 158–161. Hossmann, K. A., Mies, G., Paschen, W., Csiba, L., Bodsch, W., Rapin, J. R., et al. (1985). Multiparametric imaging of blood flow and metabolism after middle cerebral artery occlusion in cats. Journal of Cerebral Blood Flow and Metabolism, 5, 97–107. Huang, Y. Z., Edwards, M. J., Rounis, E., Bhatia, K. P., & Rothwell, J. C. (2005). Theta burst stimulation of the human motor cortex. Neuron, 45, 201–206. Huang, M., Harvey, R. L., Stoykov, M. E., Ruland, S., Weinand, M., Lowry, D., et al. (2008). Cortical stimulation for upper limb recovery following ischemic stroke: A small phase II pilot study of a fully implanted stimulator. Topics in Stroke Rehabilitation, 15, 160–172. Hummel, F., Celnik, P., Giraux, P., Floel, A., Wu, W. H., Gerloff, C., et al. (2005). Effects of non-invasive cortical stimulation on skilled motor function in chronic stroke. Brain, 128, 490–499. Hummel, F., & Cohen, L. G. (2005). Improvement of motor function with noninvasive cortical stimulation in a patient with chronic stroke. Neurorehabilitation and Neural Repair, 19, 14–19.

292 Hummel, F. C., & Cohen, L. G. (2006). Non-invasive brain stimulation: A new strategy to improve neurorehabilitation after stroke? Lancet Neurology, 5, 708–712. Jacobs, K. M., & Donoghue, J. P. (1991). Reshaping the cortical motor map by unmasking latent intracortical connections. Science, 251, 944–947. Jaillard, A., Martin, C. D., Garambois, K., Lebas, J. F., & Hommel, M. (2005). Vicarious function within the human primary motor cortex? A longitudinal fMRI stroke study. Brain, 128, 1122–1138. Jang, S. H. (2009). Review of motor recovery in patients with traumatic brain injury. NeuroRehabilitation, 24, 349–353. Jones, T. A., & Schallert, T. (1992). Overgrowth and pruning of dendrites in adult rats recovering from neocortical damage. Brain Research, 581, 156–160. Jones, T. A., & Schallert, T. (1994). Use-dependent growth of pyramidal neurons after neocortical damage. The Journal of Neuroscience, 14, 2140–2152. Kaelin-Lang, A., Luft, A. R., Sawaki, L., Burstein, A. H., Sohn, Y. H., & Cohen, L. G. (2002). Modulation of human corticomotor excitability by somatosensory input. The Journal of Physiology, 540, 623–633. Khedr, E. M., Abdel-Fadeil, M. R., Farghali, A., & Qaid, M. (2009). Role of 1 and 3 Hz repetitive transcranial magnetic stimulation on motor function recovery after acute ischaemic stroke. European Journal of Neurology, 16, 1323–1330. Khedr, E. M., Etraby, A. E., Hemeda, M., Nasef, A. M., & Razek, A. A. (2010). Long-term effect of repetitive transcranial magnetic stimulation on motor function recovery after acute ischemic stroke. Acta Neurologica Scandinavica, 121, 30–37. Kim, Y. H., You, S. H., Ko, M. H., Park, J. W., Lee, K. H., Jang, S. H., et al. (2006). Repetitive transcranial magnetic stimulation-induced corticomotor excitability and associated motor skill acquisition in chronic stroke. Stroke, 37, 1471–1476. Kleim, J. A., Barbay, S., Cooper, N. R., Hogg, T. M., Reidel, C. N., Remple, M. S., et al. (2002). Motor learningdependent synaptogenesis is localized to functionally reorganized motor cortex. Neurobiology of Learning and Memory, 77, 63–77. Kleim, J. A., Barbay, S., & Nudo, R. J. (1998). Functional reorganization of the rat motor cortex following motor skill learning. Journal of Neurophysiology, 80, 3321–3325. Kleim, J. A., Bruneau, R., Calder, K., Pocock, D., Vandenberg, P. M., MacDonald, E., et al. (2003a). Functional organization of adult motor cortex is dependent upon continued protein synthesis. Neuron, 40, 167–176. Kleim, J. A., Bruneau, R., Vandenberg, P., MacDonald, E., Mulrooney, R., & Pocock, D. (2003b). Motor cortex stimulation enhances motor recovery and reduces peri-infarct dysfunction following ischemic insult. Neurological Research, 25, 789–793.

Kleim, J. A., Hogg, T. M., Vandenberg, P. M., Cooper, N. R., Bruneau, R., & Remple, M. (2004). Cortical synaptogenesis and motor map reorganization occur during late, but not early, phase of motor skill learning. The Journal of Neuroscience, 24, 628–633. Kozlowski, D. A., James, D. C., & Schallert, T. (1996). Usedependent exaggeration of neuronal injury after unilateral sensorimotor cortex lesions. The Journal of Neuroscience, 16, 4776–4786. Lang, N., Siebner, H. R., Ward, N. S., Lee, L., Nitsche, M. A., Paulus, W., et al. (2005). How does transcranial DC stimulation of the primary motor cortex alter regional neuronal activity in the human brain? The European Journal of Neuroscience, 22, 495–504. Lansberg, M. G., Bluhmki, E., & Thijs, V. N. (2009a). Efficacy and safety of tissue plasminogen activator 3 to 4.5 hours after acute ischemic stroke: A metaanalysis. Stroke, 40, 2438–2441. Lansberg, M. G., Schrooten, M., Bluhmki, E., Thijs, V. N., & Saver, J. L. (2009b). Treatment time-specific number needed to treat estimates for tissue plasminogen activator therapy in acute stroke based on shifts over the entire range of the modified Rankin Scale. Stroke, 40, 2079–2084. Levin, M. F., Kleim, J. A., & Wolf, S. L. (2009). What do motor “recovery” and “compensation” mean in patients following stroke? Neurorehabilitation and Neural Repair, 23, 313–319. Levy, R., Ruland, S., Weinand, M., Lowry, D., Dafer, R., & Bakay, R. (2008). Cortical stimulation for the rehabilitation of patients with hemiparetic stroke: A multicenter feasibility study of safety and efficacy. Journal of Neurosurgery, 108, 707–714. Liepert, J., Bauder, H., Wolfgang, H. R., Miltner, W. H., Taub, E., & Weiller, C. (2000a). Treatment-induced cortical reorganization after stroke in humans. Stroke, 31, 1210–1216. Liepert, J., Hamzei, F., & Weiller, C. (2000b). Motor cortex disinhibition of the unaffected hemisphere after acute stroke. Muscle & Nerve, 23, 1761–1763. Liepert, J., Miltner, W. H., Bauder, H., Sommer, M., Dettmers, C., Taub, E., et al. (1998). Motor cortex plasticity during constraint-induced movement therapy in stroke patients. Neuroscience Letters, 250, 5–8. Liu, Y., & Rouiller, E. M. (1999). Mechanisms of recovery of dexterity following unilateral lesion of the sensorimotor cortex in adult monkeys. Experimental Brain Research, 128, 149–159. Lomarev, M. P., Kim, D. Y., Richardson, S. P., Voller, B., & Hallett, M. (2007). Safety study of high-frequency transcranial magnetic stimulation in patients with chronic stroke. Clinical Neurophysiology, 118, 2072–2075. Lotze, M., Markert, J., Sauseng, P., Hoppe, J., Plewnia, C., & Gerloff, C. (2006). The role of multiple contralesional motor

293 areas for complex hand movements after internal capsular lesion. The Journal of Neuroscience, 26, 6096–6102. Maeda, F., Gangitano, M., Thall, M., & Pascual-Leone, A. (2002). Inter- and intra-individual variability of paired-pulse curves with transcranial magnetic stimulation (TMS). Clinical Neurophysiology, 113, 376–382. Manganotti, P., Patuzzo, S., Cortese, F., Palermo, A., Smania, N., & Fiaschi, A. (2002). Motor disinhibition in affected and unaffected hemisphere in the early period of recovery after stroke. Clinical Neurophysiology, 113, 936–943. Mansur, C. G., Fregni, F., Boggio, P. S., Riberto, M., GallucciNeto, J., Santos, C. M., et al. (2005). A sham stimulationcontrolled trial of rTMS of the unaffected hemisphere in stroke patients. Neurology, 64, 1802–1804. Marshall, R. S., Perera, G. M., Lazar, R. M., Krakauer, J. W., Constantine, R. C., & Delapaz, R. L. (2000). Evolution of cortical activation during recovery from corticospinal tract infarction. Stroke, 31, 656–661. Mcneal, D. W., Darling, W. G., Ge, J., StilwellMorecraft, K. S., Solon, K. M., Hynes, S. M., et al. (2010). Selective long-term reorganization of the corticospinal projection from the supplementary motor cortex following recovery from lateral motor cortex injury. The Journal of Comparative Neurology, 518, 586–621. Meyer, K. L., Dempsey, R. J., Roy, M. W., & Donaldson, D. L. (1985). Somatosensory evoked potentials as a measure of experimental cerebral ischemia. Journal of Neurosurgery, 62, 269–275. Myers, W. A., Churchill, J. D., Muja, N., & Garraghty, P. E. (2000). Role of NMDA receptors in adult primate cortical somatosensory plasticity. The Journal of Comparative Neurology, 418, 373–382. Napieralski, J. A., Butler, A. K., & Chesselet, M. F. (1996). Anatomical and functional evidence for lesion-specific sprouting of corticostriatal input in the adult rat. The Journal of Comparative Neurology, 373, 484–497. Nhan, H., Barquist, K., Bell, K., Esselman, P., Odderson, I. R., & Cramer, S. C. (2004). Brain function early after stroke in relation to subsequent recovery. Journal of Cerebral Blood Flow and Metabolism, 24, 756–763. Nishibe, M., Barbay, S., Guggenmos, D., & Nudo, R. J. (2010). Reorganization of motor cortex after controlled cortical impact in rats and implications for functional recovery. Journal of Neurotrauma, 27, 2221–2232. Nitsche, M. A., Doemkes, S., Karakose, T., Antal, A., Liebetanz, D., Lang, N., et al. (2007). Shaping the effects of transcranial direct current stimulation of the human motor cortex. Journal of Neurophysiology, 97, 3109–3117. Nitsche, M. A., Fricke, K., Henschke, U., Schlitterlau, A., Liebetanz, D., Lang, N., et al. (2003). Pharmacological modulation of cortical excitability shifts induced by transcranial

direct current stimulation in humans. The Journal of Physiology, 553, 293–301. Nitsche, M. A., Seeber, A., Frommann, K., Klein, C. C., Rochford, C., Nitsche, M. S., et al. (2005). Modulating parameters of excitability during and after transcranial direct current stimulation of the human motor cortex. The Journal of Physiology, 568, 291–303. Nowak, D. A., Grefkes, C., Ameli, M., & Fink, G. R. (2009). Interhemispheric competition after stroke: Brain stimulation to enhance recovery of function of the affected hand. Neurorehabilitation and Neural Repair, 23, 641–656. Nowak, D. A., Grefkes, C., Dafotakis, M., Eickhoff, S., Kust, J., Karbe, H., et al. (2008). Effects of low-frequency repetitive transcranial magnetic stimulation of the contralesional primary motor cortex on movement kinematics and neural activity in subcortical stroke. Archives of Neurology, 65, 741–747. Nudo, R. J., & Dancause, N. (2007). Neuroscientific basis for occupational and physical therapy interventions. In N. D. Zasler, D. I. Katz & R. D. Zafonte (Eds.), Brain injury medicine: Principles and practice. New York: Demos Medical Publishing. Nudo, R. J., Jenkins, W. M., & Merzenich, M. M. (1990). Repetitive microstimulation alters the cortical representation of movements in adult rats. Somatosensory & Motor Research, 7, 463–483. Nudo, R. J., & Milliken, G. W. (1996). Reorganization of movement representations in primary motor cortex following focal ischemic infarcts in adult squirrel monkeys. Journal of Neurophysiology, 75, 2144–2149. Nudo, R. J., Milliken, G. W., Jenkins, W. M., & Merzenich, M. M. (1996a). Use-dependent alterations of movement representations in primary motor cortex of adult squirrel monkeys. The Journal of Neuroscience, 16, 785–807. Nudo, R. J., Wise, B. M., Sifuentes, F., & Milliken, G. W. (1996b). Neural substrates for the effects of rehabilitative training on motor recovery after ischemic infarct. Science, 272, 1791–1794. Papadopoulos, C. M., Tsai, S. Y., Guillen, V., Ortega, J., Kartje, G. L., & Wolf, W. A. (2009). Motor recovery and axonal plasticity with short-term amphetamine after stroke. Stroke, 40, 294–302. Patel, P. M., Drummond, J. C., Sano, T., Cole, D. J., Kalkman, C. J., & Yaksh, T. L. (1993). Effect of ibuprofen on regional eicosanoid production and neuronal injury after forebrain ischemia in rats. Brain Research, 614, 315–324. Penfield, W., & Rasmussen, T. L. (1952). The cerebral cortex of man. New York: Macmillan. Plautz, E. J., Barbay, S., Frost, S. B., Friel, K. M., Dancause, N., Zoubina, E. V., et al. (2003). Post-infarct cortical plasticity and behavioral recovery using concurrent cortical stimulation and rehabilitative training: A feasibility study in primates. Neurological Research, 25, 801–810.

294 Plautz, E. J., Milliken, G. W., & Nudo, R. J. (2000). Effects of repetitive motor training on movement representations in adult squirrel monkeys: Role of use versus learning. Neurobiology of Learning and Memory, 74, 27–55. Plow, E. B., Carey, J. R., Nudo, R. J., & Pascual-Leone, A. (2009). Invasive cortical stimulation to promote recovery of function after stroke: A critical appraisal. Stroke, 40, 1926–1931. Pomeroy, V. M., Cloud, G., Tallis, R. C., Donaldson, C., Nayak, V., & Miller, S. (2007). Transcranial magnetic stimulation and muscle contraction to enhance stroke recovery: A randomized proof-of-principle and feasibility investigation. Neurorehabilitation and Neural Repair, 21, 509–517. Rapisarda, G., Bastings, E., De Noordhout, A. M., Pennisi, G., & Delwaide, P. J. (1996). Can motor recovery in stroke patients be predicted by early transcranial magnetic stimulation? Stroke, 27, 2191–2196. Raux, M., Xie, H., Similowski, T., & Koski, L. (2010). Facilitatory conditioning of the supplementary motor area in humans enhances the corticophrenic responsiveness to transcranial magnetic stimulation. Journal of Applied Physiology, 108, 39–46. Reinecke, S., Dinse, H. R., Reinke, H., & Witte, O. W. (2003). Induction of bilateral plasticity in sensory cortical maps by small unilateral cortical infarcts in rats. The European Journal of Neuroscience, 17, 623–627. Rema, V., & Ebner, F. F. (2003). Lesions of mature barrel field cortex interfere with sensory processing and plasticity in connected areas of the contralateral hemisphere. The Journal of Neuroscience, 23, 10378–10387. Remple, M. S., Bruneau, R. M., Vandenberg, P. M., Goertzen, C., & Kleim, J. A. (2001). Sensitivity of cortical movement representations to motor experience: Evidence that skill learning but not strength training induces cortical reorganization. Behavioural Brain Research, 123, 133–141. Riecker, A., Groschel, K., Ackermann, H., Schnaudigel, S., Kassubek, J., & Kastrup, A. (2010). The role of the unaffected hemisphere in motor recovery after stroke. Human Brain Mapping, 31, 1017–1029. Rosenow, J., Das, K., Rovit, R. L., & Couldwell, W. T. (2002). Irving S. Cooper and his role in intracranial stimulation for movement disorders and epilepsy. Stereotactic and Functional Neurosurgery, 78, 95–112. Rouiller, E. M., Babalian, A., Kazennikov, O., Moret, V., Yu, X. H., & Wiesendanger, M. (1994). Transcallosal connections of the distal forelimb representations of the primary and supplementary motor cortical areas in macaque monkeys. Experimental Brain Research, 102, 227–243. Rouiller, E. M., Yu, X. H., Moret, V., Tempini, A., Wiesendanger, M., & Liang, F. (1998). Dexterity in adult monkeys following early lesion of the motor cortical hand area: The role of cortex adjacent to the lesion. The European Journal of Neuroscience, 10, 729–740.

Sakatani, K., Iizuka, H., & Young, W. (1990). Somatosensory evoked potentials in rat cerebral cortex before and after middle cerebral artery occlusion. Stroke, 21, 124–132. Schaechter, J. D., & Perdue, K. L. (2008). Enhanced cortical activation in the contralesional hemisphere of chronic stroke patients in response to motor skill challenge. Cerebral Cortex, 18, 638–647. Schallert, T., Fleming, S. M., & Woodlee, M. T. (2003). Should the injured and intact hemispheres be treated differently during the early phases of physical restorative therapy in experimental stroke or parkinsonism? Physical Medicine and Rehabilitation Clinics of North America, 14, S27–S46. Schambra, H. M., Sawaki, L., & Cohen, L. G. (2003). Modulation of excitability of human motor cortex (M1) by 1 Hz transcranial magnetic stimulation of the contralateral M1. Clinical Neurophysiology, 114, 130–133. Seitz, R. J., Hoflich, P., Binkofski, F., Tellmann, L., Herzog, H., & Freund, H. J. (1998). Role of the premotor cortex in recovery from middle cerebral artery infarction. Archives of Neurology, 55, 1081–1088. Sharp, F. R., Bolger, E., & Evans, K. (1982). Thiamine deficiency limits glucose utilization and glial proliferation in brain lesions of symptomatic rats. Journal of Cerebral Blood Flow and Metabolism, 2, 203–207. Sharp, F. R., & Ryan, A. F. (1984). Regional (14C) 2deoxyglucose uptake during forelimb movements evoked by rat motor cortex stimulation: Pons, cerebellum, medulla, spinal cord, muscle. The Journal of Comparative Neurology, 224, 286–306. Shumway-Cook, A., & Woollacoot, M. (2001). Motor control: Theory and practical applications. Baltimore, MD: Williams & Wilkins. Slavin, M. D., Laurence, S., & Stein, D. G. (1988). Another look at vicariation. In S. Finger, T. E. Levere, C. R. Almli & D. G. Stein (Eds.), Brain injury and recovery: Theoretical and controversial issues. New York: Plenum Press. Stoney, S. D., Jr. Thompson, W. D., & Asanuma, H. (1968). Excitation of pyramidal tract cells by intracortical microstimulation: Effective extent of stimulating current. Journal of Neurophysiology, 31, 659–669. Stowe, A. M., Plautz, E. J., Nguyen, P., Frost, S. B., EisnerJanowicz, I., Barbay, S., et al. (2008). Neuronal HIF-1 alpha protein and VEGFR-2 immunoreactivity in functionally related motor areas following a focal M1 infarct. Journal of Cerebral Blood Flow and Metabolism, 28, 612–620. Stroemer, R. P., Kent, T. A., & Hulsebosch, C. E. (1998). Enhanced neocortical neural sprouting, synaptogenesis, and behavioral recovery with D-amphetamine therapy after neocortical infarction in rats. Stroke, 29, 2381–2393 discussion 2393–2395. Takeuchi, N., Chuma, T., Matsuo, Y., Watanabe, I., & Ikoma, K. (2005). Repetitive transcranial magnetic

295 stimulation of contralesional primary motor cortex improves hand function after stroke. Stroke, 36, 2681–2686. Talelli, P., Greenwood, R. J., & Rothwell, J. C. (2007). Exploring theta burst stimulation as an intervention to improve motor recovery in chronic stroke. Clinical Neurophysiology, 118, 333–342. Taub, E. (1980). Somatosensory deafferentation research with monkeys. Implications for rehabilitation medicine. In L. P. Ince (Ed.), Behavioral psychology in rehabilitation medicine: Clinical implication. New York: Williams and Wilkins. Taub, E., & Morris, D. M. (2001). Constraint-induced movement therapy to enhance recovery after stroke. Current Atherosclerosis Reports, 3, 279–286. Taub, E., Uswatte, G., & Elbert, T. (2002). New treatments in neurorehabilitation founded on basic research. Nature Reviews. Neuroscience, 3, 228–236. Taub, E., Uswatte, G., & Morris, D. M. (2003). Improved motor recovery after stroke and massive cortical reorganization following constraint-induced movement therapy. Physical Medicine and Rehabilitation Clinics of North America, 14, S77–S91 ix. Taub, E., Uswatte, G., & Pidikiti, R. (1999). Constraintinduced movement therapy: A new family of techniques with broad application to physical rehabilitation—A clinical review. Journal of Rehabilitation Research and Development, 36, 237–251. Teskey, G. C., Flynn, C., Goertzen, C. D., Monfils, M. H., & Young, N. A. (2003). Cortical stimulation improves skilled forelimb use following a focal ischemic infarct in the rat. Neurological Research, 25, 794–800. Tsai, S. Y., Markus, T. M., Andrews, E. M., Cheatwood, J. L., Emerick, A. J., Mir, A. K., et al. (2007). Intrathecal treatment with anti-Nogo-A antibody improves functional recovery in adult rats after stroke. Experimental Brain Research, 182, 261–266. Turton, A., Wroe, S., Trepte, N., Fraser, C., & Lemon, R. N. (1996). Contralateral and ipsilateral EMG responses to transcranial magnetic stimulation during recovery of arm and hand function after stroke. Electroencephalography and Clinical Neurophysiology, 101, 316–328. Uswatte, G., Taub, E., Morris, D., Barman, J., & Crago, J. (2006). Contribution of the shaping and restraint components of constraint-induced movement therapy to treatment outcome. NeuroRehabilitation, 21, 147–156. Ward, N. S., Brown, M. M., Thompson, A. J., & Frackowiak, R. S. (2004). The influence of time after stroke

on brain activations during a motor task. Annals of Neurology, 55, 829–834. Wellman, C. L., Arnold, L. L., Garman, E. E., & Garraghty, P. E. (2002). Acute reductions in GABAA receptor binding in layer IV of adult primate somatosensory cortex after peripheral nerve injury. Brain Research, 954, 68–72. Whishaw, I. Q. (2000). Loss of the innate cortical engram for action patterns used in skilled reaching and the development of behavioral compensation following motor cortex lesions in the rat. Neuropharmacology, 39, 788–805. Whishaw, I. Q., Pellis, S. M., Gorny, B. P., & Pellis, V. C. (1991). The impairments in reaching and the movements of compensation in rats with motor cortex lesions: An endpoint, videorecording, and movement notation analysis. Behavioural Brain Research, 42, 77–91. Whishaw, I. Q., Piecharka, D. M., Zeeb, F., & Stein, D. G. (2004). Unilateral frontal lobe contusion and forelimb function: Chronic quantitative and qualitative impairments in reflexive and skilled forelimb movements in rats. Journal of Neurotrauma, 21, 1584–1600. Wittenberg, G. F., Chen, R., Ishii, K., Bushara, K. O., Eckloff, S., Croarkin, E., et al. (2003). Constraint-induced therapy in stroke: Magnetic-stimulation motor maps and cerebral activation. Neurorehabilitation and Neural Repair, 17, 48–57. Wolf, S. L., Winstein, C. J., Miller, J. P., Taub, E., Uswatte, G., Morris, D., et al. (2006). Effect of constraint-induced movement therapy on upper extremity function 3 to 9 months after stroke: The EXCITE randomized clinical trial. Journal of the American Medical Association, 296, 2095–2104. Woolsey, C. N., Settlage, P. H., Meyer, D. R., Spencer, W., Hamuy, T. P., & Travis, A. M. (1952). Patterns of localization in precentral and “supplementary” motor areas and their relation to the concept of a premotor area. Research Publications—Association for Research in Nervous and Mental Disease, 238–264. Yozbatiran, N., Alonso-Alonso, M., See, J., DemirtasTatlidede, A., Luu, D., Motiwala, R. R., et al. (2009). Safety and behavioral effects of high-frequency repetitive transcranial magnetic stimulation in stroke. Stroke, 40, 309–312. Yrjanheikki, J., Keinanen, R., Pellikka, M., Hokfelt, T., & Koistinaho, J. (1998). Tetracyclines inhibit microglial activation and are neuroprotective in global brain ischemia. Proceedings of the National Academy of Sciences of the United States of America, 95, 15769–15774.

Subject Index

Age-related macular degeneration (AMD) blindness in elderly, 6 plasticity cortical reorganization, 203 “filling-in” phenomenon, 202 preferred retinal location (PRL), 202–203 Amblyopia description, 223–224 training Gabor patch detection task, 223 therapy, children, 224 VA and binocular functions, 225 AMD. See Age-related macular degeneration American Spinal Injury Association (ASIA), 164 Amputation prosthetics control architecture, 66 electromyographic activity (EMG), 65–66 mechanical impedance, 66 targeted muscle reinnervation (TMR), 65–66 Areas of residual vision (ARVs) excitation-inhibition dysbalance functional silencing, 243 visual border, 243 intact hemisphere, functional silencing, 245–246 visual-field border high-resolution perimetry (HRP), 209, 210 perception quality, 209–212 perimetric methods, 209 plasticity, 212 testing, 212 Artificial Retina Project, 8–9 Artificial silicone retina (ASR), 8 ARVs. See Areas of residual vision Auditory-to-vision sensory substitution (AVSS), 22

BCI. See Brain-computer interfaces Bionic eye brain plasticity, 12–13 challenges device efficacy, 11 implanted recipients, 11–12 neuroplastic transformation, 12 sensory-substitution devices, 12 visually deprived brain, 12 technical challenges brain–machine interfaces (BMIs), 3–4 clinical therapy, 9 electrode geometry, 9–10 eye movement, 10 imaging methodology, 11 implantation, 10, 11 molecular-based therapies, 4–5 neuroprosthetic approaches, 3–4 robotic limb prosthesis, 3–4 sensory-substitution devices, 4–5 visual impairment, 4–5 visual neuroprosthetic approaches “ab externo”, 6–8 ASR, 8 Boston Retinal Implant device, 7 electrical stimulation, cortex, 9 German-based consortiums, 9 head-scanning techniques, 8–9 image processing, 8 microelectrical stimulation, 5–6 optic nerve stimulation, 9 perception neural code, 6 retinal-based approaches, 6 RP and AMD, 6 297

298

Bionic eye (Continued) signal processing, 6–8 subretinal design, 6–8 vision restoration, 5 vitreoretinal surgery technique, 8–9 Blindness, brain structure and function anatomy, human imaging techniques, 20–21 metabolic changes, 21 volume reduction, 20–21 white matter (WM) changes, 21 Braille reading, 18 cortical recognition/unmasking neuronal connection, 28 TMS impair, 28 cross-modal plasticity sensory substitution devices, 22 TDU, 22 dorsal and ventral visual stream motion-sensitive hMTþ complex, 22–24 occipital cortex, 23 shape recognition task, 24 TDU, 22–24 navigation description, 25 fMRI data and experiment, 26 TDU, 26 occipital cortex activation cognitive task, 25 olfactory processing, 24–25 somatosensory system, 24 vision, 17–18 visual cortex activation somatosensory cortex stimulation, 26–27 tactile sensation, 27–28 “tactile sensations”, 26–27 visual cortex activity, 28 visual deprivation model, animal auditory and somatosensory system, 18–19 cerebral cortex, 18–19 peripheral inputs, neocortex, 18–19 “rewired” hamster, 19, 20 Blindsight, residual vision training grating stimuli, 222 striate/extrastriate training approach, 223

unilateral occipital damage, 222 vs. visual border stimulation, 223 Blood oxygen level dependent (BOLD) response neural metabolic activity, 265–266 neurofeedback, ACC, 269 Body weight support on a treadmill training (BWSTT), 164 BOLD. See Blood oxygen level dependent Boston retinal implant device, 7 Braille reading, 18 Brain-computer interfaces (BCI) ACC metabolism, 267 control, 103–104, 110 definition, 265 ECoG and EEG recordings, 65 EEG neurofeedback, 265 fMRI applications, 266 BOLD, 265–266 definition, 265–266 mental health and psychiatric disorders, 267 subsytem, fMRI, 266 Thorndike's Law of Effect, 266–267 imagination, 103–104 pedophilia, fMRI ACC, 269–270 BCI-GUI, 268 Caucasian avatars, 267 CBT and 3D technology, 267 eye-tracking (ET) technique, 268 penile plethysmography (PPG), 268 penile tumescence and gaze behavior, 268 ROC analysis, 269 sensitivity (Se) and specificity (Sp), 268–269 VISS, 267–269 principles, 266–267 research, 65 ROI network, 266–267 single-neuron recordings, 104, 106 tuning curve, 104–105 Brain-control (BC) condition, 105 cortical activity, 104–105, 106 cursor, 105 curves record, tuning, 104–105

299

neural signal, 104–105, 106 tuning differences, 105 Brain–machine interfaces (BMI) afferent electrical simulation, 86 ICMS, 86 kinematic control information, 85 mimics natural somatosensation, 85–86 peripheral stimulation, 86 psychometric curves, 86, 87 skin tactile receptors, 86 vibration sense, 86 efferent functional electrical stimulation, 84 multielectrode recording, 84 myoelectric signals, 85 premotor cortical activity, 84–85 primary motor cortex, 84–85 TMR procedure, 85 Brain stimulation, injury recovery after stroke anodal and cathodal, 283–284 cortical, 283 fundamental rational deoxyglucose, metabolic marker, 283 electrical, 283 Parkinson's disease (PD), 283 noninvasive techniques, after stroke contralesional cortex inhibition, 286 ipsilesional M1, 286 tDCS, 285, 286 TMS, 286 rewiring contralesional cortex, 284 delta waves, 284 ipsilateral control, 284–285 synchronous neuronal activity, 284 transcranial magnetic stimulation (TMS), 284–285 stroke, human complexity, 285 phase III trial, 285

CIMT. See Constraint-induced movement therapy Cognitive behavioral therapy (CBT) deviant sexuality, 267 treatment, sex offenders, 269–270 Constraint-induced movement therapy (CIMT) arm motor function, 62 robotics, 62–63 Cooperative studies program (CSP) Food and Drug Administration (FDA), 61 intensive comparison training (ICT) protocol, 62 multisite RCT, 61 neural plasticity, 62 Wolf Motor function test, 62 Cortical plasticity EPSC amplitude, 88, 89 long-term potentiation (LTP), 88 spike-triggered electrical stimulation, 90 STDP, 88 synaptic efficacy change, 88 timed visual stimuli, 89 TMS, 89–90 Corticospinal tract (CST) contribution to normal walking EMG:EMG synchronization, 184–186 neuroimaging, 182 supraspinal centers, 181–182 TMS and TES, 183–184 enhancing recovery, after lesion, 191 involvement in adaptation and perturbation, gait CNS, 186 “force field” walking, proactive vs. reactive mechanisms, 187–188 proactive modifications, 186 reactive modifications, 186–187 walking after impairment combining electrophysiological, imaging and gait analysis, 189–191 excitability levels, early after stroke, 188 locomotor training, 188–189 CST. See Corticospinal tract

Calcium-binding proteins (CaBPs), 19 CBT. See Cognitive behavioral therapy

Deafferentation depression, 240–241 Diaschisis, 240–241

300

Diffusion tensor imaging (DTI), 20–21 Diffusion tensor tractography (DTT), 20–21 DTI. See Diffusion tensor imaging DTT. See Diffusion tensor tractography Efferent and afferent feedback, 131 Electrical stimulation alterative forced choice (AFC), 77 mechanical indentation, 77 mechanical probe intensity, 77 somatotopic map, 77–78 EMG:EMG synchronization coupling between motor units, 185–186 motor cortex activity, 184–185 short-term synchrony, 184–185 time and frequency domains, 184–185 TMS and coherence observations, 186 Evidence-based Review of Stroke Rehabilitation, 149 “Extrastriate-VRT” (eVRT), 223 Eye movement training, 224 fMRI. See Functional magnetic resonance imaging Functional magnetic resonance imaging (fMRI), 10, 263–264 Gait analysis, SCI survivors biomechanical analysis ankle and hip power profiles, 170–172 biofeedback, triceps surae, 167 BWSTT, 169 3D positions, 166 effects, pharmacological agents, 170 EMG recordings, 169–170 energy generation, 168–169 force field, AFO, 174 functional ambulators, 169 hip and ankle mean power curves, 173 hip hiking, 169–170 knee hyperextension, 166–167 late stance power generation burst, 167 mechanical moment, 169 non-maximal speed, walking, 168–169 overloading, lower limb, 175

overly sensitive exteroceptive reflexes, 166–167 overuse, hip extensors, 170 paretic side after stroke, S10 and S11 subjects, 175 plantarflexing force, 173–175 plantar flexor angle, 166 power data after stroke, 167–168 preferred gait speeds, 170 pre-injury motor patterns, 170 TA and RF, 173–175 clinical assessments, 163 internal and external forces, 162 locomotor training, 162 task-oriented approach ASIA C and D subjects, 164 BWSTT, 164 clinical indices, 165 mean differences, 164–165 measurable differences, 164 muscle strengthening exercises, 165–166 neurological classification, ASIA, 164 RCTs, 163 treadmill training results, 164 technological advances, 162–163 walking endurance, 162 Gait patterns proactive modifications, 186 reactive modifications, 186–187 g-amino butyric acid subtype a (GABAa), 275 Glaucoma vs. AMD, 203 retinal lesions, 203–204 VRT, 203 Haptic robots anthropomorphic systems, 138 arms and hands, 138, 139 biomimetic tactile sensing, 138, 140, 141 control algorithm, 138 exploratory algorithm, 141 internal representation, 141–142 Hebbian conditioning altered behavior auditory cue intensity, 95 ICMS cue-detection task, 94

301

Laplace approximation, 95 noise, 96 perceptual threshold, 95, 96 psychophysical curve, 95, 96 short-latency experiments, 95–96 IFC. See Inferred functional connectivity Haptics, mechatronic system biomimetic design, 130 definition, 129 description, 129 entities testing Bayesian decision making, 134–135 object identification, 134 parsimonious extension, 135 sensorimotor library, 133–134 sensory signals, 134–135 exteroceptive senses integration attributes, 133 sound, 133 visual information, 133 function restoration, 131 human dexterity, 129–130 internal representation, objects “critical periods”, 130–131 deformable objects, 132 materials, 133 rigid objects, 132 self identification, 131–132 surfaces, 132 tools, 133 mechatronic platform, 138 multimodel tactile information, 130 robots anthropomorphic systems, 138 arms and hands, 138, 139 biomimetic tactile sensing, 138, 140, 141 control algorithm, 138 exploratory algorithm, 141 internal representation, 141–142 sensorimotor control cortical consolidation, 137 genetic algorithm, 136 industrial robots, 135–136 interneurons, 135–136 spinal cord function, 135–136

spinal regulator design, 135–136 task performance, 136–137 “theory of computation”, 129–130 visual targeting, 138–141 Hip hiking, 169–170 ICMS. See Intracortical microstimulation techniques IFC. See Inferred functional connectivity IHT. See In-home teletherapy (IHT) Inferred functional connectivity (IFC) altered, Hebbian condition functional connectivity, 92 monosynaptic connection, 94 network activity recording, 92 spike-triggered stimulus, 92–93 targeted and non-targeted potentiation, 94 trigger-to-target pair changes, 93 neuron network baseline firing rate, 90, 91 Bayesian approach, 90–91 functional connectivity, 91–92 kernel, 90 maximum a posteriori (MAP), 90–91 stimulus-based methods, 90 In-home teletherapy (IHT) clients, 152 equipment, 148–149 exercise therapy, 149–150, 155 implementation, 148 Internet, 148 interventions, 148 outcome evaluation, fatigue effect, 155 ReJoyce workstations, 151 stimulators, 152 trail, 155–156 treatment, 148 Integrated theory of sexual offending (ITSO), 264 Internal reference frames, 131–132 Intracortical microstimulation techniques (ICMS) cortical sector, 276–277 electrode stimulation, 283 functional representation, 274 M1 hand area, 277

302

Invasive current stimulation methods congenital blindness, 225 optic nerve, 224–225 subcorticography and diagnostic stimulation, 225 visual cortex, 224 ITSO. See Integrated theory of sexual offending Lateral occipital tactile vision (LOtv), 24 Locomotor training, cortical plasticity after lesion to spinal cord, 189 after stroke, 188–189 Long-term depression (LTD) lost function restoration, neuronal, 247–248 NMDA receptors, 246–247 synaptic transmission, 248 LOtv. See Lateral occipital tactile vision Long-term potentiation (LTP) Map plasticity in motor cortex CIMT, 277 hand representations, lesions, 276–277 neurophysiologic, 276 skill-/learning-dependent, 276 training techniques, 277 motor cortex networks, effect axonal projection, 278–279 de novo cortical connection, 278–279 distal forelimb representation, 278 PMv, 277–278 stroke consequence, 279 two stage process cortico-cortical projection, 275 GABAa receptor, 275 NMDA receptor, 275–276 primary motor cortex, 275 Mechanical intensity discrimination comparison stimulus, 79–80 perceived probe indentation, 80 somatosensory cortex, 80 veridical intensity percepts, 79–80 MEPs. See Motor evoked potentials MIT-Manus robotic exercise therapy, 149–150 Motor evoked potentials (MEPs), 183–184

National Eye Institute-Visual Functioning Questionnaire (NEI-VFQ), 229–230 Neural prosthetic system across-trial firing-rate, Fano factors firing-rate variability changes, 42–44 instructed-delay task, 41–42 preparatory and neural activity, 41 RT, 42 arm movement cortical control, 34 instructed-delay task, hand measurements and EMG, 36 neural responses, 37 neurons, 36–37 reaction times (RTs), 36, 37 voluntary, 35–36 barriers, 35 design communication prostheses and “transit time”, 53 instructed-delay reach task, 52 noisy electrical activity, neuron population, 51–53 domains, 50 Fano factor and RTs, lower experiments, 96-channel electrode array, 45 positive correlation, 44 saccadic eye movements, 45 short delay duration trials, 44 state space, 44–45 intracortical electrical microstimulation, 50–51 loop, RT go cue, 49 preparatory state, 50 state space, 49–50 movement, 33–34 neural prosthetic systems, 34 neural trajectories, 51 optimal subspace hypothesis dynamical systems perspective, 37–38 motor preparation, 38 neural activity, 39 neuron firing rate, 38–39 noise source, 38 preparatory activity, 39 tuning, 39

303

perturbing neural activity, RT perturbation experiments, 46–47 PMd, 45–46 preparatory activity, 45 state space, 45, 46 reach-speed modulation delay-period activity, 40 optimal subspace hypothesis, 40 single-trial neural trajectories multiple, GPFA, 47–48 neurons, 47 single-neuron recordings, 48–49 state space, 47 state-space trajectories, 51 technologies, 34–35 trial-by-trial correlation firing rate, 41 preparatory activity, 40–41 reach-speed variant, 41 Neurobiological mechanisms, vision restoration and attention, 248–249 cellular, plasticity LTD, 248 LTP, 246–247, 248 and molecular mechanisms, 246 neurons metaplasticity and silencing, 247–248 neuron strengthening, 246 residual structures, 246 minimal residual structure hypothesis description, 238–239 dysfunction, 239 “network” plasticity, 239 occipital cortex removal, 239 Parkinson's disease, 239 plasticity and neuronal nuclei, 239 network plasticity downstream networks, 243 excitation-inhibition balance, 243–246 nuclei damage, 240 primary/secondary deafferentation, 240 RFP, deafferented brain structure, 240–243 visual structure loss, 240 neuronal synchronization, 248 within-systems plasticity cellular changes, 240

residual tissue, 239 RGCs, 240 Neuroengineering, 201 Neuroimaging muscle activity during human bicycling, 182 SPECT and PET, 182 -statistics, 183 Noninvasive current stimulation methods description, 225 electroencephalogram (EEG)-power-spectra analysis, 226–228 rtACS, 226–227 tDCS, 228 TES, 228 vision restoration, 228, 229 OAS. See Odor awareness scale OCT. See Optical coherence tomography Odor awareness scale (OAS), 24–25 Optical coherence tomography (OCT), 10, 11 Paired associative stimulation (PAS), 191 Parkinson's disease (PD), 283 PAS. See Paired associative stimulation PD. See Parkinson's disease Pedophilia BCI. See Brain-computer interfaces description, 263 diagnosis, 264 offending selfregulation, 264–265 prevalence, 263–264 and sexual arousal functional neuroanatomy ACCs, 265 treatment, 265 sexual deviance etiology, 264 Plasticity map. See Map plasticity Plasticity, recovery after injury brain stimulation. See Brain stimulation, injury recovery concept, plastic brain functional, 274 rehabilitative intervention, 273 spontaneous, 273–274 contralesional cortex inhibition

304

Plasticity, recovery after injury (Continued) motor cortex suppression, 287–288 multiple session effect, 288 contralesional hemisphere role ipsilesional cortex, 287 negative role, 287 injury rewiring, stimulation cortical, 284 EEG studies, 284–285 ipsilateral control, 284–285 map plasticity motor cortex, 276–277 two stage process, 275–276 motor cortex network effect, 277–279 noninvasive brain stimulation technique, 285–286 postlesion map change human neuroimaging, 280–281 kinematics, forelimb movement, 280 motor representation, rostral forelimb, 280 PMv reorganization, M1 lesion, 279 SMA, 279–280 stimulation electrical, 275–276 ICMS, 283 stroke, 285 topographic maps, functional index biological marker, 274–275 disuse/behavioral training, 274 treatment, stroke patient amphetamine pairing, 282 CIMT, 281 cortical stimulation, 282 EXCITE, 282 GABAa receptors, 282 pharmacological agent, 282 rehabilitation, 281 tissue plasminogen activator (t-PA), 281 upper limb representational map size, 281–282 Plasticity, visual system after optic nerve lesions animal models, 204 neuritis recovery, human, 204–206 prechiasmatic, 204

after post-chiasmatic lesions human, vision spontaneous recovery, 207–208 upstream brain region damage, 206 vision spontaneous recovery, animals, 206–207 after retinal lesions acute damage, 202 AMD, 202–203 glaucoma, 203–204 congenital and early blindness description, 208 neural reorganization, 208 plasticity, 209 transcranial magnetic stimulation (TMS), 208–209 perceptual learning practice, 201–202 tasks, 201 RF. See Receptive field plasticity synaptic. See Synaptic plasticity PMC. See Premotor cortex Population inference activity, 110 advantage, signal control, 108 brain-computer interfaces (BCIs) map, 103 correlations, neural activity, 108, 109 device control, 103 function, 110 dimensionality reduction techniques, 110 driving signals, 108 Fetz's technique, 108 firing rate predictions, 110 motor imagery, 108 neuron firing rate, 108 operant-conditioning, 110 primary motor cortex, 104 prostheses arm movements. See Prostheses, arm movements cursor imagery. See Prostheses, cursor imagery prosthetic decoding and neural activity, 110 signal control, 104

305

single-neuron recordings, 104 spike trains, neural trajectory, 108, 109 volitional control signals, 108 Premotor cortex (PMC), 191 Prostheses arm movements cortical activity changes, 104–105, 106 cortical signal and record turning curves, 104–105 data, 104 hand and brain control, 104–105 motor cortex and neural activity, 104 signal control, 104–105 cursor imagery electrocorticographic signals, 105 imagined wrist/elbow flexion, 105 interactive procedure and parameters control, 107 limitations, 107–108 motor observation, 105–107 movement data, 105 targets and decoding parameters, 105–107 Randomized controlled trials (RCTs), 61, 163 Real-time functional magnetic imaging (rt-fMRI) diagnosis, pedophilia, 264 fMRI-BCI, pedophilia. See Brain-computer interfaces functional neuroanatomy, 265 neurological determination, 264–265 psychological and financial cost, 263–264 self regulation, 264–265 sex offender, 263 sexual abuse incidence rate, 263–264 sexual deviance etiology, 264 Receptive field (RF) plasticity cats and monkeys, 241 cell loss, 240–241 cortical interneurons, 241 hot spots comparison, 242 object detection, 242 reorganization, 241 spike activity, 241–242 “vision-for-perception” tasks, 242–243 visual cortex, 241

visual-field charts, 242 VRT, 242 Repetitive transcranial magnetic stimulation (rTMS), 25 Residual vision binocular visual functions, 213 blindsight mechanism, 213 vision restoring pathways, 212–213 electrical current stimulation, activation invasive, 224–225 noninvasive, 226–228 experience, activation amblyopia, 213 CNS damage, post lesion recovery, 213–214 early recovery phase, 214 training, 214–224 training, activation animal, 214–216 human, 216–224 visual stimuli, 214 vision activation theory, 201 visual field border, 209–212 Retinitis pigmentosa (RP), 6 Rewired hamster brain anatomy and behavioral outcome, 20 auditory cortex lesions, 19 CaBPs distribution, 19 nonvisual thalamic site, 19 retinofugal projection, 19 Robotic technology, neuromotor rehabilitation amputation prosthetics, 65–66 applicability CIMT, 62 Fugl-Meyer scale, 63–64 impairment reduction, 63–64 robot therapy effect, 62–63 assistive technologies iBOR, 65 MANUS robot, 65 “personal best” recovery, 64 spinal cord injury (SCI), 64 brain-computer interfaces, 65 cost-effectiveness active area, research, 64

306

Robotic technology, neuromotor rehabilitation (Continued) benefit analysis, 64 neuromechanical system, 64 robotic equipment, 64 CSP-558, 61–62 length of stay (LOS), 59 neuro-and biomechanical factors, 60 quality, care, 60 stroke, 59–60 therapy, 60–61 Robotic therapy biomechanical integrity, 60–61 motor rehabilitation lower (LE) and upper extremity (UE), 61 movement initiation, 61 musculo-skeletal integrity, 60–61 randomized clinical trials (RCTs), 61 sensory-motor function, 60 shoulder-hand syndrome, 60–61 rt-fMRI. See Real-time functional magnetic imaging rTMS. See Repetitive transcranial magnetic stimulation Sensorimotor behavior and neuronal functional connectivity adaptation role MT stimulation, 88 neuronal and limb state mapping, 87–88 neurons, 88 pre and postsynaptic activity, 88 afferent BMI electrical simulation, 86 ICMS, 86 kinematic control information, 85 mimics natural somatosensation, 85–86 peripheral stimulation, 86 psychometric curves, 86, 87 skin tactile receptors, 86 vibration sense, 86 efferent BMI functional electrical stimulation, 84 multielectrode recording, 84 myoelectric signals, 85

premotor cortical activity, 84–85 primary motor cortex, 84–85 TMR procedure, 85 Hebbian association and cortical plasticity, 88–90 Hebbian conditioning altered behavior, 94–96 IFC, 92–94 IFC, neurons, 90–92, 99 neural rehabilitation, potential applications cortical stimulation, 98 FES, 97–98 hand and finger movement, 98 Hebbian association, 98 injury, 96–97 upper limb function recovery, 96–97 plasticity, 84 standard rehabilitation approaches, 85 Sensorimotor control consolidation, cortical, 137 cortical representation learning curves, 137 trial-and-error learning, 137–138 embryological development, 135 performance actuators activation, 136–137 trial-and-error refinement, 136–137 spinal regulator design biological regulator, 135–136 genetic algorithm, 136 industrial robots, 135–136 interneurons, 135–136 “redundancy”, 136 spinal cord function, 135–136 Sensorimotor cortex (SMC), 188 Sensory feedback, upper limb prostheses electrical substitution task psychometric function, 79 somatotopic map, 78, 79 stimulus detection, 79 functional roles, peripheral afferents controlled action/movements, 71 effective stimulation, 73–74 Golgi tendon organs, 72 joint replacement surgery, 72

307

mechanoreceptive system, 73–74 muscle force, 72 pacinian (PC) afferents, 73 paciniform corpuscles, 72 peripheral receptors, 74 Pinocchio effect, 72 rapidly adapting (RA) and slowly adapting type 1 (SA1), 73 receptor types, 71–72 sensing temperature, 74 somatosensory pathways, 74, 75 tactile motion, 73 inherent modality specificity, 70 intensity discrimination, 79–80 limb Luke Skywalker, 69–70 mechanical intensity, 77–78 motor control, 69–70 neurons stimulation, 80 perception pathways, 74–75 sensory substitution methods, 70 somatosensory cortex advantages, 77 non-human primates, 75 natural percept goal, 76 neural coding mechanisms, 76 stimulating neurons, 75–76 tonotopic representation, 76 stimulation biphasic pulses, 79 custom-written Matlab code, 78 targeted reinnervation, 70 SMC. See Sensorimotor cortex Spike-timing dependent plasticity (STDP), 88, 89 Spinal cord injury (SCI) duration and intensity, 156–157 fatigue resistance, 155 hand movements, 155 low function, 153 moderate physical activity, 155 Spinal Cord Injury Rehabilitation Evidence, 149 Spontaneous vision recovery animals cats, 206 cortical reorganization, 207 macaque monkey, 207

occipital lobectomies, 207 rats, 206 unilateral lesions, 207 visual cortex damage, 206 human cortex ischemic lesions, 208 functional neuroimaging, 208 geniculostriate lesions, 207 transmodal plasticity, 208 visual-field defects, 208 STDP. See Spike-timing dependent plasticity Synaptic plasticity, 247 TA. See Tibialis anterior Tactile communication system applications, 113–114 array, 114 benefits, 125 compensate sensory loss, 114 dense arrays, 114 devices development, skin deformation, 114 displays development, 116–117 electromechanical actuators, 116–117 mobile users, 116–117 skin and actuator/contractor, 116–117 technologies, 114, 116–117 ear and skin, insensitive changes, 116 hairy skin, deficiency, 113–114 limitations, 114 optimal configuration and location, 125 possibilities, 114 sensitive areas, 115–116 skin's capacity, 114–115 somatosensory system, 125 spatial information, 125 spectrum ranges, 114 tactons. See Tactons vibrotactile signals, 114–116 Weber's law, 115–116 Tactile-to-vision sensory substitution (TVSS), 22 Tactons creation and representation, 117 designing and language, 117 graphical depictions, 117, 118

308

Tactons (Continued) icons, 117 localization glued motor, 120–121 motor array, 119–120 moving contactor, 118, 120 site identifying, 119 skin and vibrotactile stimulation, 120–121 two-dimensional vibrotactile displays, 119–120 vibrotactile thresholds, 119 meaning and concepts, 117 pattern identification array, 121–122 ceiling effect, 122 complex cues, 121 display and schematic representation, 121–122 features, 123 Green's finding, 123–124 information transmission capabilities, 124 intermotor spacing range, 121–122 IT measures, 124 limitation and motor activation, 123 pancake motors, 116, 120, 121 responses percentage, 122, 123 static information transfer, 122 visual representation, 122 WTCU, 121 research, 117–119 signal, vibrotactile, 117 vibrotactile stimuli, 117–119 vocabulary, 117–119 Targeted muscle reinnervation (TMR), 85 tDCS. See Transcranial direct current stimulation TDU. See Tongue display unit Telerehabilitation 2D and 3D games, 154 data and audiovisual streams, 154 games advantages, 154–155 feedback, 154–155 group interactions, 154 Internet, webcams, speakers and microphones, 153 Nintendo Wii games, 154

phantom robot, 154 virtual network computing software, 153 whole-arm range motion and finger extension, 155 wireless router system, 153 Tibialis anterior (TA), 173–175, 184 TMR. See Targeted muscle reinnervation TMS. See Transcranial magnetic stimulation TMS and TES. See Transcranial magnetic and electrical stimulation Tongue display unit (TDU) fMRI data, 26 occipital cortex activation, 26–27 sensory substitution system, 18 stimulation, 22 Training, residual vision animal cats, 214 monkeys, 214–216 behavior, 214 human amblyopia, 223 behavioral stimulation, 216 blindsight, 222–223 border zone, 217–220 brain damaged patients, 216 compensatory, 224 improvements after intervention, 218 objective-subjective mismatch, 215 Transcranial direct current stimulation (tDCS), 191 Transcranial magnetic and electrical stimulation (TMS and TES) MEPs, 183–184 soleus (SOL) H-reflex, 183–184 subthreshold inhibition, TA muscle, 184, 185 Transcranial magnetic stimulation (TMS) contralesional suppression, 287–288 single pulse, 284–285 tDCS, 286 TVSS. See Tactile-to-vision sensory substitution UEA. See Utah electrode array Upper extremity rehabilitation, technology improvement

309

constraint-induced movement therapy, 148 cost and difficulties, 156–157 descriptions, 148 evidence-based treatment customization, unique injuries and obstacles, 149 designing and randomizing trials, 149 plausible, 149 spinal cord injury rehabilitation evidence, 149 standard care/placebo treatment, 149 stroke rehabilitation, 149 exercise equipment chronic stroke, 150, 151 conventional exercise therapy, 149–150 design and test instruments, 150 final implementation, ReJoyce system, 151, 152 IHT, 149–150 MIT-Manus robotic exercise therapy, 149–150 motor rehabilitation, home setting, 150 prototype ReJoyce workstations, 151–152 randomized control trail, 149–150 robotic rehabilitation devices, 149–150 sessions, boring, 149–150 subacute stroke, 150, 151 table top system, 150, 151 telescopic joystick, 150–151 exercise therapy, 148 fatigue effect and outcome evalution action research arm test, 155 IHT, 155 loss, 155 muscle fatigue, 155–156 performance, hand function test, 155, 156 rapid muscle, 156 ReJoyce automated hand function test, 155 FES, 152–153 hand function improvement, 156–157 in-home teletherapy (IHT), 148 issues absence daily supervision, 148–149 careful attention, 148–149 fatigue monitor, telerehabilitation, 148–149

FES involvement, 148–149 randomization, 148–149 require IHT equipment, 148–149 trail duration, 148–149 stroke and SCI, duration and intensity, 156–157 survivors and recovery, 147 telerehabilitation, 153–155 Utah electrode array (UEA), 77–78 VBM. See Voxel-based morphometry Ventroposterior lateral nucleus (VPL), 74–75 Virtual interactive sexual stimulus (VISS) ACC activation, 267–268 eye-tracking (ET) technique, 268 fMRI BCI, 268–269 penile tumescence and gaze behavior, 268 ROC analysis, 269 signal processing analysis and BCI–GUI, 268 Vision restoration adult brain, 231 brain, 200 dysfunctions, 200 eye movement artifact blind spot position, 232 border shifts, visual-field, 234 diagnostic testing and visual-field expansion, 232 eye-tracker adjusted visual chart, 234 fixation performance, 233 measurements, 232 problems, 232 retinal charts, 233 scotoma, 235 visual-field border nature, 232 functional improvements, 232 hot spot influencing factors, 237 issues, 200 lesion parameters age, 236 defect type, visual-field, 236 type, 236 visual-field topography, 236–238 neurobiological mechanisms

310

Vision restoration (Continued) and attention, 248–249 cellular, plasticity, 246–248 minimal residual structure hypothesis, 238–239 network plasticity, 240–246 neuronal synchronization, 248 within-systems plasticity, 239–240 neuroengineering, 201 neuroplasticity approach, 201 patient parameters, 236 perceptual learning, 231 plasticity after optic nerve lesions, 204–206 after post-chiasmatic lesions, 206–208 after retinal lesions, 202–204 congenital and early blindness, 208–209 perceptual learning, 201–202 post-lesion, 201 residual vision activation theory behavioral tasks, 250 blindsight, 212–213 description, 250–251

“hypothesis of minimal residual structures”, 249 partial blindness, 251 visual field border, 209–212 spontaneous recovery, 231–232 subjective vision and activities border training, 228–230 mismatch problem, 230–231 noninvasive current stimulation, 230 psychophysical parameters and RF plasticity, 228 visual-field loss, 228 training effects, 238 visual loss, 200 VISS. See Virtual interactive sexual stimulus Visual Basic.NET interface, 121 Visual hallucination, 244 Visual loss, 200 Voxel-based morphometry (VBM), 20–21 VPL. See Ventroposterior lateral nucleus Walking endurance, 162 Wireless tactile control unit (WTCU), 121

Other volumes in PROGRESS IN BRAIN RESEARCH Volume 149: Cortical Function: A View from the Thalamus, by V.A. Casagrande, R.W. Guillery and S.M. Sherman (Eds.) – 2005 ISBN 0-444-51679-4. Volume 150: The Boundaries of Consciousness: Neurobiology and Neuropathology, by Steven Laureys (Ed.) – 2005, ISBN 0-444-51851-7. Volume 151: Neuroanatomy of the Oculomotor System, by J.A. Büttner-Ennever (Ed.) – 2006, ISBN 0-444-51696-4. Volume 152: Autonomic Dysfunction after Spinal Cord Injury, by L.C. Weaver and C. Polosa (Eds.) – 2006, ISBN 0-444-51925-4. Volume 153: Hypothalamic Integration of Energy Metabolism, by A. Kalsbeek, E. Fliers, M.A. Hofman, D.F. Swaab, E.J.W. Van Someren and R.M. Buijs (Eds.) – 2006, ISBN 978-0-444-52261-0. Volume 154: Visual Perception, Part 1, Fundamentals of Vision: Low and Mid-Level Processes in Perception, by S. Martinez-Conde, S.L. Macknik, L.M. Martinez, J.M. Alonso and P.U. Tse (Eds.) – 2006, ISBN 978-0-444-52966-4. Volume 155: Visual Perception, Part 2, Fundamentals of Awareness, Multi-Sensory Integration and High-Order Perception, by S. Martinez-Conde, S.L. Macknik, L.M. Martinez, J.M. Alonso and P.U. Tse (Eds.) – 2006, ISBN 978-0-444-51927-6. Volume 156: Understanding Emotions, by S. Anders, G. Ende, M. Junghofer, J. Kissler and D. Wildgruber (Eds.) – 2006, ISBN 978-0-444-52182-8. Volume 157: Reprogramming of the Brain, by A.R. Mller (Ed.) – 2006, ISBN 978-0-444-51602-2. Volume 158: Functional Genomics and Proteomics in the Clinical Neurosciences, by S.E. Hemby and S. Bahn (Eds.) – 2006, ISBN 978-0-444-51853-8. Volume 159: Event-Related Dynamics of Brain Oscillations, by C. Neuper and W. Klimesch (Eds.) – 2006, ISBN 978-0-444-52183-5. Volume 160: GABA and the Basal Ganglia: From Molecules to Systems, by J.M. Tepper, E.D. Abercrombie and J.P. Bolam (Eds.) – 2007, ISBN 978-0-444-52184-2. Volume 161: Neurotrauma: New Insights into Pathology and Treatment, by J.T. Weber and A.I.R. Maas (Eds.) – 2007, ISBN 978-0-444-53017-2. Volume 162: Neurobiology of Hyperthermia, by H.S. Sharma (Ed.) – 2007, ISBN 978-0-444-51926-9. Volume 163: The Dentate Gyrus: A Comprehensive Guide to Structure, Function, and Clinical Implications, by H.E. Scharfman (Ed.) – 2007, ISBN 978-0-444-53015-8. Volume 164: From Action to Cognition, by C. von Hofsten and K. Rosander (Eds.) – 2007, ISBN 978-0-444-53016-5. Volume 165: Computational Neuroscience: Theoretical Insights into Brain Function, by P. Cisek, T. Drew and J.F. Kalaska (Eds.) – 2007, ISBN 978-0-444-52823-0. Volume 166: Tinnitus: Pathophysiology and Treatment, by B. Langguth, G. Hajak, T. Kleinjung, A. Cacace and A.R. Mller (Eds.) – 2007, ISBN 978-0-444-53167-4. Volume 167: Stress Hormones and Post Traumatic Stress Disorder: Basic Studies and Clinical Perspectives, by E.R. de Kloet, M.S. Oitzl and E. Vermetten (Eds.) – 2008, ISBN 978-0-444-53140-7. Volume 168: Models of Brain and Mind: Physical, Computational and Psychological Approaches, by R. Banerjee and B.K. Chakrabarti (Eds.) – 2008, ISBN 978-0-444-53050-9. Volume 169: Essence of Memory, by W.S. Sossin, J.-C. Lacaille, V.F. Castellucci and S. Belleville (Eds.) – 2008, ISBN 978-0-444-53164-3. Volume 170: Advances in Vasopressin and Oxytocin – From Genes to Behaviour to Disease, by I.D. Neumann and R. Landgraf (Eds.) – 2008, ISBN 978-0-444-53201-5. Volume 171: Using Eye Movements as an Experimental Probe of Brain Function—A Symposium in Honor of Jean BüttnerEnnever, by Christopher Kennard and R. John Leigh (Eds.) – 2008, ISBN 978-0-444-53163-6. Volume 172: Serotonin–Dopamine Interaction: Experimental Evidence and Therapeutic Relevance, by Giuseppe Di Giovanni, Vincenzo Di Matteo and Ennio Esposito (Eds.) – 2008, ISBN 978-0-444-53235-0. Volume 173: Glaucoma: An Open Window to Neurodegeneration and Neuroprotection, by Carlo Nucci, Neville N. Osborne, Giacinto Bagetta and Luciano Cerulli (Eds.) – 2008, ISBN 978-0-444-53256-5. Volume 174: Mind and Motion: The Bidirectional Link Between Thought and Action, by Markus Raab, Joseph G. Johnson and Hauke R. Heekeren (Eds.) – 2009, 978-0-444-53356-2. Volume 175: Neurotherapy: Progress in Restorative Neuroscience and Neurology — Proceedings of the 25th International Summer School of Brain Research, held at the Royal Netherlands Academy of Arts and Sciences, Amsterdam, The Netherlands, August 25–28, 2008, by J. Verhaagen, E.M. Hol, I. Huitinga, J. Wijnholds, A.A. Bergen, G.J. Boer and D.F. Swaab (Eds.) –2009, ISBN 978-0-12-374511-8. Volume 176: Attention, by Narayanan Srinivasan (Ed.) – 2009, ISBN 978-0-444-53426-2. Volume 177: Coma Science: Clinical and Ethical Implications, by Steven Laureys, Nicholas D. Schiff and Adrian M. Owen (Eds.) – 2009, 978-0-444-53432-3. Volume 178: Cultural Neuroscience: Cultural Influences On Brain Function, by Joan Y. Chiao (Ed.) – 2009, 978-0-444-53361-6. Volume 179: Genetic models of schizophrenia, by Akira Sawa (Ed.) – 2009, 978-0-444-53430-9. Volume 180: Nanoneuroscience and Nanoneuropharmacology, by Hari Shanker Sharma (Ed.) – 2009, 978-0-444-53431-6.

312

Other volumes in PROGRESS IN BRAIN RESEARCH

Volume 181: Neuroendocrinology: The Normal Neuroendocrine System, by Luciano Martini, George P. Chrousos, Fernand Labrie, Karel Pacak and Donald W. Pfaff (Eds.) – 2010, 978-0-444-53617-4. Volume 182: Neuroendocrinology: Pathological Situations and Diseases, by Luciano Martini, George P. Chrousos, Fernand Labrie, Karel Pacak and Donald W. Pfaff (Eds.) – 2010, 978-0-444-53616-7. Volume 183: Recent Advances in Parkinson's Disease: Basic Research, by Anders Björklund and M. Angela Cenci (Eds.) – 2010, 978-0-444-53614-3. Volume 184: Recent Advances in Parkinson's Disease: Translational and Clinical Research, by Anders Björklund and M. Angela Cenci (Eds.) – 2010, 978-0-444-53750-8. Volume 185: Human Sleep and Cognition, by Gerard A. Kerkhof and Hans P.A. Van Dongen (Eds.) – 2010, 978-0-444-53702-7. Volume 186: Sex Differences in the Human Brain, their Underpinnings and Implications, by Ivanka Savic (Ed.) – 2010, 978-0-44453630-3. Volume 187: Breathe, Walk and Chew: The Neural Challenge: Part I, by Jean-Pierre Gossard, Réjean Dubuc and Arlette Kolta (Eds.) – 2010, 978-0-444-53613-6. Volume 188: Breathe, Walk and Chew; The Neural Challenge: Part II, by Jean-Pierre Gossard, Réjean Dubuc and Arlette Kolta (Eds.) – 2011, 978-0-444-53825-3. Volume 189: Gene Expression to Neurobiology and Behaviour: Human Brain Development and Developmental Disorders, by Oliver Braddick, Janette Atkinson and Giorgio M. Innocenti (Eds.) – 2011, 978-0-444-53884-0. Volume 190: Human Sleep and Cognition Part II: Clinical and Applied Research, by Hans P.A. Van Dongen and Gerard A. Kerkhof (Eds.) – 2011, 978-0-444-53817-8. Volume 191: Enhancing Performance for Action and perception: Multisensory Integration, Neuroplasticity and Neuroprosthetics: Part I, by Andrea M. Green, C. Elaine Chapman, John F. Kalaska and Franco Lepore (Eds.) –2011, 978-0-44453752-2.