SERIES EDITORS
STEPHEN G. WAXMAN Bridget Marie Flaherty Professor of Neurology Neurobiology, and Pharmacology; Director, Center for Neuroscience & Regeneration/Neurorehabilitation Research Yale University School of Medicine New Haven, Connecticut USA
DONALD G. STEIN Asa G. Candler Professor Department of Emergency Medicine Emory University Atlanta, Georgia USA
DICK F. SWAAB Professor of Neurobiology Medical Faculty, University of Amsterdam; Leader Research team Neuropsychiatric Disorders Netherlands Institute for Neuroscience Amsterdam The Netherlands
HOWARD L. FIELDS Professor of Neurology Endowed Chair in Pharmacology of Addiction Director, Wheeler Center for the Neurobiology of Addiction University of California San Francisco, California USA
Elsevier Radarweg 29, PO Box 211, 1000 AE Amsterdam, The Netherlands Linacre House, Jordan Hill, Oxford OX2 8DP, UK 360 Park Avenue South, New York, NY 10010-1710 First edition 2011 Copyright Ó 2011 Elsevier B.V. All rights reserved No part of this publication may be reproduced, stored in a retrieval system or transmitted in any form or by any means electronic, mechanical, photocopying, recording or otherwise without the prior written permission of the publisher Permissions may be sought directly from Elsevier’s Science & Technology Rights Department in Oxford, UK: phone (þ44) (0) 1865 843830; fax (þ44) (0) 1865 853333; email:
[email protected]. Alternatively you can submit your request online by visiting the Elsevier web site at http://elsevier.com/locate/permissions, and selecting Obtaining permission to use Elsevier material Notice No responsibility is assumed by the publisher for any injury and/or damage to persons or property as a matter of products liability, negligence or otherwise, or from any use or operation of any methods, products, instructions or ideas contained in the material herein. Because of rapid advances in the medical sciences, in particular, independent verification of diagnoses and drug dosages should be made Library of Congress Cataloging-in-Publication Data A catalog record for this book is available from the Library of Congress British Library Cataloguing in Publication Data A catalogue record for this book is available from the British Library ISBN: 978-0-444-53752-2 ISSN: 0079-6123
For information on all Elsevier publications visit our website at elsevierdirect.com
Printed and bound in Great Britain 11 12 13 14
10 9 8
7
6 5
4
3
2 1
List of Contributors A.J. Bastian, Department of Motor Learning Lab, Kennedy Krieger Institute, and Neuroscience Department of Johns Hopkins School of Medicine, Baltimore, Maryland, USA M. Casadio, Department of Physiology, Northwestern University, and Sensory Motor Performance Program, Rehabilitation Institute of Chicago, Chicago, Illinois, USA F. Champoux, Centre de Recherche Interdisciplinaire en Réadaptation du Montréal Métropolitain, Institut Raymond-Dewar, Montréal, Québec, Canada O. Collignon, Département de Psychologie, Centre de Recherche en Neuropsychologie et Cognition (CERNEC), and Centre de Recherche CHU Sainte-Justine, Université de Montréal, Montréal, Québec, Canada E.K. Cressman, School of Human Kinetics, University of Ottawa, Ottawa, Ontario, Canada Z.C. Danziger, Department of Biomedical Engineering, Northwestern University, and Sensory Motor Performance Program, Rehabilitation Institute of Chicago, Chicago, Illinois, USA M. Darainy, Department of Psychology, McGill University, Montréal, Québec, Canada, and Shahed University, Tehran, Iran E. de Villers-Sidani, Department of Neurology and Neurosurgery, Montreal Neurological Institute, McGill University, Montréal, Québec, Canada R.T. Dydew, Centre for Vision Research, York University, Toronto, Ontario, Canada J. Frasnelli, Département de Psychologie, Centre de Recherche en Neuropsychologie et Cognition, Université de Montréal, Montréal, Québec, Canada L.R. Harris, Centre for Vision Research, York University, Toronto, Ontario, Canada D.Y.P. Henriques, Center for Vision Research, and School of Kinesiology and Health Science, York University, Toronto, Ontario, Canada S. Hutchins, BRAMS Laboratory and Department of Psychology, Université de Montréal, Montréal, Québec, Canada J.N. Ingram, Computational and Biological Learning Lab, Department of Engineering, University of Cambridge, Cambridge, United Kingdom H. Jenkin, Centre for Vision Research, York University, Toronto, Ontario, Canada M. Jenkin, Centre for Vision Research, York University, Toronto, Ontario, Canada A. Kral, Department of Experimental Otology, Institute of Audioneurotechnology, Medical University Hannover, Hannover, Germany S. Lacey, Department of Neurology, Emory University, Atlanta, Georgia, USA F. Lepore, Département de Psychologie, Centre de Recherche en Neuropsychologie et Cognition (CERNEC), Université de Montréal, Montréal, Québec, Canada S.G. Lomber, Department of Physiology and Pharmacology, and Department of Psychology, Centre for Brain and Mind, The University of Western Ontario, London, Ontario, Canada w
Deceased
v
vi
L. Malone, Department of Motor Learning Lab, Kennedy Krieger Institute, and Biomedical Engineering Department of Johns Hopkins School of Medicine, Baltimore, Maryland, USA A.A.G. Mattar, Department of Psychology, McGill University, Montréal, Québec, Canada M. Alex Meredith, Department of Anatomy and Neurobiology, School of Medicine, Virginia Commonwealth University, Richmond, Virginia, USA M.M. Merzenich, W.M. Keck Center for Integrative Neuroscience, Coleman Laboratory, Department of Otolaryngology, University of California, San Francisco, and Brain Plasticity Institute, San Francisco, California, USA K.M. Mosier, Department of Radiology, Section of Neuroradiology and Imaging Sciences, Indiana University, Indianapolis, Indiana, USA F.A. Mussa-Ivaldi, Department of Physiology, Department of Physical Medicine and Rehabilitation, Department of Biomedical Engineering, Northwestern University, and Sensory Motor Performance Program, Rehabilitation Institute of Chicago, Chicago, Illinois, USA S.M. Nasir, The Roxelyn and Richard Pepper Department of Communication Sciences and Disorders, Northwestern University, Evanston, Illinois, USA E. Nava, Department of Biological Psychology and Neuropsychology, University of Hamburg, Hamburg, Germany D.J. Ostry, Department of Psychology, McGill University, Montréal, Québec, Canada, and Haskins Laboratories, New Haven, Connecticut, USA E.F. Pace-Schott, Department of Psychology and Neuroscience, and Neuroscience and Behavior Program, University of Massachusetts, Amherst, USA I. Peretz, BRAMS Laboratory and Department of Psychology, Université de Montréal, Montréal, Québec, Canada B. Röder, Department of Biological Psychology and Neuropsychology, University of Hamburg, Hamburg, Germany B.A. Rowland, Department of Neurobiology and Anatomy, Wake Forest School of Medicine, WinstonSalem, North Carolina, USA P.N. Sabes, Department of Physiology, Keck Center for Integrative Neuroscience, University of California, San Francisco, California, USA K. Sathian, Department of Neurology, Department of Rehabilitation Medicine, Department of Psychology, Emory University, Atlanta, and Rehabilitation R&D Center of Excellence, Atlanta VAMC, Decatur, Georgia, USA R.A. Scheidt, Department of Biomedical Engineering, Marquette University, Milwaukee, Wisconsin, USA R.M.C. Spencer, Department of Psychology and Neuroscience, and Neuroscience and Behavior Program, University of Massachusetts, Amherst, USA B.E. Stein, Department of Neurobiology and Anatomy, Wake Forest School of Medicine, WinstonSalem, North Carolina, USA G. Torres-Oviedo, Department of Motor Learning Lab, Kennedy Krieger Institute, and Neuroscience Department of Johns Hopkins School of Medicine, Baltimore, Maryland, USA E. Vasudevan, Department of Motor Learning Lab, Kennedy Krieger Institute, and Neuroscience Department of Johns Hopkins School of Medicine, Baltimore, Maryland, USA P. Voss, Centre de Recherche en Neuropsychologie et Cognition (CERNEC), and International Laboratory for Brain, Music and Sound Research, Université de Montréal, Montréal, Québec, Canada D.M. Wolpert, Computational and Biological Learning Lab, Department of Engineering, University of Cambridge, Cambridge, United Kingdom
Preface Basic neuroscience research over the past several decades has defined many of the central neuronal mechanisms underlying the functioning of the various sensory and motor systems. These systems work together in a cooperative fashion: perception is required to plan action, while movements often serve to acquire desired sensory inputs (e.g., eye movements to capture a visual target of interest). Performance levels reach their apogee in examples ranging from high-performance athletes to concert-level pianists, with both requiring highly refined sensorimotor abilities. There is now considerable interest in how the brain adapts its functioning in health and disease. Extensive progress has been made in understanding how use and disuse influence motor and sensory performance, and the underlying mechanisms of neuronal plasticity responsible for adaptive changes. Increasingly, basic scientists working in these fields are being challenged to translate this scientific knowledge into applications that provide new and innovative methods to restore lost function in humans following injury or disease (e.g., amputation, myopathies, neuropathies, spinal cord lesions or stroke). In recent years, scientists have risen to this challenge, collaborating with engineers and clinicians to help develop novel technologies. These have progressed to the point where devices such as cochlear implants are now commonly used in clinical practice, while applications such as neuroprosthetic devices controlled by the brain are rapidly becoming a realistic possibility for restoring lost motor function. This two-volume set of books is the result of a symposium, inspired by these new initiatives, that was held at the Université de Montréal on May 10–11, 2010 (see http://www.grsnc.umontreal.ca/32s/). It was organized by the Groupe de Recherche sur le Système Nerveux Central (GRSNC) as one of a series of annual international symposia held on a different topic each year. The symposium included presentations by world-renowned experts working on the neuronal mechanisms that play critical roles in learning new motor and sensory skills in both health and disease. The objective was to provide an overview of the various neural mechanisms that contribute to learning new motor and sensory skills as well as adapting to changed circumstances including the use of devices and implants to substitute for lost sensory or motor abilities (neural prosthetics). The symposium emphasized the importance of basic science research as the foundation for innovative technological developments that can help restore function and improve the quality of life for disabled individuals. It equally emphasized how such new technologies can contribute to our basic scientific understanding of the neural mechanisms of sensorimotor control and adaptation. Many of the participants of that meeting have contributed chapters to this book, including symposium speakers and poster presenters. In addition, we invited a number of other well-known experts who could not participate in the conference itself to submit chapters. This two-volume collection of over 30 chapters can only cover a fraction of the topics and extensive range of work that pertains to adapting our motor and sensory systems to changed conditions and to the development of technologies that substitute for lost abilities. However, it does address a range of motor functions and sensory modalities; considers adaptive changes at both behavioral and neurophysiological levels; and presents perspectives on basic research, clinical approaches, and technological innovations. The result is a collection that includes chapters broadly separated into five key themes: (1) mechanisms to enhance vii
viii
motor performance, (2) mechanisms to enhance sensory perception, (3) multisensory interactions to enhance action and perception, (4) assistive technologies to enhance sensorimotor performance, and (5) neurorehabilitation. The current volume (Volume I) focuses on the basic mechanisms underlying performance changes (themes 1–3). Volume II complements this first volume by focussing on the translation of scientific knowledge into technological applications and clinical strategies that can help restore lost function and improve quality of life following injury or disease (themes 4 and 5). The conference and this book would not have been possible without the generous support of the GRSNC, the Fonds de la Recherche en Santé de Québec (FRSQ), the Canadian Institutes of Health Research (CIHR), the Institute of Neuroscience, Mental Health and Addiction (CIHR), the Institute of Musculoskeletal Health and Arthritis (CIHR), and the Faculty of Arts and Sciences and Faculty of Medicine of the Université de Montréal. We gratefully acknowledge these sponsors as well as our contributing authors who dedicated their time and effort to present their perspectives on the neural mechanisms and technological advances that enhance our performance for action and perception. Andrea M. Green C. Elaine Chapman John F. Kalaska Franco Lepore
A. M. Green, C. E. Chapman, J. F. Kalaska and F. Lepore (Eds.) Progress in Brain Research, Vol. 191 ISSN: 0079-6123 Copyright Ó 2011 Elsevier B.V. All rights reserved.
CHAPTER 1
Naturalistic approaches to sensorimotor control James N. Ingram* and Daniel M. Wolpert Computational and Biological Learning Lab, Department of Engineering, University of Cambridge, Cambridge, United Kingdom
Abstract: Human sensorimotor control has been predominantly studied using fixed tasks performed under laboratory conditions. This approach has greatly advanced our understanding of the mechanisms that integrate sensory information and generate motor commands during voluntary movement. However, experimental tasks necessarily restrict the range of behaviors that are studied. Moreover, the processes studied in the laboratory may not be the same processes that subjects call upon during their everyday lives. Naturalistic approaches thus provide an important adjunct to traditional laboratory-based studies. For example, wearable self-contained tracking systems can allow subjects to be monitored outside the laboratory, where they engage spontaneously in natural everyday behavior. Similarly, advances in virtual reality technology allow laboratory-based tasks to be made more naturalistic. Here, we review naturalistic approaches, including perspectives from psychology and visual neuroscience, as well as studies and technological advances in the field of sensorimotor control. Keywords: human sensorimotor control; natural tasks; natural behavior; movement statistics; movement kinematics; object manipulation; tool use.
consider grasping an object such as a coffee cup. In order to reach for the cup, sensory information regarding its three-dimensional location, represented initially by its two-dimensional position on the retina, must be transformed into a motor command that moves the hand from its current location to the location of the cup (Shadmehr and Wise, 2005; Snyder, 2000; Soechting and Flanders, 1992). Similarly, in order to grasp the cup, sensory information regarding its three-dimensional shape must be transformed into a motor command that configures the digits to accommodate the cup
Introduction Sensorimotor control can be regarded as a series of transformations between sensory inputs and motor commands (Craig, 1989; Fogassi and Luppino, 2005; Pouget and Snyder, 2000; Rizzolatti et al., 1998; Shadmehr and Wise, 2005; Snyder, 2000; Soechting and Flanders, 1992). For example, *Corresponding author. Tel.: þ44-1223-748-514; Fax: þ44-1223-332-662 E-mail:
[email protected] DOI: 10.1016/B978-0-444-53752-2.00016-3
3
4
(Castiello, 2005; Castiello and Begliomini, 2008; Santello and Soechting, 1998). Finally, once the cup is grasped, sensory information regarding the dynamics of the cup (such as its mass) must be used to rapidly engage the transformations that will mediate control of the arm–cup combination (Atkeson and Hollerbach, 1985; Bock, 1990, 1993; Lacquaniti et al., 1982). In many cases, the study of sensorimotor control endeavors to understand these transformations, how they are acquired and represented in the brain, how they adapt to new tasks, and how they generalize to novel task variations. When learning a new motor skill, for example, existing sensorimotor transformations may be adapted and new transformations may be learned (Haruno et al., 2001; Miall, 2002; Wolpert et al., 2001; Wolpert and Kawato, 1998). The study of motor learning can thus reveal important details about the underlying transformations (Ghahramani and Wolpert, 1997; Shadmehr, 2004). As such, many laboratory-based studies which examine sensorimotor control use adaptation paradigms in which subjects reach toward visual targets in the presence of perturbations which induce movement errors. In the case of dynamic (force) perturbations, the subject grasps the handle of a robotic manipulandum which can apply forces to the arm (see, e.g., Caithness et al., 2004; Gandolfo et al., 1996; Howard et al., 2008, 2010; Malfait et al., 2002; Shadmehr and Brashers-Krug, 1997; Shadmehr and Mussa-Ivaldi, 1994; Tcheang et al., 2007; Tong et al., 2002). Typically, the forces depend on the kinematics of the movement, such as its velocity, and cause the arm to deviate from the target. Over the course of many trials, the subject adapts to the perturbation and the deviation of the hand reduces. In the case of kinematic perturbations, the position of the subject's hand is measured and, typically, displayed as a cursor on a screen. Subjects reach toward visual targets with the cursor. A transformation (such as a rotation) can be applied to the cursor which perturbs it relative to the veridical position of
the hand (see, e.g., Ghahramani and Wolpert, 1997; Ghahramani et al., 1996; Howard et al., 2010; Kagerer et al., 1997; Krakauer et al., 1999, 2000, 2005). Once again, over the course of many trials, the subject adapts to the perturbation and the deviation of the cursor reduces. These laboratory-based perturbation studies have greatly advanced our understanding of sensorimotor control. However, because they predominantly focus on reaching movements during a limited number of perturbations, they do not capture the full range of everyday human behavior. Here, we present more naturalistic approaches. We begin by reviewing perspectives from psychology and go on to describe a naturalistic approach which has been successful in the study of the visual system. We then review studies which examine human behavior in naturalistic settings, focusing on relevant advances in technology and studies which record movement kinematics during natural everyday tasks. Because object manipulation emerges as an important component of naturalistic behavior in these studies, we finish with a review of object manipulation and tool use. Specifically, we present results from various experimental paradigms including a recent naturalistic approach in which a novel robotic manipulandum (the WristBOT) is used to simulate objects with familiar dynamics.
Naturalistic perspectives from animal psychology The animal psychologist Nicholas Humphrey published a seminal paper in 1976 in which he speculated about the function of intelligence in primates (Humphrey, 1976). The paper begins with a conundrum: how to reconcile the remarkable cognitive abilities that many primates demonstrate in laboratory-based experiments with the apparent simplicity of their natural lives, where food is abundant (literally growing on trees), predators are few, and the only demands are to “eat, sleep, and play.” He asked “What—if it exists—is the natural equivalent of the laboratory
5
test of intelligence?” He reasoned that if an animal could be shown to have a particular cognitive skill in the laboratory, that skill should have some natural application in the wild. He argued that the process of natural selection would not tolerate “needless extravagance” and that “We do not expect to find that animals possess abilities which far exceed the calls that natural living makes upon them.” The same argument could be applied to laboratory-based studies of human sensorimotor control. For example, if we observe that subjects can adapt to a particular perturbation during a controlled laboratory-based task, what does that tell us about the sensorimotor processes that humans regularly call upon during their everyday lives? In Humphrey's case, he answered the question by carefully observing the natural behavior of primates. In the endeavor to understand the human sensorimotor system, the natural behavior of our subjects may also be an important source of information. Whereas Humphrey encourages us to explore the natural everyday expression of the skills and processes we observe during laboratory-based tasks, other animal psychologists would argue that we should question the ecological relevance of the tasks themselves. For example, a particular primate species may fail on a laboratory-based task that is designed to characterize a specific cognitive ability (Povinelli, 2000; Povinelli and Bering, 2002; Tomasello and Call, 1997). In this case, the conclusion would be that the cognitive repertoire of the species does not include the ability in question. However, if the task is reformulated in terms of the natural everyday situations in which the animal finds itself (foraging for food, competing with conspecifics, etc.), successful performance can be unmasked (Flombaum and Santos, 2005; Hare et al., 2000, 2001). This issue of ecological relevance may also apply to the performance of human subjects during the laboratory-based tasks that are designed to study sensorimotor control (Bock and Hagemann, 2010). For example, despite our intuition that humans can successfully learn and
recall a variety of different motor skills and interact with a variety of different objects, experiments have shown that concurrent adaptation to distinct sensorimotor tasks can be difficult to achieve in the laboratory (Bock et al., 2001; Brashers-Krug et al., 1996; Goedert and Willingham, 2002; Karniel and Mussa-Ivaldi, 2002; Krakauer et al., 1999, 2005; Miall et al., 2004; Shadmehr and Brashers-Krug, 1997; Wigmore et al., 2002). However, in natural everyday life, different motor skills are often associated with distinct behavioral contexts. It is thus not surprising that when experiments are made more naturalistic by including distinct contextual cues, subjects can learn and appropriately recall laboratory-based tasks that would otherwise interfere (Howard et al., 2008, 2010; Lee and Schweighofer, 2009; Nozaki and Scott, 2009; Nozaki et al., 2006).
Naturalistic perspectives from human cognitive ethology The importance of a naturalistic approach is also advocated by proponents of human cognitive ethology (Kingstone et al., 2008). Ethology is the study of animal (and human) behavior in natural settings (Eibl-Eibesfeldt, 1989; McFarland, 1999). The emphasis is on the adaptive and ecological significance of behavior, how it develops during the lifetime of the individual, and how it has evolved during the history of the species. It can be contrasted with the approaches of experimental psychology, which focus on laboratorybased tasks rather than natural behavior and largely ignore questions of ecological relevance and evolution (Kingstone et al., 2008). In human cognitive ethology, studies of natural real-world behavior are regarded as an important adjunct to experimental laboratory-based approaches, with some going so far as to argue that they are a necessary prerequisite (Kingstone et al., 2008). An example of this approach is given by Kingstone and colleagues and consists of a pair of studies that examine vehicle steering behavior.
6
In the first study, the natural steering behavior of subjects was measured outside the laboratory in a real-world driving task (Land and Lee, 1994). In the second study, a laboratory-based driving simulator was then used to test a specific hypothesis regarding the sources of information that drivers use for steering (Land and Horwood, 1995). The experimental hypothesis was constrained by the real-world behavior of subjects, as measured in the first study, and the simulated roads were modeled on the real-world roads from which the natural dataset was collected. Kingstone and colleagues argue that all experiments examining human cognition should begin with a characterization of the natural manifestations of the processes involved. They warn against the implicit assumption that a process identified during a controlled laboratory-based task is the same process that is naturally engaged by subjects in the real world (see also Bock and Hagemann, 2010). Learning a novel dynamic perturbation in the laboratory, for example, may be nothing like learning to use a new tool in our home workshop. If we are interesting in the sensorimotor control of object manipulation, asking subjects to grasp the handles of robots that generate novel force fields may provide only partial answers. Ideally, we should also study the natural tool-using behavior of our subjects outside the laboratory, and inside the laboratory, we should ask them to grasp a robotic manipulandum that looks and behaves like a real tool.
A naturalistic approach to the visual system The receptive fields (RFs) of neurons in the visual system have been traditionally defined using simple artificial stimuli (for recent reviews, see Fitzpatrick, 2000; Reinagel, 2001; Ringach, 2004). For example, the circular center-surround RFs of retinal ganglion cells were originally defined using small spots of light (Hartline, 1938; Kuffler, 1953). The same method later revealed similar RFs in the lateral geniculate nucleus
(Hubel, 1960; Hubel and Wiesel, 1961). In contrast, bars of light were found to elicit the largest response from neurons in primary visual cortex (V1) (Hubel and Wiesel, 1959). This finding was pivotal because it provided the first evidence for a transformation of RFs from one visual processing area to the next (Tompa and Sáry, 2010; Wurtz, 2009). A hierarchical view of visual processing emerged, in which the RFs at each level were constructed from simpler units in the preceding level (Carpenter, 2000; Gross, 2002; Gross et al., 1972; Hubel and Wiesel, 1965; Konorski, 1967; Perrett et al., 1987; Tompa and Sáry, 2010). Within this framework, using artificial stimuli to map the RFs at all stages of the visual hierarchy was regarded as essential in the effort to understand vision (Hubel and Wiesel, 1965; Tanaka, 1996; Tompa and Sáry, 2010). However, beyond their role as abstract feature detectors contributing progressively to visual perception, there was little discussion as to why RFs had particular properties (Balasubramanian and Sterling, 2009). In contrast to traditional approaches based on artificial stimuli, the concept of efficient coding from information theory allows the properties of visual RFs to be explained in terms of natural visual stimuli (Barlow, 1961; Simoncelli, 2003; Simoncelli and Olshausen, 2001). Specifically, natural images are redundant due to correlations across both space and time (Simoncelli and Olshausen, 2001; van Hateren, 1992), and efficient coding assumes that the early stages of visual processing aim to reduce this redundancy (Barlow, 1961; van Hateren, 1992). Within such a naturalistic framework, the statistical structure of natural visual images becomes central to understanding RF properties. For example, retinal processing can be regarded as an attempt to maximize the information about the visual image that is transmitted to the brain by the optic nerve (Geisler, 2008; Laughlin, 1987). Consistent with this, center-surround RFs in the retina appear to exploit spatial correlations that exist in natural images (Balasubramanian and Sterling, 2009;
7
Srinivasan et al., 1982). Moreover, the RFs of both simple (Olshausen and Field, 1996) and complex (Földiak, 1991; Kording et al., 2004) cells in V1 appear to be based on an efficient neural representation of natural images. For example, simple cell RFs self-organize spontaneously under a learning algorithm that is optimized to find a sparse code for natural scenes (Olshausen and Field, 1996). Similarly, many features of complex cell RFs self-organize under a learning algorithm that is optimized to find stable responses to natural scenes (Kording et al., 2004). Thus, whereas traditional approaches to the visual system have used artificial stimuli to simply map the structure of visual RFs, a naturalist approach based on natural visual stimuli allows the RFs to be predicted from first principles (Simoncelli and Olshausen, 2001).
Naturalistic approaches to human behavior As reviewed in the previous section, an analysis of the natural inputs to the visual system (natural images) has been highly productive in the study of visual processing. Approaches that record the natural outputs of the sensorimotor system (everyday human behavior) may be similarly informative. Depending on the study, the data collected may include the occurrence of particular behaviors, the kinematics of movements, physical interactions with objects, social interactions with people, or the location of the subject. We briefly review studies and technologies associated with collecting behavioral data from subjects in their natural environment and then review in more detail the studies that specifically record movement kinematics during natural everyday tasks. Studies of human behavior in naturalistic settings have traditionally relied on observation or indirect measures. Examples of the use of observation include a study of human travel behavior which required subjects to keep a 6week travel diary (Schlich and Axhausen, 2003) and a study of the everyday use of the hand which
required an observer to keep a diary of the actions performed by subjects during the observation period (Kilbreath and Heard, 2005). In the case of indirect measures, examples include the use of e-mail logs to examine the statistics of discrete human behaviors (Barabasi, 2005), the use of dollar bill dispersal patterns to examine the statistics of human travel (Brockmann et al., 2006), and monitoring the usage of the computer mouse to examine the statistics of human movement (Slijper et al., 2009). Recently, mobile phones have become an important tool for collecting data relevant to everyday human behavior (Eagle and Pentland, 2006, 2009). For example, large datasets of human travel patterns can be obtained from mobile phones (Anderson and Muller, 2006; González et al., 2008). In addition, mobile phones include an increasing variety of sensors, such as accelerometers, which can be used to collect data unobtrusively from naturally behaving subjects (Ganti et al., 2010; Hynes et al., 2009). This information can be used, for example, to distinguish between different everyday activities (Ganti et al., 2010). Mobile phones can also interface with small wireless sensors worn elsewhere on the body. For example, Nokia has developed a combined three-axis accelerometer and gyroscope motion sensor the size of a wristwatch which can be worn on segments of the body (Györbíró et al., 2009). This combination of accelerometers and gyroscopes has been shown to overcome the problems associated with using accelerometers alone (Luinge and Veltink, 2005; Takeda et al., 2010). The Nokia motion sensors can stream data to the subject's mobile phone via bluetooth, providing kinematic data simultaneously from multiple body segments. In a recent study, this data was used to distinguish between different everyday activities (Györbíró et al., 2009). In general, mobile phone companies are interested in determining the user's behavioral state so that the phone can respond appropriately in different contexts (Anderson and Muller, 2006; Bokharouss et al., 2007; Devlic et al., 2009;
8
Ganti et al., 2010; Györbíró et al., 2009). However, the application of these technologies to naturalistic studies of human behavior is clear (Eagle and Pentland, 2006, 2009). The interaction of subjects with objects in the environment is also an important source of information regarding naturalistic behavior (Beetz et al., 2008; Philipose et al., 2004; Tenorth et al., 2009). For example, by attaching small inexpensive radio-frequency-identification (RFID) tags to the objects in the subject's environment, a instrumented glove can be used to record the different objects used by the subject as they go about their daily routine (Beetz et al., 2008; Philipose et al., 2004). This information can be used to distinguish between different everyday tasks, and can also distinguish different stages within each task (Philipose et al., 2004). A disadvantage of the use of RFID technology is that every object must be physically tagged. An alternative method to track a subject's interactions with objects uses a head-mounted camera and image processing software to extract hand posture and the shape of the grasped object (Beetz et al., 2008).
Naturalistic studies of movement kinematics The kinematics of a subject's movements provide an important source of information for studying sensorimotor control. However, most commercially available motion tracking systems are designed for use inside the laboratory (Kitagawa and Windor, 2008; Mündermann et al., 2006). The majority of studies which examine human movement kinematics are thus performed under laboratory conditions (for recent reviews, see Schmidt and Lee, 2005; Shadmehr and Wise, 2005). In contrast, naturalistic studies of spontaneously behaving humans require mobile, wearable systems which minimally restrict the movements of the subject. As discussed in the previous section, small wireless sensors which can stream kinematic data from multiple
segments of the body to a data logger (such as a mobile phone) may provide one solution (Györbíró et al., 2009; Lee et al., 2010). However, these technologies are not yet widely available. To date, naturalistic studies of movement kinematics have thus used commercial motion tracking systems which have been modified to make them wearable by subjects. These studies, which have examined movements of the eyes, hands, and arms, are reviewed in the following sections.
Eye movements during natural tasks Eye movements are the most frequent kind of movement that humans make, more frequent even than heartbeats (Carpenter, 2000). The oculomotor system has many features which make it an ideal model system for the study of sensorimotor control (Carpenter, 2000; Munoz, 2002; Sparks, 2002). Eye movements are relatively easy to measure (Wade and Tatler, 2005) and the neural circuitry which underlies them is well understood (Munoz, 2002; Sparks, 2002). Moreover, eye movements are intimately associated with the performance of many motor tasks (Ballard et al., 1992; Johansson et al., 2001; Land, 2009; Land and Hayhoe, 2001; Land and Tatler, 2009). They also provide a convenient behavioral marker for cognitive processes including attention (e.g., Corbetta et al., 1998) and decision making (e.g., Gold and Shadlen, 2000; Schall, 2000). It is not surprising, therefore, that a number of studies have examined eye movements during natural everyday tasks (for recent reviews, see Hayhoe and Ballard, 2005; Land, 2006, 2009; Land and Tatler, 2009). The purpose of eye movements (saccades and smooth pursuit, for a recent review, see Krauzlis, 2005) is to move the small high-acuity spotlight of foveal vision to fixate a particular object or location in the visual scene (Land, 1999; Land and Tatler, 2009; Munoz, 2002). As such, tracking the position of the eyes during a task provides a record of what
9
visual information subjects are using and when they obtain it (Ballard et al., 1992; Johansson et al., 2001; Land and Hayhoe, 2001; Land and Tatler, 2009). The importance of this information during the execution of natural everyday tasks has long been recognized (reviewed in Land and Tatler, 2009; Wade and Tatler, 2005). However, early tracking systems required that the head be fixed which limited recordings to sedentary tasks performed within the laboratory (reviewed in Land and Tatler, 2009). These tasks included reading (Buswell, 1920), typing (Butsch, 1932), viewing pictures (Buswell, 1935; Yarbus, 1967), and playing the piano (Weaver, 1943).
(a) Scene camera
Eye camera
More recently, light-weight head-free eye trackers have become available (Wade and Tatler, 2005) allowing the development of wearable, selfcontained systems (Fig. 1a; Hayhoe and Ballard, 2005; Land and Tatler, 2009; Pelz and Canosa, 2001). Typically, these systems include a camera which records the visual scene as viewed by the subject along with a cursor or cross-hair which indicates the point of fixation within the scene. Studies of eye movements during natural tasks have thus moved outside the laboratory where mobile, unrestricted subjects can engage in a wider range of behaviors (Hayhoe and Ballard, 2005; Land, 2006, 2009; Land and Tatler, 2009).
(b)
Scene monitor Recording backpack (c)
Fig. 1. Eye movements during natural tasks. Panel (a) is modified from Hayhoe and Ballard (2005). Copyright (2005), with permission from Elsevier. Panels (b) and (c) are modified from Land and Hayhoe (2001). Copyright (2001), with permission from Elsevier. (a) An example of a wearable eye-tracking system which consists of an eye camera and scene camera which are mounted on light-weight eyewear. A backpack contains the recording hardware. (b) Fixations of a typical subject while making a cup of tea. Notice the large number of fixations on objects relevant to the task (such as the electric kettle) whereas taskirrelevant objects (such as the stove) are ignored. (c) Fixations of a typical subject while making a sandwich.
10
Such behaviors include everyday activities such as driving a car (Land and Lee, 1994; Land and Tatler, 2001), tea making (Fig. 1b; Land and Hayhoe, 2001; Land et al., 1999), sandwich making (Fig. 1c; Hayhoe et al., 2003; Land and Hayhoe, 2001), and hand washing (Pelz and Canosa, 2001). Ball games such as table tennis (Land and Furneaux, 1997), cricket (Land and McLeod, 2000), catch (Hayhoe et al., 2005), and squash (Land, 2009) have also been studied. Two important findings arose from the early studies of natural eye movements during sedentary tasks. First, the pattern of eye movements is dramatically influenced by the specific requirements of the task (Yarbus, 1967), and second, eye movements usually lead movements of the arm and hand by about one second (reviewed in Land and Tatler, 2009). Contemporary laboratory-based studies have confirmed these findings by using tasks specifically designed to capture the essential features of naturalistic behavior (Ballard et al., 1992; Johansson et al., 2001; Pelz et al., 2001). One of the first studies to use this method found that, rather than relying on detailed visual memory, subjects make eye movements to gather information immediately before it is required in the task (Ballard et al., 1992). Subsequently, using the same task, it was found that subjects will even delay movements of the hand until the eye is available (Pelz et al., 2001). Contemporary studies of eye movements during natural everyday tasks have reported similar findings. For example, when subjects make a pot of tea (Land and Hayhoe, 2001; Land et al., 1999), objects are usually fixated immediately before being used in the task, with irrelevant objects being largely ignored (Fig. 1b). A similar pattern is seen during sandwich making (Hayhoe et al., 2003; Land and Hayhoe, 2001) and hand washing (Pelz and Canosa, 2001). The influence of task requirements on eye movements is particularly striking. When subjects passively view natural scenes, they selectively fixate some areas over others based on the “bottom-up” salience
of features in the scene. For example, visual attention is attracted by regions with high spatial frequencies, high edge densities or high contrast (for reviews see Henderson, 2003; Henderson and Hollingworth, 1999). In contrast, when specific tasks are imposed, the pattern of eye movements is driven by the “top-town” requirements of the task (see reviews in Ballard et al., 1992; Land and Tatler, 2009; Land, 2006). For example, while subjects are waiting for the go-signal to begin a particular task, they fixate irrelevant objects with the same frequency as the objects that are relevant to the task (Hayhoe et al., 2003). The number of irrelevant object fixations falls dramatically once the task begins. Before studies of naturalistic eye movements, it had been assumed that subjects used visual information obtained by the eyes to construct a detailed model of the visual world which could be consulted as required during task execution (Ballard et al., 1992). The study of eye movements during natural everyday tasks outside the laboratory and during laboratory-based tasks designed to be naturalistic has shown that rather than rely on memory, subjects use their eyes to obtain information immediately before it is required in the task.
Hand and arm movements during natural tasks The naturalistic studies of eye movements reviewed in the previous section have been made possible by the development of wearable, selfcontained eye-tracking systems (Hayhoe and Ballard, 2005; Land and Tatler, 2009; Pelz and Canosa, 2001). Two recent studies from our group have used wearable, self-contained systems to record hand (Ingram et al., 2008) and arm (Howard et al., 2009a) movements during natural everyday behavior. However, in contrast to studies of eye movements, which have invariably imposed specific tasks on the subject, we allowed our subjects to engage spontaneously in natural everyday behavior.
11
The statistics of natural hand movements Although the 15 joints of the hand can potentially implement 20 degrees of freedom (Jones, 1997; Stockwell, 1981), laboratory-based studies suggest that the effective dimensionality of hand movements is much less (reviewed in Jones and Lederman, 2006). For example, the ability of subjects to move the digits independently is limited (Hager-Ross and Schieber, 2000; Reilly and Hammond, 2000) due to both mechanical (Lang and Schieber, 2004; von Schroeder and Botte, 1993) and neuromuscular (Kilbreath and Gandevia, 1994; Lemon, 1997; Reilly and Schieber, 2003) factors. Moreover, the sensorimotor system is thought to employ synergies which reduce the dimensionality and thereby simplify the control problem (Mason et al., 2001; Santello et al., 1998, 2002; Schieber and Santello, 2004; Tresch et al., 2006). However, these conclusions are based on results from laboratory-based tasks which potentially constrain the variety of hand movements observed. To address this issue using a naturalistic approach, we obtained datasets of spontaneous everyday movements from the right hand of subjects who wore a self-contained motion tracking system (Ingram et al., 2008). The system consisted of an instrumented cloth glove (the commercially available CyberGlove from CyberGlove Systems) and a backpack which contained the data acquisition hardware (Fig. 2a). Subjects were fitted with the system and instructed to go about their normal daily routine. A total of 17 h of data was collected, which consisted of 19 joint angles of the digits sampled at 84 Hz. To estimate the dimensionality of natural hand movements in the dataset, we performed a principal component analysis (PCA) on joint angular velocity (Fig. 2b and c). Consistent with the reduced dimensionality discussed above, the first 10 PCs collectively explained almost all (94%) of the variance (Fig. 2c). Moreover, the first two PCs accounted for more than half of the variance (60%) and were well conserved across subjects.
The first PC explained 40% of the variance and reflected a coordinated flexion (closing) and extension (opening) of the four fingers. The second PC explained an additional 20% of the variance and also involved flexion and extension of the four fingers. Figure 2b shows how these first two PCs combine to produce a large range of hand postures. An important question arising from the current study is whether there are differences between the statistics of hand movements made during laboratory-based tasks and those made during everyday life. Previous studies have performed PCA on angular position data collected during a reach-tograsp task (Santello et al., 1998, 2002). In these previous studies, the first two PCs involved flexion and extension of the fingers and accounted for 74% of the variance. When the same analysis was repeated on our dataset, the first two PCs also involved finger flexion/extension and accounted for 70% of the variance. This similarity with previous laboratory-based studies suggests that reach-to-grasp movements and object manipulation form an important component of the natural everyday tasks performed by the hand. Consistent with this, 60% of the natural use of the hands involves grasping and manipulating objects (Kilbreath and Heard, 2005). Many previous laboratory-based studies have examined the independence of digit movements, showing that the thumb and index finger are moved relatively independently, whereas the middle and ring fingers tend to move together with the other digits (Hager-Ross and Schieber, 2000; Kilbreath and Gandevia, 1994). We quantified digit independence in our natural dataset by determining the degree to which the movements of each digit (the angular velocities of the associated joints) could be linearly predicted from the movements (angular velocities) of the remaining four digits. This measure was expressed as the percentage of unexplained variance (Fig. 2d) and was largest for the thumb, followed by the index finger, then the little and middle fingers, and was smallest for the ring
12 (b)
(a) Recording backpack
PC 2
Instrumented LED glove indicator
PC 1 (d)
(e)
90 80 70 60 50 40
80 60 40 20 0
30 2
4
6
8 10 12 14 16 18
(f) 100
% Variance explained
100 % Variance unexplained
% Variance explained
100
100 % Variance explained
(c)
80 60 40 20 0
T
I
M
R
L
80 60 40 20 0
T
I
M
R
L
T
I
M
R
L
PC number
Fig. 2. The statistics of natural hand movements. Panels (b) through (f) are reprinted from Ingram et al. (2008). Used with permission. (a) The wearable motion tracking system consisted of an instrument cloth glove (the CyberGlove from CyberGlove Systems) which measured 19 joint angles of the digits. A backpack contained the recording hardware. Subjects were told to go about their normal daily routine and return when the LED indicator stopped flashing. (b) The first two principal components (PC) explained 60% of the variance in joint angular velocity and combine to produce a range of hand postures. (c) The percent variance explained by increasing numbers of principal components. The first 10 PCs accounted for 94% of the variance in joint angular velocity. (d) The percent variance in angular velocity which remained unexplained for each digit after a linear reconstruction which was based on data from the other four digits (T ¼ Thumb, I ¼ Index, M ¼ Middle, R ¼ Ring, L ¼ Little). (e) The percent variance in angular velocity which was explained by a linear reconstruction which paired the thumb individually with the other digits. The gray 100% bar indicated self-pairing. (f) The percent variance explained for digit pairs involving the little finger, plotted as in (e).
finger. Interestingly, this pattern of digit independence was correlated with results from several previous studies, including the number of cortical sites encoding movement of each digit (Penfield and Broldrey, 1937) and a laboratory-based measure of the ability of subjects to move each digit individually (Hager-Ross and Schieber, 2000).
We also quantified coupling between pairs of digits, applying the linear reconstruction method separately to each digit paired separately with the other four digits. This measure was expressed as the percent variance that was explained. Results for the thumb (Fig. 2e) show that its movements are very difficult to predict based on
13
movements of the fingers. Results for the fingers show that the best linear reconstructions (highest coupling) are based on the movements of the immediately neighboring fingers, decreasing progressively with increasing distance (Fig. 2f shows this pattern for the little finger). Thus, whereas the thumb moves independently of the fingers, the movements of a given finger are more or less related to neighboring fingers based on the topological distance between them. These results from a naturalistic study of hand movements have generally supported those obtained from previous laboratory-based studies. However, because previous studies have employed a limited number of experimental tasks, it is important to verify their conclusions in the natural everyday behavior of subjects. Specifically, we have verified the pattern of digit independence in the everyday use of the hand and shown that many aspects of natural hand movements have been well characterized by laboratory-based studies in which subjects reach to grasp objects.
The statistics of natural arm movements Many laboratory-based studies have examined the ability of subjects to make bimanual movements with particular phase relations (Kelso, 1984, 1995; Li et al., 2005; Mechsner et al., 2001; Schmidt et al., 1993; Swinnen et al., 1998, 2002). Results indicate that not all phase relations are equally easy to perform. At a low frequency of movement, both symmetric movements (phase difference between the two arms of 0 ) and antisymmetric movements (phase difference of 180 ) are easy to perform, whereas movements with intermediate phase relations are more difficult. At higher frequencies, only symmetric movements can be performed easily and all other phase relations tend to transition to the symmetric mode (Tuller and Kelso, 1989; Wimmers et al., 1992). This “symmetry bias” has been extensively studied in laboratory-based
experiments and there has been much debate regarding its significance and underlying substrate (e.g., Mechsner et al., 2001; Treffner and Turvey, 1996). Its relevance to the everyday behavior of subjects, however, is not clear. To address this issue using a naturalistic approach, we obtained datasets of spontaneous everyday arm movements of subjects who wore a self-contained motion tracking system (Howard et al., 2009a). We hypothesized that the symmetry bias would be reflected in the natural statistics of everyday tasks. Electromagnetic sensors (the commercially available Liberty system from Polhemus) were attached to the left and right arms and the data acquisition hardware was contained in a backpack (Fig. 3a). Subjects were fitted with the system and instructed to go about their normal routine. A total of 31 h of data was collected, which consisted of the position and orientation of the sensors on the upper and lower segments of the left and right arms sampled at 120 Hz. We analyzed the phase relations between flexion/extension movements of the right and left elbow, calculating the natural incidence of different phase relations for a range of movement frequencies (Fig. 3b and c). At low movement frequencies, the distribution of phase incidence was bimodal with peaks for both symmetric and antisymmetric movements (see also Fig. 3d). At higher movement frequencies, phase incidence became unimodal and was dominated by symmetric movements. The progression of phase incidence from a bimodal to a unimodal distribution as movement frequency increases can be seen in Fig. 3b. These results provide an important adjunct to laboratory-based studies because they show that the symmetry bias is expressed in the natural everyday movements of subjects. The coordinate system in which the symmetry bias is expressed is an important issue which has been examined in laboratory-based studies (Mechsner et al., 2001; Swinnen et al., 1998). If the symmetry bias is expressed only in joint-based (intrinsic) coordinates (Fig. 3c), it may be a property of sensorimotor control or the
(a)
(b)
3.2–6.4
0.11 0.10
Transmitter
0.09 0.08 1.2–2.4 0.07
Relative incidence
Frequency bands (Hz)
Recording backpack
0.06 0.05
SR1 SR2
0.04
0.45–0.9 0
90
180
270
360
Phase relation (⬚)
(c)
(d)
(g)
qL
25 Lab task error (⬚)
qR
Relative incidence
Elbow angles (intrinsic coordinates) 0.10 0.08 0.06 0.04 0
90
180
270
20 15 10 5 0
360
Phase relation (⬚)
(e)
Lab task error (deg)
Relative incidence
0.08
0.14
(h) Wrist position (extrinsic coordinates)
cL
0.05
Natural incidence
(f)
cR
Low frequency
0.10 0.08 0.06 0.04 0
90
180
270
Phase relation (⬚)
360
120
High frequency
100 80 60 40 20 0.05
0.08
0.14
Natural incidence
Fig. 3. The statistics of natural arm movements. Panels (b) through (h) are reprinted from Howard et al. (2009a). Used with permission. (a) The wearable motion tracking system consisted of small electromagnetic sensors and a transmitter (the Liberty system from Polhemus). A backpack contained the recording hardware. The sensors were attached to the upper and lower segments of the left and right arms as shown (SR1 ¼ right upper, SR2 ¼ right lower; left arm and sensors not shown). (b) Distributions of relative phases between left and right elbow joint angles. Note bimodal distribution at low movement frequencies consisting of both symmetric (0 /360 ) and antisymmetric (180 ) phases and unimodal distribution at higher frequencies consisting of symmetric phase only. (c) Elbow angles represent an intrinsic coordinate system for representing movements of the arms. Symmetric movements are shown by homogenous left/right pairings of arrow heads (left filled with right filled or left open with right open). Antisymmetric movements are shown by heterogeneous left/right pairings of arrow heads (left filled with right open or right open with left filled). (d) Relative incidence of different phase relations at low frequencies for natural movements represented in intrinsic coordinates (as shown in (c)). (e) Wrist positions in Cartesian coordinates represent an extrinsic coordinate system for representing movements of the arms. Symmetric and antisymmetric movements are shown as in (c). (f) Relative incidence of different phase relations at low frequencies for natural movements represented in extrinsic coordinates (as shown in (e)). (g) Error during the low frequency laboratory-based tracking task for different phase relations plotted against log of the natural incidence of those phase relations. (h) Error during the high-frequency laboratory-based tracking task for different phase relations plotted against log of the natural incidence of those phase relations.
15
musculoskeletal system. However, if the symmetry bias is expressed in external (extrinsic) coordinates (Fig. 3e), it may be a property of the naturalistic tasks which humans regularly perform. For example, bimanual object manipulation is frequent during everyday life (Kilbreath and Heard, 2005) and imposes particular constraints on movements expressed in extrinsic coordinates (Howard et al., 2009a). Specifically, moving the hands together (or apart) to bimanually grasp (or release) an object requires antisymmetric movements, whereas transporting an object once it is grasped requires symmetric movements. If the constraints of bimanual object manipulation are important, then the symmetry bias should be more pronounced for movements expressed in extrinsic coordinates (relevant to the object). To examine this issue, we compared the phase incidence of natural movements defined in intrinsic space (elbow joint angle; Fig. 3c and d) with those defined in extrinsic space (the Cartesian position of sensors on the wrist; Fig. 3e and f). The distribution of phase incidence was bimodal in both cases. However, the incidence of 180 phase was much higher for the movements defined in extrinsic space, occurring as frequently in this case as symmetric movements. This suggests that natural everyday tasks are biased toward both symmetric and antisymmetric movements of the hands in extrinsic space, consistent with the constraints of bimanual object manipulation. An interesting question concerns the relationship between the level of performance on a particular task and the frequency with which that task is performed. It is well known that training improves performance, but with diminishing returns as the length of training increases (Newell and Rosenbloom, 1981). Specifically, relative performance is often related to the log of the number of training trials. This logarithmic dependence applies to a wide range of cognitive tasks including multiplication, visual search, sequence learning, rule learning, and mental rotation (Heathcote et al., 2000). We examined this issue by comparing the incidence of movement phases in
the natural movement dataset with performance on a laboratory-based bimanual tracking task. Subjects tracked two targets (one with each hand) which moved sinusoidally with various phase relations during a low- and high-frequency condition. The performance error on the task was negatively correlated with the log of the phase incidence of natural movements at both low (Fig. 3g) and high (Fig. 3h) frequencies. This demonstrates that the logarithmic training law holds between the natural incidence of everyday movements and performance on a laboratorybased task.
Naturalistic approaches to object manipulation In previous sections, object manipulation emerged as a key feature of naturalistic human behavior. For example, during everyday life, humans spend over half their time (60%) grasping and manipulating objects (Kilbreath and Heard, 2005). Not surprisingly, the statistics of natural hand (Ingram et al., 2008) and arm (Howard et al., 2009a) movements are also consistent with grasping and manipulating objects. Moreover, eye movements during natural tasks are dominated by interactions with objects (Land and Tatler, 2009). Studies which examine object manipulation should thus form an important component of naturalistic approaches to human sensorimotor control.
An ethology of human object manipulation The ability to manipulate objects and use them as tools constitutes a central theme in the study of human biology. For example, the sensorimotor development of human infants is divided into stages which are characterized by an increasing repertoire of object manipulation and tool-using skills (Case, 1985; Parker and Gibson, 1977; Piaget, 1954). Infants begin with simple prehension and manipulation of objects between
16
4–8 months and finally progress to the insightful use of objects as tools by 12–18 months. This first evidence for tool use is regarded as a milestone in human development (Case, 1985; Piaget, 1954). Similarly, the first evidence for tool use in the archeological record (2.5 million years ago) is regarded as a milestone in human evolution (Ambrose, 2001; Parker, 1974). The long evolutionary history of tool use by humans and their ancestors is thought to have influenced the dexterity of the hand (Marzke, 1992; Napier, 1980; Tocheri et al., 2008; Wilson, 1998) and the size and complexity of the brain (Ambrose, 2001; Wilson, 1998). Indeed, the oldest stone tools, although simple, required significant sensorimotor skill to use and manufacture (Pelegrin, 2005; Roche et al., 1999; Schick et al., 1999; Stout and Semaw, 2006; Toth et al., 1993). Object manipulation and tool use is also an important diagnostic feature for comparative studies of animal behavior, especially those comparing the sensorimotor and cognitive skills of humans with other primates (Parker and Gibson, 1977; Torigoe, 1985; Vauclair, 1982, 1984; Vauclair and Bard, 1983). It is known, for example, that a number of animals regularly use and even manufacture tools in their natural environments (Anderson, 2002; Brosnan, 2009; Goodall, 1963, 1968). However, the human ability and propensity for tool use far exceeds that observed in other animals (Boesch and Boesch, 1993; Povinelli, 2000; Schick et al., 1999; Toth et al., 1993; Vauclair, 1984; Visalberghi, 1993). Object manipulation is mediated by a number of interacting processes in the brain including visual object recognition (Wallis and Bulthoff, 1999), retrieval of semantic and functional information about the object (Johnson-Frey, 2004), encoding object shape for effective grasping (Castiello, 2005; Castiello and Begliomini, 2008; Santello and Soechting, 1998), and incorporating the object into the somatosensory representation of the body (Cardinali et al., 2009; Maravita and Iriki, 2004). Object manipulation also represents a challenge for sensorimotor control because
grasping an object can dramatically change the dynamics of the arm (Atkeson and Hollerbach, 1985; Bock, 1990; Lacquaniti et al., 1982). Thus, to continue moving skillfully after grasping an object, the motor commands must adapt to the particular dynamics of the object (Atkeson and Hollerbach, 1985; Bock, 1990, 1993; Johansson, 1998; Lacquaniti et al., 1982). This process is thought to be mediated by internal models of object dynamics (Flanagan et al., 2006; Wolpert and Flanagan, 2001), and a great deal of research has been devoted to understanding how internal models are acquired and represented and how they contribute to skillful object manipulation. This research has employed three main experimental approaches which are reviewed in the following sections. The first approach involves tasks in which subjects manipulate real physical objects. The remaining two approaches involve the use of robotic manipulanda to simulate virtual objects. As described below, the use of virtual objects removes the constraints associated with physical objects because the dynamics and visual feedback are under computer control.
Physical objects with familiar dynamics Laboratory-based experiments in which subjects interact with physical objects that have familiar dynamics allow the representations and skills associated with everyday object manipulation to be examined. As reviewed previously, the ability to perform skilled movements while grasping an object requires the rapid adaptation of the motor commands that control the arm to account for the dynamics associated with the grasped object. The efficacy of this process can be observed in the first movement subjects make immediately after grasping a heavy object. If the mass of the object is known, the kinematics of the first movement made with the object are essentially identical to previous movements made without it (Atkeson and Hollerbach, 1985; Lacquaniti et al., 1982). If the mass is unknown, subjects adapt rapidly
17
before the first movement is finished (Bock, 1990, 1993). Rapid adaptation is also observed when subjects grasp an object in order to lift it (reviewed in Johansson, 1998). In this case, subjects adapt both the forces applied by the digits to grasp the object (the grip force) and the forces applied by the arm to lift it. When lifting an object that is heavier or lighter than expected, for example, subjects adapt their grip force to the actual mass within just a few trials (Flanagan and Beltzner, 2000; Gordon et al., 1993; Johansson and Westling, 1988; Nowak et al., 2007). Subjects also use visual and haptic cues about the size of the object to estimate the grip force applied during lifting (Gordon et al., 1991a,b,c). For familiar everyday objects, subjects can generate appropriate forces on the very first trial (Gordon et al., 1993). Rapid adaptation is also observed when subjects lift a visually symmetric object which has an asymmetrically offset center of mass (Fu et al., 2010; Salimi et al., 2000; Zhang et al., 2010). In this case, subjects predictively generate a compensatory torque at the digits to prevent the object from tilting, a response which develops within the first few trials (Fu et al., 2010). This ability of subjects to rapidly adapt when grasping an object suggests that the sensorimotor system represents the dynamics of objects. Further evidence that subjects have knowledge of object dynamics comes from experiments which examine the perceptual abilities referred to as dynamic touch. Dynamic touch is the ability to perceive the properties of an object based on the forces and torques experienced during manipulation (Gibson, 1966; Turvey, 1996). In a typical experiment, subjects are required to perceive a particular object property after manipulating it behind a screen which occludes vision (reviewed in Turvey, 1996). For example, subjects can use dynamic touch to perceive both the length of a cylindrical rod (Solomon and Turvey, 1988) and the position along the rod at which they grasp it (Pagano et al., 1994). If the rod has a right-angle segment attached to its distal end (to make an elongated “L” shape), subjects can perceive the
orientation of the end segment (Pagano and Turvey, 1992; Turvey et al., 1992). These abilities suggest that subjects extract information from the relationship between the movements they make with an object (the kinematics) and the associated forces and torques. By combining information from dynamic touch with visual information, the perception of object properties can be made more precise (Ernst and Banks, 2002). Thus, both dynamic touch and vision are likely to contribute during naturalistic object manipulation.
Simulated objects with unfamiliar dynamics The range of experimental manipulations available during tasks that use physical objects is limited. The dynamics are constrained to rigid body physics and the precise control of visual feedback is difficult. An extensively used approach which addresses these limitations uses robot manipulanda to simulate novel dynamics combined with display systems to present computer-controlled visual feedback (see reviews in Howard et al., 2009b; Wolpert and Flanagan, 2010). In these experiments, the subject is seated and grasps the handle of a robotic manipulandum which can apply state-dependent forces to the hand. In many of these experiments, the forces depend on the velocity of the hand and are rotated to be perpendicular to the direction of movement (Caithness et al., 2004; Gandolfo et al., 1996; Howard et al., 2008, 2010; Malfait et al., 2002; Shadmehr and Brashers-Krug, 1997; Shadmehr and Mussa-Ivaldi, 1994; Tcheang et al., 2007; Tong et al., 2002). Visual targets are presented using the display system and subjects make reaching movements to the targets from a central starting position. In the initial “null” condition, the motors of the robot are turned off. In this case, subjects have no difficulty reaching the targets and make movements which are approximately straight lines. When the force field is turned on, movement paths are initially perturbed in the direction of the field. Over many trials, the
18
movements progressively return to their original kinematic form as subjects adapt to the perturbing dynamics. This progressive adaptation can be shown to be associated with the acquisition of an internal model of the dynamics. If the force field is unexpectedly turned off, for example, movement paths are perturbed in the opposite direction. This is because subjects generate the forces they expect, based on their acquired internal model of the perturbing dynamics. Dynamic perturbation studies have provided detailed information about the processes of sensorimotor adaptation and the associated representations of dynamics. However, the applicability of the results to everyday object manipulation is not clear (Lackner and DiZio, 2005). In some respects, the learned dynamics appear to be associated with an internal model of a grasped object (Cothros et al., 2006, 2009). In other respects, the learned dynamics appear to be associated with an internal model of the arm (Karniel and Mussa-Ivaldi, 2002; Malfait et al., 2002; Shadmehr and Mussa-Ivaldi, 1994). Moreover, the majority of studies have examined adaptation to novel dynamics, which occurs over tens or hundreds of trials. In contrast, as reviewed in the previous section, humans adapt to the familiar dynamics of objects they encounter during everyday life within just a few trials. In addition, the robotic devices used in most studies generate only translational forces that depend only on the translational kinematics of the hand. In contrast, naturalistic objects generate both translational forces and rotational torques that depend on the translational and rotational kinematics of the object (as well as its orientation in external space). In the next section, an approach which addresses these issues is presented.
Simulated objects with familiar dynamics Robot manipulanda can be used to simulate objects with familiar dynamics (see review in Wolpert and Flanagan, 2010), thereby combining
aspects from the two approaches reviewed above. This allows the processes associated with naturalistic object manipulation to be examined, without the constraints imposed by the physics of realworld objects. However, only a relatively small number of studies have used this approach. For example, the coordination of grip force has been examined during bimanual manipulation of a simulated object. In this case, the dynamics could be coupled or uncoupled between the left and right hands, allowing the effect of object linkage to be examined (White et al., 2008; Witney and Wolpert, 2003; Witney et al., 2000). When the dynamics were coupled, the object behaved like a single object that was grasped between the two hands (see also Howard et al., 2008). Grip force modulation has also been examined using a simulated object which is grasped between the thumb and index finger (Mawase and Karniel, 2010). In this case, the study replicated the object lifting task used in the many grip force studies reviewed above, but with the greater potential for experimental control offered by a simulated environment. Recently, we have taken a different approach by developing a novel planar robotic manipulandum (the WristBOT; Fig. 4a) which includes rotational torque control at the vertical handle (Howard et al., 2009b). Combined with a virtual reality display system, this allows us to simulate the dynamics and visual feedback of an object which can be rotated and translated in the horizontal plane (Howard et al., 2009b; Ingram et al., 2010). The object resembles a small hammer (Fig. 4b), and consists of a mass on the end of a rigid rod. Subjects manipulate the object by grasping the handle at the base of the rod (Fig. 4b). Rotating the object generates both a torque and a force. The torque depends on the angular acceleration of the object. The force can be derived from two orthogonal components. The first and major component (the tangential force) is due to the tangential acceleration of the mass and is always perpendicular to the rod. The second and minor component (the
19 (a)
Drive system cables
(b)
m
r Elbow pulley
a t F Grasp point
Handle pulley (c)
(d) 180 Respond with direction
Rotate object (5 s)
Response angle (⬚)
0° 90
0
−90 −180 −180
0 90 −90 Object orientation (⬚)
180
Fig. 4. The WristBOT robotic manipulandum, simulated object, and haptic discrimination task. Panel (a) is reprinted from Ingram et al. (2010). Copyright (2010), with permission from Elsevier. Panels (b) through (d) are reprinted from Howard et al. (2009b). Copyright (2009), with permission from Elsevier. (a) The WristBOT is a modified version of the vBOT planar two-dimensional robotic manipulandum. It includes an additional degree of freedom allowing torque control around the vertical handle. Cables and pulleys (only two of which are shown) implement the transmission system between the handle and the drive system at the rear of the manipulandum (not shown). (b) The dynamics of the virtual object were simulated as a point mass (mass m) on the end of a rigid rod (length r) of zero mass. Subjects grasped the object at the base of the rod. When rotated clockwise (as shown), the object generated a counter-clockwise torque (t) due to the angular acceleration (a) of the object. The object also generated a force (F) due to the circular motion of the mass. At the peak angular acceleration, the force was perpendicular to the rod, as shown. Importantly, the orientation of the force changes with the orientation of the object. (c) The haptic discrimination task required subjects to rotate the object for 5 s and then make a movement toward the perceived direction of the mass. The object was presented at a different orientation on every trial. Visual feedback was withheld. (d) Response angle (circular mean and circular standard error) across subjects plotted against actual orientation of the object. Solid line shows circular linear fit to subject responses and dashed line shows perfect performance.
centripetal force) is due to the circular velocity of the mass and acts along the rod toward the center of rotation. Simulations demonstrated that the
peak force acts in a direction that is close to perpendicular to the rod. Thus, as subjects rotate the object, the force experienced at the handle
20
will perturb the hand in a direction that depends on the orientation of the object. In the following sections, we review two recent studies which have used this simulated object.
Haptic discrimination task The direction of the forces associated with rotating an object provides a potential source of information regarding its orientation (or rather, the orientation of its center of mass). Previous studies of haptic perception have used physical objects and have suggested that subjects use torque to determine the orientation of the principal axis of the object (Pagano and Turvey, 1992; Turvey, 1996; Turvey et al., 1992). Specifically, the smallest torque is associated with rotating the object around its principal axis. We used a simulated haptic discrimination task (Fig. 4c) to determine if subjects can also use force direction to perceive object orientation (Howard et al., 2009b). In the case of our simulated object, force direction was the only source of information because torque is independent of orientation when rotating around a fixed axis. Subjects first rotated the simulated object back and forth for 5 s in the absence of visual feedback and then indicated the orientation of the object by making a movement toward the perceived location of the center of mass. Results showed that subjects could accurately perceive the orientation of the object based on its simulated dynamics (Fig. 4d). This suggests that the forces associated with rotating an object are an important source of information regarding object orientation.
Object manipulation task To examine the representation of dynamics associated with familiar everyday objects, we developed a manipulation task that required subjects to rotate the simulated object while keeping its handle stationary (Ingram et al.,
2010). The visual orientation and dynamics of the object could be varied from trial to trial (Fig. 5a). To successfully perform the task, subjects had to generate a torque to rotate the object as well as a force to keep the handle stationary. As described above, the direction of the force depends on the orientation of the object (see Fig. 4b). In the first experiment, the object was presented at different visual orientations (see inset of Fig. 5a). Subjects experienced the torque as they rotated the object, but not the forces. Instead, the manipulandum simulated a stiff spring which clamped the handle in place. This allowed us to measure the anticipatory forces produced by subjects in the absence of the forces normally produced by the object. Results showed that subjects produce anticipatory forces in directions that were appropriate for the visual orientation of the object (Fig. 5b). That is, subjects produce forces that are directed to oppose the forces they expect the object to produce. Importantly, subjects do this before they have experienced the full dynamics of the object, providing evidence that they have a preexisting representation of the dynamics that can be recalled based on visual information. In subsequent experiments, we examined the structure of this representation, how it adapted when exposed to the dynamics of a particular object, and how it was modulated by the visual orientation of the object. In a second experiment, we examined the time course of adaptation (Fig. 5c). Subjects first experienced the object with the forces normally generated by its dynamics turned off. After they had adapted to this zero-force object (preexposure phase in Fig. 5c), the forces were unexpectedly turned on. Although this caused large deviations of the handle on the first few trials, these errors rapidly decreased over subsequent trials as subjects adapted the magnitude of their forces to stabilize the object (exposure phase in Fig. 5c). After many trials of exposure to the normal dynamics of the object, the forces associated with rotating the object were again turned off (postexposure phase in
21 (a)
(b)
Mirror
267 Peak force angle (⬚)
0° −45°
y
−90°
x
−135° 180°
222 177 132 87 −180
Peak displacement (cm)
Exposure (full dynamics)
Pre-exposure (zero force)
1.2
0
(d)
Post-exposure (zero force)
0.8 Peak displacement (cm)
(c)
−135 −90 −45 Object orientation (⬚)
1.0 0.8 0.6 0.4
Training Transfer
0.6
0.4
0.2 1
48 49
100
272
Trials
320
−180
−90 −45 Object orientation (⬚)
0
Fig. 5. The representation of familiar object dynamics. Panel (a) (modified) and panels (b) and (d) (reprinted) are from Ingram et al. (2010). Copyright (2010), with permission from Elsevier. (a) Top view of subject showing visual feedback of the object projected over the hand. The mirror prevents subject from seeing either their hand or the manipulandum. Dashed line shows subject's midline. Inset shows the object presented at different visual orientations. (b) The angle of the peak force produced by subjects as they rotate the object (circular mean and circular standard error) plotted against the visual orientation of the object. The dashed line shows perfect performance. (c) Peak displacement of the handle of the object plotted against trial number. Peak displacement increases when the forces associated with rotating the object are unexpectedly turned on (exposure), decreasing rapidly over the next few trials to an asymptotic level. Peak displacement increases again when the forces are unexpectedly turned off (postexposure), decreasing rapidly to preexposure levels. (d) Peak displacement plotted against the orientation of the object. Subjects experience the full dynamics of the object at the training orientation (square) and are presented with a small number of probe trials at transfer orientations (circles) with the forces turned off. Peak displacement is a measure of the forces subjects produce as they rotate the object. The largest forces (displacements) are produced at the training orientation and decrease progressively as the orientation of the object increases relative to the training orientation. Solid line shows the mean of a Gaussian fit individually to each subject (mean standard deviation of Gaussian fit ¼ 34 ).
Fig. 5c). This initially caused large deviations of the handle, due to the large forces that subjects had learned to produce during the exposure phase. Once again, these errors rapidly decreased over subsequent trials as subjects adapted the magnitude of their forces to be appropriate for
the zero-force object. Importantly, these results show that the rapid adaptation characteristic of manipulating everyday objects can also occur when subjects manipulate simulated objects, provided the dynamics are familiar (see also Witney et al., 2000).
22
In a third experiment, we presented subjects with objects of three different masses to examine how this experience would influence the magnitude of the forces they produced. As expected, subjects adapted the force magnitude according to the mass of the object. Similar results have been obtained for grip force when subjects lift objects of varying mass (Flanagan and Beltzner, 2000; Gordon et al., 1993; Johansson and Westling, 1988; Nowak et al., 2007). The adaptation of force magnitude was further examined in a fourth experiment which examined generalization. Studies of generalization can reveal important details of how dynamics are represented (Shadmehr, 2004). Subjects experienced the object at a single training orientation after which force magnitude was examined at five visual orientations, including four novel orientations where the object had not been experienced. We observed a Gaussian pattern of generalization, with the largest forces produced at the training orientation, decreasing progressively as the orientation increased relative to the training orientation (Fig. 5d). Results from this experiment are consistent with multiple local representations of object dynamics because a single general representation would predict perfect generalization. In summary, using a novel robotic manipulation to simulate a familiar naturalistic object, we have shown that subjects have a preexisting representation of the associated dynamics. Subjects can recall this representation based on vision of the object and can use it for haptic perception when visual information is not available. During manipulation, adaptation of the representation to a particular object is rapid, consistent with many previous studies in which subjects manipulate physical objects. Adaptation is also context specific, being locally confined to the orientation at which the object is experienced. These results suggest that the ability to skillfully manipulate everyday objects is mediated by multiple rapidly adapting representations which capture the local dynamics associated with specific object contexts.
Conclusion The methods of sensorimotor neuroscience have traditionally involved the use of artificial laboratory-based tasks to examine the mechanisms that underlie voluntary movement. In the case of visual neuroscience, the adoption of more naturalistic approaches has involved a shift from artificial stimuli created in the laboratory to natural images taken from the real world. Similarly, the adoption of more naturalistic approaches in sensorimotor neuroscience will require a shift from artificial laboratory-based tasks to natural tasks that are representative of the everyday behavior of subjects. Fortunately, continuing advances in motion tracking, virtual reality and even mobile phone technology are making this shift ever more tractable. In the case of visual neuroscience, naturalistic approaches have required new analytical methods from information theory, statistics, and engineering and have led to new theories of sensory processing. Similarly, naturalistic approaches to human sensorimotor control will almost certainly require new analytical techniques, especially with regard to large datasets of natural behavior and movement kinematics. However, we expect that these efforts will be productive.
Acknowledgments We thank Randy Flanagan, Ian Howard, and Konrad Körding for their collaboration on various projects reviewed herein. This work was supported by the Wellcome Trust. References Ambrose, S. H. (2001). Paleolithic technology and human evolution. Science, 291, 1748–1753. Anderson, J. R. (2002). Gone fishing: Tool use in animals. Biologist (London), 49, 15–18. Anderson, I., & Muller, H. (2006). Practical context awareness for GSM cell phones. In 2006 10th IEEE international symposium on wearable computers, Montreux, Switzerland: IEEE, pp. 126–127.
23 Atkeson, C., & Hollerbach, J. (1985). Kinematic features of unrestrained vertical arm movements. The Journal of Neuroscience, 5, 2318–2330. Balasubramanian, V., & Sterling, P. (2009). Receptive fields and functional architecture in the retina. The Journal of Physiology, 587, 2753–2767. Ballard, D. H., Hayhoe, M. M., Li, F., Whitehead, S. D., Frisby, J. P., Taylor, J. G., et al. (1992). Hand-eye coordination during sequential tasks [and discussion]. Philosophical Transactions of the Royal Society B: Biological Sciences, 337, 331–339. Barabasi, A. (2005). The origin of bursts and heavy tails in human dynamics. Nature, 435, 207–211. Barlow, H. B. (1961). Possible principles underlying the transformation of sensory messages. In W. Rosenblith (Ed.), Sensory Communication (pp. 217–234). Cambridge, MA: M.I.T. Press. Beetz, M., Stulp, F., Radig, B., Bandouch, J., Blodow, N., Dolha, M., et al. (2008). The assistive kitchen—A demonstration scenario for cognitive technical systems. In 2008 17th IEEE international symposium on robot and human interactive communication, (Vols. 1 and 2, pp. 1–8). New York: IEEE. Bock, O. (1990). Load compensation in human goal-directed arm movements. Behavioural Brain Research, 41, 167–177. Bock, O. (1993). Early stages of load compensation in human aimed arm movements. Behavioural Brain Research, 55, 61–68. Bock, O., & Hagemann, A. (2010). An experimental paradigm to compare motor performance under laboratory and under everyday-like conditions. Journal of Neuroscience Methods, 193, 24–28. Bock, O., Schneider, S., & Bloomberg, J. (2001). Conditions for interference versus facilitation during sequential sensorimotor adaptation. Experimental Brain Research, 138, 359–365. Boesch, C., & Boesch, H. (1993). Diversity of tool use and tool making in wild chimpanzees. In A. Berthelet & J. Chavaillon (Eds.), The use of tools by human and nonhuman primates (pp. 158–174). Oxford: Oxford University Press. Bokharouss, I., Wobcke, W., Chan, Y. W., Limaru, A., & Wong, A. (2007). A location-aware mobile call handling assistant. In 21st international conference on advanced networking and applications workshops/symposia, Vol. 2, proceedings, Los Alamitos: IEEE Computer Society, pp. 282–289. Brashers-Krug, T., Shadmehr, R., & Bizzi, E. (1996). Consolidation in human motor memory. Nature, 382, 252–255. Brockmann, D., Hufnagel, L., & Geisel, T. (2006). The scaling laws of human travel. Nature, 439, 462–465. Brosnan, S. F. (2009). Animal behavior: The right tool for the job. Current Biology, 19, R124–R125.
Buswell, G. T. (1920). An experimental study of the eye-voice span in reading. Chicago: Chicago University Press. Buswell, G. T. (1935). How people look at pictures: A study of the psychology of perception in art. Chicago: Chicago University Press. Butsch, R. I. C. (1932). Eye movements and the eye-hand span in typewriting. Journal of Educational Psychology, 23, 104–121. Caithness, G., Osu, R., Bays, P., Chase, H., Klassen, J., Kawato, M., et al. (2004). Failure to consolidate the consolidation theory of learning for sensorimotor adaptation tasks. The Journal of Neuroscience, 24, 8662–8671. Cardinali, L., Frassinetti, F., Brozzoli, C., Urquizar, C., Roy, A. C., & Farne, A. (2009). Tool-use induces morphological updating of the body schema. Current Biology, 19, R478–R479. Carpenter, R. H. (2000). The neural control of looking. Current Biology, 10, R291–R293. Case, R. (1985). Intellectual development—Birth to adulthood. Orlando, FL: Academic Press Inc. Castiello, U. (2005). The neuroscience of grasping. Nature Reviews. Neuroscience, 6, 726–736. Castiello, U., & Begliomini, C. (2008). The cortical control of visually guided grasping. The Neuroscientist, 14, 157–170. Corbetta, M., Akbudak, E., Conturo, T., Snyder, A., Ollinger, J., Drury, H., et al. (1998). A common network of functional areas for attention and eye movements. Neuron, 21, 761–773. Cothros, N., Wong, J. D., & Gribble, P. L. (2006). Are there distinct neural representations of object and limb dynamics? Experimental Brain Research, 173, 689–697. Cothros, N., Wong, J., & Gribble, P. L. (2009). Visual cues signaling object grasp reduce interference in motor learning. Journal of Neurophysiology, 102, 2112–2120. Craig, J. J. (1989). Introduction to robotics—Mechanics and control (2nd ed.). Reading, MA: Addison-Wesley Publishing Company. Devlic, A., Reichle, R., Wagner, M., Pinheiro, M. K., Vanromplay, Y., Berbers, Y., et al. (2009). Context inference of users’ social relationships and distributed policy management. In: 2009 IEEE international conference on pervasive computing and communications, New York: IEEE. Eagle, N., & Pentland, A. (2006). Reality mining: Sensing complex social systems. Personal and Ubiquitous Computing, 10, 255–268. Eagle, N., & Pentland, A. S. (2009). Eigenbehaviors: Identifying structure in routine. Behavioral Ecology and Sociobiology, 63, 1057–1066. Eibl-Eibesfeldt, I. (1989). Human ethology. Piscataway, NY: Aldine Transaction. Ernst, M. O., & Banks, M. S. (2002). Humans integrate visual and haptic information in a statistically optimal fashion. Nature, 415, 429–433.
24 Fitzpatrick, D. (2000). Seeing beyond the receptive field in primary visual cortex. Current Opinion in Neurobiology, 10, 438–443. Flanagan, J. R., & Beltzner, M. A. (2000). Independence of perceptual and sensorimotor predictions in the size-weight illusion. Nature Neuroscience, 3, 737–741. Flanagan, J. R., Bowman, M. C., & Johansson, R. S. (2006). Control strategies in object manipulation tasks. Current Opinion in Neurobiology, 16, 650–659. Flombaum, J. I., & Santos, L. R. (2005). Rhesus monkeys attribute perceptions to others. Current Biology, 15, 447–452. Fogassi, L., & Luppino, G. (2005). Motor functions of the parietal lobe. Current Opinion in Neurobiology, 15, 626–631. Földiak, P. (1991). Learning invariance from transformation sequences. Neural Computation, 3, 194–200. Fu, Q., Zhang, W., & Santello, M. (2010). Anticipatory planning and control of grasp positions and forces for dexterous two-digit manipulation. The Journal of Neuroscience, 30, 9117–9126. Gandolfo, F., Mussa-Ivaldi, F. A., & Bizzi, E. (1996). Motor learning by field approximation. Proceedings of the National Academy of Sciences of the United States of America, 93, 3843–3846. Ganti, R. K., Srinivasan, S., & Gacic, A. (2010). Multisensor fusion in smartphones for lifestyle monitoring. Proceedings of the 2010 International Conference on Body Sensor Networks (pp. 36–43). Washington, DC: IEEE Computer Society. Geisler, W. S. (2008). Visual perception and the statistical properties of natural scenes. Annual Review of Psychology, 59, 167–192. Ghahramani, Z., & Wolpert, D. M. (1997). Modular decomposition in visuomotor learning. Nature, 386, 392–395. Ghahramani, Z., Wolpert, D. M., & Jordan, M. I. (1996). Generalization to local remappings of the visuomotor coordinate transformation. The Journal of Neuroscience, 16, 7085–7096. Gibson, J. J. (1966). The senses considered as perceptual systems. Boston, MA: Houghton Mifflin. Goedert, K. M., & Willingham, D. B. (2002). Patterns of interference in sequence learning and prism adaptation inconsistent with the consolidation hypothesis. Learning and Memory, 9, 279–292. Gold, J. I., & Shadlen, M. N. (2000). Representation of a perceptual decision in developing oculomotor commands. Nature, 404, 390–394. González, M., Hidalgo, C., & Barabási, A. (2008). Understanding individual human mobility patterns. Nature, 453, 779–782. Goodall, J. (1963). Feeding behaviour of wild chimpanzees— A preliminary report. Symposium of the Zoological Society of London, 10, 9–48.
Goodall, J. (1968). The behaviour of free-living chimpanzees in the Gombe Stream Reserve. Animal Behaviour Monographs, 1, 161–311. Gordon, A. M., Forssberg, H., Johansson, R. S., & Westling, G. (1991a). The integration of haptically acquired size information in the programming of precision grip. Experimental Brain Research, 83, 483–488. Gordon, A. M., Forssberg, H., Johansson, R. S., & Westling, G. (1991b). Integration of sensory information during the programming of precision grip: Comments on the contributions of size cues. Experimental Brain Research, 85, 226–229. Gordon, A. M., Forssberg, H., Johansson, R. S., & Westling, G. (1991c). Visual size cues in the programming of manipulative forces during precision grip. Experimental Brain Research, 83, 477–482. Gordon, A. M., Westling, G., Cole, K. J., & Johansson, R. S. (1993). Memory representations underlying motor commands used during manipulation of common and novel objects. Journal of Neurophysiology, 69, 1789–1796. Gross, C. G. (2002). Genealogy of the “Grandmother Cell” The Neuroscientist, 8, 512–518. Gross, C. G., Rochamir, C. E., & Bender, D. B. (1972). Visual properties of neurons in inferotemporal cortex of macaque. Journal of Neurophysiology, 35, 96–111. Győrbíró, N., Fábián, Á., & Hományi, G. (2009). An activity recognition system for mobile phones. Mobile Networks and Applications, 14, 82–91. Hager-Ross, C., & Schieber, M. H. (2000). Quantifying the independence of human finger movements: Comparisons of digits, hands, and movement frequencies. The Journal of Neuroscience, 20, 8542–8550. Hare, B., Call, J., Agnetta, B., & Tomasello, M. (2000). Chimpanzees know what conspecifics do and do not see. Animal Behaviour, 59, 771–785. Hare, B., Call, J., & Tomasello, M. (2001). Do chimpanzees know what conspecifics know? Animal Behaviour, 61, 139–151. Hartline, H. (1938). The response of single optic nerve fibers of the vertebrate eye to illumination of the retina. The American Journal of Physiology, 121, 400–415. Haruno, M., Wolpert, D. M., & Kawato, M. (2001). Mosaic model for sensorimotor learning and control. Neural Computation, 13, 2201–2220. Hayhoe, M., & Ballard, D. (2005). Eye movements in natural behavior. Trends in Cognitive Sciences (Regular Edition), 9, 188–194. Hayhoe, M., Mennie, N., Sullivan, B., & Gorgos, K. (2005). The role of internal models and prediction in catching balls. Proceedings of the American Association for Artificial Intelligence, Fall.
25 Hayhoe, M. M., Shrivastava, A., Mruczek, R., & Pelz, J. B. (2003). Visual memory and motor planning in a natural task. Journal of Vision, 3, 49–63. Heathcote, A., Brown, S., & Mewhort, D. J. (2000). The power law repealed: The case for an exponential law of practice. Psychonomic Bulletin and Review, 7, 185–207. Henderson, J. (2003). Human gaze control during real-world scene perception. Trends in Cognitive Sciences, 7, 498–504. Henderson, J., & Hollingworth, A. (1999). High-level scene perception. Annual Review of Psychology, 50, 243–271. Howard, I. S., Ingram, J. N., Kording, K. P., & Wolpert, D. M. (2009a). Statistics of natural movements are reflected in motor errors. Journal of Neurophysiology, 102, 1902–1910. Howard, I. S., Ingram, J. N., & Wolpert, D. M. (2009b). A modular planar robotic manipulandum with end-point torque control. Journal of Neuroscience Methods, 188, 199–211. Howard, I. S., Ingram, J. N., & Wolpert, D. M. (2008). Composition and decomposition in bimanual dynamic learning. The Journal of Neuroscience, 28, 10531–10540. Howard, I. S., Ingram, J. N., & Wolpert, D. M. (2010). Context-dependent partitioning of motor learning in bimanual movements. Journal of Neurophysiology, 104, 2082–2091. Hubel, D. (1960). Single unit activity in lateral geniculate body and optic tract of unrestrained cats. The Journal of Physiology, 150, 91–104. Hubel, D., & Wiesel, T. (1959). Receptive fields of single neurones in the cat's striate cortex. The Journal of Physiology, 148, 574–591. Hubel, D., & Wiesel, T. (1961). Integrative action in the cat's lateral geniculate body. The Journal of Physiology, 155, 385–398. Hubel, D., & Wiesel, T. (1965). Receptive fields and functional architecture in two nonstriate visual areas (18 and 19) of the cat. Journal of Neurophysiology, 28, 229–289. Humphrey, N. (1976). The social function of intellect. In P. P. G. Bateson & R. A. Hinde (Eds.), Growing points in ethology (pp. 1–8). Cambridge: Cambridge University Press. Hynes, M., Wang, H., & Kilmartin, L. (2009). Off-the-shelf mobile handset environments for deploying accelerometer based gait and activity analysis algorithms. In Conference Proceedings—IEEE Engineering in Medicine and Biology Society 2009, 5187–5190. Ingram, J. N., Howard, I. S., Flanagan, J. R., & Wolpert, D. M. (2010). Multiple grasp-specific representations of tool dynamics mediate skillful manipulation. Current Biology, 20, 618–623. Ingram, J. N., Kording, K. P., Howard, I. S., & Wolpert, D. M. (2008). The statistics of natural hand movements. Experimental Brain Research, 188, 223–236. Johansson, R. S. (1998). Sensory input and control of grip. Novartis Foundation Symposium, 218, 45–59 discussion 59–63.
Johansson, R. S., & Westling, G. (1988). Coordinated isometric muscle commands adequately and erroneously programmed for the weight during lifting task with precision grip. Experimental Brain Research, 71, 59–71. Johansson, R., Westling, G., Backstrom, A., & Flanagan, J. (2001). Eye-hand coordination in object manipulation. The Journal of Neuroscience, 21, 6917–6932. Johnson-Frey, S. H. (2004). The neural bases of complex tool use in humans. Trends in Cognitive Sciences, 8, 71–78. Jones, L. A. (1997). Dextrous hands: Human, prosthetic, and robotic. Presence, 6, 29–56. Jones, L. A., & Lederman, S. J. (2006). Human hand function. Oxford: Oxford University Press. Kagerer, F. A., Contreras-Vidal, J. L., & Stelmach, G. E. (1997). Adaptation to gradual as compared with sudden visuo-motor distortions. Experimental Brain Research, 115, 557–561. Karniel, A., & Mussa-Ivaldi, F. A. (2002). Does the motor control system use multiple models and context switching to cope with a variable environment? Experimental Brain Research, 143, 520–524. Kelso, J. A. S. (1984). Phase transitions and critical behaviour in human interlimb coordination. The American Journal of Physiology, 240, 1000–1004. Kelso, J. A. (1995). Dynamic patterns: The self-organization of brain and behavior. Cambridge, MA: The MIT Press. Kilbreath, S. L., & Gandevia, S. C. (1994). Limited independent flexion of the thumb and fingers in human subjects. Journal of Physiology, 479, 487–497. Kilbreath, S., & Heard, R. (2005). Frequency of hand use in healthy older persons. The Australian Journal of Physiotherapy, 51, 119–122. Kingstone, A., Smilek, D., & Eastwood, J. D. (2008). Cognitive ethology: A new approach for studying human cognition. British Journal of Psychology, 99, 317–340. Kitagawa, M., & Windor, B. (2008). MoCap for artists. Amsterdam: Focal Press. Konorski, J. (1967). Integrative activity of the brain: An interdisciplinary approach. Chicago, IL: University of Chicago Press. Kording, K., Kayser, C., Einhauser, W., & Konig, P. (2004). How are complex cell properties adapted to the statistics of natural stimuli? Journal of Neurophysiology, 91, 206–212. Krakauer, J. W., Ghez, C., & Ghilardi, M. F. (2005). Adaptation to visuomotor transformations: Consolidation, interference, and forgetting. The Journal of Neuroscience, 25, 473–478. Krakauer, J. W., Ghilardi, M. F., & Ghez, C. (1999). Independent learning of internal models for kinematic and dynamic control of reaching. Nature Neuroscience, 2, 1026–1031. Krakauer, J. W., Pine, Z. M., Ghilardi, M. F., & Ghez, C. (2000). Learning of visuomotor transformations for vectorial
26 planning of reaching trajectories. The Journal of Neuroscience, 20, 8916–8924. Krauzlis, R. J. (2005). The control of voluntary eye movements: New perspectives. The Neuroscientist, 11, 124–137. Kuffler, S. (1953). Discharge patterns and functional organization of mammalian retina. Journal of Neurophysiology, 16, 37–68. Lackner, J. R., & DiZio, P. (2005). Motor control and learning in altered dynamic environments. Current Opinion in Neurobiology, 15, 653–659. Lacquaniti, F., Soechting, J., & Terzuolo, C. (1982). Some factors pertinent to the organization and control of arm movements. Brain Research, 252, 394–397. Land, M. (1999). Motion and vision: Why animals move their eyes. Journal of Comparative Physiology A: Neuroethology, Sensory, Neural, and Behavioral Physiology, 185, 341–352. Land, M. F. (2006). Eye movements and the control of actions in everyday life. Progress in Retinal and Eye Research, 25, 296–324. Land, M. F. (2009). Vision, eye movements, and natural behavior. Visual Neuroscience, 26, 51–62. Land, M. F., & Furneaux, S. (1997). The knowledge base of the oculomotor system. Philosophical Transactions of the Royal Society of London Series B: Biological Sciences, 352, 1231–1239. Land, M., & Hayhoe, M. (2001). In what ways do eye movements contribute to everyday activities? Vision Research, 41, 3559–3565. Land, M., & Horwood, J. (1995). Which parts of the road guide steering? Nature, 377, 339–340. Land, M., & Lee, D. (1994). Where we look when we steer. Nature, 369, 742–744. Land, M., & McLeod, P. (2000). From eye movements to actions: How batsmen hit the ball. Nature Neuroscience, 3, 1340–1345. Land, M., Mennie, N., & Rusted, J. (1999). The roles of vision and eye movements in the control of activities of daily living. Perception, 28, 1311–1328. Land, M., & Tatler, B. (2001). Steering with the head: The visual strategy of a racing driver. Current Biology, 11, 1215–1220. Land, M., & Tatler, B. (2009). Looking and acting—Vision and eye movements in natural behaviour. Oxford, England: Oxford University Press. Lang, C. E., & Schieber, M. H. (2004). Human finger independence: Limitations due to passive mechanical coupling versus active neuromuscular control. Journal of Neurophysiology, 92, 2802–2810. Laughlin, S. (1987). Form and function in retinal processing. Trends in Neurosciences, 10, 478–483. Lee, G. X., Low, K. S., & Taher, T. (2010). Unrestrained measurement of arm motion based on a wearable wireless sensor network. In IEEE transactions on instrumentation and measurement, 59(5), 1309–1317.
Lee, J. Y., & Schweighofer, N. (2009). Dual adaptation supports a parallel architecture of motor memory. The Journal of Neuroscience, 29, 10396–10404. Lemon, R. N. (1997). Mechanisms of cortical control of hand function. The Neuroscientist, 3, 389–398. Li, Y., Levin, O., Forner-Cordero, A., & Swinnen, S. P. (2005). Interactions between interlimb and intralimb coordination during the performance of bimanual multijoint movements. Experimental Brain Research, 163, 515–526. Luinge, H., & Veltink, P. (2005). Measuring orientation of human body segments using miniature gyroscopes and accelerometers. Medical and Biological Engineering and Computing, 43, 273–282. Malfait, N., Shiller, D. M., & Ostry, D. J. (2002). Transfer of motor learning across arm configurations. The Journal of Neuroscience, 22, 9656–9660. Maravita, A., & Iriki, A. (2004). Tools for the body (schema). Trends in Cognitive Sciences, 8, 79–86. Marzke, M. W. (1992). Evolutionary development of the human thumb. Hand Clinics, 8, 1–8. Mason, C. R., Gomez, J. E., & Ebner, T. J. (2001). Hand synergies during reach-to-grasp. Journal of Neurophysiology, 86, 2896–2910. Mawase, F., & Karniel, A. (2010). Evidence for predictive control in lifting series of virtual objects. Experimental Brain Research, 203, 447–452. McFarland, D. (1999). Animal behaviour: Psychobiology, ethology and evolution (3rd ed.). Harlow, England: Pearson Education Limited. Mechsner, F., Kerzel, D., Knoblich, G., & Prinz, W. (2001). Perceptual basis of bimanual coordination. Nature, 414, 69–73. Miall, C. (2002). Modular motor learning. Trends in Cognitive Sciences, 6, 1–3. Miall, R. C., Jenkinson, N., & Kulkarni, K. (2004). Adaptation to rotated visual feedback: A re-examination of motor interference. Experimental Brain Research, 154, 201–210. Mündermann, L., Corazza, S., & Andriacchi, T. (2006). The evolution of methods for the capture of human movement leading to markerless motion capture for biomechanical applications. Journal of Neuroengineering and Rehabilitation, 3, 1–11. Munoz, D. P. (2002). Commentary: Saccadic eye movements: Overview of neural circuitry. Progress in Brain Research, 140, 89–96. Napier, N. (1980). Hands. New York: Pantheon Books. Newell, A., & Rosenbloom, P. S. (1981). Mechanisms of skill acquisition and the law of practice. In J. R. Anderson (Ed.), Cognitive skills and their acquisition (pp. 1–55). Hillsdale, NJ: Erlbaum. Nowak, D. A., Koupan, C., & Hermsdorfer, J. (2007). Formation and decay of sensorimotor and associative memory in object lifting. European Journal of Applied Physiology, 100, 719–726.
27 Nozaki, D., Kurtzer, I., & Scott, S. H. (2006). Limited transfer of learning between unimanual and bimanual skills within the same limb. Nature Neuroscience, 9, 1364–1366. Nozaki, D., & Scott, S. H. (2009). Multi-compartment model can explain partial transfer of learning within the same limb between unimanual and bimanual reaching. Experimental Brain Research, 194, 451–463. Olshausen, B., & Field, D. (1996). Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature, 381, 607–609. Pagano, C. C., Kinsella-Shaw, J. M., Cassidy, P. E., & Turvey, M. T. (1994). Role of the inertia tensor in haptically perceiving where an object is grasped. Journal of Experimental Psychology. Human Perception and Performance, 20, 276–285. Pagano, C. C., & Turvey, M. T. (1992). Eigenvectors of the inertia tensor and perceiving the orientation of a hand-held object by dynamic touch. Perception and Psychophysics, 52, 617–624. Parker, C. (1974). The antecedents of man the manipulator. Journal of Human Evolution, 3, 493–500. Parker, S. T., & Gibson, K. R. (1977). Object manipulation, tool use and sensorimotor intelligence as feeding adaptations in cebus monkeys and great apes. Journal of Human Evolution, 6, 623–641. Pelegrin, J. (2005). Remarks about archaelogical techniques and methods of knapping: Elements of a cognitive approach to stone knapping. In V. Roux & B. Bril (Eds.), Stone knapping: The necessary conditions for a uniquely human behaviour (pp. 23–34). Cambridge: McDonald Institute for Archaelogical Research. Pelz, J., & Canosa, R. (2001). Oculomotor behavior and perceptual strategies in complex tasks. Vision Research, 41, 3587–3596. Pelz, J., Hayhoe, M., & Loeber, R. (2001). The coordination of eye, head, and hand movements in a natural task. Experimental Brain Research, 139, 266–277. Penfield, W., & Broldrey, E. (1937). Somatic motor and sensory representation in the cerebral cortex of man as studied by electrical stimulation. Brain, 60, 389–443. Perrett, D., Mistlin, A., & Chitty, A. (1987). Visual neurones responsive to faces. Trends in Neurosciences, 10, 358–364. Philipose, M., Fishkin, K., Perkowitz, M., Patterson, D., Fox, D., Kautz, H., & Hahnel, D. (2004). Inferring activities from interactions with objects. In IEEE pervasive computing (pp. 50–57). New York, NY: IEEE Communications Society. Piaget, J. (1954). Construction of reality in the child. New York: Bellantine Books. Pouget, A., & Snyder, L. (2000). Computational approaches to sensorimotor transformations. Nature Neuroscience, 3, 1192–1198. Povinelli, D. (2000). Folk physics for apes. Oxford: Oxford University Press.
Povinelli, D., & Bering, J. (2002). The mentality of apes revisited. Current Directions in Psychological Science, 11, 115–119. Reilly, K. T., & Hammond, G. R. (2000). Independence of force production by digits of the human hand. Neuroscience Letters, 290, 53–56. Reilly, K. T., & Schieber, M. H. (2003). Incomplete functional subdivision of the human multitendoned finger muscle flexor digitorum profundus: An electromyographic study. Journal of Neurophysiology, 90, 2560–2570. Reinagel, P. (2001). How do visual neurons respond in the real world? Current Opinion in Neurobiology, 11, 437–442. Ringach, D. (2004). Mapping receptive fields in primary visual cortex. Journal of Physiology (London), 558, 717–728. Rizzolatti, G., Luppino, G., & Matelli, M. (1998). The organization of the cortical motor system: New concepts. Electroencephalography and Clinical Neurophysiology, 106, 283–296. Roche, H., Delagnes, A., Brugal, J., Feibel, C., Kibunjia, M., Mourre, V., et al. (1999). Early hominid stone tool production and technical skill 2.34 Myr ago in west Turkana, Kenya. Nature, 399, 57–60. Salimi, I., Hollender, I., Frazier, W., & Gordon, A. M. (2000). Specificity of internal representations underlying grasping. Journal of Neurophysiology, 84, 2390–2397. Santello, M., Flanders, M., & Soechting, J. F. (1998). Postural hand synergies for tool use. The Journal of Neuroscience, 18, 10105–10115. Santello, M., Flanders, M., & Soechting, J. F. (2002). Patterns of hand motion during grasping and the influence of sensory guidance. The Journal of Neuroscience: The Official Journal of the Society for Neuroscience, 22, 1426–1435. Santello, M., & Soechting, J. F. (1998). Gradual molding of the hand to object contours. Journal of Neurophysiology, 79, 1307–1320. Schall, J. D. (2000). From sensory evidence to a motor command. Current Biology, 10, R404–R406. Schick, K., Toth, N., & Garufi, G. (1999). Continuing investigations into the stone tool-making and tool-using capabilities of a Bonobo (Pan paniscus). Journal of Archaeological Science, 26, 821–832. Schieber, M. H., & Santello, M. (2004). Hand function: Peripheral and central constraints on performance. Journal of Applied Physiology, 96, 2293–2300. Schlich, R., & Axhausen, K. (2003). Habitual travel behaviour: Evidence from a six-week travel diary. Transportation, 30, 13–36. Schmidt, R. C., & Lee, T. D. (2005). Motor control and learning—A behavioral emphasis (4th ed.). Champaign, IL: Human Kinetics. Schmidt, R. C., Shaw, B. K., & Turvey, M. T. (1993). Coupling dynamics in interlimb coordination. Journal of Experimental Psychology. Human Perception and Performance, 19, 397–415.
28 Shadmehr, R. (2004). Generalization as a behavioral window to the neural mechanisms of learning internal models. Human Movement Science, 23, 543–568. Shadmehr, R., & Brashers-Krug, T. (1997). Functional stages in the formation of human long-term motor memory. The Journal of Neuroscience, 17, 409–419. Shadmehr, R., & Mussa-Ivaldi, F. A. (1994). Adaptive representation of dynamics during learning of a motor task. The Journal of Neuroscience, 14, 3208–3224. Shadmehr, R., & Wise, S. P. (2005). The computational neurobiology of reaching and pointing: A foundation for motor learning. Cambridge, MA: The MIT Press. Simoncelli, E. P. (2003). Vision and the statistics of the visual environment. Current Opinion in Neurobiology, 13, 144–149. Simoncelli, E., & Olshausen, B. (2001). Natural image statistics and neural representation. Annual Review of Neuroscience, 24, 1193–1216. Slijper, H., Richter, J., Over, E., Smeets, J., & Frens, M. (2009). Statistics predict kinematics of hand movements during everyday activity. Journal of Motor Behavior, 41, 3–9. Snyder, L. H. (2000). Coordinate transformations for eye and arm movements in the brain. Current Opinion in Neurobiology, 10, 747–754. Soechting, J., & Flanders, M. (1992). Moving in three-dimensional space: Frames of reference, vectors, and coordinate systems. Annual Review of Neuroscience, 15, 167–191. Solomon, H. Y., & Turvey, M. T. (1988). Haptically perceiving the distances reachable with hand-held objects. Journal of Experimental Psychology. Human Perception and Performance, 14, 404–427. Sparks, D. L. (2002). The brainstem control of saccadic eye movements. Nature Reviews. Neuroscience, 3, 952–964. Srinivasan, M. V., Laughlin, S. B., & Dubs, A. (1982). Predictive coding: A fresh view of inhibition in the retina. Proceedings of the Royal Society B: Biological Sciences, 216, 427–459. Stockwell, R. A. (1981). G. J. Romanes (Ed.), Cunningham's textbook of anatomy. Oxford: Oxford University Press, (pp. 211–264). Stout, D., & Semaw, S. (2006). Knapping skill of the earliest stone tool-makers: Insights from the study of modern human novices. In N. Toth & K. Schick (Eds.), The Oldowan: Case studies into the earliest stone age. Gosport, IN: Stone Age Institute Press, (pp. 307–320). Swinnen, S. P., Dounskaia, N., & Duysens, J. (2002). Patterns of bimanual interference reveal movement encoding within a radial egocentric reference frame. Journal of Cognitive Neuroscience, 14, 463–471. Swinnen, S. P., Jardin, K., Verschueren, S., Meulenbroek, R., Franz, L., Dounskaia, N., et al. (1998). Exploring interlimb constraints during bimanual graphic performance: Effects of muscle grouping and direction. Behavioural Brain Research, 90, 79–87.
Takeda, R., Tadano, S., Natorigawa, A., Todoh, M., & Yoshinari, S. (2010). Gait posture estimation using wearable acceleration and gyro sensors. Journal of Biomechanics, 42, 2486–2494. Tanaka, K. (1996). Inferotemporal cortex and object vision. Annual Review of Neuroscience, 19, 109–139. Tcheang, L., Bays, P. M., Ingram, J. N., & Wolpert, D. M. (2007). Simultaneous bimanual dynamics are learned without interference. Experimental Brain Research, 183, 17–25. Tenorth, M., Bandouch, J., & Beetz, M. (2009). The TUM kitchen data set of everyday manipulation activities for motion tracking and action recognition. In: Workshop on Tracking Humans for the Evaluation of their Motion in Image Sequences (ICCV). Tocheri, M. W., Orr, C. M., Jacofsky, M. C., & Marzke, M. W. (2008). The evolutionary history of the hominin hand since the last common ancestor of Pan and Homo. Journal of Anatomy, 212, 544–562. Tomasello, M., & Call, J. (1997). Primate cognition. Oxford: Oxford University Press. Tompa, T., & Sáry, G. (2010). A review on the inferior temporal cortex of the macaque. Brain Research Reviews, 62, 165–182. Tong, C., Wolpert, D. M., & Flanagan, J. R. (2002). Kinematics and dynamics are not represented independently in motor working memory: Evidence from an interference study. The Journal of Neuroscience, 22, 1108–1113. Torigoe, T. (1985). Comparison of object manipulation among 74 species of non-human primates. Primates, 26, 182–194. Toth, N., Schick, K., Savage-Rumbaugh, E. S., Sevcik, R. A., & Rumbaugh, D. M. (1993). Pan the tool-maker: Investigations into the stone tool-making and tool-using capabilities of a bonobo (Pan paniscus). Journal of Archaeological Science, 20, 81–91. Treffner, P. J., & Turvey, M. T. (1996). Symmetry, broken symmetry, and handedness in bimanual coordination dynamics. Experimental Brain Research, 107, 463–478. Tresch, M. C., Cheung, V. C. K., & d'Avella, A. (2006). Matrix factorization algorithms for the identification of muscle synergies: Evaluation on simulated and experimental data sets. Journal of Neurophysiology, 95, 2199–2212. Tuller, B., & Kelso, J. A. (1989). Environmentally-specified patterns of movement coordination in normal and splitbrain subjects. Experimental Brain Research, 75, 306–316. Turvey, M. T. (1996). Dynamic touch. The American Psychologist, 51, 1134–1152. Turvey, M. T., Burton, G., Pagano, C. C., Solomon, H. Y., & Runeson, S. (1992). Role of the inertia tensor in perceiving object orientation by dynamic touch. Journal of Experimental Psychology. Human Perception and Performance, 18, 714–727. van Hateren, J. H. (1992). Real and optimal neural images in early vision. Nature, 360, 68–70.
29 Vauclair, J. (1982). Sensorimotor intelligence in human and non-human primates. Journal of Human Evolution, 11, 257–264. Vauclair, J. (1984). Phylogenetic approach to object manipulation in human and ape infants. Human Development, 27, 321–328. Vauclair, J., & Bard, K. (1983). Development of manipulations with objects in ape and human infants. Journal of Human Evolution, 12, 631–645. Visalberghi, E. (1993). Capuchin monkeys: A window into tool use in apes and humans. In K. R. Gibson & T. Ingold (Eds.), Tools, language and cognition in human evolution (pp. 138–150). Cambridge: Cambridge University Press. von Schroeder, H. P., & Botte, M. J. (1993). The functional significance of the long extensors and juncturae tendinum in finger extension. The Journal of Hand Surgery, 18A, 641–647. Wade, N. J., & Tatler, B. (2005). The moving tablet of the eye: The origins of modern eye movement research. Oxford: Oxford University Press. Wallis, G., & Bulthoff, H. (1999). Learning to recognize objects. Trends in Cognitive Sciences, 3, 22–31. Weaver, H. E. (1943). A study of visual processes in reading differently constructed musical selections. Psychological Monographs, 55, 1–30. White, O., Dowling, N., Bracewell, R. M., & Diedrichsen, J. (2008). Hand interactions in rapid grip force adjustments are independent of object dynamics. Journal of Neurophysiology, 100, 2738–2745. Wigmore, V., Tong, C., & Flanagan, J. R. (2002). Visuomotor rotations of varying size and direction compete for a single internal model in motor working memory. Journal of
Experimental Psychology. Human Perception and Performance, 28, 447–457. Wilson, F. R. (1998). The hand—How its use shapes the brain, language, and human culture. New York: Pantheon Books. Wimmers, R. H., Beek, P. J., & Vanwieringen, P. C. W. (1992). Phase-transitions in rhythmic tracking movements—A case of unilateral coupling. Human Movement Science, 11, 217–226. Witney, A. G., Goodbody, S. J., & Wolpert, D. M. (2000). Learning and decay of prediction in object manipulation. Journal of Neurophysiology, 84, 334–343. Witney, A. G., & Wolpert, D. M. (2003). Spatial representation of predictive motor learning. Journal of Neurophysiology, 89, 1837–1843. Wolpert, D. M., & Flanagan, J. R. (2001). Motor prediction. Current Biology, 11, R729–R732. Wolpert, D. M., & Flanagan, J. R. (2010). Q&A: Robotics as a tool to understand the brain. BMC Biology, 8, 92. Wolpert, D. M., Ghahramani, Z., & Flanagan, J. R. (2001). Perspectives and problems in motor learning. Trends in Cognitive Sciences, 5, 487–494. Wolpert, D., & Kawato, M. (1998). Multiple paired forward and inverse models for motor control. Neural Networks, 11, 1317–1329. Wurtz, R. H. (2009). Recounting the impact of Hubel and Wiesel. The Journal of Physiology, 587, 2817–2823. Yarbus, A. (1967). Eye movements and vision. New York: Plenum Press. Zhang, W., Gordon, A. M., Fu, Q., & Santello, M. (2010). Manipulation after object rotation reveals independent sensorimotor memory representations of digit positions and forces. Journal of Neurophysiology, 103, 2953–2964.
A. M. Green, C. E. Chapman, J. F. Kalaska and F. Lepore (Eds.) Progress in Brain Research, Vol. 191 ISSN: 0079-6123 Copyright Ó 2011 Elsevier B.V. All rights reserved.
CHAPTER 2
Sensory change following motor learning Andrew A. G. Mattar{, Sazzad M. Nasirk, Mohammad Darainy{,{ and David J. Ostry{,},* {
k
Department of Psychology, McGill University, Montréal, Québec, Canada { Shahed University, Tehran, Iran } Haskins Laboratories, New Haven, Connecticut, USA The Roxelyn and Richard Pepper Department of Communication Sciences and Disorders, Northwestern University, Evanston, Illinois, USA
Abstract: Here we describe two studies linking perceptual change with motor learning. In the first, we document persistent changes in somatosensory perception that occur following force field learning. Subjects learned to control a robotic device that applied forces to the hand during arm movements. This led to a change in the sensed position of the limb that lasted at least 24 h. Control experiments revealed that the sensory change depended on motor learning. In the second study, we describe changes in the perception of speech sounds that occur following speech motor learning. Subjects adapted control of speech movements to compensate for loads applied to the jaw by a robot. Perception of speech sounds was measured before and after motor learning. Adapted subjects showed a consistent shift in perception. In contrast, no consistent shift was seen in control subjects and subjects that did not adapt to the load. These studies suggest that motor learning changes both sensory and motor function. Keywords: motor learning; sensory plasticity; arm movements; proprioception; speech motor control; auditory perception.
the human motor system and, likewise, to skill acquisition in the adult nervous system. Here, we summarize two studies in which we have examined the hypothesis that motor learning, which is associated with plastic changes to motor areas of the brain, leads to changes in sensory perception. We have investigated motor learning in the context of reaching movements and in speech motor control. We have examined the
Introduction To what extent is plasticity in motor and sensory systems linked? Neuroplasticity in sensory and motor systems is central to the development of *Corresponding author. Tel.: þ1-514-398-6111; Fax: þ1-514-398-4896 E-mail:
[email protected] DOI: 10.1016/B978-0-444-53752-2.00015-1
31
32
extent to which motor learning modifies somatosensory perception and the perception of speech. Our findings suggest that plasticity in motor systems does not occur in isolation, but it results in changes to sensory systems as well. Our studies examine sensorimotor learning in both an arm movement task and a speech task. It is known that there are functional connections linking brain areas involved in the sensory and motor components of these tasks. There are known ipsilateral corticocortical projections linking somatosensory cortex with motor areas of the brain (Darian-Smith et al., 1993; Jones et al., 1978). Activity in somatosensory cortex varies systematically with movement (AgeraniotiBélanger and Chapman, 1992; Chapman and Ageranioti-Bélanger, 1991; Cohen et al., 1994; Prud'homme and Kalaska, 1994; Prud'homme et al., 1994; Soso and Fetz, 1980), and the sensory signals arising from movement can result in changes to somatosensory receptive fields (Jenkins et al., 1990; Recanzone et al., 1992a,b; Xerri et al., 1999). Likewise, auditory processing recruits activity in motor areas of the brain (Chen et al., 2008; Pulvermüller et al., 2006), and auditory and somatosensory inputs converge within auditory cortex (Foxe et al., 2002; Fu et al., 2003; Kayser et al., 2005; Shore and Zhou, 2006). In addition, there are a number of pieces of evidence suggesting perceptual changes related to somatosensory input, movement, and learning. These include proprioceptive changes following visuomotor adaptation in reaching movements and in manual tracking (Cressman and Henriques, 2009, 2010; Cressman et al. 2010; Malfait et al., 2008; Simani et al., 2007; van Beers et al., 2002) and visual and proprioceptive changes following force field learning (Brown et al., 2007; Haith et al., 2008). They also include changes to auditory perception that are caused by somatosensory input (Ito et al., 2009; Jousmäki and Hari, 1998; Murray et al., 2005; Schürmann et al., 2004). These studies thus suggest that via the links between motor, somatosensory, and auditory areas of the brain, an effect of motor learning on perception may be likely.
Below, we describe a study involving human arm movement that tests the idea that sensory function is modified by motor learning. Specifically, we show that learning to correct for forces that are applied to the limb by a robot results in durable changes to the sensed position of the limb. We report a second study in which we test the hypothesis that speech motor learning, and in particular the somatosensory inputs associated with learning, affect the classification of speech sounds. In both studies, we observe perceptual changes following learning. These findings suggest that motor learning affects not only the motor system but also involves changes to sensory areas of the brain.
The effect of motor learning on somatosensory perception of the upper limb Subjects made movements to a target in a standard force field learning procedure. In this task, subjects make reaching movements to a visual target while holding the handle of a robotic device that is programmed to apply forces to the subject's hand (Fig. 1a). Studies employing this technique have been used to document learning and plasticity in motor systems (Gribble and Scott, 2002; Shadmehr and Holcomb, 1997; Shadmehr and Mussa-Ivaldi, 1994). Figure 1b shows the experimental sequence. We interleaved blocks of trials in which we estimated the sensed position of the limb (shown in gray) with blocks of force field learning trials. We tested sensory perception twice before and once after force field learning. We also tested for the persistence of changes in sensory perception after the effects of motor learning were eliminated using washout trials. We obtained estimates of the sensed position of the limb using an iterative procedure known as PEST (parameter estimation by sequential testing; Taylor and Creelman, 1967). The PEST procedure was done in the absence of vision. On each movement in the testing sequence, the limb was displaced laterally using a force channel (Fig. 1c). At
33 (b)
0 −10 −20 −30 −40 −50
Aftereffects Perceptual test
Force field training Perceptual test
Null field
Perceptual test
50 40 30 20 10
Perceptual test
Movement curvature (mm)
(a)
Right Left 1
50
100
150
1
50
100
1
150
50
Trial number (c)
Final PEST Final PEST trial trial
(d) 0
200
PEST trial Before FF training
Hand position in Y (mm)
10 160 120 80 40
20 30 40 50
1 2 3 4 5 6 7 8 9 PEST trial
0 −40
−20
0
0
20
Hand position in X (mm)
40
60 −40 −20
0
10
30
Lateral displacement (mm)
Fig. 1. Force field learning and the perception of limb position. (a) Subjects held the handle of a robotic device, when making movements to targets and during perceptual testing. The robot was capable of applying forces to the hand. Targets were presented on a horizontal screen that occluded vision of the hand, arm, and robot. (b) Subjects learn to compensate for velocitydependent mechanical loads that displace the limb to the right or the left. Perceptual tests (gray bars) of the sensed limb position are interleaved with force field training. Average movement curvature ( SE) is shown throughout training. (c) An iterative procedure known as PEST estimates the perceptual boundary between left and right. A computer-generated force channel laterally displaced the limb, and subjects are required to indicate whether the limb has been deflected to the right. Individual PEST runs starting from left and right, respectively, are shown. The sequence is indicated by the shading of the PEST trials beginning at the right. (d) A sequence of six PEST runs (starting from the top) with the horizontal axis showing the lateral position of the hand and the PEST trial number on the vertical. The shaded sequence of trials shown at the top is the same as is shown on the right side of (c). PEST runs alternately start from the right and the left and end on a similar estimate of the perceptual boundary. Note that the horizontal axis highlights lateral hand positions between 0 and 10 mm.
the end of each movement the subject gave a “yes” or “no” response indicating whether the limb had been deflected to the right. Over the course of several trials, the magnitude of the deflection was modified based on the subject's responses in order
to determine the perceptual boundary between left and right. Figure 1b shows a sequence of PEST trials for a representative subject, prior to force field learning. The left panel shows a PEST sequence that began with a leftward deflection;
34
the right panel shows a sequence for the same subject beginning from the right. Figure 1d shows a sequence of six PEST runs. Each run converges on a stable estimate of the perceptual boundary between left and right. In the motor learning phase of the experiment, subjects made movements in a clockwise or counterclockwise force field applied by a robot arm (Fig. 1a), whose actions were to push the hand to the right or to the left. Performance over the course of training was quantified by computing the maximum perpendicular distance (PD) from a straight line joining movement start and end. Figure 1b shows movement curvature (PD values), averaged over subjects, for each phase of the experiment. Under null conditions, subjects move straight to the target. Upon the introduction of the force field, movements are deflected laterally but over the course of training they straighten to near null field levels. The reduction in curvature from the initial 10 movements to the final 10 movements was reliable for both force field directions. Curvature on initial aftereffect movements is opposite to the curvature on initial force field movements reflecting the adjustment to motor commands needed to produce straight movements in the presence of load. Curvature at the end of the washout trials differs from initial null field trials; movements remain curved in a direction opposite to that of the applied force. On a per-subject basis, we quantified perceptual performance by fitting a logistic function to the set of lateral limb positions and the associated binary responses that were obtained over successive PEST runs. For example, the sequence of PEST trials shown in Fig. 1d would lead to a single psychometric function relating limb position to the perceptual response. For visualization purposes, Fig. 2a shows binned response probabilities, averaged across subjects, and psychometric functions fit to the means for the rightward and leftward force fields. Separate curves are shown for estimates obtained before and after learning. The psychometric curve, and hence the perceptual boundary between left and right
shifts in a direction opposite to the applied load. If the force field acts to the right (Fig. 2a, right panel), the probability of responding that the hand was pushed to the right increases following training. This means that following force field learning, the subject feels as if the hand is located farther to the right. Figure 2b shows the position of the perceptual boundary in each of the four test sessions. The perceptual boundary was computed as the 50% point on the psychometric curve. For each subject separately, we computed the shift in the perceptual boundary as a difference between the final null condition estimate and the estimate following force field training. We computed the persistence of the shift as the difference between the final null condition estimate and the estimate following aftereffect trials. The shifts are shown in Fig. 2c. It can be seen that immediately after force field training there was a shift in the sensed position of the limb that was reliably different than zero. The shift decreased following washout but remained different than zero. The magnitude of the shift was no different for both force field directions. Thus, the sensed position of the limb changes following force field learning, and the shift persists even after the kinematic effects of learning have been washed out. In a control study, we examined the persistence of the perceptual change. Subjects were tested in a procedure that was identical to the main experiment, but it included an additional perceptual test 24 h following learning. The results are shown in Fig. 2c. It can be seen that the force field led to a reliable shift in the perceptual boundary that was no different across the three estimates. Thus, periods of force field learning lasting 10 min result in shifts in the perceptual boundary that persist for at least 24 h. We conducted a second control experiment to determine the extent to which the observed perceptual changes are tied to motor learning. We used methods that were identical to those in the main experiment, except that the force field learning phase was replaced with a task that did
35 (a)
Leftward field
Rightward field
100 Probability of “right” response (%)
Before learning After learning
Before learning After learning
80 60 40 20 0
−20 −10 Left
0
10
−20 −10 Left
20 Right
0
10
20 Right
Position (mm) (c) 8
3
7
Change in perceptual boundary (mm)
Perceptual boundary (mm)
(b)
6 5 4 3 2 1 e1
B
e2
FF AE er er Aft Aft Perceptual test
lin
lin
e as
B
e as
Left Right
2 1 0 −1 −2 er
Aft
FF
h AE 24 er er t f Aft A Perceptual test Left Right Passive control 24 h left
Fig. 2. The perceptual boundary shifts in a direction opposite to the applied force following motor learning. (a) Binned response probabilities averaged over subjects (SE) before (light gray) and after (black or dark gray) learning. Fitted psychometric functions reflect the perceptual classification for each force field direction. (b) Mean perceptual boundary between left and right (SE) for baseline estimates (baseline 1 and baseline 2), estimates following force field learning (after FF), and estimates following aftereffect trials (after AE). The sensed position of the limb changes following learning, and the change persists following aftereffect trials. (c) The direction of the perceptual shift depends on the force field (left vs. right). The perceptual shift persists for at least 24 h (24 h left). A perceptual shift is not observed when the robot passively moves the hand through the same sequence of positions and velocities as in the left condition such that subjects do not experience motor learning (passive control).
not involve motor learning. In the null field and aftereffect phases of the experiment, subjects moved actively. The force field learning phase was replaced with a passive task in which subjects
held the robot handle as it reproduced the movements of subjects in the leftward force field condition of the main experiment. Under positionservo control, the robot produced this series of
36
movements and the subject's arm was moved along the mean trajectory for each movement in the training sequence. Thus, subjects experienced a series of movements with the same kinematics as those in main experiment, but importantly they did not experience motor learning. The upper panel of Fig. 3 shows the mean movement curvature (PD) for subjects tested in the passive control experiment and for subjects tested in the original experiment. The lower panel shows the average difference between PD in the passive control condition and PD in the original leftward force field. The lower panel of Fig. 3 shows that in the null phase, movement kinematics were well matched when subjects in both conditions made active movements. In the force field phase of the experiment, the near-zero values indicate that subjects in the passive control experiment experienced kinematics that closely
matched the mean trajectory in the original experiment. The nonzero values at the start of the aftereffect phase indicate that in the main experiment, training in the force field resulted in aftereffects and hence motor learning that was greater than following training in the passive control experiment. Figure 2c shows measures of perceptual change for subjects trained in the original experiment, as well as for subjects trained in the passive control. Perceptual shifts depended on whether or not subjects experienced motor learning. As described above, subjects in the original experiment who learned the leftward force field showed perceptual shifts that were reliably different than zero both immediately after learning and after washout trials. In contrast, subjects tested in the passive control experiment showed shifts that did not differ from zero at either time point.
Null field
Force field training
Aftereffects
20 10 0
20
Left Passive control Left minus passive control
Perceptual test
Difference in movement curvature (mm)
−30
Perceptual test
−20
Perceptual test
−10
Perceptual test
Movement Curvature (mm)
30
10 0 −10 1
50
100
150
1
50
100
150
1
50
Trial number Fig. 3. The perceptual shift depends on motor learning. In a control experiment, subjects experience the same trajectories as individuals that display motor learning. Subjects move actively in the null and aftereffect phases of the study. In the force field training phase, the robot moves the arm to replicate the average movement path of subjects that learned the leftward force field. The top panel shows mean movement curvature ( SE) for subjects in the original leftward condition (black) and the passive control condition (light gray). The bottom panel gives the difference between active and passive movements (dark gray). Movement aftereffects are not observed in the passive condition (light gray) indicating there is no motor learning.
37
In the perceptual tests, the subject had to identify whether an auditory stimulus chosen at random from a synthesized eight step spectral continuum sounded more like the word head or had (Fig. 4). A psychometric function was fitted to the data and gave the probability of identifying the word as had. We focused on whether motor learning led to changes to perceptual performance. Sensorimotor learning was evaluated using a composite measure of movement curvature. Curvature was assessed on a per-subject basis, in null condition trials, at the start and at the end of learning. Statistically reliable adaptation was observed in 17 of the 23 subjects. This is typical of studies of speech motor learning in which about a third of all subjects fail to adapt (Nasir
The effect of speech motor learning on the perception of speech sounds In order to evaluate the idea that speech motor learning affects auditory perception, we trained healthy adults in a force field learning task (Lackner and Dizio, 1994; Shadmehr and MussaIvaldi, 1994) in which a robotic device applied a mechanical load to the jaw as subjects repeated aloud test utterances that were chosen randomly from a set of four possibilities (bad, had, mad, sad; Fig. 4). The mechanical load was velocitydependent and acted to displace the jaw in a protrusion direction, altering somatosensory but not auditory feedback. Perception of speech sounds was assessed before and after force field training. (a)
Protocol
(b)
Force field training/word repetition
Perceptual test
Perceptual test
Time
Frequency (Hz)
(c) 3000 2000 1000 0
Head
Had Auditory continuum
Fig. 4. Experimental set-up, protocol, and auditory test stimuli for the speech experiment. (a) A velocity-dependent load was delivered to the jaw by a robotic device. (b) Subjects completed an auditory identification task before and after motor learning. Control subjects repeated the same set of utterances but were not attached to the robot. (c) During perceptual testing, subjects indicated whether a given auditory test stimulus sounded more like head or had.
38
and Ostry, 2006, 2008; Purcell and Munhall, 2006; Tremblay et al., 2003). Figure 5a shows a representative sagittal plane view of jaw trajectories during speech for a subject that adapted to the load. Movements are straight in the absence of load; the jaw is displaced in a protrusion direction when the load is first applied; curvature decreases with training. Figure 5b shows movement curvature measures for the same subject, for individual trials, over the course of the entire experiment. (a)
As shown in Fig. 5a, movement curvature was low in the null condition, increased with the introduction of load and then progressively decreased with training. The auditory psychometric function for this subject shifted to the right following training (Fig. 5c). This indicates that words sounded more like head after learning. Figure 6a shows perceptual psychometric functions for adapted subjects before and after force field training. A rightward shift following (b) 2.5
0
2
−10 Curvature
Elevation (mm)
Null condition
−20
1
0
−10
0
10
0
Protrusion (mm) (c)
1.5
0.5
−30
Identification probability
Training Null condition
End training Initial training
1
50
100
150 200 Trials
250
300
350
Posttraining Pretraining
0.8 0.6 0.4 0.2 0 1 2 Head
3
4
5
6
7
8 Had
Auditory continuum Fig. 5. Speech motor learning and changes in speech perception. (a) Sagittal view of jaw movement paths for a representative subject who adapted to the load. Movements were straight in the absence of load (light gray). The jaw was deflected in the protrusion direction when the load was introduced (black). Curvature decreased with training (dark gray). (b) Scatter plot showing movement curvature over the course of training for the same subject as in (a). The vertical axis shows movement curvature; the horizontal axis gives trial number. Curvature is low on null trials (light gray) increases when the load is introduced and decreases over the course of training (black). (c) The psychometric function depicting identification probability for had before (light gray) and after (black) training. A perceptual shift toward head was observed following learning.
39 (a)
(b)
1
1
Pretraining
0.8
Identification probability
Identification probability
Posttraining
0.6 0.4 0.2 0
1 Head
2
3
4
5
6
7
0.8 0.6 0.4 0.2 0
8 Had
1 2 Head
Auditory continuum
(c)
(d)
Adapted subjects Nonadapted and control subjects
Perceptual shift (%)
Probability
Perceptual shift (%)
6
7
8 Had
20
0.3 0.2 0.1 0
5
Auditory continuum
4
0
4
(e)
0.5 0.4
8
3
−20
−10 0 10 Perceptual shift (%)
20
10 0 −10 0
10
20 30 40 50 Adaptation (%)
60
Fig. 6. Perception of speech sounds changes following speech motor learning. (a) The average psychometric functions for adapted subjects reveal a perceptual shift to the right following training (light gray: pretraining, black: posttraining). (b) There is no perceptual shift for nonadapted and control subjects. (c) The perceptual shift for adapted subjects (black) was reliably greater than the shift observed in nonadapted and control subjects (light gray), which was not different than zero. (d) Histograms showing the perceptual change for adapted (black) and nonadapted/control subjects (light gray). (e) The perceptual shift was correlated with adaptation. Subjects that showed greater adaptation also had greater perceptual shifts.
training is evident. A measure of probability, that was used to assess perceptual change, was obtained by summing each subject's response probabilities for individual stimulus items and dividing the total by a baseline measure that was obtained before learning. The change in identification probability from before to after training was used to gauge the perceptual shift. In 15 of the 17 subjects that adapted to the force field, we found a rightward shift in the psychometric function following training. This rightward
perceptual shift means that after force field learning the auditory stimuli are more likely to be classified as head. In effect, the perceptual space assigned to head increased with motor learning. The remaining six subjects who failed to adapt did not show any consistent pattern in their perceptual shifts. We evaluated the possibility that the perceptual shift might be due to factors other than motor learning by testing a group of control subjects who completed the entire experiment without
40
force field training. This control study included the entire sequence of several hundred speech movements. For control subjects, the perceptual shift, computed in the same manner as for the experimental subjects, was not different than zero (Fig. 6c). Moreover, we found that perceptual shifts obtained for the nonadapted subjects in the main experiment did not differ from the shifts obtained from control subjects. Figure 6b shows the psychometric functions averaged over nonadapted and control subjects combined, before and after word repetition (or force field training for the nonadapted subjects). No difference can be seen in the psychometric functions of the subjects that did not experience motor learning. Statistical tests were conducted on the perceptual probability scores before and after training. The analysis compared the scores of adapted subjects with those of control subjects and nonadapted subjects combined. The test thus compared the perceptual performance of subjects that successfully learned the motor task with those that did not. For adapted subjects, we found that identification scores were significantly different after training than before. For subjects that did not show motor learning, the difference in the two perceptual tests was nonsignificant. Thus, speech motor learning in a force field environment modifies perception of speech sounds. Word repetition alone cannot explain the observed perceptual effects. In order to characterize further the pattern of perceptual shifts, we obtained histograms giving the distribution of shifts for both the adapted and the combined nonadapted and control groups (Fig. 6d). The histogram for the adapted group is to the right of the histogram for the nonadapted subjects. We also examined the possibility that subjects that showed greater learning would also show a greater perceptual shift. We calculated an index of learning for each adapted subject by computing the reduction in curvature over the course of training divided by the curvature due to the introduction of load. A value of 1.0 indicates complete adaptation. Computed in this
fashion, adaptation ranged from 0.05 to 0.55 and when averaged across subjects and test words, it was 0.29 0.03 (mean SE). Figure 6e shows the relationship between the amount of adaptation and the associated perceptual shift. We found that adapted subjects showed a small, but significant, correlation of 0.53 between the extent of adaptation and the measured perceptual shift. We assessed the possibility that there are changes in auditory input over the course of force field training that might contribute to motor learning and also to the observed perceptual shift. Acoustical effects related to the application of load and learning were evaluated by computing the first and second formant frequencies of the vowel /æ/ immediately following the initial consonant in each of the test utterances. A statistical analysis found no reliable differences in either formant frequency over the course of the experiment. This suggests that there were no changes in auditory input over the course of adaptation.
Discussion In the limb movement study, we showed that motor learning results in changes in the sensed position of the limb. The passive control experiment reveals that changes in somatosensory perception depend on motor learning. The perceptual change is robust, in that it persists for periods lasting at least 24 h. In the absence of movement, sensory experience results in a selective expansion of the specific regions of somatosensory cortex that are associated with the sensory exposure, and it also results in changes in the size of sensory receptive fields that reflect the characteristics of the adaptation (Recanzone et al., 1992a,b). Changes to receptive field size in somatosensory cortex are observed when sensory training is combined with motor tasks that require precise contact with a rotating disk (Jenkins et al., 1990) or finger and forearm movements to remove food from a narrow well (Xerri et al., 1999). In these latter cases,
41
it is not clear whether it is the sensory experience, the motor experience, or both factors in combination that leads to changes in the sensory system. This issue is clarified by the findings summarized here. Changes in sensory perception depend on active movement and learning. Control subjects who experienced the same movements but did not experience motor learning showed no perceptual change. This points to a central role of motor learning in somatosensory plasticity. The idea that sensory perception depends on both sensory and motor systems has been proposed by other researchers (Feldman, 2009; Haith et al., 2008). One possibility is that the central contribution to position sense involves motor commands that are adjusted by adaptation (see Feldman, 2009, for a recent review of central and afferent contributions to position sense). In effect, sensory signals from receptors are measured in a motoric reference frame that can be modified by learning. Another possibility is that the learning recalibrates both sensory and motor processes. Haith et al. propose that changes in performance that are observed in the context of learning depend on changes to both motor and sensory function that are driven by error (Haith et al., 2008). In a second study, we found that the perceptual classification of speech sounds was modified by speech motor learning. There was a systematic change such that following learning, speech sounds on a continuum ranging from head to had were more frequently classified as head. Moreover, the perceptual shift varied with learning; the perceptual change was greater in subjects that showed greater adaptation during learning. The perceptual shift was not observed in subjects who failed to adapt to the forces applied by the robot, nor was it observed in control subjects who repeated the same words but did not undergo force field learning. This suggests a link between motor learning and the perceptual change. The findings thus indicate that speech learning modifies not only the motor system but also the perception of speech sounds.
The sensory basis of the auditory perceptual effect was somatosensory in nature. Force field training modified the motion path of the jaw and hence somatosensory feedback, but it did not affect the acoustical patterns of speech at any point during training. Hence, there was no change in auditory information that might result in perceptual modification. Thus the sensory basis of both the motor learning and the perceptual recalibration is presumably somatosensory but not auditory. This conclusion is supported by the observation that adaptation to mechanical load occurs when subjects perform the speech production task silently, indicating that it is not dependent upon explicit acoustical feedback (Tremblay et al., 2003). It is also supported by the finding that profoundly deaf adults who are tested with their assistive hearing devices turned off can still adapt to mechanical loads applied during speech (Nasir and Ostry, 2008). The perceptual shift we observed is in the same direction as in previous studies of perceptual adaptation (Cooper and Lauritsen, 1974; Cooper et al., 1976). Cooper and colleagues observed that after listening to repetitions of a particular consonant–vowel stimulus, the probability that subjects would report hearing this same stimulus in subsequent perceptual testing was reduced. The effect reported here is similar to that observed by Cooper, but there are important differences suggesting the effects are different in origin. We found no perceptual shift in nonadapted subjects who repeatedly said or heard a given test stimulus. Moreover, control subjects also repeated and listened to the same set of utterances but did not show a reliable perceptual change. Both of these facts are consistent with the idea that motor learning, but not repeated experience with the speech stimuli, is the source of the perceptual change. Influences of somatosensory input on auditory perception have been documented previously. There is somatosensory input to the cochlear nucleus, and there are known bidirectional interactions between auditory and somatosensory
42
cortex (Foxe et al., 2002; Fu et al., 2003; Jousmäki and Hari, 1998; Kayser et al., 2005; Murray et al., 2005; Schürmann et al., 2006; Shore and Zhou, 2006). In addition, there are reports that somatosensory inputs affect auditory perceptual function in cases involving speech (Gillmeister and Eimer, 2007; Ito et al., 2009; Schürmann et al., 2004). The present example of somatosensory–auditory interaction is intriguing because subjects receive somatosensory input when producing speech but not when perceiving speech sounds produced by others. Indeed, the involvement of somatosensory information in the perceptual processing of speech would be consistent with the idea that speech perception is mediated by the mechanisms of speech production (Hickok and Poeppel, 2000; Libermann and Mattingly, 1985). This view is supported by other studies demonstrating that electromyographic responses evoked by transcranial magnetic stimulation (TMS) to primary motor cortex are facilitated by watching speech movements and listening to speech sounds (Fadiga et al., 2002; Watkins et al., 2003), and that speech perception is affected by repetitive TMS to premotor cortex (Meister et al., 2007). However, the perceptual effects described here may well occur differently, resulting from the direct effects of somatosensory input on auditory cortex (Hackett et al., 2007). In summary, in both of the studies described above, we have found that motor learning leads to changes in perceptual function. In both cases, the perceptual change was grounded in motor learning; sensory experience on its own was not sufficient for changes in perception. These findings suggest that plasticity in sensory and motor systems is linked, and that changes in each system may not occur in isolation. References Ageranioti-Bélanger, S. A., & Chapman, C. E. (1992). Discharge properties of neurones in the hand area of primary somatosensory cortex in monkeys in relation to the
performance of an active tactile discrimination task. II. Area 2 as compared to areas 3b and 1. Experimental Brain Research, 91, 207–228. Brown, L. E., Wilson, E. T., Goodale, M. A., & Gribble, P. L. (2007). Motor force field learning influences visual processing of target motion. The Journal of Neuroscience, 27, 9975–9983. Chapman, C. E., & Ageranioti-Bélanger, S. A. (1991). Discharge properties of neurones in the hand area of primary somatosensory cortex in monkeys in relation to the performance of an active tactile discrimination task. I. Areas 3b and 1. Experimental Brain Research, 87, 319–339. Chen, J. L., Penhune, V. B., & Zatorre, R. J. (2008). Listening to musical rhythms recruits motor regions of the brain. Cerebral Cortex, 18, 2844–2854. Cohen, D. A., Prud'homme, M. J., & Kalaska, J. F. (1994). Tactile activity in primate primary somatosensory cortex during active arm movements: Correlation with receptive field properties. Journal of Neurophysiology, 71, 161–172. Cooper, W. E., & Lauritsen, M. R. (1974). Feature processing in the perception and production of speech. Nature, 252, 121–123. Cooper, W. E., Billings, D., & Cole, R. A. (1976). Articulatory effects on speech perception: A second report. Journal of Phonetics, 4, 219–232. Cressman, E. K., & Henriques, D. Y. (2009). Sensory recalibration of hand position following visuomotor adaptation. Journal of Neurophysiology, 102, 3505–3518. Cressman, E. K., & Henriques, D. Y. (2010). Reach adaptation and proprioceptive recalibration following exposure to misaligned sensory input. Journal of Neurophysiology, 103, 1888–1895. Cressman, E. K., Salomonczyk, D., & Henriques, D. Y. (2010). Visuomotor adaptation and proprioceptive recalibration in older adults. Experimental Brain Research, 205, 533–544. Darian-Smith, C., Darian-Smith, I., Burman, K., & Ratcliffe, N. (1993). Ipsilateral cortical projections to areas 3a, 3b, and 4 in the macaque monkey. The Journal of Comparative Neurology, 335, 200–213. Fadiga, L., Craighero, L., Buccino, G., & Rizzolati, G. (2002). Speech listening specifically modulates the excitability of tongue muscles: A TMS study. The European Journal of Neuroscience, 15, 399–402. Feldman, A. G. (2009). New insights into action–perception coupling. Experimental Brain Research, 194, 39–58. Foxe, J. J., Wylie, G. R., Martinez, A., Schroeder, C. E., Javitt, D. C., Guilfoyle, D., et al. (2002). Auditory-somatosensory multisensory processing in auditory association cortex: An fMRI study. Journal of Neurophysiology, 88, 540–543. Fu, K. M., Johnston, T. A., Shah, A. S., Arnold, L., Smiley, J., Hackett, T. A., et al. (2003). Auditory cortical neurons respond to somatosensory stimulation. The Journal of Neuroscience, 23, 7510–7515.
43 Gillmeister, H., & Eimer, M. (2007). Tactile enhancement of auditory detection and perceived loudness. Brain Research, 1160, 58–68. Gribble, P. L., & Scott, S. H. (2002). Overlap of internal models in motor cortex for mechanical loads during reaching. Nature, 417, 938–941. Hackett, T. A., Smiley, J. F., Ulbert, I., Karmos, G., Lakatos, P., de la Mothe, L. A., et al. (2007). Sources of somatosensory input to the caudal belt areas of auditory cortex. Perception, 36, 1419–1430. Haith, A., Jackson, C., Miall, R., & Vijayakumar, S. (2008). Unifying the sensory and motor components of sensorimotor adaptation. Advances in Neural Information Processing Systems, 21, 593–600. Hickok, G., & Poeppel, D. (2000). Toward functional neuroanatomy of speech perception. Trends in Cognitive Science, 4, 131–138. Ito, T., Tiede, M., & Ostry, D. J. (2009). Somatosensory function in speech perception. Proceedings of the National Academy of Sciences of the United States of America, 106, 1245–1248. Jenkins, W. M., Merzenich, M. M., Ochs, M. T., Allard, T., & Guíc-Robles, E. (1990). Functional reorganization of primary somatosensory cortex in adult owl monkeys after behaviorally controlled tactile stimulation. Journal of Neurophysiology, 63, 82–104. Jones, E. G., Coulter, J. D., & Hendry, S. H. (1978). Intracortical connectivity of architectonic fields in the somatic sensory, motor and parietal cortex of monkeys. The Journal of Comparative Neurology, 181, 291–347. Jousmäki, V., & Hari, R. (1998). Parchment-skin illusion: Sound-biased touch. Current Biology, 8, RC190. Kayser, C., Petkov, C. I., Augath, M., & Logothetis, N. K. (2005). Integration of touch and sound in auditory cortex. Neuron, 48, 373–384. Lackner, J. R., & Dizio, P. (1994). Rapid adaptation to coriolis force perturbations of arm trajectory. Journal of Neurophysiology, 72, 299–313. Libermann, A. M., & Mattingly, I. G. (1985). The motor theory of speech perception revised. Cognition, 21, 1–36. Malfait, N., Henriques, D. Y., & Gribble, P. L. (2008). Shape distortion produced by isolated mismatch between vision and proprioception. Journal of Neurophysiology, 99, 231–243. Meister, I. G., Wilson, S. M., Deblieck, C., Wu, A. D., & Iacoboni, M. (2007). The essential role of premotor cortex in speech perception. Current Biology, 17, 1692–1696. Murray, M. M., Molholm, S., Michel, C. M., Heslenfeld, D. J., Ritter, W., Javitt, D. C., et al. (2005). Grabbing your ear: Rapid auditory-somatosensory multisensory interactions in low-level sensory cortices are not constrained by stimulus alignment. Cerebral Cortex, 15, 963–974.
Nasir, S. M., & Ostry, D. J. (2006). Somatosensory precision in speech production. Current Biology, 16, 1918–1923. Nasir, S. M., & Ostry, D. J. (2008). Speech motor learning in profoundly deaf adults. Nature Neuroscience, 11, 1217–1222. Prud'homme, M. J., & Kalaska, J. F. (1994). Proprioceptive activity in primate primary somatosensory cortex during active arm reaching movements. Journal of Neurophysiology, 72, 2280–2301. Prud'homme, M. J., Cohen, D. A., & Kalaska, J. F. (1994). Tactile activity in primate primary somatosensory cortex during active arm movements: Cytoarchitectonic distribution. Journal of Neurophysiology, 71, 173–181. Pulvermüller, F., Huss, M., Kherif, F., Moscoso del Prado Martin, F., Hauk, O., & Shtyrov, Y. (2006). Motor cortex maps articulatory features of speech sounds. Proceedings of the National Academy of Sciences of the United States of America, 103, 7865–7870. Purcell, D. W., & Munhall, K. G. (2006). Adaptive control of vowel formant frequency: Evidence from real-time formant manipulation. The Journal of the Acoustical Society of America, 119, 2288–2297. Recanzone, G. H., Merzenich, M. M., Jenkins, W. M., Grajski, K. A., & Dinse, H. R. (1992a). Topographic reorganization of the hand representation in cortical area 3b owl monkeys trained in a frequency-discrimination task. Journal of Neurophysiology, 67, 1031–1056. Recanzone, G. H., Merzenich, M. M., & Jenkins, W. M. (1992b). Frequency discrimination training engaging a restricted skin surface results in an emergence of a cutaneous response zone in cortical area 3a. Journal of Neurophysiology, 67, 1057–1070. Schürmann, M., Caetano, G., Jousmäki, V., & Hari, R. (2004). Hands help hearing: Facilitatory audiotactile interaction at low sound-intensity levels. The Journal of the Acoustical Society of America, 115, 830–832. Shadmehr, R., & Holcomb, H. H. (1997). Neural correlates of motor memory consolidation. Science, 277, 821–825. Shadmehr, R., & Mussa-Ivaldi, F. A. (1994). Adaptive representation of dynamics during learning of a motor task. The Journal of Neuroscience, 14, 3208–3224. Shore, S. E., & Zhou, J. (2006). Somatosensory influence on the cochlear nucleus and beyond. Hearing Research, 216–217, 90–99. Simani, M. C., McGuire, L. M., & Sabes, P. N. (2007). Visualshift adaptation is composed of separable sensory and taskdependent effects. Journal of Neurophysiology, 98, 2827–2841. Soso, M. J., & Fetz, E. E. (1980). Responses of identified cells in postcentral cortex of awake monkeys during comparable active and passive joint movements. Journal of Neurophysiology, 43, 1090–1110.
44 Taylor, M. M., & Creelman, C. D. (1967). PEST: Efficient estimates on probability functions. The Journal of the Acoustical Society of America, 41, 782–787. Tremblay, S., Shiller, D. M., & Ostry, D. J. (2003). Somatosensory basis of speech production. Nature, 423, 866–869. van Beers, R. J., Wolpert, D. M., & Haggard, P. (2002). When feeling is more important than seeing in sensorimotor adaptation. Current Biology, 12, 834–837.
Watkins, K. E., Strafella, A. P., & Paus, T. (2003). Seeing and hearing speech excites the motor system involved in speech production. Neuropsychologia, 41, 989–994. Xerri, C., Merzenich, M. M., Jenkins, W., & Santucci, S. (1999). Representational plasticity in cortical area 3b paralleling tactual-motor skill acquisition in adult monkeys. Cerebral Cortex, 9, 264–276.
A. M. Green, C. E. Chapman, J. F. Kalaska and F. Lepore (Eds.) Progress in Brain Research, Vol. 191 ISSN: 0079-6123 Copyright Ó 2011 Elsevier B.V. All rights reserved.
CHAPTER 3
Sensory motor remapping of space in human–machine interfaces Ferdinando A. Mussa-Ivaldi{,{,},},*, Maura Casadio{,}, Zachary C. Danziger},}, Kristine M. Mosierk and Robert A. Scheidt# {
{
Department of Physiology, Northwestern University, Chicago, Illinois, USA Department of Physical Medicine and Rehabilitation, Northwestern University, Chicago, Illinois, USA } Department of Biomedical Engineering, Northwestern University, Chicago, Illinois, USA } Sensory Motor Performance Program, Rehabilitation Institute of Chicago, Chicago, Illinois, USA k Department of Radiology, Section of Neuroradiology and Imaging Sciences, Indiana University, Indianapolis, Indiana, USA # Department of Biomedical Engineering, Marquette University, Milwaukee, Wisconsin, USA
Abstract: Studies of adaptation to patterns of deterministic forces have revealed the ability of the motor control system to form and use predictive representations of the environment. These studies have also pointed out that adaptation to novel dynamics is aimed at preserving the trajectories of a controlled endpoint, either the hand of a subject or a transported object. We review some of these experiments and present more recent studies aimed at understanding how the motor system forms representations of the physical space in which actions take place. An extensive line of investigations in visual information processing has dealt with the issue of how the Euclidean properties of space are recovered from visual signals that do not appear to possess these properties. The same question is addressed here in the context of motor behavior and motor learning by observing how people remap hand gestures and body motions that control the state of an external device. We present some theoretical considerations and experimental evidence about the ability of the nervous system to create novel patterns of coordination that are consistent with the representation of extrapersonal space. We also discuss the perspective of endowing human–machine interfaces with learning algorithms that, combined with human learning, may facilitate the control of powered wheelchairs and other assistive devices. Keywords: motor learning; space; dimensionality reduction; human-machine interface; braincomputer interface.
*Corresponding author. Tel.: þ1-312-238-1230; Fax: 1-312-238-2208 E-mail:
[email protected] DOI: 10.1016/B978-0-444-53752-2.00014-X
45
46
Introduction
Motor learning
Human–machine interfaces (HMIs) come in several different forms. Sensory interfaces transform sounds into cochlear stimuli (Loeb, 1990), images into phosphenene-inducing stimuli to the visual cortex (Zrenner, 2002), or into electrical stimuli to the tongue (Bach-y-Rita, 1999). Various attempts, old and recent, have aimed at the artificial generation of proprioceptive sensation by stimulating the somatosensory cortex (Houweling and Brecht, 2007; Libet et al., 1964; Romo et al., 2000). Motor interfaces may transform electromyographic (EMG) signals into commands for a prosthetic limb (Kuiken et al., 2009), electroencephalogram (EEG) signals into characters on a computer screen, multiunit recordings from cortical areas into a moving cursor (Wolpaw and McFarland, 2004), or upper body movements into commands for a wheelchair (Casadio et al., 2010). Sensory and motor interfaces both implement novel transformations between the external physical world and internal neural representations. In a sensory interface, neural representations result in perceptions. In a motor interface, the neural representations reflect movement goals, plans, and commands. In a motor HMI, the problem of forming a functional map between neural signals and external environment is similar to remapping problems studied in earlier works, focused on the adaptation to force fields (Lackner and Dizio, 1994; Shadmehr and Mussa-Ivaldi, 1994) and dynamical loads. There, the environment imposed a transformation upon the relationship between the state of motion of the arm and forces experienced at the hand. The neural representation that formed through learning was an image in the brain of this new external relation in the environment. This image allows the brain to recover a desired movement of the hand by counteracting the disturbing force. Here, we take a step toward a more fundamental understanding of how space, “ordinary” space, is remapped through motor learning.
Recently, a simple and powerful idea has changed our view of motor learning. Motor learning is not only a process in which one improves performance in a particular act. Rather, it is a process through which the brain acquires knowledge about the environment. However, this is not the ordinary kind of knowledge (explicit knowledge) such as when we learn an equation or a historical fact. It is implicit knowledge that may not reach our consciousness, and yet it informs and influences our behaviors, especially those expressed in the presence of a novel situation. The current focus of most motor learning studies is on “generalization”; that is, on how experience determines behavior beyond what one has been exposed to. The mathematical framework for the concept of generalization comes from statistical theory (Poggio and Smale, 2003), where data points and some a priori knowledge determine the value of a function at new locations. If the new location is within the domain of the data, we have the problem of interpolation, whose solutions are generally more reliable than those of extrapolation problems, that is, when the predictions are made outside the domain of the data. In the early 1980s, Morasso (1981) and Soechting and Lacquaniti (1981) independently made the deceivingly simple observation that when we reach to a target, our hands tend to move along quasi-rectilinear pathways, following bell-shaped speed profiles. This simplicity or “regularity” of movement is evident only when one considers motion of the hand: In contrast, the shoulder and elbow joints engage in coordinated patterns of rotations that may or may not include reversals in the sign of angular velocities depending on the direction of movement. These observations gave rise to an intense debate between two views. One view suggested that the brain deliberately plans the shape of hand trajectories and coordinates muscle activities and joint motions accordingly (Flash and Hogan, 1985; Morasso, 1981). The opposing view suggested that the shape of the
47
observed kinematics is a side effect of dynamic optimization (Uno et al., 1989), such as the minimization of the rate of change of torque. By considering how the brain learns to perform reaching movements in the presence of perturbing forces (Lackner and Dizio, 1994; Shadmehr and Mussa-Ivaldi, 1994), studies of motor adaptation to force fields provided a means to address, if not completely resolve, this debate. Such studies have two key features in common. First, perturbing forces were not applied randomly but instead followed some strict deterministic rule. This rule established a force field wherein the amount and direction of the external force depended upon the state of motion of the hand (i.e., its position and velocity). The second important element is that subjects were typically instructed to move their hand to some target locations but were not instructed on what path the hand should have followed. If the trajectory followed by the hand to reach a target were the side effect of a process that seeks to optimize a dynamic quantity such as the muscle force or the change in joint torque rate, then moving against a force field would lead to systematically different trajectories than if hand path kinematics were deliberately planned. Contrary to the dynamic optimization prediction, many force-field adaptation experiments have shown that after an initial disturbance to the trajectory, the hand returns to its original straight motion (Fig. 1). Moreover, if the field is suddenly removed, an aftereffect is transiently observed demonstrating that at least a portion of the response is a preplanned (feedforward) compensatory response. Importantly, Dingwell et al. (2002, 2004) observed similar adaptations when subjects controlled the movement of a virtual mass connected to the hand via a simulated spring. In this case, adaptation led to rectilinear motions of the virtual mass and more complex movements of the hand. These findings demonstrate that the trajectory of the controlled “endpoint”—whether the hand or a hand-held object—is not a side effect of some dynamic optimization. Instead, endpoint
trajectories reflect explicit kinematic goals. As we discuss next, these goals reflect the geometrical properties of the space in which we move. What is “ordinary space”? We form an intuitive understanding of the environment in which we move through our sensory and motor experiences. But what does it mean to have knowledge of something as fundamental as space itself? Scientists and engineers have developed general mathematical notions of space. They refer to “signal space” or “configuration space.” These are all generalizations of the more ordinary concept of space. If we have three signals, for example, the surface EMG activities measured over three muscles, we can form a three-dimensional (3D) Cartesian space with three axes, each representing the magnitude of EMG activity measured over one muscle. Together, the measured EMG signals map onto a single point moving in time along a trajectory through this 3D space. While this mapping provides us with an intuitive data visualization technique, signal spaces are not typically equivalent to the physical space around us, the so-called ordinary space. In particular, ordinary space has a special property not shared by all signal spaces. In the ordinary space, the rules of Euclidean geometry and, among these Pythagoras’ theorem, support a rigorous and meaningful definition of both the minimum distance between two points (the definition of vector length) and the angle between two such vectors. Although we can draw a line joining two points in the EMG space described above, the distance between EMG points will carry little meaning. Moreover, what it means to “rotate” EMG signals by a given angle in this space is even less clear.1
1
Sometimes we carry out operations on signal spaces, like principal component analysis (PCA), which imply a notion of distance and angle. But in such cases, angles and distances are mere artifacts carrying no clear geometrical meaning.
48
F=
(a)
−10.1 −11.2 . r −11.2 11.1
(b)
Aftereffects
(c)
(d)
5 cm 5 cm
Fig. 1. Adaptation of arm movements to an external force field. Top-left: Experimental apparatus. The subject holds the handle of a two-joint robot manipulandum. Targets are presented on a computer monitor, together with a cursor representing the position of the hand. Top-middle: unperturbed trajectories, observed at the beginning of the experiment, with the motors turned off. Topright: velocity-dependent force field. The perturbing force is a linear function of the instantaneous hand velocity. In this case, the transfer matrix has a negative (stable) and a positive (unstable) eigenvalue. The force pattern in the space of hand velocity is shown under the equation. At the center (zero velocity) the force is zero. Bottom-left panels (A–D): evolution of hand trajectories in four successive epochs, while the subject practiced moving against the force field. The trajectories are averaged over repeated trials. The gray shadow is the standard deviation. In the final set, the trajectories are similar to those executed before the perturbation was turned on. Bottom-right: Aftereffects observed when the field was unexpectedly turned off at the end of training (modified from Shadmehr and Mussa-Ivaldi, 1994).
Euclidean properties of ordinary space The ordinary space within which we move is Euclidean (a special kind of inner product space). The defining feature of a Euclidean space is that
basic operations performed on vectors in one region of space (e.g., addition, multiplication by a scalar) yield identical results in all other regions of space. That is, Euclidean space is flat, not curved like Riemannian spaces: if a stick
49
measures 1 m in one region of Euclidean space, then it measures 1 m in all other regions of space. Although length and distance can be calculated in many ways, there is only one distance measure— the “Euclidean norm”—that satisfies Pythagoras’ theorem (a necessary condition for the norm to arise from the application of an inner product). The Euclidean norm is the distance measure we obtain by adding the squares of the projections of the line joining the two points over orthogonal axes. So, if we represent a point A in an N-dimensional space as a vector a ¼ [a1, a2, . . ., aN]T and a point B as a vector b ¼ [b1, b2, . . ., bN]T, then the Euclidean distance between a and b is qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi distða; bÞ ¼ ða bÞT ða bÞ qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi ¼ ða1 b1 Þ2 þ ða2 b2 Þ2 þ þ ðaN bN Þ2 ð1Þ We are familiar with this distance in 2D and 3D space. But the definition of Euclidean distance is readily extended to N dimensions. The crucial feature of this metric, and this metric only, is that distances are conserved when the points in space are subject to any transformation of the Euclidean group, including rotations, reflections, and translations. The invariance by translations of the origin is immediately seen. Rotations and reflections are represented by orthogonal matrices that satisfy the condition RT R ¼ I
ð2Þ
(i.e., the inverse of an orthogonal matrix is its transpose). For example, if we rotate a line segment by R, the new distance in Euclidean space is equal to the old distance, since qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi ½Rða bÞT Rða bÞ ¼ ða bÞT RT Rða bÞ qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi ¼ ða bÞT ða bÞ ð3Þ
In summary, in the ordinary Euclidean space: 1. Distances between points obey Pythagoras’ theorem and are calculated by a sum of squares. 2. Distances (and therefore the size of objects) do not change with translations, rotations, and reflections. Or, stated otherwise, vector direction and magnitude are mutually independent entities.
Intrinsic geometry of sensorimotor signals in the central nervous system Sensory and motor signals in the nervous system appear to be endowed with neither of the above two properties with respect to the space within which we move. For example, the EMG activities giving rise to movement of our hand would generally change if we execute another movement in the same direction and with the same amplitude starting from a new location. Likewise, the firing rates of limb proprioceptors undoubtedly change if we make a movement with the same amplitude from the same starting location, but now oriented in a different direction. Nevertheless, we easily move our hand any desired distance along any desired direction from any starting point inside the reachable workspace. It therefore seems safe to conclude that our brains are competent to understand and represent the Euclidean properties of space and that our motor systems are able to organize coordination according to these properties. From this perspective, the observation of rectilinear and smooth hand trajectories has a simple interpretation. Straight segments are natural geometrical primitives of Euclidean spaces: they are geodesics (i.e., paths of minimum length). The essential hypothesis, then, is that the brain constructs and preserves patterns of coordination that are consistent with the geometrical features of the environment in which it operates.
50
Encoding the metric properties of Euclidean space Early studies of adaptation of reaching movements to force fields demonstrated the stability of planned kinematics in the face of dynamical perturbations (Shadmehr and Mussa-Ivaldi, 1994), suggesting that the brain develops an internal representation of the dynamics of the limb and its environment, which it uses to plan upcoming movements. The observation that subjects preferentially generate straight-line endpoint motions (Dingwell et al., 2002, 2004) further suggests that the nervous system also develops an internal representation of the environment within which movement occurs. Both representations are necessary to support the kind of learning involved in the operation of HMIs: Different HMIs require their users to learn the geometrical transformation from a set of internal signals endowed with specific metric properties (EEGs, multiunit activities, residual body motions, etc.) into control variables that drive a physical system with potentially significant dynamics (the orientation of a robotic arm, the position of a cursor, the speed and direction of a wheelchair, etc.). We next describe experiments that sought to test whether the brain constructs and preserves patterns of coordination consistent with the geometrical features of the environment using a noninvasive experimental approach with immediate relevance to the application of adaptive control in HMIs. Mosier et al. (2005) and colleagues (Liu and Scheidt, 2008; Liu et al., 2011) studied how subjects learn to remap hand gestures for controlling the motion of a cursor on a computer screen. In their experiments, subjects wore a data glove and sat in front of a computer monitor. A linear transformation A mapped 19 sensor signals from the data glove into two coordinates of a cursor on a computer screen: ax;1 ax;2 . . . ax;19 x0 x P¼ ¼ ay;1 ay;2 . . . ay;19 y0 y ð4Þ ½ h1
h2
...
h19
1 T ¼ AH
Subjects were required to smoothly transition between hand gestures so as to reach a set of targets on the monitor. This task had some relevant features, namely: 1. It was an unusual task. It was practically impossible that a subject had previous exposure to the transformation from hand gestures to cursor positions. 2. The hand and the cursor were physically uncoupled. Vision was therefore the only source of feedback information about the movement of the cursor available to the subjects. 3. There was a dimensional imbalance between the degrees of freedom of the controlled cursor (2) and the degrees of freedom of the hand gestures measured by the data glove (19). 4. Most importantly, there was a mismatch between the metric properties of the space in which the cursor moves and the space of the hand gestures. Specifically, the computer monitor defines a 2D Euclidean space with a well-defined concept of distance between points, whereas there is no clear metric structure for hand gestures. These features are shared by brain–machine interfaces that map neural signals into the screen coordinates of a computer cursor or the 3D position of a robotic arm. The hand-shaping task provides a simple noninvasive paradigm wherein one can understand and address the computational and learning challenges of brain–machine interfaces.
Learning an inverse geometrical model of space A linear mapping A from data-glove “control” signals to the two coordinates of the cursor creates a natural partition of the glove-signal space into two complementary subspaces. One is the 2D (x, y) task-space within which the cursor moves, HT ¼ AþAH [where Aþ ¼ AT(A AT) 1 is the Moore–Penrose (MP) pseudoinverse of A]. The second is its 17D null-space, HN ¼ (I19 AþA)H (where I19 is the 19D identity
51
matrix), which is everywhere orthogonal to the task-space (Fig. 2). Note that both task- and null-spaces are embedded in 19 dimensions. Given a point on the screen, the null-space of that point contains all glove-signal configurations that project onto that point under the mapping A (i.e., the null-space of a cursor position is the inverse image of that position under the handto-cursor linear map). Consider a hand gesture that generates a glove-signal vector B and suppose that this vector maps onto cursor position P. Because of the mismatch in dimensionality between the data-glove signal and cursor vectors (often referred to as “redundancy of control”), one can reach a new position Q in an infinite number of ways. In Fig. 2, the glove-signal space is depicted as a simplified 3D space. In this case, the null-space at
q is a line (because 3 signal dimensions 2 monitor dimensions ¼ 1 null-space dimension). Thus, one can reach Q with any configuration (C, D, E, F, etc.) on this line. However, the configuration C is special because it lies within the taskspace including B and thus, the movement BC is the movement with the smallest Euclidean norm (in the glove-signal space). In this simplified representation, the hand-to-cursor linear map partitions the signal space into a family of parallel planes orthogonal at each point to the corresponding null-space. While visualizing this in more than three dimensions is impossible, the geometrical representation remains generally correct and insightful. Consider now the problem facing the subjects in the experiments of Mosier et al. (2005). Subjects were presented with a target on the
(a)
(b)
LP
h3
h3
LQ
P
F E
B
P Q
D
B C h2
h2
h1
h1 A+ = AT (A·AT)−1 Definitions
T(A) = A+·A N(A) = IK −T(A)
Projections
Null-space component = hN = N(A)·h Task-space component = hT = T(A)·h h = hN + hT hNT ·h = 0
Fig. 2. Geometrical representation. (a) The “hand space,” H, is represented in reduced dimension as a 3D space. The matrix, A, establishes a linear map from three glove signals to a 2D computer monitor. T(A) and N(A) are the task-space and the nullspace of A. The line, LP, contains all the points in H that map onto the same point P on the screen. This line is the “null-space” of A at P. A continuous family of parallel planes, all perpendicular to the null-space and each representing the screen space, fills the entire signal space. (b) The starting hand configuration, B, lies on a particular plane in H and maps to the cursor position, P. All the dotted lines in H leading from B to LQ produce the line shown on the monitor. The “null-space component” of a movement guiding the cursor from P to Q is its projection along LQ. The “task-space component” is the projection on the plane containing BC. Bottom: The mathematical derivation of the null-space and task-space components generated by the transformation matrix A (from Mussa-Ivaldi and Danziger, 2009).
52
screen and were required to shape their hand so that the cursor could reach the target as quickly and accurately as possible. A number of investigators have proposed that in natural movements, the brain exploits kinematic redundancy for achieving its goal with the highest possible precision in task-relevant dimensions. Redundancy would allow disregarding performance variability in degrees of freedom that do not affect performance in task-space. This is a venerable theory, first published by Bernstein (1967) and more recently formalized as the “uncontrolled manifold” theory (Latash et al., 2001, 2002; Scholz and Schoner, 1999) and as “optimal feedback control” (Todorov and Jordan, 2002). These different formulations share the prediction that the motor system will transfers motor variability (or motor noise) to glove-signal degrees of freedom that do not affect the goal, so that performance variability at the goal—that is, at the target—is kept at a minimum. This is not a mere speculation; in a number of empirical cases the prediction matches observed behavior, as in Bernstein's example of hitting a nail with a hammer. However, in the experiments of Mosier et al. (2005) things turned out differently. As subjects became expert in the task of moving the cursor by shaping their hand, they displayed three significant trends with practice that were spontaneous and not explicitly instructed: 1. They executed increasingly straighter trajectories in task-space (Fig. 3a). 2. They reduced the amount of motion in the null-space of the hand-to-cursor map (Fig. 3b). 3. They reduced variability of motion in both the null-space and the task-space (Fig. 3c). Taken together, these three observations suggest that during training, subjects were learning an inverse geometric model of task-space. Consider that among all the possible right inverses of A, the MP pseudoinverse 1 ð5Þ Aþ ¼ AT AAT
selects the glove-signal solution with minimum Euclidean norm. This is the norm calculated as a sum of squares: qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi ð6Þ jjhjj ¼ h21 þ h22 þ þ h219 Passing through each point B in the signal space (Fig. 2), there is one and only one 2D plane that contains all inverse images of the points in the screen that are at a minimum Euclidean distance from B. The subjects in the experiment of Mosier et al. (2005) demonstrated a learning trend to move over these planes and to reduce the variance orthogonal to them—both at the targets and along the movement trajectory. We consider this to be evidence that the learning process is not only driven by the explicit goal of reaching the targets but also by the goal of forming an inverse model of the target space and its metric properties. This internal representation of space is essential to generalize learning beyond the training set. In a second set of experiments, Liu and Scheidt (2008) controlled the type and amount of taskrelated visual feedback available to different groups of subjects as they learned to move the cursor using finger motions. Subjects rapidly learned to associate certain screen locations with desired hand shapes when cued by small pictures of hand postures at screen locations defined by the mapping A. Although these subjects were also competent to form the gestures with minimal error when cued by simple spatial targets (small discs at the same locations as the pictures), they failed to generalize this learning to untrained target locations (pictorial cue group; Fig. 4). Subjects in a second group also learned to reduce taskspace errors when provided with knowledge of results in the form of a static display of final cursor position at the end of each movement; however, this learning also failed to generalize beyond the training target set (terminal feedback group; Fig. 4). Only subjects provided with continuous visual feedback of cursor motion learned to generalize beyond their training set
53
0.45
(b)
0.4
Null motion (G.S.U.)
Aspect ratio
(a)
0.35 0.3 0.25 0.2
1
2
3
240 220 200 180
24%
160 140
4
1
(c)
3
4
40
60
35
55 D1
45 40
D2
35
D3
30
D4
30 STD (G.S.U.)
50 STD (G.S.U.)
2 Day
Day
D1
25
D2 D3
20 15
D4 10
25 5
20 15
0
0.2
0.4 0.6 Null-space
0.8
1
0
0
0.2
0.4 0.6 Task-space
0.8
1
Fig. 3. Behavioral results of the hand-to-cursor mapping experiment. (a) Subjects execute progressively straighter trajectories of the cursor on the screen. This is measured by the aspect ratio, the maximum perpendicular excursion from the straight-line segment joining the start and end of the movement divided by the length of that line segment. The aspect ratio of perfectly straight lines is zero. (b) Length of subject movements in the null-space of the task, hand motion that does not contribute to cursor movement, decreases through training. (c) Average variability of hand movements over four consecutive days (D1, D2, D3, D4). Left: average standard deviation across subjects of the null-space component over the course of a single movement. Right: average standard deviation across subjects of the task-space component over a single movement. Standard deviations are in glove-signal units (G.S.U.), that is, the numerical values generated by the CyberGlove sensors, each ranging between 0 and 255. The x axes units are normalized time (0: movement start; 1: movement end). The overall variance decreases with practice both in the task- and in the null-space (from Mosier et al., 2005).
(continuous feedback group; Fig. 4) and so, visual feedback of endpoint motion appears necessary for learning an inverse geometrical model of the space of cursor motion. Of all the feedback conditions tested, only continuous visual feedback
provides explicit gradient information that can ^ of the facilitate estimation of an inverse model B hand-to-screen mapping A. Liu and colleagues further examined the learning of an inverse geometric representation
140 Euclidean distance [pixel]
Generalization target performance
54
110
80
Continuous feedback Pictorial cues Terminal feedback
50
3
9
15
21
27
33
Trial cycle number
Fig. 4. The ability to generalize beyond the trained target set depends on the type and amount of task-related visual feedback available during practice in moving the cursor using finger motions. Subjects performed 33 cycles of six movements, wherein a cycle consisted of one movement to each of five training targets (performed with visual feedback) plus a movement to one of three generalization targets (performed entirely without visual feedback). Each generalization target was visited once every three cycles. Each trace represents the across-subject average generalization error for subjects provided with continuous visual feedback of target capture errors (black squares), subjects provided with feedback of terminal target capture errors only (gray diamonds), and subjects provided with pictorial cues of desired hand shapes (gray circles). Error bars represent 1 SEM. We evaluated whether performance gains in generalization trials were consistent with the learning of an inverse hand-to-screen mapping or whether the different training conditions might have promoted another form of learning, such as the formation of associations between endpoint targets and hand gestures projecting onto them (i.e., a look-up table). Look-up table performance was computed as the across-subject average of the mean distance between the three generalization targets and their nearest training target on the screen. Because each subject's A matrix was unique, the locations of generalization and training targets varied slightly from one subject to the next. The gray band indicates the predicted mean 1 SD look-up table performance. Only those subjects provided with continuous visual feedback of cursor motion demonstrated generalization performance consistent with learning an inverse map of task-space (adapted from Liu and Scheidt, 2008).
of task-space by studying how subjects reorganize finger coordination patterns while adapting to rotation and scaling distortions of a newly learned hand-to-screen mapping (Liu et al., 2011). After learning a common hand-to-screen mapping A by practicing a target capture task on one day and refreshing that learning early on the next day, subjects were then exposed to either a rotation y of cursor motion about the origin (TR): xrotated cosðyÞ sinðyÞ x ¼ TR P ¼ PT ¼ sinðyÞ cosðyÞ y yrotated ð7Þ or a scaling k of cursor motion in task-space (TS): xscaled k 0 x PT ¼ ¼ TS P ¼ ð8Þ yscaled 0 k y The distortion parameters y and k were selected such that uncorrected error magnitudes were identical on initial application of T in both cases. The question Liu and colleagues asked was whether step-wise application of the two task-space distortions would induce similar or different reorganization of finger movements. Both distortions required a simple reweighting of the finger coordination patterns acquired during initial learning of A (Fig. 5a), while neither required reorganization of null-space behavior. Because A is a rectangular matrix with 2 rows and 19 columns, it does not have a unique inverse; rather, there are infinite 19 2 matrices B such that AB¼I2
ð9Þ
where I2 is the 2 2 unit matrix. These are “right inverses” of A, each one generating a particular glove-signal vector H mapping onto a common screen coordinate P. Liu et al. (2011) estimated ^ used the inverse hand-to-screen transformation B to solve the target acquisition task before and after adaptation to TR and TS by a least squares fit to the data:
55
^ ¼ HPT PPT 1 B
ð10Þ
^ obtained after They then evaluated how well B adaptation (BADAPT) was predicted by rotation ^ obtained just prior (TR) or scaling (TS) of the B to imposing the distortion (BBEFORE) by computing a difference magnitude DBADAPT: DBADAPT ¼ jjBADAPT BBEFORE T1 jj
ð11Þ
They compared this to the difference magnitude obtained from data collected in two separate time intervals during baseline training on the second day (i.e., before imposing the distortion; BL1 and BL2). Here, T 1 of Eq. (9) is assumed to be the identity matrix: DBNOISE ¼ jjBBL1 BBL2 jj
ð12Þ
Importantly, Liu and colleagues found that adaptation to the rotation induced a significant change in the subject's inverse geometric model of Euclidean task-space whereas adaptation to a scaling did not (Fig. 5b). Because the magnitude of initial exposure error was virtually identical in the two cases, the different behaviors cannot be accounted for by error magnitude. Instead, the results provide compelling evidence that in the course of practicing the target capture task, subjects learned to invoke categorically different compensatory responses to errors of direction and extent. To do so, they must have internalized the inner product structure imposed by the linear hand-to-screen mapping, which establishes the independence of vector concepts of movement direction and extent in task-space. Under the assumption that the brain minimizes energetic costs in addition to kinematic errors (see Shadmehr and Krakauer, 2008 for a review), subjects in the current study should at all times have used their baseline inverse map to constrain command updates to only those degrees of freedom contributing to task performance. This was not the case. The findings were also inconsistent with the general proposition that once the “structure” of a redundant task is learned, such dimensionality reduction is used to improve the
efficiency of learning in tasks sharing a similar structure (Braun et al., 2010). Instead, the findings of Mosier et al. (2005) and colleagues (Liu and Scheidt, 2008) demonstrate that as the subjects learned to remap the function of their finger movements for controlling the motion of the cursor, they also did something that was not prescribed by their task instructions. They formed a motor representation of the space in which cursor was moving and, in the process of learning, they imported the Euclidean structure of the computer monitor into the space of their control signals. This differs sharply from the trend predicted by the uncontrolled manifold theory, where a reduction in the variance at the target should have been accompanied by no such decrease in performance variance in redundant degrees of freedom. The experimental observations of Bernstein, Scholz, Latash, and others (Bernstein, 1967; Latash et al., 2001; Scholz and Schoner, 1999) can be reconciled with the observations of Mosier and colleagues if one considers that the glove task is completely novel, whereas tasks such as hitting a nail with a hammer are performed within the domain of a well learned control system. Because the purpose of learning is to form a map for executing a given task over a broad target space in many different situational contexts, it is possible that once a baseline competency and confidence in the mapping is established, the abundance of degrees of freedom becomes an available resource to achieve a more flexible performance, with higher variability in the null-space.
The dual-learning problem A HMI sets a relation from body-generated signals to control signals or commands for an external device. This relation does not need to be fixed. Intuition suggests that it should be possible to modify the map implemented by the interface so as to facilitate the learning process. In this spirit, Taylor et al. (2002) have employed a
56 (a)
Baseline
Early
Late
Early
Late
Rotation
Scaling
(b)
1.0
^ || DB || (GSU/pixel)
Adaptation
0.5
Washout
Rotation
DB D B NO IS D ADA E iff P er T en ce
DB D B NO IS D ADA E iff P er T en ce
0
Scaling
Fig. 5. Adaptation to rotation and scaling distortions of task-space 1 day after initially learning the manual target capture task. (a) Patterns of cursor trajectory errors are similar to those typically observed in studies of horizontal planar reaching with the arm. Here, we show data from representative subjects exposed to a ROTATION (top) or SCALING (bottom) of task-space during baseline (left) adaptation (early and late) as well as washout (early and late) blocks of trials. Shading indicates the adaptation block of trials. During preadaptation practice with the baseline map, cursor trajectories were well directed to the target. Imposing the step-wise counterclockwise (CCW) rotation caused cursor trajectories to deviate CCW initially but later “hook back” to the desired final position (Fig. 4a, top). With practice under the altered map, trajectories regained their original rectilinearity. When the baseline map was suddenly restored, initial trajectories deviated clockwise (CW) relative to trajectories made at the start of Session 2, indicating that subjects used an adaptive feedforward strategy to compensate for the rotation. These aftereffects were eliminated by the end of the washout period. Similarly, initial exposure to a step-wise increase in the gain of the hand-to-screen map resulted in cursor trajectories that far overshot their goal. Further practice under the altered map reduced these extent errors. Restoration of the baseline map resulted in initial cursor movements that undershot their goal. These targeting errors were virtually eliminated by the end of the washout period. (b) Adaptation to the rotation induced a significant change in the subject's inverse geometric model of Euclidean task-space whereas adaptation to a scaling did not. ^ is our measure of reorganization within the redundant articulation space, for subjects exposed to a rotation (red) and Here, DB scaling (black) of task-space, both before (solid bars) and after (unfilled bars) visuomotor adaptation. For the subjects exposed ^ after adaptation could not reasonably be characterized as a rotated version of the baseline map to the rotation distortion, B because DBADAPT far exceeded DBNOISE for these subjects. The within-subject difference between DBADAPT and DBNOISE was 0.44 0.32 G.S.U./pixel (red solid bar), from which we conclude that the rotational distortion induced these subjects to form a new inverse hand-to-screen map during adaptation. In contrast, DBADAPT did not exceed DBNOISE, for scaling subjects
57
where ^s ¼ ½l1 ; l2 ; x0 ; y0 T is a constant parameter vector that includes the link lengths and the origin of the shoulder joint. The virtual arm was not displayed except for the arm's endpoint, which was represented by a 0.5-cm-radius circle. Subjects
h1 aq1,1 aq1,2
…
aq2,1 aq2,2
…
aq1,19 aq2,19
.
h2 …
coadaptive movement prediction algorithm in rhesus macaques to improve cortically controlled 3D cursor movements. Using an extensive set of empirically chosen parameters, they updated the system weights through a normalized balance between the subject's most successful trials and their most recent errors, resulting in quick initial error reductions of about 7% daily. After significant training with exposure to the coadaptive algorithm, subjects performed a series of novel point-to-point reaching movements. They found that subjects’ performance in the new task was not appreciably different from the training task. This is evidence of successful generalization. Danziger et al. (2009) modified the glove-cursor paradigm by introducing a nonlinear transformation between the hand signals and the cursor (Fig. 6). In their experiment, the 19D vector of sensor values was mapped to the position of a cursor presented on a computer monitor. First, the glove signals were multiplied by a 2 19 transformation matrix to obtain a pair of angles. These angles then served as inputs to a forward kinematics equation of a simulated 2-link planar arm to determine the end-effector location: 2 3 h1 6 h2 7 y1 6 7 ¼ A6 .. 7 y2 4 . 5 ð13Þ h19 x cosðy1 Þ cosðy1 þ y2 Þ 1 0 ^s ¼ sinðy2 Þ sinðy1 þ y2 Þ 0 1 y
q1 =
z(q,s^)
q2
x y
h19 A
q2 q1
Fig. 6. Hand posture represented as a point in “hand space,” h, is mapped by a linear transformation matrix, A, into twojoint angles of a simulated planar revolute-joint kinematic arm on a monitor. The endpoint of the simulated arm was determined by the nonlinear forward kinematics, z. Subjects placed the arm's endpoint into displayed targets through controlled finger motions. During training, the elements of the A matrix were updated to eliminate movement errors and assist subjects in learning the task (from Danziger et al., 2009).
were given no information about the underlying mapping of hand movement to cursor position. The mapping matrix, A, was initially determined by having the subject generate four preset hand postures. Each one of these postures was placed in correspondence with a corner of a rectangle inside the joint angle workspace. The A matrix was then calculated as, A ¼ Y Hþ, where Y is a 2 4 matrix of angle pairs that represent the corners of the rectangle, and Hþ is the MP pseudoinverse of H (Ben-Israel and Greville, 1980), the 19 4 matrix whose columns are signal vectors corresponding to the calibration postures. Using the MP pseudoinverse corresponded to
(black gradient bars; p ¼ 0.942), yielding an average within-subject difference between DBADAPT and DBNOISE of 0.03 0.10 G.S.U./ pixel (black solid bar). We, therefore, found no compelling reason to reject the hypothesis that after adaptation, scaling subjects simply contracted their baseline inverse map to compensate for the imposed scaling distortion. Taken together, the results demonstrate that applying a rotational distortion to cursor motion initiated a search within redundant degrees of freedom for a new solution to the target capture task whereas application of the scaling distortion did not (adapted from Liu et al., 2010).
58
minimizing the norm of the A matrix in the Euclidean metric. As a result of this redundant geometry, each point of the workspace was reachable by many anatomically attainable hand postures. Danziger et al. asked subjects to shape their hands so as to move the tip of the simulated arm into a number of targets. The experiment proceeded in sets of training epochs. In each epoch, the mapping between the hand joint angles and the arm's free-moving tip (the “endeffector”) was updated so as to cancel the mean endpoint error in the previous set of movements. This was done in two ways by two separate subject groups: (a) by a least mean squares (LMS) gradient descent algorithm which takes steps in the direction of the negative gradient of the endpoint error function, or (b) by applying the MP pseudoinverse which offers an analytical solution for error elimination while minimizing the norm of the mapping. LMS (Widrow and Hoff, 1960) is an iterative procedure, which seeks to minimize the square of the performance error norm by iteratively modifying the elements of the A matrix in Eq. (13). The minimization procedure terminated when the difference between the old and the new matrix exceeded a preset threshold. In contrast, the MP procedure was merely a recalibration of the A matrix, which canceled the average error after each epoch. Therefore, both LMS and MP algorithms had identical goals, to abolish the mean error in each training epoch, and each method found a different solution. The result was that subjects exposed to the LMS adaptive update outperformed their control counterparts who had a constant mapping. But, surprisingly, the MP update procedure was a complete failure, and subjects exposed to this method failed to improve their skill levels at all (Fig. 7, left). We hypothesize that this was because the LMS procedure finds local solutions to the error elimination problem (because it is a gradient decent algorithm), while the MP update may lead to radically different A-matrices across epochs. This finding highlights a trade-off
between maintaining a constant structure of the map and altering the structure of the map so as to assist subjects in their learning. But perhaps the most important finding in that study was a negative result. In spite of the more efficient learning over the training set, subjects in the LMS group did not show any significant improvement over the control group on a different set of targets, which were not practiced during the training session (Fig. 7, right). The implication is that the LMS algorithms facilitated subjects’ creation of an associative map from the training targets to a set of corresponding hand configurations. However, this did not improve learning the geometry of the control space itself. Had this been the case, we would expect to see greater improvement in generalization. Finding machine learning methods that facilitate “space learning” as distinct from improving performance over a training set remains an open and important research goal in human–machine interfacing.
A clinical perspective: the body–machine interface The experiments of Mosier et al. (2005) and Danziger et al. (2009) demonstrated the ability of the motor system to reorganize motor coordination so as to match the low-dimensional geometrical structure of a novel control space. Subjects learned to redistribute the variance of the many degrees of freedom in their fingers over a 2D space that was effectively an inverse image of the computer monitor under the hand-to-cursor map. We now consider in the same framework the problem of controlling a powered wheelchair by coordinated upper body motions. People suffering from paralysis, such as spinal cord injury (SCI) survivors are offered a variety of devices for operating electrically powered wheelchairs. These include joysticks, head and neck switches, sip-and-puff devices, and other interfaces. All these devices are designed to match the motor control functions that are available to their users.
59 Normalized average endpoint reaching error MP compared with LMS and control performance
Movement errors in control and LMS subjects during generalization averages over many subjects
1.6 MP Control LMS
1.4
1.2 Control LMS
Average normalized error
1.1 Normalized error
1.2 1 0.8 0.6
1 0.9 0.8 0.7 0.6 0.5 0.4
0.4
0.3 0.2
1
2
3
4 5 6 Epoch number
7
8
0.2
1
2
3
4 5 6 Epoch number dashed lines separate days
7
8
9
Fig. 7. (Left) Average normalized movement errors for three subject groups in the experiment outlined in Fig. 6. The mapping for MP subjects was updated to minimize prior movement errors by an analytical method, which resulted in large mapping changes. The mapping for LMS subjects was also updated to minimize prior error but with a gradient descent algorithm that resulted in small mapping changes. Control subjects had a constant mapping. LMS subjects outperformed controls, while MP subjects failed to learn the task at all. (Right) Movement errors on untrained targets for control and LMS groups show that adaptive mapping updates does not facilitate spatial generalization (from Danziger et al., 2008, 2009).
However, they have a fixed structure and ultimately they present the users with challenging learning problems (Fehr et al., 2000). In general, the lack of customizability of these devices creates various difficulties across types and levels of disability (Hunt et al., 2004) and subjects with poor control of the upper body are at a greater risk of incurring accidents. Decades of research and advances in robotics and machine learning offer now the possibility to shift the burden of learning from the human user to the device itself. In a simple metaphor, instead of having the user of the wheelchair learning how to operate a joystick, we may have the wheelchair interface looking at the user's body as if it were a joystick. The controller of a powered wheelchair is a 2D device, setting the forward speed and the rotation about a vertical axis. Most paralyzed SCI
survivors have residual mobility much in excess of 2 degrees of freedom. Therefore, from a computational point of view one can see the control problem as a problem of embedding a 2D control surface within a higher-dimensional “residual motor space.” This is analogous to the problem of embedding the control space of a robotic arm within the signal space associated with a multiunit neural signal from a cortical area. From a geometrical standpoint, the embedding operation is facilitated by the ability of the motor control system to learn Euclidean metrics in a remapping operation, as shown in Mosier et al. (2005). While control variables may have a nonEuclidean Riemannian structure, a powerful theorem by Nash (1956) states that any Riemannian surface can be embedded within a Euclidean space of higher dimension. A simple way to
60
construct a Euclidean space from body motions is by principal component analysis (PCA; Jolliffe, 2002). This is a standard technique to represent a multidimensional signal in a Cartesian reference frame, whose axes are ordered by decreasing variance. Using PCA, Casadio et al. (2010) developed a camera-based system to capture upper body motions and control the position of a cursor on a computer monitor (Fig. 8). Both SCI injured subjects—at or above C5—and unimpaired control subjects participated in this study. Four small cameras monitored the motions of four small infrared active markers that were placed on the subjects’ upper arms and shoulders. Since each marker had a 2D image on a camera, the net signal was an 8D vector of marker coordinates. This vector defined the “body space.” The control space was defined by the two coordinates (x, y) of the cursor on the monitor. Unlike the hand-to-cursor map of the previous study, the body-to-cursor map was not based on a set of predefined calibration points. Instead, in the first part of the experiment subjects performed free
Fig. 8. Controlling a cursor by upper-body motion: experimental apparatus. Four infrared cameras capture the movements of four active markers attached to the subject's arm and shoulder. Each camera outputs the instantaneous x, y coordinates of a marker. The eight coordinates from the four cameras are mapped by linear transformation into the coordinates of a cursor, presented as a small dot on the monitor. The subject is asked to move the upper body so as to guide the dot inside a target (from Casadio et al., 2010).
upper body motions for 1 min. This was called the “dance” calibration. A rhythmic music background facilitated the subjects’ performance in this initial phase. The purpose of the dance was to evaluate how subjects naturally distributed motor variance over the signal space. The two principal component vectors, generating the highest variance of the calibration signals, defined two Cartesian axes over the signal space. In the calibration phase, subjects could scale the axis to compensate for the difference in variance associated with them. They were also allowed to rotate and/or reflect the axis to match the natural right-left, front-back directions of body space. After the calibration, subjects were engaged in a set of reaching movements. Both control and SCI subjects learned to execute efficiently the required motions of the cursor on the computer monitor by controlling their upper body movements (Fig. 9). Learning in terms of error reduction, increase in movement speed, and trajectory smoothness was evident both in controls and SCI subjects. In particular, all SCI subjects were able to use their shoulder movements for piloting the cursor for about 1 h. Importantly, subjects did not merely learn to track the cursor on the monitor. Instead, they acquired the broader skill of organizing their upper-body motions in “feedforward” motor programs, analogous to the natural reaching by hand. No statistically significant effect of vision could be detected, as well as no interaction between vision and practice when comparing movement executed under continuous visual feedback of the cursor, with movements where the cursor feedback was suppressed. Moreover, PCA succeeded in capturing the main characteristics of the upper-body movements for both control and SCI subjects. During the calibration phase, for all high-level SCI subjects it was possible to extract at least two principal components with significant variance from the 8D signals. Their impairment constrained and shaped the movements. Compared to control, they had on average a bigger variance associated with the first component and
61 Early Training
Late Training
Control
SCI 1
SCI 2
SCI 3
SCI 4
Fig. 9. Movement trajectories in early (left) phases of learning, for a control subject subjects. Calibration lines on bottom right panel: 1 cm on the computer screen (from 2010).
and late (right) and four SCI corner of each Casadio et al.,
smaller variances associated with the second through fourth components. Otherwise stated, the SCI subjects had a lower-dimensional upper body motor space. At the end of training, for all subjects the first three principal components accounted for more than 95% of the overall variance. Furthermore, the variance accounted for (VAF) by the two first principal components slightly increased with practice. However, there was a significant difference
between controls and SCI subjects. Controls mainly changed the movements associated with their degrees of freedom in order to use two balanced principal movements. They learned to increase the variance associated with the second principal component (Fig. 10), thus achieving a better balance between the variance explained by the first two components. This behavior was consistent with the consideration that subjects practiced a 2D task, with a balanced on-screen excursion in both dimensions. In contrast, at the end of the training, SCI subjects maintained the predominance of the variance explained by the first component: they increased the variance explained by the first component and decreased the fourth. Their impairment effectively constrained their movements during the execution of the reaching task as well as during the free exploration of the space. The most relevant findings of Casadio et al. (2010) concerned the distribution of variance across task-relevant and task-irrelevant dimensions. For control subjects, the VAF by the task-space with respect to the overall variance significantly increased with practice. In spite of the reduced number of training movements, the same trend was present in most SCI subjects. Therefore, as in the hand-cursor glove experiments of Mosier et al. (2005), subjects learned to reduce the variance that did not contribute to the motion of the cursor and demonstrated the ability to form an inverse model of the body-to-cursor transformation. As subjects reduced the dimensionality of their body motions, they also showed a marked tendency to align their movement subspace with the 2D space established by the body-cursor map (Fig. 11). It is important to observe that this was by no means an expected result. In principle, one could be successful at the task while confining one's movements to a 2D subspace that differs from the 2D subspace defined by the calibration. To see this, consider the task of drawing on a wall with the shadow of your hand. You can move the hand on any invisible surface with any orientation
62 80
80 First target set Last target set
First target set Last target set 60 Variance (%)
Variance (%)
60
40
20
20
0
40
1
2
3
4
5
6
7
8
0
1
2
3
4
5
6
7
8
Space dimensions
Principal component
Fig. 10. Distribution of motor variance across learning. Left panel: Results of principal component analysis on the first (gray) and last movement set (black) for control subjects (mean þ SE). In the first movement set (gray) more than 95% of variance was explained by four principal components. At the end of the training session (black), unimpaired controls mainly tended to increase the variance associated with the second principal component. Right panel: Control subjects (mean þ SE). Results of the projection of the data of the first (gray) and last movement set (black) over the 8D space defined by the body-cursor map. This transformation defines an orthonormal basis, where the “task-space” components a1, a2 determine the cursor position on the screen, and the orthogonal vectors a3, . . ., a8 represent the “null-space” components that do not change the control vector. For most of the control subjects, the fraction of movement variance in the null-space decreased with training in favor of the variance associated in the task-space (from Casadio et al., 2010).
(except perpendicular to the wall!). The result of Casadio and collaborators is analogous to finding that one would prefer to move the hand on an invisible plane parallel to the wall. Taken together, these results indicate that subjects were able to capture the structure of the task-space and to align their movements with it.
a3
a3
a1
a2
a1
a2
Conclusions Fig. 11. Matching the plane of the task. The limited number of dimensions involved in the task allowed us to project the body movement signals in a 3D subspace where the vectors a1,a2 define the “task-space” and a3 is the most significant nullspace component in terms of variance accounted for. In the first movement set (early phase of learning, left panel) there was a relevant movement variance associated with the nullspace dimension a3. That component was strongly reduced in the last target set (late phase of learning, right panel) where the movement's space became more planar, with the majority of the movement variance accounted by the task-space components a1, a2 (from Casadio et al., 2010).
The concept of motor redundancy has attracted consistent attention since the early studies of motor control. Bernstein (1967) pioneered the concept of “motor equivalence” at dawn of the past century by observing the remarkable ability of the motor system to generate a variety of movements achieving a single well-defined goal. As aptly suggested by Latash (2000), the very term “redundancy” is a misnomer as it implies
63
an excess of elements to be controlled instead of a fundamental resource of biological systems. We agree with Latash, and stick opportunistically with the term redundancy simply because it is commonly accepted and well understood. There is a long history of studies that have addressed the computational tasks associated with kinematic redundancy while others have considered the advantage of large kinematic spaces in providing ways to improve accuracy in the reduced space defined by a task. Here, we have reviewed a new point of view on this issue. We considered how the abundance of degrees of freedom may be a fundamental resource in the learning and remapping problems that are encountered in human–machine interfacing. We focused on two distinctive features: 1. The HMI often poses new learning problems and these problems may be burdensome to users that are already facing the challenges of disability. 2. By creating an abundance of signals—either neural recordings or body motions—one can cast a wide net over which a lower-dimensional control space can be optimally adapted. Work on remapping of finger and body movements over 2D task-spaces have highlighted the existence of learning mechanisms that capture the structure of a novel map relating motor commands to their effect on task-relevant variables. Both unimpaired and severely paralyzed subjects were able with practice not only to perform what they were asked to do but they also adapted their movements to match the structure of the novel geometrical space over which they operated. This may be seen as “suboptimal” with respect to a goal of maximal accuracy. Subjects did not shift their variance from the low-dimensional task to the null-space (or uncontrolled manifold). Instead, as learning progressed, variance in the null-space decreased as well as variance in the task-relevant variables. This is consistent with the hypothesis that through learning, the motor system strives to form an inverse map of
the task. This must be a function from the lowdimensional target space to the high-dimensional space of control variables. It is only after such a map is formed that a user may begin to exploit the possibility of achieving the same goals through a multitude of equivalent paths.
Acknowledgments This work was supported by the NINDS grants 1R21HD053608 and 1R01NS053581-01A2, by Neilsen Foundation, and Brinson Foundation.
References Bach-y-Rita, P. (1999). Theoretical aspects of sensory substitution and of neurotransmission-related reorganization in spinal cord injury. Spinal Cord, 37, 465–474. Ben-Israel, A., & Greville, T. N. E. (1980). Generalized inverses: Theory and application. New York, NY: John Wiley and Sons. Bernstein, N. (1967). The coordination and regulation of movement. Oxford: Pegammon Press. Braun, D., Mehring, C., & Wolpert, D. (2010). Structure learning in action. Behavioural Brain Research, 206, 157–165. Casadio, M., Pressman, A., Fishbach, A., Danziger, Z., Acosta, S., Chen, D., et al. (2010). Functional reorganization of upper-body movement after spinal cord injury. Experimental Brain Research, 207, 233–247. Danziger, Z., Fishbach, A., & Mussa-Ivaldi, F. (2008). Adapting Human-Machine Interfaces to User Performance. IEEE EMBC. Canada: Vancouver British Columbia. Danziger, Z., Fishbach, A., & Mussa-Ivaldi, F. A. (2009). Learning algorithms for human–machine interfaces. IEEE Transactions on Biomedical Engineering, 56, 1502–1511. Dingwell, J. B., Mah, C. D., & Mussa-Ivaldi, F. A. (2002). Manipulating objects with internal degrees of freedom: Evidence for model-based control. Journal of Neurophysiology, 88, 222–235. Dingwell, J. B., Mah, C. D., & Mussa-Ivaldi, F. A. (2004). An experimentally confirmed mathematical model for human control of a non-rigid object. Journal of Neurophysiology, 91, 1158–1170. Fehr, L., Langbein, W. E., & Skaar, S. B. (2000). Adequacy of power wheelchair control interfaces for persons with severe disabilities: A clinical survey. Journal of Rehabilitation Research and Development, 37, 353–360.
64 Flash, T., & Hogan, N. (1985). The coordination of arm movements: An experimentally confirmed mathematical model. The Journal of Neuroscience, 5, 1688–1703. Houweling, A. R., & Brecht, M. (2007). Behavioural report of single neuron stimulation in somatosensory cortex. Nature, 451, 65–68. Hunt, P. C., Boninger, M. L., Cooper, R. A., Zafonte, R. D., Fitzgerald, S. G., & Schmeler, M. R. (2004). Demographic and socioeconomic factors associated with disparity in wheelchair customizability among people with traumatic spinal cord injury. Archives of Physical Medicine and Rehabilitation, 85, 1859–1864. Jolliffe, I. T. (2002). Principal component analysis. New York, NY: Springer. Kuiken, T. A., Li, G., Lock, B. A., Lipcshutz, R. D., Miller, L. A., Subblefield, K. A., et al. (2009). Targeted muscle reinnervation for real-time myoelectric control of multifunction artificial arms. JAMA, 301, 619–628. Lackner, J., & Dizio, P. (1994). Rapid adaptation to Coriolis force perturbations of arm trajectory. Journal of Neurophysiology, 72, 299–313. Latash, M. (2000). There is no motor redundancy in human movements. There is motor abundance. Motor Control, 4, 259–261. Latash, M. L., Scholz, J. F., Danion, F., & Schoner, G. (2001). Structure of motor variability in marginally redundant multifinger force production tasks. Experimental Brain Research, 141, 153–165. Latash, M. L., Scholz, J. P., & Schoner, G. (2002). Motor control strategies revealed in the structure of motor variability. Exercise and Sport Sciences Reviews, 30, 26–31. Libet, B., Alberts, W. W., & Wright, E. W. (1964). Production of threshold levels of conscious sensation by electrical stimulation of human somatosensory cortex. Journal of Neurophysiology, 27, 546. Liu, X., & Scheidt, R. (2008). Contributions of online visual feedback to the learning and generalization of novel finger coordination patterns. Journal of Neurophysiology, 99, 2546–2557. Liu, X., Mosier, K. M., Mussa-Ivaldi, F. A., Casadio, M., & Scheidt, R. A. (2011). Reorganization of finger coordination patterns during adaptation to rotation and scaling of a newly learned sensorimotor transformation. Journal of Neurophysiology, 105, 454–473. Loeb, G. E. (1990). Cochlear prosthetics. Annual Review of Neuroscience, 13, 357–371. Morasso, P. (1981). Spatial control of arm movements. Experimental Brain Research, 42, 223–227.
Mosier, K. M., Scheidt, R. A., Acosta, S., & MussaIvaldi, F. A. (2005). Remapping hand movements in a novel geometrical environment. Journal of Neurophysiology, 94, 4362–4372. Mussa-Ivaldi, F. A., & Danziger, Z. (2009). The remapping of space in motor learning and human-machine interfaces. Journal of Physiology-Paris, 103(3-5), 263–275. Nash, J. (1956). The imbedding problem for Riemannian manifolds. Annals of Mathematics, 63, 20–63. Poggio, T., & Smale, S. (2003). The mathematics of learning: Dealing with data. Notices of the American Mathematical Society, 50, 537–544. Romo, R., Hernandez, A., Zainos, A., Brody, C. D., & Lemus, L. (2000). Sensing without touching: Psychophysical performance based on cortical microstimulation. Neuron, 26, 273–278. Scholz, J. P., & Schoner, G. (1999). The uncontrolled manifold concept: Identifying control variables for a functional task. Experimental Brain Research, 126, 289–306. Shadmehr, R., & Krakauer, J. (2008). A computational neuroanatomy for motor control. Experimental Brain Research, 185, 359–381. Shadmehr, R., & Mussa-Ivaldi, F. A. (1994). Adaptive representation of dynamics during learning of a motor task. The Journal of Neuroscience, 14, 3208–3224. Soechting, J. F., & Lacquaniti, F. (1981). Invariant characteristics of a pointing movement in man. The Journal of Neuroscience, 1, 710–720. Taylor, D. M., Tillery, S. I., & Schwartz, A. B. (2002). Direct cortical control of 3D neuroprosthetic devices. Science, 296, 1829–1832. Todorov, E., & Jordan, M. I. (2002). Optimal feedback control as a theory of motor coordination. Nature Neuroscience, 5, 1226–1235. Uno, Y., Kawato, M., & Suzuki, R. (1989). Formation and control of optimal trajectory in human multijoint arm movement. Biological Cybernetics, 61, 89–101. Widrow, B., & Hoff, M. (1960). Adaptive switching circuits. WESCON Conv Rec, 4, 99. Wolpaw, J. R., & McFarland, D. J. (2004). Control of a two-dimensional movement signal by a noninvasive brain–computer interface in humans. Proceedings of the National Academy of Sciences of the United States of America, 101, 17849–17854. Zrenner, E. (2002). Will retinal implants restore vision? Science, 295, 1022–1025.
A. M. Green, C. E. Chapman, J. F. Kalaska and F. Lepore (Eds.) Progress in Brain Research, Vol. 191 ISSN: 0079-6123 Copyright Ó 2011 Elsevier B.V. All rights reserved.
CHAPTER 4
Locomotor adaptation Gelsy Torres-Oviedo{,{, Erin Vasudevan{,{,1, Laura Malone{,} and Amy J. Bastian*,{,{ {
Department of Motor Learning Lab, Kennedy Krieger Institute, Baltimore, Maryland, USA Neuroscience Department of Johns Hopkins School of Medicine, Baltimore, Maryland, USA Biomedical Engineering Department of Johns Hopkins School of Medicine, Baltimore, Maryland, USA {
}
Abstract: Motor learning is an essential part of human behavior, but poorly understood in the context of walking control. Here, we discuss our recent work on locomotor adaptation, which is an error driven motor learning process used to alter spatiotemporal elements of walking. Locomotor adaptation can be induced using a split-belt treadmill that controls the speed of each leg independently. Practicing split-belt walking changes the coordination between the legs, resulting in storage of a new walking pattern. Here, we review findings from this experimental paradigm regarding the learning and generalization of locomotor adaptation. First, we discuss how split-belt walking adaptation develops slowly throughout childhood and adolescence. Second, we demonstrate that conscious effort to change the walking pattern during split-belt training can speed up adaptation but worsens retention. In contrast, distraction (i.e., performing a dual task) during training slows adaptation but improves retention. Finally, we show the walking pattern acquired on the split-belt treadmill generalizes to natural walking when vision is removed. This suggests that treadmill learning can be generalized to different contexts if visual cues specific to the treadmill are removed. These findings allow us to highlight the many future questions that will need to be answered in order to develop more rational methods of rehabilitation for walking deficits. Keywords: locomotion; motor learning; adaptation; generalization of learning; rehabilitation.
Walking is a fundamental motor act. As such, it must be flexible enough to accommodate different environments, yet automatic enough so that we do not have to consciously focus on every step. Recently, we, and others, have been exploring the adaptability of locomotion with an eye toward improving rehabilitation of walking for people with brain lesions (e.g., Choi et al., 2009;
*Corresponding author. Tel.: þ443-923-2718; Fax: þ443-923-2715 E-mail:
[email protected] 1 Present address: Moss Rehabilitation Research Institute, Pennsylvania, USA.
DOI: 10.1016/B978-0-444-53752-2.00013-8
65
66
Reisman et al., 2007, 2009). This review will focus on what we know about adaptive processes for human walking control, and perhaps more importantly, what we do not know. Adaptive processes allow us to modify our locomotor patterns to suit changing environments. Since this is a critical ability for navigating the world, it is possible that adaptation develops at a very early age in humans. Conversely, the development of adaptation could follow a more protracted time course. This may be particularly true in human children, since they take much longer to learn how to walk independently than most other mammals. While humans typically begin walking 1 year after birth, many other mammals (e.g., horses, elephants) walk on the day that they are born. However, a recent study suggests that the late onset of human walking might be related to large brain mass, which takes extra time to develop (Garwicz et al., 2009). Indeed, if one considers the time from conception (rather than birth) to onset of walking, mammals with large brains relative to their body take longest to walk: humans ( 19–25 months) and elephants ( 22 months). Both animals have large brains—an adult human brain weighs 1350 g and an adult elephant brain weighs 4400 g. However, the percentage of brain mass with respect to the body is larger in humans than in elephants. Thus, brain development seems to be an important influence in dictating the onset of walking in mammals. Given the dependence of onset of walking on brain development, we wondered if other elements of walking control would follow a protracted developmental time course in humans as the nervous system matures. Specifically, we have been interested in understanding whether children can learn novel walking patterns through adaptive learning mechanisms. Although children are able to walk independently, we predicted that processes to adapt locomotor patterns would not be fully developed since human brain development continues well after birth, through childhood, and even into adulthood (LeBel et al., 2008).
We use a motor learning paradigm to study walking adaptation involving a split-belt treadmill, with independent belts under each leg (Reisman et al., 2005). Using this device, we can study people walking with the belts moving at the same speed, or “tied,” and with the belts moving at different speeds, or “split.” Figure 1a illustrates the general paradigm that is used for these studies. We have previously reported that adults adapt their walking pattern when walking in the split-belt condition over the course of 10–20 min. They specifically change step symmetry (i.e., the normalized difference in step sizes of the two legs; Fig. 1b), using both spatial and temporal strategies as described in Fig. 1c and d. When returning to tied belts, they show aftereffects in both domains, indicating that the nervous system learned and stored a new locomotor pattern that had to be actively unlearned. Recent work in our lab suggests that young children can adapt their walking pattern, but appear to show different developmental patterns for spatial versus temporal adaptation of walking (Vasudevan et al., 2011). Our initial intuition was that children might be more flexible in their ability to learn and, therefore, might adapt faster or more completely. Instead, we found that 3- to 5-year-old children adapt step symmetry slowly (Fig. 2a), and this ability does not fully develop until after age 12–14. Similar findings were present for the center of oscillation difference, which is defined as the difference in the midpoint position between heel strike (HS) and toe-off of each leg. Since the center of oscillation is dependent upon where the foot is placed at HS and where it is lifted off at toe-off, this measure reflects spatial locomotor control (Fig. 2b). In contrast, all ages could adapt the temporal parameter of phase at normal rates (Fig. 2c). Our interpretation of this finding is that the ability to adapt spatial control of walking depends on brain functions that are still developing through adolescence. Candidate sites are the cerebellum and motor cortex, though we consider the former to be more likely (Morton and Bastian, 2006).
67 (a)
Shoulder Pelvis Hip Limb angle
Knee
Baseline tied
Adaptation split
Postadapt tied
0.5 m/s 1.5 m/s
0.5 and 1.5 m/s
0.5 m/s
16 min
5 min
2 min
+
2 min
Ankle MT
Step symmetry
(c)
Center of oscillation shift 30
Degrees
Slow HS
0
−30
Step sym =
Phase shift 30
Slow HS Fast HS
0
−30
1s Fast step
(d)
Fast HS Degrees
(b)
1s
Slow step Fast step − Slow step Fast step + Slow step
Fast step
Slow step
Fast step
Slow step
Fig. 1. (a) Diagram of marker locations and an example of the paradigm structure. Limb angle convention is shown on the stick figure (left panel). Panel on the right shows an example experimental paradigm indicating the periods of split and tied-belt walking. The walking pattern is first recorded during a baseline period in which both treadmill belts move at the same speed. Then, changes to the walking pattern are recorded during an adaptation period in which one belt moves two to four times faster than the other. Finally, stored changes to the walking pattern are assessed during a deadaptation period in which the treadmill belts move at the same speed as in the baseline period. (b) An example of kinematic data of two consecutive steps is shown. Kinematic data for every two steps were used to calculate step symmetry, defined as the difference in step lengths normalized by the step lengths sum. (c) Figure adapted from Malone and Bastian (2010). Limb angle trajectories plotted as a function of time in late split-belt adaptation—two cycles are shown. Gray trajectory represents the movement in the slow limb in early adaptation. Positive limb angles are when the limb is in front of the trunk (flexion). Two time points are marked—slow heel strike (HS) in black and fast HS in gray. The spread between the limb angles is directly proportional to the step lengths shown in the bottom. Step lengths can be equalized by changing the position of the foot at landing (i.e., the “spatial” placement of the foot). This spatial strategy is known as a shift in the center of oscillation difference since subjects change midpoint angle around which each leg oscillates, with respect to the other leg. (d) Step lengths can also be equalized by changing the timing of foot landing, as shown by the change in phasing of the slow limb from the gray trajectory (early adaptation) to the black trajectory. This purely temporal strategy is known as phase shift since subjects equalize step lengths by changing the timing of foot landings with respect to each other.
This result is interesting and raises many issues about development of movement adaptability. First, it suggests that the nervous system gains some adaptive abilities in late childhood. This is counter to the belief that, because children are developing, they are “more plastic” and should
adapt faster. Of course, an important question is whether there are advantages to adapting slower as a child—since children adapt more slowly, do they also deadapt slower and does this make them retain more from day to day, for example? A second issue is whether this result would be
68
Step symmetry
Adaptation
Deadaptation
0 0.2
−0.1
0.1
−0.2
0
−0.3
10
3–5 yo 12–14 yo Adults
Center of oscillation
0 5 −5 0 −10 0.08
0.02
Phasing
0.06
0
0.04
−0.02
0.02
−0.04
0
−0.06
−0.02 0
100
200
300
400
−0.08 500 0 Number of steps
100
200
300
Fig. 2. Rates of adaptation (left column) and deadaptation (right column) in 3- to 5-year olds (red; n ¼ 10), 12- to 14-year olds (blue; n ¼ 10), and adults (black; n ¼ 10). Step symmetry data are shown in the top row, center of oscillation difference in the middle and phasing on the bottom. Shaded regions indicate standard error. Data were fit with linear, single-exponential, or double-exponential functions depending on which fit resulted in the highest r2 values. For 3- to 5-year-old step symmetry and center of oscillation difference, linear fits were best; double-exponential fits were best for the phasing data. A single exponential fit was used for 12to 14-year-old center of oscillation difference adaptation data and all remaining 12- to 14-year-old data were best fit by doubleexponential functions. All adult data were fit by double-exponential functions.
observed in adaptation of other kinds of movements, such as finger control. Clearly, there are differences in which brain areas are involved in these different kinds of movements. Walking heavily engages brainstem circuits, which may make its control more unique. Along this line, a third question is what neural substrates are important for adapting temporal versus spatial control of walking and do they control other movements (i.e., reaching)? We are particularly interested in knowing whether spinal circuits are involved in this adaptive process. Previously, we have shown that the cerebellum is necessary for
walking adaptation (Morton and Bastian, 2006), but have not been able to probe spinal contributions directly. Finally, do children learn better or faster when trained for longer periods of time (days rather than minutes)? This would obviously be more relevant for rehabilitation, since training is done over days to weeks. Another set of recent studies from our group has used a similar split-belt treadmill paradigm in healthy adults to explore whether we can change the rate of walking adaptation, and whether we can promote generalization of the adapted pattern to overground walking. These
69
questions are important not only to understand the adaptive process but also to determine how best to leverage this type of learning for rehabilitation. We would like to optimize the amount of adaptation, how long it lasts, and its transfer to more natural walking conditions. We first tested whether adaptation and deadaptation rates could be altered by (a) asking people to consciously correct their walking pattern to aid adaptation, or (b) distracting them with a dual task during adaptation (Malone and Bastian, 2010). Figure 3a shows the basic paradigm—subjects were tested in baseline tied-belt conditions with no instruction. We then asked each of the three groups to (1) consciously correct their step sizes to be equal by watching their feet on a video screen, (2) perform a secondary task while watching a video, or (3) simply walk with no instructions or distraction. Here, we assessed the adaptation and deadaptation rates. The deadaptation rate is perhaps more interesting in this particular study because all manipulations (e.g., distraction, conscious corrections) were removed in the deadaptation period. Figure 3b illustrates the main result from this study—adaptation and deadaptation of step symmetry were faster with conscious corrections and slower with distraction (Malone and Bastian, 2010). Thus, conscious corrections during adaptation sped the process up, but this did not lead to better retention in deadaptation. In contrast, distraction slowed the adaptation process, but resulted in better retention since deadaptation was also slower. This demonstrates that the conditions under which the nervous system learns are important, as they strongly influence the pattern of unlearning. In this work, we also found that the conscious correction and distraction effects were due to changes in the rate of adapting the spatial pattern, but not the temporal pattern (Fig. 3c and d). In other words, conscious corrections to change the step size were implemented by changing where the foot was placed, and not when it was moved there. Interestingly, distraction slowed spatial adaptation
only, despite the fact that there was no indication of what to change in this condition—subjects could have changed either the spatial or temporal components of walking. These results suggest that adaptation of spatial locomotor control is more flexible and accessible than temporal control. One interpretation of this finding is that different neural structures are involved in these two control processes, and that spatial control is more easily accessed using conscious cerebral resources. However, timing control may operate at a lower level in the nervous system, such as the brainstem or spinal cord, and is therefore less accessible through cerebral resources. The cerebellum, which is known to influence both spatial and temporal control, has projections to both cerebral motor areas and brainstem regions. Thus, there may be distinct anatomical circuits for these adaptive learning processes. These results bring up several important questions. First, does distraction lead to better day-to-day retention of newly learned movement patterns? In other words, if a person is distracted during training, will the effects last longer? Second, in rehabilitation, people are often instructed how to move and asked to “try” to move in the desired way. However, our results suggest that patients would retain more of what they learn if they do not use conscious or voluntary resources. Therefore, it is possible that a more effective rehabilitation strategy may be to put patients into a situation that drives the learning of a new pattern without having to use voluntary effort. In other words, perhaps patients would learn better if they were not “trying” so hard. Given our interest in patient rehabilitation, a third interesting question is whether similar effects of conscious correction versus distraction would be observed in patient populations. Can people who have a cerebral stroke, for example, benefit in any way from distraction? Do they even respond in the same way to conscious efforts? In sum these issues have important significance for rehabilitation of walking. Another important aspect of motor learning is how well the adapted pattern transfers to
70 Baseline tied
Adaptation split
Deadapt tied
0.5 m/s
1.5 m/s
0.5 and 1.5 m/s
0.5 m/s
2 min
2 min
16 min
5 min
control: no instructions conscious correction: feedback corrections distraction: dual task
all groups: no instructions
all groups: no instructions
Adaptation
(b)
Deadaptation
0
−0.2 −0.3 −0.4 −0.5 −0.6
(c)
400
Plateau *
300 200 100 0
Plateau 200 150 100
*
50 0
14 12
Center of oscillation difference (degrees)
Control Conscious correction Distraction
Number of strides
Step symmetry
−0.1
0.6 0.4 0.2 0 0.6 0.4 0.2 0 0.6 0.4 0.2 0
Number of strides
(a)
0
200
−5
150
−10
100
100
0
50
0
−5
0
400
*
300
10 200
8 6
−10
4
0
2
−5 −10
0
(d)
0.60
Phasing
0.56
200
400
0.50
300
0.45
200
0.40
100
0.50
50
0
0.45
0
0.58
0.54 0.52
150 100
0.40 0.50
0.50 0.48
0.45 0.40 60 120 180 240 300 360 420 480 540
30 60 90 120
Stride number
Fig. 3. (a) Experimental paradigm showing the periods of split-belt walking and conditions. In baseline, tied walking all groups were given no specific instructions. Subjects were divided into three groups for adaptation (split belts). The conscious correction group (N ¼ 11) was instructed on how to step more symmetrically and given intermittent visual feedback of their stepping during adaptation. The distraction group (N ¼ 11) was given an auditory and visual dual-task they were asked to focus on. The control
71
untrained environments or situations. The amount of transfer, or generalization, indicates how much of the adapted circuit is used in different situations. This question of generalization of device-induced motor learning across different environments has been addressed in recent studies (e.g., Berniker and Kording, 2008; Cothros et al., 2009; Kluzik et al., 2008; McVea and Pearson, 2007; Wolpert et al., 1998 ). Here, we discuss it in the context of human locomotion. Our prior work has shown that healthy subjects transfer little of the split-belt adaptation to overground walking (Reisman et al., 2009). Instead, it seems that they link the adapted pattern to the context of being on the treadmill. Given our interest in using split-belt treadmills to rehabilitate walking patterns in people with brain lesions, we wanted to understand if we could improve the generalization of split-belt treadmill adaptation to more natural walking situations. We hypothesized that treadmill walking has some unique features that provide very strong contextual cues to people as they walk on it, the main one being the mismatch between vision and proprioception. Specifically, when walking on a treadmill, proprioception tells us that we are moving, but vision tells us that we are not. This is a highly unusual situation, and the nervous system may therefore link the adapted pattern to this particular context. We tested whether removing vision during split-belt treadmill adaptation could improve overground transfer of the new walking pattern
(Torres-Oviedo and Bastian, 2010). Subjects walked with or without vision during an adaptation and transfer experiment. Figure 4a illustrates the basic paradigm—subjects walked overground and on the treadmill before and after split-belt adaptation. They were given a “catch” trial of tied-belt walking during adaptation so that we could assess how much they had learned prior to testing the transfer of adaptation effects to overground walking. Figure 4b shows individual subject data for step symmetry from periods of this experiment. Both subjects adapted, though the aftereffects during the catch trial in the subject from the no-vision group were larger than the one from the vision group, indicating that this first subject learned more. Transfer to overground walking was also markedly different between these subjects—the one without vision transferred much more than the one with vision. When subjects returned to the treadmill there was again a striking difference—the subject with no vision showed much greater washout of the adapted pattern compared to the subject with vision. Group data for step symmetry are shown in Fig. 4c–e. Similar changes were observed in phasing (i.e., temporal control). This work demonstrates that altering the sensory context can change the extent to which treadmill learning transfers to natural overground walking. We speculate that this could be for a couple of reasons. One possibility is that it changes a person's perception of the source of the error during adaptation (i.e., credit
group (N ¼ 11) was given no specific instructions. In deadaptation (tied belts), all groups walked under “Control” conditions, where the visual feedback and distracter were removed. (b) Adaptation and deadaptation curves for step symmetry. Average adaptation curves for the three groups, with standard errors indicated by the shaded area. Baseline values are subtracted out from curves (i.e., symmetry is indicated by a value of 0). Average deadaptation curves for the three groups. Recall that all groups deadapted under the same condition (no feedback or distraction). Curves are shown individually to more clearly illustrate the plateau level. Bar graphs represent group averages for adaptation and deadaptation rate, assessed by the number of strides until plateau is reached (i.e., behavior is level and stable). Note that with step symmetry, the conscious correction group adapted faster, and the Distraction group adapted slower. However, retention was improved with the Distraction group because they took longer to deadapt, despite removal of the distracter. (c) Adaptation and deadaptation curves for the center of oscillation difference. Average adaptation curves for the three groups plotted as in (b). Trends seen in the center of oscillation difference are comparable to those seen in step symmetry. (d) Average adaptation and deadaptation curves for phasing, plotted as similar to (b). Note that our interventions did not significantly affect the rate of adaptation or deadaptation of phasing.
72 (a)
Paradigm
Baseline over ground
baseline tied
self-selected speed all groups
Adaptation split
0.5 m/s
1.0 m/s
0.5 and 1.0 m/s
2 min
2 min
10 min
all groups
5 min
group1: Vision group 2: noVision
Postadapt over ground
Postadapt tied
self-selected speed
0.5 m/s
all groups
all groups
5 min
Catch (tied-belts) 0.5 m/s, 10 sec (b)
Sample subjects Baseline OG TM
Catch TM
Postadaptation OG TM Vision No-vision
Step symmetry
0.8 0.6 0.4 0.2 0 −0.2 0 30 Steps
(c)
2472 Steps
Learning
0.6
(d)
*
2640
Transfer
3780
(e)
Washout
100
100
80
80
*
% Washout
% Transfer
0.4
40
60
40
si Vi
si ovi N
si Vi
si
on si Vi
on
0 on
0 on
20
ovi
N
ov
is
io n
0
20
on
0.2
N
Step sym catch
* 60
Fig. 4. (a) Overall paradigm. In all groups, baseline behavior was recorded overground (OG) and subsequently on the treadmill with the two belts moving at 0.7 m/s. Then subjects were adapted for a total of 15 min, during which one belt was moving at 0.5 m/s and the other belt at 1 m/s. After 10 min of adaptation, a 10-s catch trial was introduced, in which both belts moved at 0.7 m/s. Subjects were readapted (i.e., belts’ ratio at 2:1) for five more minutes before they were asked to walk OG, where we tested the transfer of treadmill adaptation to natural walking. Subjects were transported on a wheelchair to a 6-m walkway where they walked back-and-forward 15 times. All steps on the walkway were recorded except for those when subjects were
73
assignment) from the treadmill to the person (Berniker and Kording, 2008). If this were the case, the person would learn to associate the newly learned calibration to one's faulty movements, rather than to being on the treadmill. A second is that closing the eyes may have led to an upweighting of proprioceptive information. It is possible that errors derived from proprioceptive signals encode learning in intrinsic (i.e., body centered) coordinates and thus learning could be more easily carried with the person when they move off of the treadmill. These results also lead to several questions. First, is it necessary to actually remove vision to improve transfer, or can this be done through other means? For example, if visual and proprioceptive information were congruent during splitbelt adaptation, would transfer to overground walking improve? We have started to study this using optic flow patterns displayed to the individuals as they walk. We can manipulate optic flow to match or oppose the proprioceptive signals and would like to be able to understand how these two sources of information are integrated. If it is important to upweight proprioceptive information from the legs to improve transfer to natural walking, adding congruent vision may not help. However, if it is important to remove the sensory mismatch and make the
adaptation context more similar to natural walking situations, then adding optic flow may improve it. Another important question is whether individuals with stroke will show a similar effect from changing the sensory context during split-belt treadmill adaptation. Our previous work has shown that people with cerebral lesions caused by stroke (e.g., middle cerebral artery distribution), can adapt their walking pattern and show better transfer to overground walking than controls (Reisman et al., 2009), even with eyes open. Will changing the visual information to match the proprioceptive inputs improve this transfer? We think that it is unrealistic to adapt stroke patients without vision and, therefore, would like to use visual displays to manipulate visual information during this task. Finally, it is not understood whether credit assignment or the ability to assign errors to the environment versus the body is developed throughout childhood. Therefore, we would like to know how children transfer split-belt treadmill adaptation. Does an immature nervous system transfer newly adapted patterns more readily? If so, does this mean that they have difficulty learning context-dependent walking calibrations? These questions are important for reaching our ultimate goal of optimizing this process for long-term training of adults and children with brain damage.
turning to return to the initial position. Finally, subjects returned to the treadmill where they walked for 5–10 min at 0.7 m/s to determine form the remaining aftereffects the extent to which walking without the device washed out the learning specific to the treadmill. (b) Spatial symmetry (i.e., symmetry in step lengths of the two legs) of sample subjects of the vision and no-vision group when walking on the treadmill (TM) and OG during baseline, catch, and deadaptation periods. Behavior of two sample subjects is shown: one walking with vision (gray trace) and one walking without vision (black trace). Lines represent the running average using a three-step window SD (shaded area). No differences in step symmetry were observed preadaptation when subjects walked with and without vision on the treadmill or OG. However, the subject that walked without vision had larger aftereffects on the treadmill during the catch trial (i.e., more learning), more transfer of treadmill learning to OG walking, and more washout of learning specific to the treadmill than subject that walked with vision. (c) Aftereffects on treadmill during catch trial for vision and no-vision groups. Subjects that trained without vision had significantly larger aftereffects—greater learning, than subjects that trained with vision. Bars’ height indicates the averaged aftereffects of the first three steps during the catch trial across subjects SE. (d) Transfer of adaptation effects to OG walking. (e) Washout of treadmill spatial aftereffects following OG walking. Removing vision during training had a significant effect on the washout of step symmetry aftereffects specific to the treadmill. Step symmetry transfer and washout are expressed as a percentage of the aftereffects on the treadmill during catch. Bars’ height indicates the average across subjects SE of % transfer and % washout for the first three steps OG or when returning to the treadmill. Figures in all panels were adapted from Torres-Oviedo and Bastian (2010). *p
Best unisensory response (mean impulses)
Fig. 4. The dependence of SC multisensory integration on cortex. The top-left figure shows the placement of cryogenic coils in the relevant cortical areas. Cold fluid circulated through the coils reduces the temperature of the surrounding cortex and inhibits activity. The top-right figure shows the area deactivated and then reactivated in this procedure (shaded region) and sample responses from a visual–auditory neuron to visual (V), auditory (A), and spatiotemporally concordant visual–auditory stimuli. Prior to cooling (control), the neuron shows an enhanced response to the visual–auditory stimulus complex. However, when cortex is cooled (deactivate AES), the multisensory response is no longer statistically greater than the best unisensory response. Reactivating cortex (reactivate AES) returns the neuron's integrative capabilities. The bottom-left figure plots the multisensory response versus the best unisensory response for a population of similar visual–auditory neurons before deactivation (green), when only one subregion of AES is deactivated (red, FAES; blue, AEV), or when both are deactivated (yellow). The bottomright plots the enhancement index (percent difference between the multisensory and best unisensory response) for these four conditions against the best unisensory response. The results of this study indicate that there is a true “synergy” between the subregions of AES cortex in producing multisensory integration in the SC: deactivating one or the other subregion often yields results equivalent to deactivating both.
152
properties of these neurons are well-studied and provide a benchmark for sensory development (Stein, 1984). The newborn cat has poor motor control and is both blind and deaf, and its SC contains only tactile-responsive neurons (Stein et al., 1973), which presumably aid the neonate in suckling (Larson and Stein, 1984). SC responses to auditory stimuli are first apparent at 5 dpn (days postnatal; Stein, 1984) and visual responses in the multisensory (deep) layers after several additional weeks (Kao et al., 1994). Obviously, prior to this time, these neurons cannot engage in multisensory integration. It is necessary to be specific about the appearance of visual sensitivity in the multisensory layers of the SC, because the overlying superficial layers, which are purely visual, develop their visual sensitivity considerably earlier (Stein, 1984; Stein et al., 1973). Although superficial layer neurons are not directly involved in multisensory processes (their function is believed to more closely approximate that of neurons in the primary projection pathway), this superficial-deep developmental lag is still somewhat surprising because superficial layer neurons provide some of the visual input to the multisensory layers (Behan and Appell, 1992; Grantyn and Grantyn, 1984; Moschovakis and Karabelas, 1985). Apparently, the functional coupling of superficial neurons with their deep layer target neurons has not yet developed. The maturational distinction between visually responsive neurons within the same structure underscores a key difference between unisensory neurons and those that will be involved in integrating inputs from different senses. The chronology of multisensory neurons parallels but is delayed with respect to the chronology of unisensory development. The earliest multisensory neurons are somatosensory-auditory, appearing at 10–12 days after birth. The first visual–nonvisual neurons take 3 weeks to appear (Kao et al., 1994; Stein et al., 1973; Wallace and Stein, 1997). However, the incidence of these multisensory neurons does not reach adult-
like proportions until many weeks later. Visual, auditory, and somatosensory receptive fields are all initially very large and contract significantly over months of development, thereby enhancing the resolution of their individual maps, the concordance among the maps, and of special importance in this context, the spatial concordance of the multiple receptive fields of individual neurons (Fig. 5). The changes are accompanied by increases in the vigor of neuronal responses to sensory stimuli, increases in response reliability, decrease in response latency, and an increase in the ability to respond to successive stimuli (Kao et al., 1994; Stein et al., 1973; Wallace and Stein, 1997). These functional changes reflect the maturation of the intrinsic circuitry of the structure, as well as the maturation and selection of its afferents resulting from selective strengthening and pruning of synapses. However, these neonatal multisensory neurons are incapable of integrating their multiple sensory inputs. SC neurons do not show multisensory integration until at least a month of age, long after they have developed the capacity to respond to more than one sensory modality (Wallace and Stein, 1997). In other words, they respond to crossmodal stimulations as if only one (typically the more effective) stimulus is present. Once multisensory integration begins to appear, only a few neurons show it at first. Gradually, more and more multisensory neurons begin to show integration, but it takes many weeks before the normal complement of neurons capable of multisensory integration is achieved. The inability of neonatal multisensory neurons to integrate their different sensory inputs is not limited to the kitten, nor is it restricted to altricial species. The Rhesus monkey is much more mature at birth than is the cat, and already has many multisensory SC neurons. Apparently, the appearance of mulitsensory neurons during development does not depend on postnatal experience, but on developmental stage, an observation we will revisit below. However, the multisensory neurons in the newborn primate,
153 Visual
Auditory
Somatosensory
Visual
22 dpn
N
20 dpn S
S
-
N
I
S
T
N
I
I
S
S
T
42 dpn S
N
T
42 dpn
N
I
T
N
I
I
S
S
T
135 dpn S
N
I
-
135 dpn
N
I
T
N
T
I
Fig. 5. Receptive fields of SC neurons shrink during development. Illustrated are the receptive fields of two types of neurons (left: visual–auditory, right: visual–somatosensory) at different developmental ages (increasing from top to bottom). As depicted, the receptive fields shrink in size as the animal ages, increasing their specificity and also their spatial concordance across the senses. Adapted from Wallace and Stein (1997).
just like those in the cat, are incapable of integrating their different sensory inputs, and, in this regard, are distinctly different than their adult counterparts (Wallace and Stein, 2001; Wallace et al., 1996). Presumably this is because they have not yet had the requisite experience with crossmodal events. Recent observations in human subjects (Gori et al., 2008; Neil et al., 2006; Putzar et al., 2007) also suggest that there is a gradual postnatal acquisition of this capability, but there is no unequivocal information regarding the newborn. However, this does not mean that newborns have no multisensory processing capabilities, only that
they cannot use crossmodal information in a synergistic way (i.e., do not engage “multisensory integration” as defined above). Those studying human development sometimes include other multisensory processes under this umbrella. The best example of this is crossmodal matching, a capacity that appears to be present early in life. However, as noted earlier, this process does not yield an integrated product. While it is clearly a multisensory process, it is not an example of multisensory integration (see Stein et al., 2010 for more discussion). But, some caution should still be exercised here, as the brains and/or behaviors of only a limited number of species
154
have been studied thus far. It may still turn out that some examples of multisensory integration, such as those involving chemical senses, and/or species that are rarely examined in the laboratory, develop prenatally and independent of experience. After all, it is probably an inherent characteristic of single-celled organisms that have multiple receptors embedded in the same membrane.
1988). Presumably, the inability of neonatal SC multisensory neurons to integrate their crossmodal inputs is because the AES–SC synaptic coupling is not properly functional (just as those from superficial layers are not). This is only a supposition, for at this point we know little about how this projection changes over time. Some of the AES inputs to the SC certainly become functional at about 1 month of age, for soon after individual SC neurons exhibit multisensory integration, this capability can be blocked by deactivating AES (see Fig. 6; Wallace and Stein, 2000). These relationships strengthen over the next few months. This is also a period during which the brain is exposed to a variety of sensory stimuli, some of which are linked to the same event and some of which are not. Crossmodal cues that are derived
How experience changes the circuit for multisensory integration Inputs from AES have already reached the multisensory SC at birth, even before its constituent neurons become multisensory (McHaffie et al.,
15–20 weeks
75
control AES deactivated **
(12)
7–10 weeks
50
(7)
* (5)
4–6 weeks
(15)
200
25
Multisensory enhancement (%)
% of deep SC sensory population
11–14 weeks
** (10)
150
(12) 100
(11) **
50
(9)
0
Multisensory integrating neurons Multisensory integrating neurons affected by AES deactivation
(8)
−50
0 0
4
6
8
10
12
14
16
20
Adult
Postnatal age (weeks) Fig. 6. The developmental appearance o f multisensory integration coincides with the development of AES–SC influences. There is a progressive increase in the percent of SC neurons exhibiting multisensory integration capabilities as revealed by the graph. Note that whenever a neuron with integrating capabilities was located, the effect of AES deactivation was examined. Regardless of age, nearly all neurons lost this capability during cryogenic block of AES activity (numbers in parentheses show the number of neurons examined). Presumably, those SC neurons that were not affected by AES blockade were dependent on adjacent areas (e.g., rostral lateral suprasylvian cortex, see Jiang et al., 2001). From Wallace and Stein (2000).
155
from the same event often occur in spatiotemporal concordance, while unrelated events are far less tightly linked in space and time. Presumably, after sufficient experience, the brain has learned the statistics of those sensory events, which, via Hebbian learning rules, have been incorporated into the neural architecture underlying the capacity to integrate different sensory inputs. Such experience provides the foundation for a principled way of perceiving and interacting with the world so that only some stimulus configurations will be integrated and yield response enhancement or response depression. Put another way, the experience leads the animal to expect that certain crossmodal physical properties covary (e.g., the timing and/or spatial location of visual and auditory stimuli) and this “knowledge” is used to craft the principles for discriminating between those stimuli derived from the same event and those derived from different events. The first test of this hypothesis was aimed at determining whether experience is essential for the maturation of this process. Visual–nonvisual experiences were precluded by rearing animals in darkness from birth to well after the maturation of multisensory integration is normally achieved (i.e., 6 months or more). Interestingly, this rearing condition did not prevent the development of visually responsive neurons. In fact, in addition to unisensory neurons, each of the crossmodal convergence and response patterns characteristic of normal animals was evident in neurons within the SC of dark-reared animals, though their incidence was slightly different (Wallace et al., 2001, 2004). This parallels the observations in monkey, which is born later in development than the cat but already has visual–nonvisual SC neurons. Visual experience is obviously not essential for the appearance of such neurons. The receptive fields of these neurons in darkreared cats were very large, more like neonatal SC neurons than those in the adult. Like neonatal neurons, they could not integrate their crossmodal inputs and their responses to crossmodal pairs
of visual–nonvisual stimuli were no more vigorous than were their responses to the best of the modality-specific component stimuli (Fig. 7). As postulated, experience with visual–nonvisual stimuli proved to be necessary to develop the capacity to engage in multisensory integration. This is also consistent with observations in human subjects who had congenital cataracts removed during early life. Their vision seemed reasonably normal, but they were compromised in their ability to integrate visual and nonvisual cues, despite having years of experience after surgical correction (Putzar et al., 2007). The next test of this hypothesis was to rear animals in conditions in which the spatiotemporal relationships of crossmodal stimuli were altered from “normal” experience, in which they are presumably in spatiotemporal concordance when derived from the same event. If crossmodal experience determines the governing principles of multisensory integration, then changing it should change the principles. This possibility was examined after rearing animals in special dark environments in which their only experience with simultaneous visual and auditory stimuli was when they were spatially displaced (Wallace and Stein, 2007). They were raised to 6 months or more in this condition and then the multisensory integration characteristics of SC neurons were examined. Similar to simply dark-reared animals, these animals possessed the full range of multisensory convergence patterns and there were many visual–nonvisual neurons. However, the properties of visually responsive neurons were atypical: their receptive fields were very large, and many were unable to integrate visual–nonvisual cues. There was, however, a sizable minority of visual–auditory neurons that were fully capable of multisensory integration, but the stimulus configurations eliciting response enhancement or no integration were significantly different from those of normally reared animals (Fig. 8). Their receptive fields, unlike those of many of their neighbors, had contracted partially, but were in
156 NORMAL
Mean impulses/trial
VA
A
V
10
A
VA
V
0
A VA
100
10 Mean impulses/trial
V
Visual–auditory
Sum 100
0
Auditory
200 +165%
5
DARK-REARED Visual
**
15
Interaction (%)
Visual–auditory
Sum 5
50
Interaction (%)
Auditory
Visual
+8% 0
V
A
VA
0
Fig. 7. Comparison between normal and dark-reared animals. The sample neuron illustrated on top was recorded from the SC of a normally reared animal. Its visual and auditory receptive fields are relatively small and in good spatial register with one another. The summary figure on the right indicates its responses to visual, auditory, and spatiotemporally concordant visual–auditory stimuli, which yields typical multisensory response enhancement. The sample neuron on the bottom was recorded from the SC of an animal reared in the dark. Its receptive fields are much larger, and while it responds to both visual and auditory stimuli, its response to a spatiotemporally concordant visual–auditory stimulus complex is statistically no greater than the response to the visual stimulus alone. Adapted from Wallace et al. (2004).
poor spatial alignment with one another. Some were totally out of register, a feature that is exceedingly rare in normal animals, but one that clearly reflects the early experience of these animals with visual–auditory cues. Most important in the current context is that those neurons integrated spatially disparate stimuli to produce response enhancement—not spatially concordant stimuli. This is because their receptive fields were misaligned and only spatially disparate stimuli could fall simultaneously within them. Taken together, the dark rearing and disparity rearing conditions demonstrate that not only is experience critical for the maturation of multisensory integration, but that the nature of the experience directs formation of the neural circuits that engage in this process. In both normal and disparity-reared animals, the basis for multisensory response enhancement is defined by early experience. Whether this reflects a simple adaptation to
specific crossmodal stimulus configurations, or the general statistics of multisensory experience, is a subject of ongoing experimentation. Parallel experiments in AES cortex revealed that multisensory integration develops more slowly in cortex than in the SC. These multisensory neurons in AES populate the border regions between its visual (AEV), auditory (FAES), and somatosensory (SIV) subregions. This is perhaps not surprising, as in general, the development of the cortex is thought to be more protracted than that of the midbrain. These multisensory cortical neurons are involved in a circuit independent of the SC, as they do not project into the cortico-SC pathway (Wallace et al., 1992). Despite this, they have properties very similar to those found in the SC. They too fail to show multisensory integration capabilities during early neonatal stages, and develop this capacity gradually, and after SC neurons (Wallace
157 Visual receptive field
A
V
A
A V
35
A V
V
35
30
30
30
25
25
25
20
20
20
15
15
15
10
10
10
5
5
5
0
0
0
Mean impulses/trial
A V
A V
ns
8
100
A V
P < 001
200
16
+144%
interaction (%)
Impulses
35
Auditory receptive field
sum ns
+16% 0
0 V
A
VA
V
A
VA
-3%
V
A
VA
Fig. 8. Rearing animals in environments with spatially disparate visual–auditory stimulus configurations yields abnormal multisensory integration. Illustrated is a sample neuron recorded from the SC of an animal reared in an environment in which simultaneous visual and auditory stimuli were always spatially displaced. This neuron developed spatially misaligned receptive fields (i.e., the visual receptive field is central while the auditory receptive field is in the periphery). When presented with spatiotemporally concordant visual–auditory stimuli in the visual (left plots) or auditory (center plots) receptive fields, the multisensory response is no larger than the largest unisensory response (the identity of which is determined by which receptive field served as the stimulus location). However, if temporally concordant but spatially disparate visual and auditory stimuli are placed within their respective receptive fields, the multisensory response shows significant enhancement. In other words, this neuron appears to integrate spatially disparate crossmodal cues as a normal animal integrates spatially concordant cues. It fails to integrate spatially concordant, just as a normal animal might fail to integrate spatially discordant cues, an apparent “reversal” of the spatial principle. Adapted from Wallace and Stein (2007).
et al., 2006). Just as is the case for SC neurons, these AES neurons also require sensory experience and fail to develop multisensory integration capabilities when animals are raised in the dark (Carriere et al., 2007). Although the above observations suggest that the development of multisensory integration in the SC and cortex is dependent on exposure to
crossmodal stimuli and its principles adapt to their configurations, they provide no insight as to the underlying circuitry governing its development and adaptation. However, for multisensory SC neurons, the cortical deactivation studies described above coupled with the maturational time course of the AES–SC projection suggests that AES cortex is likely to play a critical role.
158
Evaluating this idea began with experiments in which chronic deactivation of association cortex (both AES and its adjacent area rLS) was induced on one side of the brain for 8 weeks (between 4 and 12 weeks postnatal) during the period in which multisensory integration normally develops (Stein and Stanford, 2008; Wallace and Stein, 1997), thereby rendering them unresponsive to sensory (in particular, crossmodal) experience. The deactivation was induced with muscimol, a GABAa agonist. It was embedded in a polymer that was placed over association cortex from which it was slowly released over this period. After the polymer released its stores of muscimol or was physically removed, the cortex became active and responsive to environmental stimuli. Animals were then tested behaviorally and physiologically when adults (1 year of age), long after cortex had reactivated. These experiments are still ongoing, but preliminary results are quite clear. Their ability of these animals to detect and locate visual stimuli was indistinguishable from that of normal animals, and was equally good in both hemifields. Further, behavioral performance indicated that they significantly benefited from the presentation of spatiotemporally concordant but task-irrelevant auditory stimuli in the ipsilateral hemifield (as do normal animals). However, in the contralateral hemifield, they were abnormal: responses to spatiotemporally concordant visual–auditory stimuli were no better than when the visual stimulus was presented alone. Apparently, deactivating ipsilateral association cortex during early life disrupted the maturation of multisensory integration capabilities in the contralateral hemifield. SC neurons in these animals also appeared incapable of synthesizing spatiotemporally concordant crossmodal stimuli to produce multisensory response enhancement. These data strongly support the hypothesis that the AES–SC projection is principally engaged in the instantiation of multisensory integration in the SC. The fact that the deficits in multisensory integration were observed
long after the deprivation period, regardless of whether they were induced by dark rearing or chronic cortical deactivation, suggests that there is a “critical” or “sensitive” period for acquiring this capacity. Such a period would demarcate the period in which the capacity could be acquired. Multisensory plasticity in adulthood Despite these observations, it is possible that multisensory integration and its principles may be plastic in adulthood, but may operate on different time scales or be sensitive to different types of experience. Animals involved in the studies described above entailing chronic deactivation during early life were retained, and were available for experimentation several years later. The results were striking: whereas they had shown deficits before, now they appeared to show normal multisensory integration capabilities both in behavior and physiology on both sides of space. It is possible that experience was gradually incorporated into the AES–SC projection over such a long period of time. Another possibility is that entirely new circuits, not involving the AES, formed to support the instantiation of multisensory integration in the SC, although this seems less likely, as such circuits do not form in response to neonatal cortical ablation of cortex (Jiang et al., 2006). Ongoing experiments are investigating this issue. However, it is possible that multisensory integration might also be plastic on shorter time scales in the adult under the proper conditions. Yu et al. (2009) examined whether multisensory SC neurons in anesthetized animals would alter their responses if repeatedly presented with temporally proximal sequential crossmodal stimuli. Because the stimuli were separated by hundreds of milliseconds, they initially generated what would be generally regarded as two distinct unisensory responses (separated by a “silent” period) rather than a single integrated response. They found
159
that SC neurons rapidly adapted to the repeated presentation of these stimulus configurations (Yu et al., 2009). After only a few minutes of exposure to the repeated sequence, the initial silent period between the two responses began to be populated by impulses. Soon it appeared as if the two responses were merging (see Fig. 9). This resulted from an increase in the magnitude and duration of the first response and a shortening of the latency of the second response when they were presented in sequence. The stimuli were either visual or auditory, but it did not seem to matter which was the first or second in the sequence. Interestingly, similar sequences of stimuli belonging to the same modality did not generate the same consistent results. These observations confirmed the presence of plasticity in adult multisensory SC neurons, and revealed that this plasticity could be induced even when the animal was anesthetized. Presumably, similar changes could be induced by temporally proximate crossmodal cues like those used in studies examining multisensory integration. The results also raise the question of whether a darkreared animal's SC neurons could acquire the capacity to develop multisensory integration capabilities even after the animal is raised to adulthood in the dark. To test this possibility, Yu et al. (2010) raised animals from birth to maturity in darkness and then provided them with spatiotemporally concordant visual–auditory stimuli during daily exposure periods (Fig. 10). Once again, the animals were anesthetized during these exposure periods. Comparatively soon, SC neurons began showing multisensory integration capabilities, and the magnitude of these integrated responses increased over time to reach the level of normal adults (Yu et al., 2010). Of particular interest was the speed of acquisition of this capability. It was far more rapid than the acquisition in normally reared animals, suggesting that much of the delay in normal maturation is related to the development of the neural architecture that
encodes these experiences. Interestingly, with only a few exceptions, only those neurons that had both receptive fields encroaching on the exposure site acquired this capability. This finding indicates that the crossmodal inputs to the neuron had to be activated together for this experience to have influence; that is, the influence of experience was not generalized across the population of neurons. However, within a given neuron, the experience was generalized to other overlapping areas of the receptive field, even those that did not exactly correspond to the exposure site. It is not clear from these observations whether this is a general finding or one specific to the exact stimuli and stimulus configurations (e.g., the fact that the exposure stimuli have precise spatiotemporal relationships) used to initiate the acquisition in multisensory integration. This may be the reason that the cats given chronic cortical deactivation do not develop multisensory integration capabilities even as young adults, and humans with congenital cataracts that have undergone corrective surgery do not immediately develop this capacity (Putzar et al., 2007). Though seemingly reasonable, this supposition requires empirical validation. The continued plasticity of multisensory integration into adulthood also suggests that its characteristics may be adaptable to changes in environmental circumstances, specifically, changes in crossmodal statistics. This promises to be an exciting issue of future exploration. The possibility that it is not too late to acquire this fundamental capacity during late childhood or adulthood promises an ability to ameliorate the dysfunctions in this capacity induced by early deprivation via congenital vision and/or early hearing impairments. Perhaps by better understanding the requirements for its acquisition, better rehabilitative strategies can be developed. It may also be possible to significantly enhance the performance of people with normal developmental histories, especially in circumstances in which detection and localization of events is of critical importance.
(a)
(b)
mean response (impulses)
1.0
0 1.0
1.0
The first 15 trials
0 1.0
The last 15 trials
0 -1000
-500
The first 15 trials
0
500
1000
0 -500
1500
180
The last 15 trials
0
500
1000
1500
0
500
1000
1500
150
Sequential cross-modal exposure (trials)
150 120
100
90 60
50
30 0 -1000
-500
0
500
1000
0 -500
1500
Time after stimulus onset (ms)
(c)
(d)
mean response (impulses)
1.6
0 1.6
2
The first 15 trials
The first 15 trials
0 2
The last 15 trials
0 -500
0
500
1000
1500
0 -500
The last 15 trials
0
500
1000
1500
0
500
1000
1500
100 160 140
Sequential cross-modal exposure (Trials)
80
120 60
100 80
40 60 40
20
20 0 -500
0
500
1000
1500
0 -500
Time after stimulus onset (ms)
Fig. 9. A repeated sequence of auditory and visual stimuli in either order led to a merging of their responses. Shown are the responses of four neurons. In each display, the responses are ordered from bottom to top. The stimuli are represented by bars above the rasters: the short one refers to the auditory stimulus and the long one to the visual stimulus. The first and last series of trials (n ¼ 15 in each) are shaded in the rasters and displayed at the top as peristimulus time histograms (20 ms bin width). Arrows indicate the time period between the responses to the stimuli. Note that the period of relative quiescence between the two distinct unisensory responses is lost after a number of trials and the responses begin to merge. This is most obvious when comparing the activity in the first and last 15 trials. From Yu et al. (2009).
3600 exposure trials (30°)
(b)
A V
A
250 ms V
A
A V
37,800 exposure trials (30°)
(c) V
50,400 exposure trials (0°)
A
A V
Mean impulses/trial
V
8
Mean impulses/trial
(a)
12
Mean impulses/trial
161
12
4
0
6
0
46.0% *
V A VA
164.7% **
V A VA
224.1% **
6
0
V A VA
Fig. 10. Exposure to spatiotemporally concordant visual–auditory stimuli leads to the maturation of multisensory integration capabilities in dark-reared animals. Shown above are the receptive fields, exposure sites, and responses of three SC neurons (a–c). Left: receptive fields (visual, black; auditory, gray) are shown on schematics of visual–auditory space. The numbers below refer to crossmodal exposure trials provided before testing the neuron's multisensory integration capability. The exposure site (0 or 30 ) is also on the schematic and designated by a light gray square. Both receptive fields of each neuron overlapped the exposure site. Middle: each neuron responded to the crossmodal stimuli with an integrated response that exceeded the most robust unisensory response and, in 2/3 cases, exceeded their sum. Right: the summary bar graphs compare the average unisensory and multisensory responses. From Yu et al. (2010).
Acknowledgments The research described here was supported in part by NIH grants NS36916 and EY016716. References Alvarado, J. C., Rowland, B. A., Stanford, T. R., & Stein, B. E. (2008). A neural network model of multisensory integration also accounts for unisensory integration in superior colliculus. Brain Research, 1242, 13–23. Alvarado, J. C., Stanford, T. R., Rowland, B. A., Vaughan, J. W., & Stein, B. E. (2009). Multisensory integration in the superior colliculus requires synergy among corticocollicular inputs. The Journal of Neuroscience, 29, 6580–6592. Alvarado, J. C., Stanford, T. R., Vaughan, J. W., & Stein, B. E. (2007). Cortex mediates multisensory but not unisensory
integration in superior colliculus. The Journal of Neuroscience, 27, 12775–12786. Angelaki, D., Gu, Y., & Deangelis, G. (2011). Visual and vestibular cue integration for heading perception in extrastriate visual cortex. Journal of Physiology, 589, 825–833. Behan, M., & Appell, P. P. (1992). Intrinsic circuitry in the cat superior colliculus: Projections from the superficial layers. The Journal of Comparative Neurology, 315, 230–243. Bell, A. H., Meredith, M. A., Van Opstal, A. J., & Munoz, D. P. (2005). Crossmodal integration in the primate superior colliculus underlying the preparation and initiation of saccadic eye movements. Journal of Neurophysiology, 93, 3659–3673. Burnett, L. R., Stein, B. E., Chaponis, D., & Wallace, M. T. (2004). Superior colliculus lesions preferentially disrupt multisensory orientation. Neuroscience, 124(3), 535–547. Burnett, L., Stein, B. E., & Wallace, M. T. (2000). Ibotenic acid lesions of the superior colliculus disrupt multisensory orientation behaviors. Society for Neuroscience, Abstracts, 26, 1220. New Orleans, LA: Society for Neuroscience.
162 Calvert, G. A., Spence, C., & Stein, B. E. (2004). The handbook of multisensory processes. Cambridge, MA: MIT Press. Carriere, B. N., Royal, D. W., Perrault, T. J., Morrison, S. P., Vaughan, J. W., Stein, B. E., et al. (2007). Visual deprivation alters the development of cortical multisensory integration. Journal of Neurophysiology, 98, 2858–2867. Edwards, S. B., Ginsburgh, C. L., Henkel, C. K., & Stein, B. E. (1979). Sources of subcortical projections to the superior colliculus in the cat. The Journal of Comparative Neurology, 184, 309–329. Ernst, M. O., & Banks, M. S. (2002). Humans integrate visual and haptic information in a statistically optimal fashion. Nature, 415, 429–433. Frens, M. A., & Van Opstal, A. J. (1998). Visual-auditory interactions modulate saccade-related activity in monkey superior colliculus. Brain Research Bulletin, 46, 211–224. Fuentes-Santamaria, V., Alvarado, J. C., McHaffie, J. G., & Stein, B. E. (2009). Axon morphologies and convergence patterns of projections from different sensory-specific cortices of the anterior ectosylvian sulcus onto multisensory neurons in the cat superior colliculus. Cerebral Cortex, 19, 2902–2915. Fuentes-Santamaria, V., Alvarado, J. C., Stein, B. E., & McHaffie, J. G. (2008). Cortex contacts both output neurons and nitrergic interneurons in the superior colliculus: Direct and indirect routes for multisensory integration. Cerebral Cortex, 18, 1640–1652. Gaither, N. S., & Stein, B. E. (1979). Reptiles and mammals use similar sensory organizations in the midbrain. Science, 205, 595–597. Gillmeister, H., & Eimer, M. (2007). Tactile enhancement of auditory detection and perceived loudness. Brain Research, 1160, 58–68. Gingras, G., Rowland, B. A., & Stein, B. E. (2009). The differing impact of multisensory and unisensory integration on behavior. The Journal of Neuroscience, 29, 4897–4902. Gori, M., Del, V. M., Sandini, G., & Burr, D. C. (2008). Young children do not integrate visual and haptic form information. Current Biology, 18, 694–698. Grantyn, A., & Grantyn, R. (1984). Generation of grouped discharges by tectal projection cells. Archives of Italian Biology, 122, 59–71. Hartline, P. H., Kass, L., & Loop, M. S. (1978). Merging of modalities in the optic tectum: Infrared and visual integration in rattlesnakes. Science, 199, 1225–1229. Huerta, M. F., & Harting, J. K. (1984). The mammalian superior colliculus: Studies of its morphology and connections. In H. Vanegas (Ed.), Comparative neurology of the optic tectum (pp. 687–773). New York, NY: Plenum Publishing Corporation. Jiang, W., Jiang, H., Rowland, B. A., & Stein, B. E. (2007). Multisensory orientation behavior is disrupted by neonatal cortical ablation. Journal of Neurophysiology, 97, 557–562. Jiang, W., Jiang, H., & Stein, B. E. (2002). Two corticotectal areas facilitate multisensory orientation behavior. Journal of Cognitive Neuroscience, 14, 1240–1255.
Jiang, W., Jiang, H., & Stein, B. E. (2006). Neonatal cortical ablation disrupts multisensory development in superior colliculus. Journal of Neurophysiology, 95, 1380–1396. Jiang, W., Wallace, M. T., Jiang, H., Vaughan, J. W., & Stein, B. E. (2001). Two cortical areas mediate multisensory integration in superior colliculus neurons. Journal of Neurophysiology, 85, 506–522. Kadunce, D. C., Vaughan, J. W., Wallace, M. T., Benedek, G., & Stein, B. E. (1997). Mechanisms of within- and crossmodality suppression in the superior colliculus2. Journal of Neurophysiology, 78, 2834–2847. Kadunce, D. C., Vaughan, J. W., Wallace, M. T., & Stein, B. E. (2001). The influence of visual and auditory receptive field organization on multisensory integration in the superior colliculus1. Experimental Brain Research, 139, 303–310. Kao, C. Q., Stein, B. E., & Coulter, D. A. (1994). Postnatal development of excitatory synaptic function in deep layers of superior colliculus. Society for Neuroscience Abstract, 20, 1186. King, A. J., & Palmer, A. R. (1985). Integration of visual and auditory information in bimodal neurones in the guinea-pig superior colliculus. Experimental Brain Research, 60, 492–500. Larson, M. A., & Stein, B. E. (1984). The use of tactile and olfactory cues in neonatal orientation and localization of the nipple. Developmental Psychobiology, 17, 423–436. Lomber, S. G., Payne, B. R., & Cornwell, P. (2001). Role of the superior colliculus in analyses of space: Superficial and intermediate layer contributions to visual orienting, auditory orienting, and visuospatial discriminations during unilateral and bilateral deactivations. The Journal of Comparative Neurology, 441, 44–57. McHaffie, J. G., Kruger, L., Clemo, H. R., & Stein, B. E. (1988). Corticothalamic and corticotectal somatosensory projections from the anterior ectosylvian sulcus (SIV cortex) in neonatal cats: An anatomical demonstration with HRP and 3H-leucine. The Journal of Comparative Neurology, 274, 115–126. Meredith, M. A., Nemitz, J. W., & Stein, B. E. (1987). Determinants of multisensory integration in superior colliculus neurons. I. Temporal factors. Journal of Neuroscience, 7, 3215–3229. Meredith, M. A., & Stein, B. E. (1983). Interactions among converging sensory inputs in the superior colliculus. Science, 221, 389–391. Meredith, M. A., & Stein, B. E. (1986). Spatial factors determine the activity of multisensory neurons in cat superior colliculus. Brain Research, 365, 350–354. Meredith, M. A., & Stein, B. E. (1996). Spatial determinants of multisensory integration in cat superior colliculus neurons. Journal of Neurophysiology, 75, 1843–1857. Moschovakis, A. K., & Karabelas, A. B. (1985). Observations on the somatodendritic morphology and axonal trajectory of intracellularly HRP-labeled efferent neurons located in the deeper layers of the superior colliculus of the cat. The Journal of Comparative Neurology, 239, 276–308.
163 Neil, P. A., Chee-Ruiter, C., Scheier, C., Lewkowicz, D. J., & Shimojo, S. (2006). Development of multisensory spatial integration and perception in humans. Developmental Science, 9, 454–464. Peck, C. K., & Baro, J. A. (1997). Discharge patterns of neurons in the rostral superior colliculus of cat: Activity related to fixation of visual and auditory targets. Experimental Brain Research, 113, 291–302. Putzar, L., Goerendt, I., Lange, K., Rosler, F., & Roder, B. (2007). Early visual deprivation impairs multisensory interactions in humans. Nature Neuroscience, 10, 1243–1245. Rowland, B. A., Stanford, T. R., & Stein, B. E. (2007). A model of the neural mechanisms underlying multisensory integration in the superior colliculus. Perception, 36, 1431–1443. Royal, D. W., Carriere, B. N., & Wallace, M. T. (2009). Spatiotemporal architecture of cortical receptive fields and its impact on multisensory interactions. Experimental Brain Research, 198, 127–136. Spence, C., Pavani, F., & Driver, J. (2004). Spatial constraints on visual-tactile cross-modal distractor congruency effects. Cognitive, Affective and Behavioral Neuroscience, 4, 148–169. Stanford, T. R., & Stein, B. E. (2007). Superadditivity in multisensory integration: Putting the computation in context. Neuroreport, 18, 787–792. Stein, B. E. (1984). Development of the superior colliculus. In Annual review of neuroscience (pp. 95–125). Palo Alto, CA: Annual Reviews, Inc. Stein, B. E., Burr, D., Constantinidis, C., Laurienti, P. J., Alex, M. M., Perrault, T. J. Jr., et al. (2010). Semantic confusion regarding the development of multisensory integration: A practical solution. The European Journal of Neuroscience, 31, 1713–1720. Stein, B. E., & Gaither, N. S. (1981). Sensory representation in reptilian optic tectum: Some comparisons with mammals. The Journal of Comparative Neurology, 202, 69–87. Stein, B. E., Huneycutt, W. S., & Meredith, M. A. (1988). Neurons and behavior: The same rules of multisensory integration apply. Brain Research, 448, 355–358. Stein, B. E., Labos, E., & Kruger, L. (1973). Sequence of changes in properties of neurons of superior colliculus of the kitten during maturation. Journal of Neurophysiology, 36, 667–679. Stein, B. E., & Meredith, M. A. (1993). The merging of the senses. Cambridge, MA: MIT Press. Stein, B. E., & Stanford, T. R. (2008). Multisensory integration: Current issues from the perspective of the single neuron. Nature Reviews. Neuroscience, 9, 255–266. Stein, B. E., Stanford, T. R., Ramachandran, R., Perrault, T. J., Jr., & Rowland, B. A. (2009). Challenges in quantifying multisensory integration? Alternative criteria, models, and inverse effectiveness. Experimental Brain Research, 198, 113–126.
Wallace, M. T., Carriere, B. N., Perrault, T. J., Jr. Vaughan, J. W., & Stein, B. E. (2006). The development of cortical multisensory integration. The Journal of Neuroscience, 26, 11844–11849. Wallace, M. T., Hairston, W. D., & Stein, B. E. (2001). Longterm effects of dark-rearing on multisensory processing. Program No. 511.6. Wallace, M. T., Meredith, M. A., & Stein, B. E. (1992). Integration of multiple sensory modalities in cat cortex. Experimental Brain Research, 91, 484–488. Wallace, M. T., Meredith, M. A., & Stein, B. E. (1993). Converging influences from visual, auditory, and somatosensory cortices onto output neurons of the superior colliculus. Journal of Neurophysiology, 69, 1797–1809. Wallace, M. T., Perrault, T. J., Jr., Hairston, W. D., & Stein, B. E. (2004). Visual experience is necessary for the development of multisensory integration1. The Journal of Neuroscience, 24, 9580–9584. Wallace, M. T., & Stein, B. E. (1994). Cross-modal synthesis in the midbrain depends on input from cortex. Journal of Neurophysiology, 71, 429–432. Wallace, M. T., & Stein, B. E. (1997). Development of multisensory neurons and multisensory integration in cat superior colliculus. The Journal of Neuroscience, 17, 2429–2444. Wallace, M. T., & Stein, B. E. (2000). Onset of cross-modal synthesis in the neonatal superior colliculus is gated by the development of cortical influences. Journal of Neurophysiology, 83, 3578–3582. Wallace, M. T., & Stein, B. E. (2001). Sensory and multisensory responses in the newborn monkey superior colliculus. The Journal of Neuroscience, 21, 8886–8894. Wallace, M. T., & Stein, B. E. (2007). Early experience determines how the senses will interact. Journal of Neurophysiology, 97, 921–926. Wallace, M. T., Wilkinson, L. K., & Stein, B. E. (1996). Representation and integration of multiple sensory inputs in primate superior colliculus. Journal of Neurophysiology, 76, 1246–1266. Wilkinson, L. K., Meredith, M. A., & Stein, B. E. (1996). The role of anterior ectosylvian cortex in cross-modality orientation and approach behavior. Experimental Brain Research, 112, 1–10. Yu, L., Rowland, B. A., & Stein, B. E. (2010). Initiating the development of multisensory integration by manipulating sensory experience. The Journal of Neuroscience, 30, 4904–4913. Yu, L., Stein, B. E., & Rowland, B. A. (2009). Adult plasticity in multisensory neurons: Short-term experience-dependent changes in the superior colliculus. The Journal of Neuroscience, 29, 15910–15922. Zahar, Y., Reches, A., & Gutfreund, Y. (2009). Multisensory enhancement in the optic tectum of the barn owl: Spike count and spike timing. Journal of Neurophysiology, 101, 2380–2394.
A. M. Green, C. E. Chapman, J. F. Kalaska and F. Lepore (Eds.) Progress in Brain Research, Vol. 191 ISSN: 0079-6123 Copyright Ó 2011 Elsevier B.V. All rights reserved.
CHAPTER 11
Multisensory object representation: Insights from studies of vision and touch Simon Lacey{ and K. Sathian{,{,},},* { }
{ Department of Neurology, Emory University, Atlanta, Georgia, USA Department of Rehabilitation Medicine, Emory University, Atlanta, Georgia USA } Department of Psychology, Emory University, Atlanta, Georgia, USA Rehabilitation R&D Center of Excellence, Atlanta VAMC, Decatur, Georgia, USA
Abstract: Behavioral studies show that the unisensory representations underlying within-modal visual and haptic object recognition are strikingly similar in terms of view- and size-sensitivity, and integration of structural and surface properties. However, the basis for these attributes differs in each modality, indicating that while these representations are functionally similar, they are not identical. Imaging studies reveal bisensory, visuo-haptic object selectivity, notably in the lateral occipital complex and the intraparietal sulcus, that suggests a shared representation of objects. Such a multisensory representation could underlie visuo-haptic cross-modal object recognition. In this chapter, we compare visual and haptic within-modal object recognition and trace a progression from functionally similar but separate unisensory representations to a shared multisensory representation underlying cross-modal object recognition as well as view-independence, regardless of modality. We outline, and provide evidence for, a model of multisensory object recognition in which representations are flexibly accessible via top-down or bottom-up processing, the choice of route being influenced by object familiarity and individual preference along the object–spatial continuum of mental imagery. Keywords: haptic; cross-modal; visual imagery; fMRI. by Klatzky, Lederman, and their colleagues (e.g., Klatzky and Lederman, 1995; Klatzky et al., 1985, 1987; Lederman and Klatzky, 1987), there is now a substantial literature on haptic object recognition. The representations underlying visual and haptic within-modal object recognition are strikingly similar: each, for example, is sensitive to changes in orientation, size, and surface properties. However,
Introduction Object recognition research has typically focused on the visual modality but, following pioneering work
*Corresponding author. Tel.: þ1-404-712-1366; Fax: þ1-404-727-3157 E-mail:
[email protected] DOI: 10.1016/B978-0-444-53752-2.00006-0
165
166
these similarities should not be taken to mean that vision and haptics necessarily access only a single, shared representation: the basis for similar representational characteristics differs between modalities. It is a first theme of this chapter that, though functionally similar, within-modal visual and haptic recognition are supported at a basic level by separate, unisensory representations. Vision and haptics do, however, converge on a shared representation in the service of higher-order recognition. We will review studies indicating that a single, shared representation supports within-modal, view-independent recognition in both vision and touch, and also visuo-haptic, cross-modal recognition. A shared representation suggests a shared neural substrate between vision and touch; this and the implications for the nature of the underlying representation constitute a second theme of this chapter. The final part of the chapter links these earlier themes by outlining a model of multisensory object recognition in which visuo-haptic access to multisensory representations is modulated by object familiarity and individual differences on the object–spatial dimension of mental imagery. (a)
View-dependence A major challenge for object recognition is achieving perceptual constancy, which insulates it from potentially disruptive transformations in the sensory input caused by changes in orientation, size, etc. Visual object recognition is viewdependent under certain circumstances, since rotating an object away from its original orientation impairs subsequent recognition (see Peissig and Tarr, 2007, for a review). Although the hands can explore an object from different sides simultaneously and therefore might be expected to be capable of acquiring information about different “views” at the same time, several studies have now shown, counter-intuitively, that haptic object recognition is also view-dependent (Craddock and Lawson, 2008, 2010; Lacey et al., 2007, 2009a; Lawson, 2009; Newell et al., 2001). The extent to which visual recognition is impaired by changes in orientation appears to depend on the axis around which an object is rotated (Gauthier et al., 2002; Lacey et al., 2007). Rotations in depth, about the x- and y-axes (Fig. 1), have more (b)
y-axis x-axis z-axis (c)
(d)
Fig. 1. Example 3D unfamiliar object shown in the original orientation (a) and rotated 180º about the z-axis (b), x-axis (c), and y-axis (d): rotation about the x- and y-axes are rotations in depth, rotation about the z-axis is a rotation in the picture plane. From Lacey et al., 2007.
167
disruptive effects than rotations in the picture plane, about the z-axis, resulting in slower and less accurate visual recognition for both 2D (Gauthier et al., 2002) and 3D stimuli (Lacey et al., 2007). However, haptic recognition is equally impaired by rotation about any axis (Lacey et al., 2007). This suggests that, although visual and haptic recognition are similar in being view-dependent, the basis for this is different in each modality. A possible explanation is that, unless the observer physically changes position relative to an object (e.g., Pasqualotto et al., 2005; Pasqualotto and Newell, 2007), a change in orientation typically means that visual recognition has to contend not only with a transformation in the perceptual shape but also with the occlusion of one or more surfaces. For example, compare Fig. 1a to c: here, rotation about the x-axis means that the object is turned upside-down and that the former top surface becomes occluded. But the hands are free to move over all surfaces of an object, and to manipulate it into different orientations relative to the hand, so that no surface is necessarily occluded in any given orientation. Haptic recognition therefore only has to deal with a shape transformation; thus, no single axis of rotation should be more or less disruptive than another due to surface occlusion. Further work is required to examine whether this explanation is, in fact, correct. The studies reviewed so far have largely concentrated on unfamiliar objects. As objects become more familiar, however, visual recognition becomes view-independent (Bülthoff and Newell, 2006; Tarr and Pinker, 1989). Many familiar objects are typically seen in one particular orientation known as a canonical view, for example, the front view of a house (Palmer et al., 1981). View-independence may hold for a range of orientations around the canonical view, but when objects are presented in radically noncanonical views, for example, an upside-down house, visual recognition can be impaired (Bülthoff and Newell, 2006; Palmer et al., 1981; Tarr and Pinker, 1989). Similarly, haptic
recognition of familiar objects is also view-independent, with unusual, non-canonical orientations incurring a cost in that they are recognized more slowly (Craddock and Lawson, 2008). However, what constitutes a canonical view depends on the modality: visually, a 3/4 view is preferred in which the object is aligned at 45 to the observer (Palmer et al., 1981). But in haptic canonical views, objects are generally aligned either parallel or orthogonal to the body midline (Woods et al., 2008). Canonical views may facilitate viewindependent recognition either because they provide the most structural information about an object or because they most closely match a stored representation, but the end result is the same for both vision and haptics (Craddock and Lawson, 2008; Woods et al., 2008). Haptic representations of familiar objects also maintain object constancy across changes in orientation even where there is a change in the hand used to explore the object (Craddock and Lawson, 2009a). In contrast to within-modal recognition, we found that visuo-haptic cross-modal recognition is view-independent even for unfamiliar objects that are highly similar and lack distinctive parts (see Fig. 1), regardless of the axis of rotation and whether visual study is followed by haptic test or vice versa (Lacey et al., 2007, 2010b). Cross-modal view-independence has also been demonstrated for familiar objects when haptic study was followed by visual test, although recognition was viewdependent in the reverse condition (Lawson, 2009). The reason for this asymmetry is not clear, but the familiar objects employed were a mixture of scale models of larger objects (e.g., bed, bath, and shark) and more or less actual size objects (e.g., jug, pencil). Possibly, some of these objects might have been more familiar visually than haptically, contributing to uncertainty when visually familiar objects had to be recognized by touch. To the best of our knowledge, however, the effect of differential familiarity depending on modality has not been investigated. There are two ways in which cross-modal viewindependence could arise. The simplest possibility
168
is that the view-dependent visual and haptic unisensory representations are directly integrated into a view-independent multisensory representation (Fig. 2a). Alternatively, cross-modal view-independence might be gated through unisensory view-independent representations, separately in vision and haptics (Fig. 2b). We sought to distinguish between these two possibilities in a perceptual learning study (Lacey et al., 2009a). First, we established that view-independence induced by learning in one modality transferred completely and symmetrically to the other; thus, within-modal view-independence, whether visual or haptic, is supported by a single view-independent representation. Second, we addressed whether this representation was the same as that underlying cross-modal (a)
Bisensory (visuo-haptic) View-independent
Unisensory (visual) View-dependent
(b)
Unisensory (haptic) View-dependent
Bisensory (visuo-haptic) View-independent
Unisensory (visual) View-independent
Unisensory (haptic) View-independent
Unisensory (visual) View-dependent
Unisensory (haptic) View-dependent
Fig. 2. Visuo-haptic view-independence might be (a) derived from directly integrating the unisensory view-dependent representations, alternatively (b) unisensory viewindependence might be necessary for bisensory viewindependence. From Lacey et al. (2009a,b).
view-independence. Since both visual and haptic within-modal view-independence were acquired following training on cross-modal object recognition (whether haptic-visual or visual-haptic), we concluded that visuo-haptic view-independence relies on a single multisensory representation that directly integrates the unisensory view-dependent representations (Fig. 2a), similar to models that have been proposed for vision (Riesenhuber and Poggio, 1999). Thus, the same representation underlies both cross-modal recognition and viewindependence, even if view-independence is tested within-modally.
Size-dependence In addition to achieving object constancy across orientation changes, the visual system also has to recognize the physical size of an object across variations in the size of the retinal image that arise from changing object–observer distance: the same object can produce retinal images that vary in size depending on whether it is near to or far from the observer. Presumably, this is compensated by cues arising from depth or motion perception, accounting for the fact that a change in size does not disrupt visual object identification (Biederman and Cooper, 1992; Uttl et al., 2007). However, size change does produce a cost in visual recognition for both unfamiliar (Jolicoeur, 1987) and familiar objects (Jolicoeur, 1987; Uttl et al., 2007). Interestingly, changes in retinal size due to movement of the observer result in better size constancy than those due to movement of the object (Combe and Wexler, 2010). Haptic perception of size is a product of both cutaneous (contact area and force) and proprioceptive (finger spread and position) information at initial contact (Berryman et al., 2006). Integration of these information sources achieves initial size constancy in that we do not perceive the size of an object as changing simply by gripping it harder, which increases contact area, or altering the spread of the fingers (Berryman et al., 2006),
169
thus, in touch, physical size is perceived directly. Thus, both haptic (Craddock and Lawson, 2009b,c) and cross-modal (Craddock and Lawson, 2009c) recognition are size-dependent. Haptic representations may store a canonical size for familiar objects, as has recently been proposed for visual representations (Konkle and Oliva, 2011), and deviations from this could impair recognition. Further work is required to examine this and to investigate perceived object constancy across size changes of unfamiliar objects.
Integration of structural and surface properties Although visual shape, color, and texture are processed in different cortical areas (Cant et al., 2009; Cant and Goodale, 2007), behavioral evidence suggests that visual object representations integrate structural and surface properties because changing the color of an object or its part-color combinations between study and test resulted in longer response times for a shape recognition task (Nicholson and Humphrey, 2003). Since altering the background color against which objects were presented did not impair recognition, this effect could be isolated to the object representation and indicated that this contained information about color as well as shape (Nicholson and Humphrey, 2003). Both visual and haptic within-modal object discrimination are impaired when surface texture is altered, showing first that information about surface properties in visual representations is not limited to modality-specific properties like color, and second that haptic representations also integrate structural and surface properties (Lacey et al., 2010b). However, the question whether surface properties are integrated into the multisensory representation underlying cross-modal object discrimination does not have a straightforward answer. We conducted a study requiring object discrimination across changes in orientation (in order to ensure that participants were accessing the view-independent multisensory
representation), texture, or both. While object discrimination was view-independent when texture did not change, replicating earlier findings, performance reduced to chance levels with a change in texture, whether orientation also changed or not (Lacey et al., 2010b). However, performance was heterogeneous, with some participants being more affected by the texture changes than others. We wondered whether this reflected the recent description of two kinds of visual imagery: “object imagery” (images that are pictorial and deal with the literal appearance of objects in terms of shape, color, brightness, etc.) and “spatial imagery” (more schematic images dealing with the spatial relations of objects, their component parts, and their spatial transformations; Blajenkova et al., 2006; Kozhevnikov et al., 2002, 2005). Both visual and haptic imagery can potentially be subdivided into object imagery dealing with the appearance or feel of objects, and spatial imagery dealing with spatial relationships between objects or between parts of objects. Hence, this object–spatial dimension of imagery might also apply to haptically derived representations. A major difference between object and spatial imagery is that the former includes surface property information while the latter does not. Further analysis of the texture-change condition showed that performance was indeed related to imagery preference such that object imagers were more likely to be impaired by texture changes than spatial imagers; thus, surface properties are likely only integrated into the multisensory representation by object imagers (Lacey et al., 2011). In a follow-up experiment, we asked participants to discriminate shape across changes in texture and texture across changes in shape in both visual and haptic within-modal conditions. As expected, spatial imagers were able to discriminate shape despite concomitant changes in texture but not vice versa, presumably because they abstract away from surface properties. By contrast, object imagers could discriminate texture despite concomitant changes in shape, but not
170
the reverse. Importantly, there was no significant difference between visual and haptic performance on either task. Thus, we concluded that the object–spatial imagery dimension occurs in haptics as well as vision, and that individual variations along this dimension affect the extent to which surface properties are integrated into object representations (Lacey et al., 2011).
Multisensory cortical processing Many studies have shown that visual cortical areas are functionally involved during haptic tasks (reviewed in Amedi et al., 2005; Sathian and Lacey, 2007). The lateral occipital complex (LOC) in the ventral visual pathway responds selectively to objects (Malach et al., 1995) and a subregion responds selectively to objects in both vision and touch (Amedi et al., 2001, 2002; Stilla and Sathian, 2008). Tactile responsiveness in the LOC has been found for both 3D (Amedi et al., 2001; Stilla and Sathian, 2008; Zhang et al., 2004) and 2D stimuli (Prather et al., 2004; Stoesz et al., 2003). The LOC does not respond during conventional auditory object recognition triggered by object-specific sounds (Amedi et al., 2002), but it does respond to shape information created by a visual–auditory sensory substitution device (SSD) (Amedi et al., 2007). SSDs convert visual information into an auditory stream or “soundscape” via a specific algorithm that conveys the visual horizontal axis via auditory time and stereo panning, the visual vertical axis by varying tone frequency, and pixel brightness by varying tone loudness: both sighted and blind humans can learn to recognize objects by extracting shape information from such soundscapes (Amedi et al., 2007). However, for participants trained in the use of the SSD, the LOC only responds to soundscapes created according to the algorithm and not to soundscapes associated arbitrarily with specific objects through learning (Amedi et al., 2007). Thus, LOC can be regarded as processing
geometric shape information, regardless of the sensory modality used to acquire it. Parietal cortical regions also show multisensory shape-selectivity: In particular, the intraparietal sulcus (IPS) is involved in perception of both the shape and location of objects, with coactivation of LOC during shape discrimination and the frontal eye fields during location discrimination (Sathian et al., 2011; Stilla and Sathian, 2008). Visuo-haptic shape selectivity has also been reported in the postcentral sulcus (PCS; Stilla and Sathian, 2008), corresponding to BA2 of primary somatosensory cortex (S1; Grefkes et al., 2001). This is an area that is generally thought to be exclusively somatosensory; however, our observation of multisensory shape selectivity in this region (Stilla and Sathian, 2008) is consistent with earlier neurophysiological studies that suggested visual responsiveness in parts of S1 (Iriki et al., 1996; Zhou and Fuster, 1997). Some case studies suggest that multisensory convergence in the LOC is necessary for both visual and haptic shape perception. For example, one patient with bilateral lesions of the LOC was unable to recognize novel objects by either vision or touch (James et al., 2006) while another, with a lesion to the left occipito-temporal cortex that likely included the LOC, exhibited both tactile and visual agnosia although somatosensory cortex and basic somatosensation were spared (Feinberg et al., 1986). Another patient with a ventrolateral somatosensory lesion showed tactile but not visual agnosia (Reed et al., 1996). These case studies are consistent with the behavioral literature reviewed above, indicating the existence of separate visual and haptic unisensory representations, with evidence for the shared multisensory representation being in the LOC. An important question is whether multisensory responses in the LOC and elsewhere reflect visuo-haptic integration at the neuronal level or separate, interdigitated populations of unisensory neurons receiving information from either visual or haptic inputs. To investigate this, Tal and
171
Amedi (2009) used an fMRI-based adaptation paradigm (fMR-A). fMR-A takes advantage of the repetition suppression effect, that is, attenuation of the blood-oxygen-level-dependent signal when the same stimulus is repeated; since similar repetition suppression is observed in single neurons, this method can be used to reveal neuronal selectivity profiles (reviewed by Grill-Spector et al., 2006; Krekelberg et al., 2006). Robust cross-modal adaptation from vision to touch was observed not only in the LOC and anterior IPS but also bilaterally in the precentral sulcus (corresponding to ventral premotor cortex) and the right anterior insula, suggesting that these areas have multisensory responses at the neuronal level. Multisensory regions which did not show fMR-A included the PCS and posterior parts of the IPS, suggesting that multisensory convergence in these zones arises from separate unisensory populations. Note, however, the concern that fMR-A effects may not necessarily reflect neuronal selectivity (Mur et al., 2010); thus, converging evidence using other methods would be helpful to confirm the conclusions from the study of Tal and Amedi (2009). The cortical localization of the multisensory, view-independent representation is not known. For visual stimuli, the LOC has been reported to show view-dependent responses in some studies and view-independent responses in other studies. As might be expected, view-dependence has been observed for unfamiliar objects (Gauthier et al., 2002) and view-independence for familiar objects (Eger et al., 2008a; Pourtois et al., 2009; Valyear et al., 2006). However, view-dependence has been found in the LOC even for familiar objects in one study, although in this study there was positionindependence (Grill-Spector et al., 1999), whereas another study found view-independence for both familiar and unfamiliar objects (James et al., 2002a). A recent study using transcranial magnetic stimulation (TMS) suggests that the LOC is causally involved in view-independent recognition, at least for 2D shape (Silvanto et al., 2010). However, further work is required to substantiate this since
only two rotations (20 and 70 ) were tested and TMS effects were only observed for the smaller rotation. View-dependent responses to visual stimuli have been reported in parietal regions, for example, the IPS (James et al., 2002a) and parieto-occipital junction (PO; Valyear et al., 2006). This is perhaps not surprising since these regions are in the dorsal pathway, which is concerned more with object location and sensory processing for action rather than object identity and sensory processing for perceptual recognition (Goodale and Milner, 1992; Ungerleider and Mishkin, 1982). Changes in orientation might be expected to affect associated actions, and indeed, the lateral PO demonstrates view-dependent responses for graspable, but not for nongraspable objects (Rice et al., 2007). Superior parietal cortex exhibits view-dependent responses during mental rotation but not during visual object recognition (Gauthier et al., 2002). Haptic and multisensory processing of stimuli across changes in orientation have not been examined in regard to cortical responses. Although James et al. (2002b) varied object orientation, their study concentrated on haptic-to-visual priming effects rather than the cross-modal response to same versus different orientations. It will be interesting to examine the effect of orientation changes when shape information is derived from the auditory soundscapes produced by SSDs and also when orientation changes alter the affordances and possibilities for haptically interacting with an object. Although haptic and multisensory processing of stimuli across changes in size have also not been investigated, visual size-independence has been consistently observed in the LOC (Eger et al., 2008a,b; Ewbank et al., 2005; Grill-Spector et al., 1999), with anterior regions showing more size-independence than posterior regions (Eger et al., 2008b; Sawamura et al., 2005). What role does visual imagery play? Haptic activation of visual cortex might arise either because haptic exploration of an object
172
evokes visual imagery (Sathian et al., 1997), presumably processed in the LOC, or because the LOC can be directly activated by somatosensory input. A recent electrophysiological study of tactile discrimination of simple geometric shapes applied to the fingerpad shows that activity propagates from somatosensory cortex to LOC as early as 150 ms after stimulus onset, a time frame which is consistent with “bottom-up” somatosensory projections to the LOC (Lucan et al., 2010). In addition, a recent case study examined a patient with visual agnosia arising from bilateral ventral occipito-temporal lesions, but with sparing of the dorsal part of the LOC that likely included the multisensory subregion (Allen and Humphreys, 2009). This patient's haptic object recognition was intact and was associated with activation of the intact dorsal part of the LOC, suggesting that this region can be activated directly by somatosensory input (Allen and Humphreys, 2009). Consistent with the visual imagery hypothesis, however, many studies have demonstrated LOC activity during visual imagery. For example, left LOC activity was observed in both blind and sighted participants during auditorily cued mental imagery of familiar object shape, where shape information would stem mainly from haptic experience in the case of the blind and mainly from visual experience in the sighted (De Volder et al., 2001). The left LOC is also active during a task requiring retrieval of geometric and material object properties from memory (Newman et al., 2005). In the right LOC, individual differences in haptic shape-selective activation magnitude were highly correlated with individual differences in ratings of visual imagery vividness (Zhang et al., 2004). By contrast, a lesser role for visual imagery has been suggested because LOC activity was substantially lower during visual imagery compared to haptic shape perception (Amedi et al., 2001), although there was no attempt to verify that participants maintained their images online during the imaging session. Some researchers have concluded that visual imagery does not explain
haptically evoked LOC activity because early- as well as late-blind individuals show shape-related LOC activation via both touch (reviewed in Pascual-Leone et al., 2005; Sathian, 2005; Sathian and Lacey, 2007) and hearing using SSDs (Amedi et al., 2007; Arno et al., 2001; Renier et al., 2004, 2005). While this argument is clearly true for the early blind, it does not necessarily exclude visual imagery as an explanation in the sighted, given the extensive evidence for cross-modal plasticity demonstrated in studies of visual deprivation (reviewed in Pascual-Leone et al., 2005; Sathian, 2005; Sathian and Lacey, 2007). The weight of the evidence is therefore that visual imagery is likely involved; in the next section, we show that this involvement depends on an interaction with object familiarity.
A model of multisensory object representation In this section, we draw together some of the threads reviewed earlier by outlining, and reviewing the evidence for, a preliminary conceptual model of visuo-haptic multisensory object representation that we detailed previously (Lacey et al., 2009b). In this model, the LOC contains a representation of object form that can be flexibly accessed either bottom-up or top-down, independently of the input modality, but with the choice of the bottom-up versus top-down route depending on object familiarity. For haptic recognition of unfamiliar objects, global shape has to be computed by exploring the entire object and relating the component parts to one another. The model therefore incorporates a bottom-up pathway from somatosensory cortex to the LOC, together with recruitment of the IPS to compute the spatial relationships between component parts and arrive at global shape, facilitated by spatial imagery processes. For haptic recognition of familiar objects, global shape can be computed more easily, perhaps from only a subset of parts or one diagnostic part, and we suggest that haptic sensing rapidly acquires enough information to
173
trigger a visual image of the object and generate a hypothesis about its identity, as has been proposed for visual sensing (e.g., Bar, 2007). The model therefore calls for top-down processing from prefrontal cortex into the LOC, associated with object imagery processes (this does not, however, exclude spatial imagery for familiar objects, e.g., in enabling view-independent recognition). In a pair of recent papers that provide evidence for this model, we examined the roles of visual object imagery and object familiarity in haptic shape perception using analyses of correlations of activation magnitude between visual object imagery and haptic shape perception (Lacey et al., 2010a) and analyses of effective connectivity (Deshpande et al., 2010). In the imagery task, participants heard pairs of words and decided whether the objects represented by those words had similar or different shapes. In contrast to previous studies, this ensured that participants were engaging in visual imagery throughout the scan, verifiable by reference to their recorded task performance. In a separate session, participants performed a haptic shape discrimination task using either familiar or unfamiliar objects. If haptic perception of shape depends on visual object imagery, we expected that activation magnitudes during the imagery task would be correlated with activation magnitudes during haptic perception of shape, with effective connectivity analyses showing similar top-down networks for object imagery and haptic shape perception. However, a lack of correlation between activation magnitudes for visual object imagery and haptic shape perception, and different networks in the two tasks reflecting bottom-up paths for haptic shape perception but top-down paths for imagery, would argue against imagery mediation of haptic shape perception. We found that object familiarity modulated both intertask correlations and effective connectivity. Visual object imagery and both haptic shape perception tasks activated the LOC bilaterally; familiar but not unfamiliar haptic shape perception recruited prefrontal cortical areas. Importantly, imagery activation
magnitudes in the LOC correlated with those for haptic shape perception for familiar, but not unfamiliar objects, and there were more regions showing such intertask correlations with imagery for the familiar compared to the unfamiliar shape task (Lacey et al., 2010a). The effective connectivity analyses showed similar networks for visual object imagery and haptic perception of familiar shape, dominated by top-down connections from prefrontal cortical regions into the LOC (Deshpande et al., 2010). Haptic perception of unfamiliar shape engaged a network that was very different from either of these; consistent with earlier connectivity analyses (Deshpande et al., 2008; Peltier et al., 2007), this network showed mainly bottom-up connections to the LOC from somatosensory cortex (PCS) (Deshpande et al., 2010). Thus, we have evidence for the first part of the model; in ongoing work, we are investigating the interaction between object familiarity and spatial imagery in order to provide similar evidence for the second part.
Summary In this chapter, we have shown that visual and haptic within-modal object recognition initially rely on separate representations that are nonetheless functionally similar in being view- and size-dependent. Further work is required to investigate the different mechanisms by which these similarities arise in each modality. These unisensory representations feed forward into a multisensory, view-independent representation that supports cross-modal object recognition. Here, further work is needed to examine the neural basis of both haptic and cross-modal view-independence. Finally, cross-modal object recognition depends on complex interactions between modalities, object and spatial dimensions of imagery, and object familiarity. A fruitful avenue for future research will be to examine how these interactions differ between sighted and blind individuals.
174
Acknowledgments Support to K. S. from the National Eye Institute at the NIH, the National Science Foundation, and the Veterans Administration is gratefully acknowledged. References Allen, H. A., & Humphreys, G. W. (2009). Direct tactile stimulation of dorsal occipito-temporal cortex in a visual agnosic. Current Biology, 19, 1044–1049. Amedi, A., Jacobson, G., Hendler, T., Malach, R., & Zohary, E. (2002). Convergence of visual and tactile shape processing in the human lateral occipital complex. Cerebral Cortex, 12, 1202–1212. Amedi, A., Malach, R., Hendler, T., Peled, S., & Zohary, E. (2001). Visuo-haptic object-related activation in the ventral visual pathway. Nature Neuroscience, 4, 324–330. Amedi, A., Stern, W. M., Camprodon, J. A., Bermpohl, F., Merabet, L., Rotman, S., et al. (2007). Shape conveyed by visual-to-auditory sensory substitution activates the lateral occipital complex. Nature Neuroscience, 10, 687–689. Amedi, A., von Kriegstein, K., van Atteveldt, N. M., Beauchamp, M. S., & Naumer, M. J. (2005). Functional imaging of human crossmodal identification and object recognition. Experimental Brain Research, 166, 559–571. Arno, P., De Volder, A. G., Vanlierde, A., WanetDefalque, M.-C., Streel, E., Robert, A., et al. (2001). Occipital activation by pattern recognition in the early blind using auditory substitution for vision. Neuroimage, 13, 632–645. Bar, M. (2007). The proactive brain: Using analogies and associations to generate predictions. Trends in Cognitive Sciences, 11, 280–289. Berryman, L. J., Yau, J. M., & Hsiao, S. S. (2006). Representation of object size in the somatosensory system. Journal of Neurophysiology, 96, 27–39. Biederman, I., & Cooper, E. E. (1992). Size invariance in visual object priming. Journal of Experimental Psychology: Human Perception and Performance, 18, 121–133. Blajenkova, O., Kozhevnikov, M., & Motes, M. A. (2006). Object-spatial imagery: A new self-report imagery questionnaire. Applied Cognitive Psychology, 20, 239–263. Bülthoff, I., & Newell, F. N. (2006). The role of familiarity in the recognition of static and dynamic objects. Progress in Brain Research, 154, 315–325. Cant, J. S., Arnott, S. R., & Goodale, M. A. (2009). fMR-adaptation reveals separate processing regions for the perception of form and texture in the human ventral stream. Experimental Brain Research, 192, 391–405.
Cant, J. S., & Goodale, M. A. (2007). Attention to form or surface properties modulates different regions of human occipitotemporal cortex. Cerebral Cortex, 17, 713–731. Combe, E., & Wexler, M. (2010). Observer movement and size constancy. Psychological Science, 21, 667–675. Craddock, M., & Lawson, R. (2008). Repetition priming and the haptic recognition of familiar and unfamiliar objects. Perception and Psychophysics, 70, 1350–1365. Craddock, M., & Lawson, R. (2009a). Do left and right matter for haptic recognition of familiar objects? Perception, 38, 1355–1376. Craddock, M., & Lawson, R. (2009b). The effect of size changes on haptic object recognition. Attention, Perception and Psychophysics, 71, 910–923. Craddock, M., & Lawson, R. (2009c). Size-sensitive perceptual representations underlie visual and haptic object recognition. PLoS ONE, 4, e8009. doi:10.1371/journal.pone.0008009. Craddock, M., & Lawson, R. (2010). The effects of temporal delay and orientation on haptic object recognition. Attention, Perception and Psychophysics, 72, 1975–1980. De Volder, A. G., Toyama, H., Kimura, Y., Kiyosawa, M., Nakano, H., Vanlierde, A., et al. (2001). Auditory triggered mental imagery of shape involves visual association areas in early blind humans. Neuroimage, 14, 129–139. Deshpande, G., Hu, X., Lacey, S., Stilla, R., & Sathian, K. (2010). Object familiarity modulates effective connectivity during haptic shape perception. Neuroimage, 49, 1991–2000. Deshpande, G., Hu, X., Stilla, R., & Sathian, K. (2008). Effective connectivity during haptic perception: A study using Granger causality analysis of functional magnetic resonance imaging data. Neuroimage, 40, 1807–1814. Eger, E., Ashburner, J., Haynes, J.-D., Dolan, R. J., & Rees, G. (2008a). fMRI activity patterns in human LOC carry information about object exemplars within category. Journal of Cognitive Neuroscience, 20, 356–370. Eger, E., Kell, C. A., & Kleinschmidt, A. (2008b). Graded size-sensitivity of object-exemplar-evoked activity patterns within human LOC regions. Journal of Neurophysiology, 100, 2038–2047. Ewbank, M. P., Schluppeck, D., & Andrews, T. J. (2005). fMR-adaptation reveals a distributed representation of inanimate objects and places in human visual cortex. Neuroimage, 28, 268–279. Feinberg, T. E., Rothi, L. J., & Heilman, K. M. (1986). Multimodal agnosia after unilateral left hemisphere lesion. Neurology, 36, 864–867. Gauthier, I., Hayward, W. G., Tarr, M. J., Anderson, A. W., Skudlarski, P., & Gore, J. C. (2002). BOLD activity during mental rotation and view-dependent object recognition. Neuron, 34, 161–171.
175 Goodale, M. A., & Milner, A. D. (1992). Separate visual pathways for perception and action. Trends in Neuroscience, 15, 20–25. Grefkes, C., Geyer, S., Schormann, T., Roland, P., & Zilles, K. (2001). Human somatosensory area 2: Observer-independent cytoarchitectonic mapping, interindividual variability, and population map. Neuroimage, 14, 617–631. Grill-Spector, K., Henson, R., & Martin, A. (2006). Repetition and the brain: Neural models of stimulus-specific effects. Trends in Cognitive Sciences, 10, 14–23. Grill-Spector, K., Kushnir, T., Edelman, S., Avidan, G., Itzchak, Y., & Malach, R. (1999). Differential processing of objects under various viewing conditions in the human lateral occipital complex. Neuron, 24, 187–203. Iriki, A., Tanaka, M., & Iwamura, Y. (1996). Attentioninduced neuronal activity in the monkey somatosensory cortex revealed by pupillometrics. Neuroscience Research, 25, 173–181. James, T. W., Humphrey, G. K., Gati, J. S., Menon, R. S., & Goodale, M. A. (2002a). Differential effects of view on object-driven activation in dorsal and ventral streams. Neuron, 35, 793–801. James, T. W., Humphrey, G. K., Gati, J. S., Servos, P., Menon, R. S., & Goodale, M. A. (2002b). Haptic study of three-dimensional objects activates extrastriate visual areas. Neuropsychologia, 40, 1706–1714. James, T. W., James, K. H., Humphrey, G. K., & Goodale, M. A. (2006). Do visual and tactile object representations share the same neural substrate? In M. A. Heller & S. Ballesteros (Eds.), Touch and blindness: Psychology and neuroscience (pp. 139–155). Mahwah, NJ: Lawrence Erlbaum Associates. Jolicoeur, P. (1987). A size-congruency effect in memory for visual shape. Memory and Cognition, 15, 531–543. Klatzky, R. L., & Lederman, S. J. (1995). Identifying objects from a haptic glance. Perception and Psychophysics, 57, 1111–1123. Klatzky, R. L., Lederman, S. J., & Metzger, V. A. (1985). Identifying objects by touch: An ‘expert system’. Perception and Psychophysics, 37, 299–302. Klatzky, R. L., Lederman, S. J., & Reed, C. L. (1987). There's more to touch than meets the eye: The salience of object attributes for haptics with and without vision. Journal of Experimental Psychology: General, 116, 356–369. Konkle, T., & Oliva, A. (2011). Canonical visual size for realworld objects. Journal of Experimental Psychology: Human Perception and Performance, 37, 23–37. Kozhevnikov, M., Hegarty, M., & Mayer, R. E. (2002). Revising the visualiser-verbaliser dimension: Evidence for two types of visualisers. Cognition and Instruction, 20, 47–77. Kozhevnikov, M., Kosslyn, S. M., & Shephard, J. (2005). Spatial versus object visualisers: A new characterisation of cognitive style. Memory and Cognition, 33, 710–726.
Krekelberg, B., Boynton, G. M., & van Wezel, R. J. A. (2006). Adaptation: From single cells to BOLD signals. Trends in Neurosciences, 29, 250–256. Lacey, S., Flueckiger, P., Stilla, R., Lava, M., & Sathian, K. (2010a). Object familiarity modulates the relationship between visual object imagery and haptic shape perception. Neuroimage, 49, 1977–1990. Lacey, S., Hall, J., & Sathian, K. (2010b). Are surface properties integrated into visuo-haptic object representations? The European Journal of Neuroscience, 31, 1882–1888. Lacey, S., Lin, J. B., & Sathian, K. (2011). Object and spatial imagery dimensions in visuo-haptic representations. Experimental Brain Research doi: 10.1007/s00221-011-2623-1 (in press). Lacey, S., Pappas, M., Kreps, A., Lee, K., & Sathian, K. (2009a). Perceptual learning of view-independence in visuo-haptic object representations. Experimental Brain Research, 198, 329–337. Lacey, S., Peters, A., & Sathian, K. (2007). Cross-modal object representation is viewpoint-independent. PLoS ONE, 2(9), e890. doi:10.1371/journal.pone0000890. Lacey, S., Tal, N., Amedi, A., & Sathian, K. (2009b). A putative model of multisensory object representation. Brain Topography, 21, 269–274. Lawson, R. (2009). A comparison of the effects of depth rotation on visual and haptic three-dimensional object recognition. Journal of Experimental Psychology: Human Perception and Performance, 35, 911–930. Lederman, S. J., & Klatzky, R. L. (1987). Hand movements: A window into haptic object recognition. Cognitive Psychology, 19, 342–368. Lucan, J. N., Foxe, J. J., Gomez-Ramirez, M., Sathian, K., & Molholm, S. (2010). Tactile shape discrimination recruits human lateral occipital complex during early perceptual processing. Human Brain Mapping, 31, 1813–1821. Malach, R., Reppas, J. B., Benson, R. R., Kwong, K. K., Jiang, H., Kennedy, W. A., et al. (1995). Object-related activity revealed by functional magnetic resonance imaging in human occipital cortex. Proceedings of the National Academy of Science of the United States of America, 92, 8135–8139. Mur, M., Ruff, D. A., Bodurka, J., Bandettini, P. A., & Kriegeskorte, N. (2010). Face-identity change activation outside the face system: “Release from adaptation” may not always indicate neuronal selectivity. Cerebral Cortex, 20, 2027–2042. Newell, F. N., Ernst, M. O., Tjan, B. S., & Bülthoff, H. H. (2001). View dependence in visual and haptic object recognition. Psychological Science, 12, 37–42. Newman, S. D., Klatzky, R. L., Lederman, S. J., & Just, M. A. (2005). Imagining material versus geometric properties of objects: An fMRI study. Cognitive Brain Research, 23, 235–246.
176 Nicholson, K. G., & Humphrey, G. K. (2003). The effect of colour congruency on shape discriminations of novel objects. Perception, 32, 339–353. Palmer, S. E., Rosch, E., & Chase, P. (1981). Canonical perspective and the perception of objects. In J. Long & A. Baddeley (Eds.), Attention and performance IX (pp. 135–151). Hillsdale, NJ: Lawrence Erlbaum Associates. Pascual-Leone, A., Amedi, A., Fregni, F., & Merabet, L. B. (2005). The plastic human brain. Annual Review of Neuroscience, 28, 377–401. Pasqualotto, A., Finucane, C., & Newell, F. N. (2005). Visual and haptic representations of scenes are updated with observer movement. Experimental Brain Research, 166, 481–488. Pasqualotto, A., & Newell, F. N. (2007). The role of visual experience on the representation and updating of novel haptic scenes. Brain and Cognition, 65, 184–194. Peissig, J. J., & Tarr, M. J. (2007). Visual object recognition: Do we know more now than we did 20 years ago? Annual Review of Psychology, 58, 75–96. Peltier, S., Stilla, R., Mariola, E., LaConte, S., Hu, X., & Sathian, K. (2007). Activity and effective connectivity of parietal and occipital cortical regions during haptic shape perception. Neuropsychologia, 45, 476–483. Pourtois, G., Schwarz, S., Spiridon, M., Martuzzi, R., & Vuilleumier, P. (2009). Object representations for multiple visual categories overlap in lateral occipital and medial fusiform cortex. Cerebral Cortex, 19, 1806–1819. Prather, S. C., Votaw, J. R., & Sathian, K. (2004). Task-specific recruitment of dorsal and ventral visual areas during tactile perception. Neuropsychologia, 42, 1079–1087. Reed, C. L., Caselli, R. J., & Farah, M. J. (1996). Tactile agnosia: Underlying impairment and implications for normal tactile object recognition. Brain, 119, 875–888. Renier, L., Collignon, O., Poirier, C., Tranduy, D., Vanlierde, A., Bol, A., et al. (2005). Cross modal activation of visual cortex during depth perception using auditory substitution of vision. Neuroimage, 26, 573–580. Renier, L., Collignon, O., Tranduy, D., Poirier, C., Vanlierde, A., Veraart, C., et al. (2004). Visual cortex activation in early blind and sighted subjects using an auditory visual substitutiondevicetoperceivedepth.Neuroimage, 22, S1. Rice, N. J., Valyear, K. F., Goodale, M. A., Milner, A. D., & Culham, J. C. (2007). Orientation sensitivity to graspable objects: An fMRI adaptation study. Neuroimage, 36, T87–T93. Riesenhuber, M., & Poggio, T. (1999). Hierarchical models of object recognition in cortex. Nature Neuroscience, 2, 1019–1025. Sathian, K. (2005). Visual cortical activity during tactile perception in the sighted and the visually deprived. Developmental Psychobiology, 46, 279–286. Sathian, K., & Lacey, S. (2007). Journeying beyond classical somatosensory cortex. Canadian Journal of Experimental Psychology, 61, 254–264.
Sathian, K., Lacey, S., Stilla, R., Gibson, G. O., Deshpande, G., Hu, X., LaConte, S. & Glielmi, C. (2011). Dual pathways for haptic and visual perception of spatial and texture information. NeuroImage, doi: 10.1016/j.neuroimage.2011.05.011 (in press). Sathian, K., Zangaladze, A., Hoffman, J. M., & Grafton, S. T. (1997). Feeling with the mind's eye. Neuroreport, 8, 3877–3881. Sawamura, H., Georgieva, S., Vogels, R., Vanduffel, W., & Orban, G. A. (2005). Using functional magnetic resonance imaging to assess adaptation and size invariance of shape processing by humans and monkeys. The Journal of Neuroscience, 25, 4294–4306. Silvanto, J., Schwarzkopf, D. S., Gilaie-Dotan, S., & Rees, G. (2010). Differing causal roles for lateral occipital complex and occipital face area in invariant shape recognition. The European Journal of Neuroscience, 32, 165–171. Stilla, R., & Sathian, K. (2008). Selective visuo-haptic processing of shape and texture. Human Brain Mapping, 29, 1123–1138. Stoesz, M., Zhang, M., Weisser, V. D., Prather, S. C., Mao, H., & Sathian, K. (2003). Neural networks active during tactile form perception: Common and differential activity during macrospatial and microspatial tasks. International Journal of Psychophysiology, 50, 41–49. Tal, N., & Amedi, A. (2009). Multisensory visual-tactile object related network in humans: Insights gained using a novel crossmodal adaptation approach. Experimental Brain Research, 198, 165–182. Tarr, M. J., & Pinker, S. (1989). Mental rotation and orientation dependence in shape recognition. Cognitive Psychology, 21, 233–282. Ungerleider, L. G., & Mishkin, M. (1982). Two cortical visual systems. In D. J. Ingle, M. A. Goodale & R. J. W. Mansfield (Eds.), Analysis of visual behavior (pp. 549–586). Cambridge, MA: MIT Press. Uttl, B., Graf, P., & Siegenthaler, A. L. (2007). Influence of object size on baseline identification, priming, and explicit memory. Scandinavian Journal of Psychology, 48, 281–288. Valyear, K. F., Culham, J. C., Sharif, N., Westwood, D., & Goodale, M. A. (2006). A double dissociation between sensitivity to changes in object identity and object orientation in the ventral and dorsal streams: A human fMRI study. Neuropsychologia, 44, 218–228. Woods, A. T., Moore, A., & Newell, F. N. (2008). Canonical views in haptic object perception. Perception, 37, 1867–1878. Zhang, M., Weisser, V. D., Stilla, R., Prather, S. C., & Sathian, K. (2004). Multisensory cortical processing of object shape and its relation to mental imagery. Cognitive, Affective and Behavioral Neuroscience, 4, 251–259. Zhou, Y.-D., & Fuster, J. M. (1997). Neuronal activity of somatosensory cortex in a cross-modal (visuo-haptic) memory task. Experimental Brain Research, 116, 551–555.
A. M. Green, C. E. Chapman, J. F. Kalaska and F. Lepore (Eds.) Progress in Brain Research, Vol. 191 ISSN: 0079-6123 Copyright Ó 2011 Elsevier B.V. All rights reserved.
CHAPTER 12
Adaptation and maladaptation: insights from brain plasticity Elena Nava* and Brigitte Röder Department of Biological Psychology and Neuropsychology, University of Hamburg, Hamburg, Germany
Abstract: Evolutionary concepts such as adaptation and maladaptation have been used by neuroscientists to explain brain properties and mechanisms. In particular, one of the most compelling characteristics of the brain, known as neuroplasticity, denotes the ability of the brain to continuously adapt its functional and structural organization to changing requirements. Although brain plasticity has evolved to favor adaptation, there are cases in which the same mechanisms underlying adaptive plasticity can turn into maladaptive changes. Here, we will consider brain plasticity and its functional and structural consequences from an evolutionary perspective, discussing cases of adaptive and maladaptive plasticity and using examples from typical and atypical development. Keywords: crossmodal plasticity; maladaptive plasticity; phantom limb pain; tinnitus; cochlear implants; evolution. Lessons from evolution
brain, and its particular adaptive properties, into a broader evolutionary perspective, according to which some structural and functional properties of an individual's brain are considered to be the result of natural selection. In this context, we will discuss the capacity of the brain to change its functional and structural organization (called plasticity or neuroplasticity) and particularly the resulting beneficial (adaptive) as well as possible detrimental (maladaptive) outcomes. Adaptation defines a dynamic process in structure, function, and behavior by which a species or individual improves its chance of survival in a specific environment as a result of natural
A number of terms used to characterize the evolutionary process have also been adopted by neuroscientists to define brain mechanisms, processes, and abilities. The following short definitions of fundamental evolutionary terms will aid in understanding the context which “inspired” neuroscientists in defining their own terms. In drawing some parallels between these commonly adopted terms, our attempt will be to put the *Corresponding author. Tel.: þ49-40-428385838; Fax: þ49-40-428386591 E-mail:
[email protected] DOI: 10.1016/B978-0-444-53752-2.00005-9
177
178
selection. While the term adaptation speaks for the evolutionary process, an adaptive trait is an aspect of the developmental pattern of the organism that enables or enhances the probability of that organism to survive and reproduce during certain stages of the lifespan (Dobzhansky, 1956). Adaptation became the root concept of Darwin's theory (1859), in that it provided the mechanism to explain why things change in the course of time, and how these affect all aspects of the life of an organism. Natural selection acts on phenotypes (i.e., an observable trait of an organism, which includes physiological as well as behavioral changes), and a particular trait will survive if best suited to the environment. Most importantly, though, only a change in genotype (i.e., the complete set of genes within an organism) will define evolution. Natural selection typically produces fitness, a commonly used but nonetheless controversial term that describes how successful an organism has been at passing its genes. Adaptive traits have continuously evolved as a response to environmental demands. The mechanism underpinning all environmentally induced phenotypic variations is called phenotypic plasticity (Via et al., 1995). This mechanism allows a single genotype to produce more than one response (in terms of morphology, physiological state, etc.) to environmental changes, including learned behaviors as well as reaction to diseases. When an organism produces a phenotype that can continuously change as a function of environmental change (e.g., the ability of the marine snail to increase shell thickness in response to new predators; see Trussell and Smith, 2000), the relationship between these two is termed reaction norm. These reactions can be flexible or more inflexible with the term flexible indicating the ability of the phenotypic trait to change throughout the organism's lifespan. In contrast, the term inflexible indicates an inability to change so that any determined characteristic remains fixed. Phenotypic plasticity likely evolved to allow different organisms a greater chance of survival in their
ever-changing surroundings. Finally, it is as a result of plasticity that the environment directly influences which phenotypes are exposed to selection. In our view, brain plasticity can be seen as an example of phenotypic plasticity. In particular, its many possible outcomes can be seen as phenotypes that react to the environmental changes. Changes in behavior occur at an ontogenetic level, but plasticity itself may have evolved phylogenetically. At the same time, the importance of phenotypic plasticity in driving genetic evolution (Price et al., 2003) suggests the importance of considering brain plasticity within the larger framework of evolutionary processes. The vision from the brain The term plasticity, as is true of most scientific terms, has undergone debates and revisions for the past 100 years (Berlucchi and Buchtel, 2009). In his seminal paper entitled “Réflexions sur l'usage du concept de plasticité en neurobiologie,” Paillard (1976; see Will et al., 2008 for the English translation and commentaries) stated that not every change in the neural system should be considered plastic. Only those resulting from a structural and functional change should be considered as such. Also, structural and functional changes should be long-lasting and not transient events (to distinguish plasticity from “elasticity”). Finally, only changes resulting from an adaptation of the system to environmental pressures should be considered plastic, therefore excluding those mechanisms responsible for the “natural” maturation of the early developing system. Recently, Lövdén et al. (2010), presenting a new theoretical framework for the study of adult plasticity and inspired by Paillard's ideas, has proposed that plasticity occurs as a consequence of a prolonged mismatch between supply (i.e., the actual capacities of the brain resulting from biological constraints and environmental influences) and environmental demands. Plasticity is then the ability of the brain to react to this mismatch
179
undergoing anatomical as well as functional changes to best fit an adaptive demand. In this view, the resulting structural and functional change that accompanies plasticity can be seen as a phenotypic plastic change. Adaptation In referring to brain mechanisms, adaptation commonly refers not only to plasticity, which is the capacity of the brain to change to suit external environmental as well as inner changes, but also to any experience acquired throughout development (for reviews, see Kolb et al., 2003; PascualLeone et al., 2005). Adaptive plasticity is also known as experiencedependent plasticity (Greenough et al., 1987). This type of plasticity refers to the ability of the brain to learn throughout its lifespan by means of processes involving structural and functional changes. Although experience-dependent plasticity refers to the ability to learn any new perceptual, motor, or cognitive skill, a particularly spectacular example is provided by musicians, whose extensive practice on a particular task (i.e., playing an instrument) has been shown to modify tactile, motor, and auditory brain regions (for reviews, see Johansson, 2006; Münte et al., 2002). Most of these studies were conducted on adult musicians, leaving the question of whether these structural brain changes could be innate (therefore predisposing the individual to learn music) or acquired through training (i.e., “real” plastic adaptation of the brain to the greater use of particular regions). Recently, some studies (Hyde et al., 2009; Moreno et al., 2009) have precisely addressed this question by investigating structural brain and behavioral changes in children trained on music skills compared to nontrained children. Hyde et al. (2009) trained fifteen 6-year-old children for 15 months on playing the keyboard, while the control group consisted of age-matched children who only had weekly music classes in school. Both groups were tested on behavioral
tasks as well as scanned with MRI before and after training. Results showed that trained children had increased activity in motor hand areas and primary auditory areas compared to controls, which correlated with behavioral improvements on motor and auditory-musical tasks. The fact that no structural brain difference was found between the two groups before training strongly suggests that training itself triggers adaptive changes. Although studies on adults and children have not directly tested whether these plastic changes can persist longer in life even if musical training is suspended, there may be a sensitive period (which refers to the limited period during development in which effects of experience are particularly strong in shaping the brain, see Knudsen, 2004) in childhood in which musical practice may result in long-lasting benefits in performance later in life. For example, brain-imaging studies highlighting plastic changes occurring as a consequence of musical training have found that the degree of these changes appears to decrease as a function of age, so that musical training experienced very early in life triggers larger plastic changes (Elbert et al., 1995). Given the particular nature of early developmental plasticity (Greenough et al., 1987; Knudsen, 2004), it could be hypothesized that musical training early in life changes the brain structurally and functionally in a hierarchical and long-lasting fashion. Although only investigated by means of a behavioral task, Watanabe et al. (2007) addressed this question by comparing performance of two groups of adults who started their musical training at different ages: early (i.e., before 7 years of age) or late (i.e., after 7 years of age). Participants of the two groups were matched for years of musical experience and practice, so that they only differed in the age when training began. The task consisted in learning to reproduce a temporally complex motor sequence by tapping in synchrony with sequentially presented visual stimuli. Results showed that early-trained musicians had an overall better performance compared to late-trained
180
musicians, suggesting that musical training started early in life (i.e., during sensitive periods) can have long-term effects on the ability to learn novel motor tasks. While the case of musicians speaks for the ability of the typically developing brain to change as a function of increased demand, there are cases in which changes in supply (i.e., the brain) cause plasticity to take place to functionally adapt to the new environment. In other words, in the case of direct or indirect brain insult (i.e., brain lesions or sensory loss, respectively), plasticity will act to reorganize the brain. In particular, plastic changes after sensory deafferentation (i.e., blindness, deafness) trigger the system to reorganize in a compensatory fashion to enable sensory-deprived individuals to better suit new environmental pressures. The following section will discuss this particular type of plasticity mechanism, which we will compare to an evolutionary concept known as exaptation. Crossmodal plasticity after sensory deafferentation: a case of exaptation? Exaptation refers to the shifts in functions of a trait during evolution, so that one trait originally serving a particular function may evolve and serve another one, achieving complete fitness for that trait (Gould, 1991; Gould and Lewontin, 1979). The classical example is bird feathers, which initially evolved for temperature regulation and only later were adapted for flight. Moreover, Gould (1991) suggested that there are two types of exaptation. The first type characterizes features that evolved by natural selection to process one function but are then co-opted for another function (i.e., the example of the bird's feathers); the second type refers to features that did not evolve as adaptations through natural selection but are rather side effects of adaptive processes, features that Gould defined spandrels. Arguing against the rigidity of concepts such as adaptation and natural selection, which cannot fully explain the complexity of some human
behaviors, he described the concept of spandrels making a parallel from the architectural spandrels present in the church of San Marco in Venice: “Every fan-vaulted ceiling must have a series of open spaces along the midline of the vault, where the sides of the fans intersect between the pillars. As the spaces must exist, they are often used for ingenious ornamental effect.” In other words, those spaces between vaults, which originally had purely structural functions, ended up being used to enhance esthetic characteristics (i.e., a by-product of their original function).
Spandrels in the brain The term exaptation, if considered in its conceptual form, well suits a particular type of plasticity called crossmodal plasticity. The term crossmodal plasticity has been adopted particularly when describing compensatory plasticity that emerges in some cases of sensory deprivation, such as blindness and profound deafness (Bavelier and Neville, 2002; Pascual-Leone et al., 2005). In particular, some studies have suggested that the absence of the stream of information coming from one sensory modality causes the brain to reorganize in a crossmodal fashion, so that the deafferented cortex responds to input coming from the intact sensory modalities. These types of changes have been also called intermodal changes in both animal (Rauschecker, 1995) and human studies (Röder et al., 1999) because of their “between-senses” interactions. In this view, intermodal changes share commonalities with the concept of exaptation, in that regions subserving the deafferented modality take over new functions originally exclusively mediated by other brain areas. Specifically, a subset of the neurons that are usually responsive to a particular stimulation in a region of the brain will now respond to stimulation of another modality or in the context of a new function. The rationale behind drawing parallels between crossmodal plasticity after sensory deprivation and the concept of exaptation is that the former
181
has been enthusiastically advanced as the mechanism responsible for the enhanced performance found, for example, in tactile and auditory tasks in blind individuals (Amedi et al., 2010; Gougoux et al., 2005; Röder et al., 1999). However, this explanation has been challenged by the diversity of tasks eliciting visual cortex activation after congenital blindness (differing in modality and complexity, see, e.g., Pavani and Röder, 2011) and by studies that found similar crossmodal activity in sighted individuals blindfolded a few days only (Pascual-Leone and Hamilton, 2001), suggesting that this process may not exclusively emerge as a consequence of early sensory deprivation per se. Where does the idea of functional crossmodal plasticity come from? Around 20 years ago, animal studies began to address the question of whether the functional properties of cortical tissue are determined by the inputs they receive rather than being innate. In these experiments, input from one sensory modality was rerouted to the primary cortex of another modality (Sur et al., 1990; von Melchner et al., 2000). For instance, Sur et al. (1990) rerouted retinal axons of newborn ferrets into the auditory pathway by removing ascending auditory projections through deafferentation of the medial geniculate nucleus (MGN) (and by removing the visual cortical targets by ablating visual cortex). This caused retinal fibers to innervate the MGN, so that MGN was now “invaded” by visual input. These inputs were then transferred to auditory cortex via intact MGN projections. The physiological and anatomical consequence of this rerouting was the development of visual networks in auditory cortex, so that a map of visual space emerged in the auditory cortex (i.e., a change in receptive field properties including the development of visual orientation-selective cells). How were these structural changes then interpreted by the animal? In other words, were the rewired projections interpreted as a visual input or an auditory one? If the behavioral role of a cortical area is independent of its input, then activation of the auditory cortex by any stimulus
would be interpreted as auditory. In contrast, if the nature of the input has a role in determining the function of a cortical area, then rewired animals should interpret visual activation in the auditory cortex as a visual stimulus. Von Melchner et al. (2000) addressed this question by training neonatal ferrets to discriminate between visual and auditory stimuli. A group of ferrets were “rewired” by directing their retinal axons to the left MGN, thus providing visual information to the auditory cortex in the left hemisphere. When the auditory cortex in the left hemisphere was lesioned, these animals were no longer able to discriminate visual stimuli, indicating that they became blind in the right visual field because the auditory cortex had mediated visual processing for this part of visual field. These experiments suggest that visual inputs routed to the auditory thalamus are capable of inducing auditory pathways to mediate vision, which crucially means that cortical areas process their functions under the input control. The fact that rewired cortices functionally mediate functions originally belonging to another region leads to the suggestion that even after sensory deprivation (i.e., without artificial rerouting), crossmodal plasticity may take place. In addition, would crossmodal plasticity correspond to an enhancement in performance in some behavioral tasks? To address this issue, Rauschecker and Kniepert (1994) tested visually deprived cats in a localization task in which animals had to walk toward a target sound source that was continuously manipulated in azimuth location. Deprived cats showed better auditory localization abilities compared to nondeprived cats, particularly for lateral and more peripheral locations, suggesting that compensatory plastic changes could underlie enhanced performance in the intact modality after sensory deprivation. Similar findings also come from King and Parsons (1999), who investigated auditory spatial acuity in visually deprived ferrets and documented improved performance in the lateral sound field for both juvenile and adult animals that were deprived early in life. However, these
182
studies might also be partially explained by intramodal changes, for example, by a higher functionality of cortical networks associated with the auditory system. Therefore, they did not provide complete answers to the functional meaning of the deafferented cortical activity. Recently, in reviewing their experiments on deaf cats conducted over several years, Lomber et al. (2010) have advanced a new hypothesis on crossmodal reorganization after sensory deprivation. In a number of experiments, the performance of congenitally deaf cats and hearing controls was compared for a number of visual psychophysical tasks (i.e., visual localization, movement detection, orientation and velocity discrimination, and visual acuity). Deaf cats were found to have enhanced performance only on the visual localization task (particularly for peripheral locations) and on the movement detection task. To investigate which cortical area could mediate the enhanced visual abilities, portions of auditory cortex were deactivated by means of a cryoloop device, which applied cold temperatures to a specific region of the brain and temporarily inactivated its functions. Interestingly, results showed that cooling of different areas could undermine the enhanced performance of deaf cats selectively for one task only, suggesting that perceptual enhancements were processed in specific cortical areas. In sum, crossmodal reorganization does not seem to be a unitary process involving reorganization of the whole (auditory) cortex; rather, it seems to involve changes in specific cortical loci. What are the characteristics of these reorganized loci? Why should they be so “special”? Lomber et al. (2010) suggested that only those regions subserving supramodal functions might undergo reorganization, while leaving modality-specific functions unaltered. In other words, skills that are shared across senses have greater potential to undergo enhancement and reorganization. For example, while color discrimination is an exclusively visual ability, and pitch discrimination an exclusively auditory ability, information on the spatial location of an object is
provided by both vision and audition. In this supramodal view, auditory deprivation will lead to crossmodal changes in those regions that “naturally” engage multisensory processing, thus leaving unchanged regions that functionally process a modality-specific feature (such as color or tone). Interestingly, crossmodal plasticity after auditory deprivation in humans appears to have a similar behavioral pattern as shown in Lomber et al. (2010). For instance, from a behavioral point of view, deaf individuals show enhanced performance in highly task-specific contexts, suggesting that not all aspects of the visual system are reorganized following sensory loss (for reviews, see Bavelier et al., 2006). In particular, deaf individuals have proven to have comparable performance to hearing controls in most visual tasks involving accuracy and sensitivity thresholds. These include brightness discrimination (Bross, 1979), visual contrast sensitivity (Finney and Dobkins, 2001), luminance change detection (Bavelier et al., 2000, 2001), motion direction (Bosworth and Dobkins, 2002a, b), motion velocity (Brozinsky and Bavelier, 2004), and temporal order perception (Nava et al., 2008). By contrast, deaf individuals appear to have enhanced performance for detection or discrimination of stimuli presented in the periphery of the visual field (Bavelier et al., 2000; Loke and Song, 1991; Neville and Lawson, 1987; but see Bottari et al., 2010 for contrasting results). In addition, found enhanced tactile sensitivity in congenitally deaf individuals when detecting suprathreshold tactile changes within a monotonous sequence of vibratory stimuli (Levänen et al., 1998; Levänen and Hamdorf, 2001). In contrast, studies in blind individuals have shown more consistent results with regards to enhanced performance compared to sighted controls in several different domains (Amedi et al., 2010; Collignon et al., 2009; Gougoux et al., 2009; Röder et al., 1996), For example, blind individuals outperform sighted controls on tactile tasks (Amedi et al., 2010; Sadato et al., 1996), auditory tasks (Rauschecker, 1995; Röder et al., 1996), sound localization tasks (Collignon
183
et al., 2009; Rauschecker, 1995; Voss et al., 2004), spatial imagery (Röder et al., 1997; Vanlierde et al., 2003), voice perception (Gougoux et al., 2009), and language perception (Röder et al., 2002). Some studies have put forward the possibility that the enhanced performance in deaf and blind individuals may be a result of recruitment of the deafferented sensory cortices by the intact senses to functionally compensate for the loss (Cohen et al., 1997; Levänen et al., 1998; Sadato et al., 1996). However, these studies remain very controversial due to several possible confounding factors (i.e., different experimental paradigms, individuals’ high variability). The most important factor concerns the limited spatial resolution of the employed neuroimaging techniques, which may not be sufficiently precise to identify the subregions of the deafferented cortex involved. In sum, the data discussed above show that the functional meaning of the cortical activity in the sensory-deprived cortex still needs to be further investigated. However, they also suggest that at least a portion of the cortical tissue that has become dominated by the intact senses may reorganize to now subserve functions of the intact modalities. In this sense, the possibility that brain regions that originally evolved to process specific modalities may partially take on new functions to better suit the environment can be seen as a case of exaptation; namely, as a mechanism that has new biological functions different from the ones that caused the original selection of that mechanism. The following section will discuss how these same spandrels can sometimes lead to maladaptive changes, therefore suggesting that plasticity may have mixed consequences: “positive” ones and “negative” ones. Maladaptation So far, plasticity has been viewed as a highly evolved feature of the brain to allow the organism to best adapt to the challenges imposed by the
environment. However, the same mechanisms that promote adaptation can sometimes turn into maladaptive changes in structure and behavior. In evolutionary biology, maladaptation has been defined as a deviation from adaptive peaks (Crespi, 2000). Adaptive peaks refer to the notion of an adaptive landscape introduced by Sewell Wright in 1931. The metaphor of the adaptive landscape was adopted to graphically summarize a theory concerning population genetics, by which “hills” represent the fittest populations (in terms of combination of genes) and the “valleys” represent the less fit populations. Natural selection tends to move the populations toward the peaks of the hills, but as the environment continuously changes, the populations are forced to adapt to these changes to maintain or build fitness. Assuming, hypothetically, that plasticity may be encoded in a group of genes, its phenotypic expression can be either adaptive or maladaptive. In this view, maladaptive plasticity can be seen as a phenotype placed in a valley of the adaptive landscape. Thus, it could be hypothesized that adaptive plasticity has evolved while leaving behind maladaptive plasticity. However, the following paragraphs will show that in some cases, the same mechanisms allowing adaptive changes can sometimes lead to maladaptive changes, thus narrowing the border between adaptive and maladaptive plasticity. Maladaptive brain plasticity, the other side of the coin Adaptive plastic changes in the cases we have described in the previous paragraph have a positive nature, in that they aid typically and atypically developing brains to functionally best fit the environment. However, there is also the other side of the coin of plasticity, which Elbert and Heim (2001) called “the dark side” of cortical reorganization, and what is commonly known as maladaptive plasticity. This can be seen as an excess of brain reorganization but might actually
184
consist of only a small structural change. In both cases, the outcomes are highly dysfunctional. If seen in the perspective of the mismatch between supply and demand, maladaptive changes even go beyond this mismatch, in that the supply (i.e., the brain) abnormally interprets the environmental demands and does not adjust to a more suitable and optimal condition. Curiously, in some cases, the same adaptive plastic changes that have aided the brain to best suit the environment are also those that can lead to maladaptive changes. For example, musicians, whose differences in brain structure with respect to nonmusicians may likely represent plastic brain adaptations in response to skill acquisition and repetitive rehearsal of those skills, can also sometimes develop the so-called musician's cramp, which is very similar to the well-known “writer's cramp” (Quartarone et al., 2003). Both maladaptive syndromes lead to focal dystonia, a movement disorder that causes the muscles to contract and spasm involuntarily. This debilitating disease finds its explanation in a dysfunctional reorganization of the brain (Tamura et al., 2009), particularly in the reorganization of the digits in the primary somatosensory cortex in these cases. More precisely, the topographic map represented in the somatosensory cortex is altered during the learning of sensorimotor skills, and those parts of the body (i.e., fingers, hand) that are stimulated the most drive the homologous cortical representations to expand (for classical animal studies, see Kaas, 1991). In support of the findings that cases of focal dystonia are triggered by maladaptive plastic changes, Candia et al. (2003) have developed a new treatment for focal hand dystonia in musicians based on the assumption that if the dysfunction arises as a consequence of maladaptive shifts of cortical reorganization, retuning the sensorimotor representations could likely treat these patients. During this training, dystonic patients have one or more nondystonic fingers immobilized in a splint device. The therapy consists in making sequential movements of two or three digits in extension, including the dystonic
finger, for a prolonged period and increasing time of training each day. In particular, in their fMRI experiment, Candia et al. (2003) showed a reduction in distances between cortical finger representations, suggesting a normalization of functional topography associated with the therapy. Most importantly, this cortical shift correlated with behavioral motor benefits, thus corroborating the notion that the underlying maladaptive mechanisms of dystonia may find their roots in cortical reorganization. The following paragraphs will focus on three particular cases for which plasticity operates in a maladaptive fashion: pain following amputation, tinnitus following hearing loss, and absence of benefits following cochlear implantation. While for the first two cases, the notion of maladaptive plasticity has a more intuitive connotation, maladaptive plasticity after cochlear implantation has a different nature. Nonetheless, all three cases represent the other side of the coin of beneficial adaptive changes, suggesting that plasticity can exert its influence in different ways. Phantoms after sensory deafferentation: phantom limb pain and tinnitus Phantom limb pain and tinnitus share common characteristics that allow, to some extent, for a direct comparison. First of all, both syndromes are characterized by a “phantom” sensation, sometimes very painful, arising from a lesion (in case of amputation) or a disease (in some cases of tinnitus following hearing loss). This, in turn, results in perceived pain although no stimulus is actively triggering it. Also, both maladaptive sensations are subjective and can change in quality throughout life, and for both conditions, similar recent training procedures have been shown to provide beneficial effects (Flor and Diers, 2009). In particular, the rationale behind the training is the assumption that pain is triggered, in both cases, by a reorganization of cortical maps, and by an “expansion” of some frequencies (in tinnitus) or somatosensory representations (in phantom pain) at the expense of others.
185
Phantom limb pain After amputation of a body part, the sensation of the presence of the missing part is reported by almost all amputees. The reported prevalence of phantom pain varies considerably in the literature, but most studies agree that around 60–80% of all amputees experience phantom pain following amputation. Phantom pain seems to be independent of age, gender, and cause of amputation. Very interestingly, phantom limb pain mostly occurs in late-amputated individuals (i.e., amputated in adulthood), being instead very infrequent in amputated children and almost absent in congenital amputees (for reviews, see Flor, 2002; Flor et al., 2006). The mechanisms underlying phantom limb pain are not fully understood and may involve complex interactions between morphologic, physiologic, and chemical changes at central and/or peripheral levels (Flor et al., 2006). However, similarly to the musicians’ case, the experience of pain correlates with reorganization of the somatosensory map. The possibility that pain, the maladaptive component following amputation, could be directly related to cortical reorganization of the primary somatosensory cortex, has only recently found major acceptance in the literature. As plastic reorganization has commonly been seen (as discussed in the previous paragraphs) as a beneficial and functional response of the brain to adaptive needs, the possibility that the same mechanism could trigger maladaptive outcomes has somehow been viewed as counterintuitive. However, nearly 15 years ago, along with other causal mechanisms that can explain phantom limb pain, the possibility that this maladaptive plastic change could additionally result from cortical reorganization started emerging (Birbaumer et al., 1997; Flor et al., 1995; Knecht et al., 1996). The relationship between cortical reorganization and phantom limb pain started with the notion that deafferentation of digits or the hand leads to plastic changes in the somatosensory
cortex (Pons et al., 1991). In addition, findings on chronic back pain revealed a strong correlation between cortical alteration and pain (Flor et al., 1997), with patients exhibiting more cortical reorganization as a function of felt pain. These two factors led researchers to point to cortical reorganization as a structural correlate of phantom limb pain. For example, Flor et al. (1995) and Birbaumer et al. (1997) determined cortical reorganization in a group of adult amputees by means of neuroelectric source imaging (a technique that combines evoked potential recordings with structural magnetic resonance imaging). In particular, Birbaumer et al. (1997) compared the representations of hand and mouth in both hemispheres of the somatosensory cortex. As amputees without pain were found to have mirrored representations of mouth and hand, any asymmetry found in amputees with pain would become a marker of cortical reorganization. As hypothesized, the cortical representation in amputees with pain showed a shift of the lip representation into the cortical region, which previously belonged to the amputated hand. An intriguing explanation of phantom limb pain has also been put forward, namely, the possibility that the maladaptive outcome could be elicited by the memory of the pain experienced prior to amputation (Flor, 2002; Katz and Melzack, 1990). In other words, if the preamputated limb had received prolonged and intense noxious input, it would have developed enhanced excitability for pain and therefore exhibited an alteration in cortical somatosensory processing. Subsequent amputation and invasion of the cortical region by neighboring inputs would then activate cortical neurons coding for pain, leading to the perception of pain. In support to this view, Nikolajsen et al. (1997) have shown that pain experienced before amputation can sometimes even predict phantom limb pain after deafferentation, supporting the importance of the memory of pain in making the phantom persist over time.
186
A particularly interesting finding concerns the lack of reorganization of somatosensory cortical maps in congenital amputees, which also correlates with their lack of reported pain (though the sensation of the missing limb persists in many cases). However, only in recent times has this correlation been investigated. So, for example, Flor et al. (1998) investigated cortical reorganization in primary somatosensory cortex in a group of congenital amputees and a group of traumatic amputees with or without pain determined by neuromagnetic source imaging. Results showed that the most cortically reorganized individuals were the traumatic amputees reporting pain. In contrast, the congenital amputees and amputees without pain presented very little reorganization and the small amounts of reorganization observed in each case were similar. In addition, phantom limb pain was found to positively correlate with cortical reorganization and with no other factor (i.e., time as amputation) or sensation (i.e., phantom limb sensation per se). The fact that congenitally limb-deprived individuals do not experience pain and do not present cortical reorganization opens an additional issue concerning adaptive and maladaptive plasticity that should be further explored, namely, the possibility that these two outcomes are influenced by development. In other words, while congenital or early deprivation may favor overall adaptation, deprivation experienced in adulthood may lead to maladaptation. Curiously, the presence or absence of beneficial versus detrimental cortical reorganization differs between types of developmental deprivations, as the following section will suggest.
Adaptation early in life: a comparison between congenitally deprived sensory modalities While congenital amputees have been shown to have a lack of cortical reorganization compared to late amputees, some studies in blind individuals show the opposite pattern (Cohen et al., 1999; Sadato et al., 2002). For example, Fieger et al.
(2006) compared his results in late-blind individuals with the findings of Röder et al. (1999) in congenitally blind individuals and showed that despite comparable performance, the brain mechanisms differed between the two groups. While a more precise spatial tuning of early auditory processes was observed in the congenitally blind (indexed by the event-related potential (ERP) called N1), later processing stages (indexed by the ERP called P3) seemed to mediate the improved behavior in the late blind. Overall, these results showed that the neural mechanisms underlying crossmodal changes differ in the developing and adult brain, further corroborating the notion that plastic changes that occur early in life can lead to functional advantages throughout life. In sum, in congenital blindness, the presence of crossmodal reorganization appears to be functionally adaptive, while in congenitally limbdeafferented individuals, the absence of crossmodal reorganization appears to be one of the preconditions for avoiding maladaptive outcomes (i.e., pain). What can this differential pattern of plasticity suggest? A hypothesis could be that plastic changes early in life as a consequence of congenital deafferentation may be more adaptive compared to changes at later developmental stages. In other words, the flexibility of the brain after either direct or indirect damage during early development may be the expression of normal ontogenetic mechanisms that instead of “repairing” (as in the case of adult brains) simply make the young brain optimally adjust to the insult. The fact that positive adaptive plasticity is expressed differentially (i.e., reorganization vs. nonreorganization) in the two cases (i.e., blindness vs. phantom limb pain) could possibly be due to the specific type of damage or exceptional experience.
Tinnitus Tinnitus can be “objective” or “subjective.” The former refers to a perceived sensation of sound
187
elicited by internal stimulation (i.e., abnormal blood flow pulsations or muscle contraction) that can be heard (therefore objectively measured) by a clinician (e.g., by placing a stethoscope over the patient's external auditory canal). Here, we will focus on subjective tinnitus, which causes the affected person to experience “phantom sounds,” commonly reported to be ringing noises, buzzes, clicks, pure tones, and even songs. Tinnitus has many different causes, otologic, neurologic, and drug related, making the understanding and treatment of the disease difficult to handle (for a clinical review of tinnitus, see Lockwood et al., 2002; M ller et al., 2010). The prevailing opinion is that tinnitus is generated as a consequence of altered patterns of intrinsic neural activity generated along the central auditory pathway following damage to peripheral auditory structures (Eggermont and Roberts, 2004), making it a prevailing symptom following hearing loss. But what does this altered neural activity precisely refer to? Electrophysiological recordings in animals have identified three types of abnormal activity in the auditory system following sensory deprivation, which could also account for causes of tinnitus when associated with hearing loss (for a comparison between animal and human studies, see Adjamian et al., 2009). The first type refers to changes in the spontaneous neural firing rate, by which neurons at rest fire even in the absence of sound stimulation (Seki and Eggermont, 2003). The second type refers to changes in the temporal firing pattern of a single neuron as well as the synchronous activity between neurons. After highnoise exposure or hearing loss, their impulses tend to become pathologically synchronous. This synchronic firing would then become more salient compared to more dispersed firing and be interpreted by the brain as a real sound. Moreover, it is precisely this prolonged synchronization that would induce the perception of tinnitus (Noreña and Eggermont, 2003; Seki and Eggermont, 2003; Weisz et al., 2005, 2007). Weisz et al. (2007) have proposed that gamma band
activity, which is increased in tinnitus patients, may reflect the synchronous firing of neurons within the auditory cortex and constitute the neural code of tinnitus. The reason why gamma band activity has been viewed with such excitement in explaining tinnitus is because a series of previous studies have shown that gamma band synchronous oscillations of neuroelectrical activity may be a mechanism used by the brain to generate and bind conscious sensations to represent distinct objects (for a review, see Sauvé, 1999). This functional significance of gamma band activity would, therefore, explain why tinnitus patients consciously experience a phantom sensation. Finally, the third type of abnormal activity in auditory system following sensory the deafferentation has been shown to result in reorganization of the cortical tonotopic representation. This third type clearly parallels mechanisms of cortical reorganization reviewed for phantom limb pain. As in the latter case, the tonotopic map becomes distorted for those sound frequencies where the hearing loss occurred. This results in an expansion of the representation of the frequencies that border on the lost frequencies, so that the deprived neurons now become responsive to frequencies adjacent to those at which hearing loss has taken place. To investigate this issue, Mühlnickel et al. (1998) used magnetoencephalographic recordings on 10 individuals with tinnitus to establish whether there could be any reorganization of the tonotopic map in the auditory cortex. The rationale was to observe whether tinnitus could be related to a shift of frequency representations in the auditory cortex. To this end, four sets of pure tones above an individual's hearing level were selected and presented to each ear to form a trajectory representing the tonotopic map in healthy controls. For tinnitus patients, three tones were distant from the tinnitus frequency and the fourth was close to the tinnitus frequency. The three tones served to reconstruct the tonotopic map of each patient. Results showed that the tinnitus frequency had “invaded” the neighboring
188
frequency regions. Further, this invasion correlated with tinnitus strength, so that patients reporting more symptoms were also the ones who presented more cortical reorganization. It is worth noting that the three types of changes described seldom occur independently of each other, as suggested by animal (Seki and Eggermont, 2003) and human studies (Weisz et al., 2005, 2007), pointing to their correlational rather than causal nature. That these three factors may be simultaneously present has been highlighted in studies that are investigating which treatments can exert the most beneficial and prolonged effects on tinnitus. In other words, several studies have particularly manipulated cortical reorganization with the assumption that, as in the case of dystonic patients, retuning the tonotopic maps could relieve patients of the phantom sensation. Recently, Okamoto et al. (2010) exposed eight chronic tinnitus patients to music they chose themselves and which they were asked to listen to for 12 months regularly. The music was then frequency modified, so that it did not contain frequencies in the range neighboring the tinnitus frequency. After a 1-year exposure, tinnitus patients reported a reduction in tinnitus loudness. There was also a corresponding decrease in evoked activity in auditory cortex areas corresponding to the tinnitus frequency. The authors speculated that lateral inhibition from the neighboring parts of the tonotopic map were responsible for the beneficial effects on tinnitus. “Rewiring” cortical reorganization through prostheses: to what extent is plasticity malleable? Considering the lessons learned from maladaptive plastic changes strictly linked to cortical reorganization, one could ask whether restoring sensory input to the deafferented region by means of a prosthesis would provide substantial relief to tinnitus and phantom limb pain patients. The rationale behind reafferentation is that either tactile
(for phantom limb pain) or auditory (for tinnitus) stimulation will expand the cortical representation of the stimulated body region, thus “rewiring” cortical maps back to their original state. According to this view, prostheses for phantom limb pain and cochlear implants for tinnitus patients could potentially help in “blocking” or even “rewiring” the effects of maladaptive plasticity. A cochlear implant is a neuroprosthetic device consisting of a microelectrode array inserted in the cochlea that directly stimulates the auditory nerve (for reviews, see Moore and Shannon, 2009; Rauschecker and Shannon, 2002). Although limb prostheses and cochlear implants cannot be directly compared because they are based on different principles (i.e., on somatosensory feedback in the former case, and on nerve stimulation in the latter), they nonetheless represent good models to investigate how and to what extent the brain learns to interpret new information. In particular, several studies have shown that these devices can, in some cases, relieve phantom limb pain and tinnitus. For example, Lotze et al. (1999) examined the effects of the use of a myoelectric device in a group of unilateral amputees using fMRI. The groups were split into myoelectric versus nonmyoelectric users based on the extent of wearing time and average usage. The myoelectric users showed a symmetrical lip representation in the somatosensory cortex (in accordance with previous studies showing that symmetrical body representations are an index of a lack of cortical reorganization), which correlated with a reduction of phantom limb pain. In contrast, the nonmyoelectric users showed the exact opposite pattern, namely, a reported intense pain that correlated with massive cortical reorganization. Similarly, for tinnitus patients, several studies have documented a reduction of tinnitus after cochlear implantation (Miyamoto et al., 1997; Ruckenstein et al., 2001). However, it should be noted that results for both treatments are controversial, in that not all patients have systematically reported benefits. To date, it is not
189
known whether this difficulty in “undoing” or “rewiring” previous plastic changes relates to the technical limits of the devices and/or to the limits of plasticity itself. It is likely, though, that both factors interact to make reafferentation a challenging issue. The particular case of cochlear implants failing to suppress and reduce tinnitus leads to our discussion of the last example of maladaptive plasticity. In which sense can a cochlear implant be maladaptive? As cochlear implantation has become routine therapy for partially restoring auditory function in profoundly deaf individuals, most studies have emphasized the beneficial outcomes of this device following auditory deprivation (Litovsky et al., 2006; Svirsky et al., 2000; Van Hoesel, 2004). The extent to which a cochlear implant exerts its benefits on single individuals appears to be determined by several factors. These factors include the age at which implantation takes place (Sharma et al., 2002, 2005), and the previous experience with auditory cues (Nava et al., 2009a,b). Clearly, cochlear implantation per se does not create any phantom sensation, so that a direct comparison to tinnitus and phantom limb pain is not feasible. However, the outcome of a cochlear implant is related to the amount of cortical reorganization that has taken place prior to implantation. In other words, precisely what we have defined as “spandrels” after sensory deafferentation may be detrimental in case of reafferentation. The following examples show that some plastic changes can be maladaptive because they do not allow the brain to “rewire” once the reafferented sensory cortices have been taken over by other modalities. Lee et al. (2001) were the first to suggest such a possibility by examining glucose metabolism (used as an index of brain activity) in a group of prelingually deafened individuals before cochlear
implantation. The degree of hypometabolism before implantation correlated with the hearing abilities achieved after implantation, so that those patients with higher hypometabolism in temporal areas (including auditory cortex) were also the ones who gained more from auditory restoration. Conversely, those with lower hypometabolism did not achieve the same auditory capabilities, as measured with a speech perception test administered at several follow-up sessions after implantation. Results were interpreted as a being related to a possible increase in visual or somatosensory afferents to these temporal regions due to auditory deafferentation. Therefore, if crossmodal plasticity takes place in the auditory cortex before implantation, improvement in hearing after implantation will be less pronounced. Beneficial outcomes after cochlear implantation have commonly been measured by evaluating speech recognition over time (for review, see Peterson et al., 2010). Reasoning that responses to visual stimulation in cochlear implant recipients may be related to their speech recognition abilities; Doucet et al. (2006) compared visual processing in two groups of cochlear implant recipients. The subjects were divided into “good” and “bad” performers according to their auditory speech perception skills, in that the former were able to recognize speech without visual cues, and the latter only relied on sign language and lip-reading to communicate efficiently. All participants were simply asked to fixate a visual stimulus presented several times while evoked potentials were recorded. Results showed that, while “good” performers had similar activation compared to hearing controls (i.e., evoked activity measured with ERPs was circumscribed around the primary visual cortex), “bad” performers exhibited extended cortical activity, suggesting recruitment of auditory cortical areas for visual processing. This result further suggests that once crossmodal plastic changes have taken place, speech perception performance after cochlear implantation might be undermined as a consequence of cortical reorganization.
190
The fact that crossmodal changes can undermine the good outcome of cochlear implants is relevant to the issue of when (in terms of age) plastic changes take place, and therefore when a device should be implanted. In this view, the existence of sensitive periods early in life for the typical development of the auditory system suggests that crossmodal plasticity may occur within these phases, and only to a lesser extent, or not at all, later in life. For example, Sharma et al. (2002) examined P1 latencies in congenitally deaf children who received a cochlear implant and found that those implanted before 3.5 years of age had normal P1 latencies, while children who received their implant after 7 years of age had abnormal latencies. This suggests a sensitive period for central auditory development that persists up to 3.5 years of age. In a further study, Sharma et al. (2005) assessed the time course of central auditory development in early and late congenitally deaf children implanted unilaterally either before 3.5 years of age or after 7 years of age. The results showed a different pattern of P1 development for early and late implanted children. While early implanted children reached almost normal P1 latencies within a week of implant use, late implanted children showed atypical response that remained atypical until the 18-month follow-up. Overall, these results suggest that, in line with what we have previously mentioned for congenitally blind individuals, plastic changes that occur within sensitive periods early in life might be particularly strong and long-lasting, therefore preventing the brain from reorganizing at a later time. In this sense, some plastic changes can be maladaptive from the perspective of reafferenting the auditory pathways later in life. Finally, it should be mentioned that, comparable to the case of phantom limb pain after amputation later in life, crossmodal changes in the auditory cortex can occur also as a function of years of deprivation. For example, Lee et al. (2003) showed that there is a correlation between years of auditory deprivation and cortical reorganization that goes beyond sensitive periods. In his
study (Lee et al., 2003), a group of postlingually deafened adults with years of auditory deprivation ranging from 2 months to 23 years underwent PET scans to evaluate their regional cerebral metabolism (similar to Lee et al., 2001). Results showed that glucose metabolism in the auditory cortex decreased after auditory deprivation, but increased as a function of years of deprivation, suggesting that functional crossmodal reorganization also takes place in the adult brain. What does this study suggest? First, it is compatible with the view that plasticity and crossmodal changes can also occur during adulthood (Pascual-Leone and Hamilton, 2001; Voss et al., 2004). Second, it corroborates the criterion expressed by Lövdén et al. (2010) by which adult plasticity is driven by a prolonged mismatch between supply and demand. The longer the mismatch is, the higher the probability that the change will result in a plastic change. Final remarks We started this review by defining some evolutionary terms adopted by neuroscientists to highlight some properties, mechanisms, and behaviors of the brain. As much as phenotypic plasticity represents an important factor in evolution, has a genetic basis, and may be altered by natural selection (Price et al., 2003), we suggest that brain plasticity could mimic this evolutionary pattern, so that it becomes worth asking why and how this characteristic of the brain has evolved. Here, we have discussed how adaptive plasticity can lead the brain to structural and functional changes, in typical and atypical development, to best suit environmental demands. However, we have also challenged the view that plasticity consists only of beneficial adaptive changes, by emphasizing how it can sometimes result in highly dysfunctional outcomes that we have generally described here as being maladaptive. From an evolutionary perspective, maladaptive plasticity arises as a phenotype that has reduced
191
fitness or is distant from an adaptive peak (Ghalambor et al., 2007). Brain plasticity has likely evolved to accommodate continuous environmental changes, suggesting that what we define as adaptive or maladaptive at any given time could also exchange roles as a function of changing environmental demands. An additional important consideration is whether in the modern era making a distinction between adaptive and maladaptive plasticity is actually relevant. Advances in technology and medicine have clearly increased our chances of survival and have, therefore, changed the pressure of natural selection on our genes by changing the extent to which we must adapt to environmental demands. In this context of less selective pressures, an adaptive landscape may be more difficult to draw, as “hills” and “valleys” effectively become less distinct. In conclusion, the environmental manipulations carried out by humans may slowly shape natural selection, may even change the rate of evolutionary dynamics, and finally also the trait of plasticity. References Adjamian, P., Sereda, M., & Hall, D. A. (2009). The mechanisms of tinnitus: Perspectives from human functional neuroimaging. Hearing Research, 253, 15–31. Amedi, A., Raz, N., Azulay, H., Malach, R., & Zohary, E. (2010). Cortical activity during tactile exploration of objects in blind and sighted humans. Restorative Neurology and Neuroscience, 28, 143–156. Bavelier, D., Brozinsky, C., Tomann, A., Mitchell, T., Neville, H., & Liu, G. (2001). Impact of early deafness and early exposure to sign language on the cerebral organization for motion processing. The Journal of Neuroscience, 21, 8931–8942. Bavelier, D., Dye, M. W. G., & Hauser, P. C. (2006). Do deaf individuals see better? Trends in Cognitive Sciences, 10, 512–518. Bavelier, D., & Neville, H. J. (2002). Cross-modal plasticity: Where and how? Nature Reviews. Neuroscience, 3, 443–452. Bavelier, D., Tomann, A., Hutton, C., Mitchell, T., Corina, D., Liu, G., et al. (2000). Visual attention to the periphery is enhanced in congenitally deaf individuals. The Journal of Neuroscience, 20, 1–6.
Berlucchi, G., & Buchtel, H. A. (2009). Neuronal plasticity: Historical roots and evolution of meaning. Experimental Brain Research, 192, 307–319. Birbaumer, N., Lutzenberger, W., Montoya, P., Larbig, W., Unertl, K., Töpfner, S., et al. (1997). Effects of regional anesthesia on phantom limb pain are mirrored in changes in cortical reorganization. The Journal of Neuroscience, 17, 5503–5508. Bosworth, R. G., & Dobkins, K. R. (2002a). The effect of spatial attention on motion processing in deaf signers, hearing signers, and hearing non-signers. Brain and Cognition, 4, 152–169. Bosworth, R. G., & Dobkins, K. R. (2002b). Visual field asymmetries for motion processing in deaf and hearing signers. Brain and Cognition, 49, 170–181. Bottari, D., Nava, E., Ley, P., & Pavani, F. (2010). Enhanced reactivity to visual stimuli in deaf individuals. Restorative Neurology and Neuroscience, 28, 167–179. Bross, M. (1979). Residual sensory capacities of the deaf: A signal detection analysis of a visual discrimination task. Perceptual and Motor Skills, 48, 187–194. Brozinsky, C. J., & Bavelier, D. (2004). Motion velocity thresholds in deaf signers: Changes in lateralization but not in overall sensitivity. Cognitive Brain Research, 21, 1–10. Candia, V., Wienbruch, C., Elbert, T., Rockstroh, B., & Ray, W. (2003). Effective behavioral treatment of focal hand dystonia in musicians alters somatosensory cortical organization. Proceedings of the National Academy of Sciences of the United States of America, 100, 7942–7946. Cohen, L., Celnik, P., Pascual-Leone, A., Corwell, B., Faiz, L., Dambrosia, J., et al. (1997). Functional relevance of crossmodal plasticity in blind humans. Nature, 389, 180–183. Cohen, L. G., Weeks, R. A., Sadato, N., Celnik, P., Ishii, K., & Hallett, M. (1999). Period of susceptibility for cross-modal plasticity in the blind. Annals of Neurology, 45, 451–460. Collignon, O., Voss, P., Lassonde, M., & Lepore, F. (2009). Cross-modal plasticity for the spatial processing of sounds in visually deprived subjects. Experimental Brain Research, 192, 343–358. Crespi, B. J. (2000). The evolution of maladaptation. Heredity, 84, 623–629. Darwin, C. (Ed.), (1859). On the origin of species by means of natural selection, or the preservation of favoured races in the struggle for life. Albemarle Street, London: John Murray. Dobzhansky, T. (1956). What is an adaptive trait? The American Naturalist, 90, 337–347. Doucet, M. E., Bergeron, F., Lassonde, M., Ferron, P., & Lepore, F. (2006). Cross-modal reorganization and speech perception in cochlear implant users. Brain, 129, 3376–3383. Eggermont, J. J., & Roberts, L. E. (2004). The neuroscience of tinnitus. Trends in Neurosciences, 27, 676–682. Elbert, T., & Heim, S. (2001). A light and a dark side. Nature, 411, 139.
192 Elbert, T., Pantev, C., Wienbruch, C., Rockstroh, B., & Taub, E. (1995). Increased cortical representation of the fingers of the left hand in string players. Science, 270, 305–307. Fieger, A., Röder, B., Teder-Sälejärvi, W., Hillyard, S. A., & Neville, H. J. (2006). Auditory spatial tuning in late-onset blindness in humans. Journal of Cognitive Neuroscience, 18, 149–157. Finney, E. M., & Dobkins, K. R. (2001). Visual contrast sensitivity in deaf versus hearing populations: Exploring the perceptual consequences of auditory deprivation and experience with a visual language. Cognitive Brain Research, 11, 171–183. Flor, H. (2002). Phantom-limb pain: Characteristics, causes, and treatment. Lancet Neurology, 1, 182–189. Flor, H., Braun, C., Elbert, T., & Birbaumer, N. (1997). Extensive reorganization of primary somatosensory cortex in chronic back pain patients. Neuroscience Letters, 224, 5–8. Flor, H., & Diers, M. (2009). Sensorimotor training and cortical reorganization. NeuroRehabilitation, 25, 19–27. Flor, H., Elbert, T., Knecht, S., Wienbruch, C., Pantev, C., Birbaumer, N., et al. (1995). Phantom-limb pain as a perceptual correlate of cortical reorganization following arm amputation. Nature, 375, 482–484. Flor, H., Elbert, T., Mühlnickel, W., Pantev, C., Wienbruch, C., & Taub, E. (1998). Cortical reorganization and phantom phenomena in congenital and traumatic upper-extremity amputees. Experimental Brain Research, 119, 205–212. Flor, H., Nikolajsen, L., & Jensen, T. S. (2006). Phantom limb pain: A case of maladaptive CNS plasticity? Nature Reviews. Neuroscience, 7, 873–881. Ghalambor, C. K., McKay, J. K., Carroll, S. P., & Reznick, D. N. (2007). Adaptive versus non-adaptive phenotypic plasticity and the potential for contemporary adaptation in new environments. Functional Ecology, 21, 394–407. Gougoux, F., Belin, P., Voss, P., Lepore, F., Lassonde, M., & Zatorre, J. R. (2009). Voice perception in blind persons: A functional magnetic resonance imaging study. Neuropsychologia, 47, 2967–2974. Gougoux, F., Zatorre, R. J., Lassonde, M., Voss, P., & Lepore, F. (2005). A functional neuroimaging study of sound localization: Visual cortex activity predicts performance in early-blind individuals. PLoS Biology, 3, e27. Gould, S. J. (1991). Exaptation: A crucial tool for an evolutionary psychology. Journal of Social Issues, 47, 43–65. Gould, S. J., & Lewontin, R. C. (1979). The sprandels of San Marco and the Panglossian Paradigm: A critique of the adaptationist programme. Proceedings of the Royal Society of London, 205, 581–598. Greenough, W. T., Black, J. E., & Wallace, C. S. (1987). Experience and brain development. Child Development, 58, 539–559.
Hyde, K. L., Lerch, J., Norton, A., Forgeard, M., Winner, E., Evans, A. C., et al. (2009). Musical training shapes structural brain development. The Journal of Neuroscience, 29, 3019–3025. Johansson, B. B. (2006). Music and brain plasticity. European Review, 14, 49–64. Kaas, J. H. (1991). Plasticity of sensory and motor maps in adult mammals. Annual Review of Neuroscience, 14, 137–167. Katz, J., & Melzack, R. (1990). Pain ‘memories’ in phantom limbs: Review and clinical observations. Pain, 43, 319–336. King, A. J., & Parsons, C. (1999). Improved auditory spatial acuity in visually deprived ferrets. The European Journal of Neuroscience, 11, 3945–3956. Knecht, S., Henningsen, H., Elbert, T., Flor, H., Hohling, C., Pantev, C., et al. (1996). Reorganizational and perceptional changes after amputation. Brain, 119, 1213–1219. Knudsen, E. I. (2004). Sensitive periods in the development of the brain and behavior. Journal of Cognitive Neuroscience, 16, 1412–1425. Kolb, B., Gibb, R., & Robinson, T. E. (2003). Brain plasticity and behavior. Current Directions in Psychological Science, 12, 1–5. Lee, D. S., Lee, J. S., Kim, S. H., Kim, J. W., Chung, J. K., Lee, M. C., et al. (2001). Cross-modal plasticity and cochlear implants. Nature, 409, 149–150. Lee, J. S., Lee, D. S., Oh, S. H., Kim, C. S., Kim, J. W., Hwang, C. H., et al. (2003). PET evidence of neuroplasticity in adult auditory cortex of postlingual deafness. Journal of Nuclear Medicine, 9, 1435–1439. Levänen, S., & Hamdorf, D. (2001). Feeling vibrations: Enhanced tactile sensitivity in congenitally deaf humans. Neuroscience Letters, 301, 75–77. Levänen, S., Jousmaki, V., & Hari, R. (1998). Vibrationinduced auditory-cortex activation in a congenitally deaf adult. Current Biology, 8, 869–873. Litovsky, R. Y., Johnstone, P. M., & Godar, S. P. (2006). Benefits of bilateral cochlearimplants and/or hearing aids in children. International Journal of Audiology, 45, 78–91. Lockwood, A. H., Salvi, R. J., & Burkhard, R. F. (2002). Tinnitus. The New England Journal of Medicine, 347, 904–910. Loke, W. H., & Song, S. (1991). Central and peripheral visual processing in hearing and nonhearing individuals. Bulletin of the Psychonomic Society, 29, 437–440. Lomber, S. G., Meredith, M. A., & Kral, A. (2010). Crossmodal plasticity in specific auditory cortices underlies visual compensations in the deaf. Nature Neuroscience, 13, 1421–1427. Lotze, M., Grodd, W., Birbaumer, N., Erb, M., Huse, E., & Flor, H. (1999). Does use of a myoelectric prosthesis prevent cortical reorganization and phantom limb pain? Nature, 2, 501–502.
193 Lövdén, M., Bäckman, L., Lindenberger, U., Schaefer, S., & Schmiedek, F. (2010). A theoretical framework for the study of adult cognitive plasticity. Psychological Bulletin, 136, 659–676. Miyamoto, R. T., Wynne, M. K., McKnight, C., & Bichey, B. (1997). Electrical suppression of tinnitus via cochlear implants. The International Tinnitus Journal, 3, 35–38. M ller, A. R., Langguth, B., DeRidder, D., & Kleinjung, T. (Eds.), (2010). Textbook of Tinnitus. Springer: New York, USA. Moore, D. R., & Shannon, R. V. (2009). Beyond cochlear implants: Awakening the deafened brain. Nature Neuroscience, 12, 686–691. Moreno, S., Marques, C., Santos, A., Santos, M., Castro, S. L., & Besson, M. (2009). Musical training influences linguistic abilities in 8-year-old children: More evidence for brain plasticity. Cerebral Cortex, 19, 712–723. Mühlnickel, W., Elbert, T., Taub, E., & Flor, H. (1998). Reorganization of auditory cortex in tinnitus. Proceedings of the National Academy of Sciences of the United States of America, 95, 10340–10343. Münte, T. F., Altenmüller, E., & Jäncke, L. (2002). The musician's brain as a model of neuroplasticity. Nature Neuroscience. Reviews, 3, 473–478. Nava, E., Bottari, D., Bonfioli, F., Beltrame, M. A., & Pavani, F. (2009a). Spatial hearing with a single cochlear implant in late-implanted adults. Hearing Research, 255, 91–98. Nava, E., Bottari, D., Portioli, G., Bonfioli, F., Beltrame, M. A., Formigoni, P., et al. (2009b). Hearing again with two ears: Recovery of spatial hearing after bilateral cochlear implantation. Neuropsychologia, 47, 928–932. Nava, E., Bottari, D., Zampini, M., & Pavani, F. (2008). Visual temporal order judgment in profoundly deaf individuals. Experimental Brain Research, 190, 179–188. Neville, H. J., & Lawson, D. S. (1987). Attention to central and peripheral visual space in a movement detection task: An event related potential and behavioral study. II. Congenitally deaf adults. Brain Research, 405, 268–283. Nikolajsen, L., Ilkjaer, S., Kroner, K., Christensen, J. H., & Jensen, T. S. (1997). The influence of preamputation pain on postamputation stump and phantom pain. Pain, 72, 393–405. Noreña, A. J., & Eggermont, J. J. (2003). Changes in spontaneous neural activity immediately after an acoustic trauma: Implications for neural correlates of tinnitus. Hearing Research, 183, 137–153. Okamoto, H., Stracke, H., Stoll, W., & Pantev, C. (2010). Listening to tailor-made notched music reduces tinnitus loudness and tinnitus-related auditory cortex activity. Proceedings of the National Academy of Sciences of the United States of America, 107, 1207–1210. Pascual-Leone, A., Amedi, A., Fregni, F., & Merabet, L. B. (2005). The plastic human brain cortex. Annual Review of Neuroscience, 28, 377–401.
Pascual-Leone, A., & Hamilton, R. (2001). The metamodal organization of the brain. Progress in Brain Research, 134, 427–445. Pavani, F., & Röder, B. (2011). Crossmodal plasticity as a consequence of sensory loss: Insights from blindness and deafness. In B. E. Stein (Ed.), The new handbook of multisensory processes. Cambridge, MA: MIT Press. Peterson, N. R., Pisoni, D. B., & Miyamoto, R. T. (2010). Cochlear implants and spoken language processing abilities: Review and assessment of the literature. Restorative Neurology and Neuroscience, 28, 237–250. Pons, T. P., Garraghty, P. E., Ommaya, A. K., Kaas, J. H., Taub, E., & Mishkin, M. (1991). Massive cortical reorganization after sensory deafferentation in adult macaques. Science, 28, 1857–1860. Price, T. D., Qvarnström, A., & Irwin, D. E. (2003). The role of phenotypic plasticity in driving genetic evolution. Proceedings: Biological Sciences, 270, 1433–1440. Quartarone, A., Bagnato, S., Rizzo, V., Siebner, H. R., Dattola, V., Scalfari, A., Morgante, F., Battaglia, F., Romano, M., & Girlanda, P. (2003). Abnormal associative plasticity of the human motor cortex in writer's cramp. Brain, 126, 2586–2596. Rauschecker, J. P. (1995). Compensatory plasticity and sensory substitution in the cerebral cortex. Trends in Neurosciences, 18, 36–43. Rauschecker, J. P., & Kniepert, U. (1994). Auditory localization behaviour in visually deprived cats. The European Journal of Neuroscience, 6, 149–160. Rauschecker, J. P., & Shannon, R. V. (2002). Sending sound to the brain. Science, 295, 1025–1029. Röder, B., Rösler, F., & Henninghausen, E. (1997). Different cortical activation patterns in blind and sighted humans during encoding and transformation of haptic images. Psychophysiology, 34, 292–307. Röder, B., Rösler, F., Henninghausen, E., & Näcker, F. (1996). Event-related potentials during auditory and somatosensory discrimination in sighted and blind human subjects. Cognitive Brain Research, 4, 77–93. Röder, B., Stock, O., Bien, S., Neville, H., & Rösler, F. (2002). Speech processing activates visual cortex in congenitally blind humans. The European Journal of Neuroscience, 16, 930–936. Röder, B., Teder-Sälejärvi, W., Sterr, A., Rösler, F., Hillyard, S. A., & Neville, H. J. (1999). Improved auditory spatial tuning in blind humans. Nature, 400, 162–166. Ruckenstein, M. J., Hedgepeth, C., Rafter, K. O., Montes, M. L., & Bigelow, D. C. (2001). Tinnitus suppression in patients with cochlear implants. Otology & Neurotology, 22, 200–204. Sadato, N., Okada, T., Honda, M., & Yonekura, Y. (2002). Critical period for cross-modal plasticity in blind humans: A functional MRI study. Neuroimage, 16, 389–400.
194 Sadato, N., Pascual-Leone, A., Grafman, J., Ibañez, V., Deiber, M. P., Dold, G., et al. (1996). Activation of the primary visual cortex by Braille reading in blind subjects. Nature, 380, 526–528. Sauvé, K. (1999). Gamma-band synchronous oscillations: Recent evidence regarding their functional significance. Consciousness and Cognition, 8, 213–224. Seki, S., & Eggermont, J. J. (2003). Changes in spontaneous firing rate and neural synchrony in cat primary auditory cortex after localized tone-induced hearing loss. Hearing Research, 180, 28–38. Sharma, A., Dorman, M. F., & Kral, A. (2005). The influence of a sensitive period on central auditory development in children with unilateral and bilateral cochlear implants. Hearing Research, 203, 134–143. Sharma, A., Dorman, M. F., & Spahr, A. J. (2002). A sensitive period for the development of the central auditory system in children with cochlear implants: Implications for age of implantation. Ear and Hearing, 23, 532–539. Sur, M., Pallas, S. L., & Roe, A. (1990). Cross-modal plasticity in cortical development: Differentiation and specification of sensory neocortex. Trends in Neurosciences, 13, 227–233. Svirsky, M. A., Robbins, A. M., Iler Kirk, K., Pisoni, D. B., & Miyamoto, R. T. (2000). Language development in profoundly deaf children with cochlear implants. Psychological Science, 11, 153–158. Tamura, Y., Ueki, Y., Lin, P., Vorbach, S., Mima, T., Kakigi, R., et al. (2009). Disordered plasticity in the primary somatosensory cortex in focal hand dystonia. Brain, 132, 749–755. Trussell, G. C., & Smith, L. D. (2000). Induced defenses in response to an invading crab predator: An explanation of historical and geographic phenotypic change. Proceedings of the National Academy of Sciences of the United States of America, 97, 2123–2127.
Van Hoesel, R. J. M. (2004). Exploring the benefits of bilateral cochlear implants. Audiology & Neurotology, 9, 234–246. Vanlierde, A., De Volder, A. G., Wanet-Defalque, M. C., & Veraart, C. (2003). Occipito-parietal cortex activation during visuo-spatial imagery in early blind humans. Neuroimage, 19, 698–709. Via, S., Gomulkiewicz, R., De Jong, G., Scheiner, S. M., Schlichting, C. D., & Van Tienderen, P. H. (1995). Adaptive phenotypic plasticity: Consensus and controversy. Trends in Ecology & Evolution, 10, 212–217. Von Melchner, L., Pallas, S. L., & Sur, M. (2000). Visual behaviour mediated by retinal projections directed to the auditory pathway. Nature, 404, 871–876. Voss, P., Lassonde, M., Gougoux, F., Fortin, M., Guillemot, J. P., & Lepore, F. (2004). Early- and late-onset blind individuals show supra-normal auditory abilities in far-space. Current Biology, 14, 1734–1738. Watanabe, D., Savion-Lemieux, T., & Penhune, V. B. (2007). The effect of early musical training on adult motor performance: Evidence for a sensitive period in motor learning. Experimental Brain Research, 176, 332–340. Weisz, N., Müller, S., Schlee, W., Dohrmann, K., Hartmann, T., & Elbert, T. (2007). The neural code of auditory phantom perception. The Journal of Neuroscience, 27, 1479–1484. Weisz, N., Wienbruch, C., Dohrmann, K., & Elbert, T. (2005). Neuromagnetic indicators of auditory cortical reorganization of tinnitus. Brain, 128, 2722–2731. Will, B., Dalrymple-Alford, J., Wolff, M., & Cassel, J. C. (2008). The concept of brain plasticity—Paillard's systemic analysis and emphasis on structure and function (followed by the translation of a seminal paper by Paillard on plasticity). Behavioral Brain Research, 192, 2–7.
A. M. Green, C. E. Chapman, J. F. Kalaska and F. Lepore (Eds.) Progress in Brain Research, Vol. 191 ISSN: 0079-6123 Copyright Ó 2011 Elsevier B.V. All rights reserved.
CHAPTER 13
Sensory integration for reaching: Models of optimality in the context of behavior and the underlying neural circuits Philip N. Sabes* Department of Physiology, Keck Center for Integrative Neuroscience, University of California, San Francisco, California, USA
Abstract: Although multisensory integration has been well modeled at the behavioral level, the link between these behavioral models and the underlying neural circuits is still not clear. This gap is even greater for the problem of sensory integration during movement planning and execution. The difficulty lies in applying simple models of sensory integration to the complex computations that are required for movement control and to the large networks of brain areas that perform these computations. Here I review psychophysical, computational, and physiological work on multisensory integration during movement planning, with an emphasis on goal-directed reaching. I argue that sensory transformations must play a central role in any modeling effort. In particular, the statistical properties of these transformations factor heavily into the way in which downstream signals are combined. As a result, our models of optimal integration are only expected to apply “locally,” that is, independently for each brain area. I suggest that local optimality can be reconciled with globally optimal behavior if one views the collection of parietal sensorimotor areas not as a set of task-specific domains, but rather as a palette of complex, sensorimotor representations that are flexibly combined to drive downstream activity and behavior. Keywords: sensory integration; reaching; neurophysiology; parietal cortex; computational models; vision; proprioception. parameter, for example, when one can feel and see an object touching one's arm. Understanding how the brain combines these signals has been an active area of research. As described below, models of optimal integration have been successful at capturing psychophysical performance in a variety of tasks. Further, network models have shown how optimal integration could be instantiated in neural circuits.
Introduction Multiple sensory modalities often provide “redundant” information about the same stimulus *Corresponding author. Tel.: þ415-476-0364; Fax: þ415-502-4848 E-mail:
[email protected] DOI: 10.1016/B978-0-444-53752-2.00004-7
195
196
However, strong links have yet to be made between these bodies of work and neurophysiological data. Here we address how models of optimal integration apply to the context of a sensory-guided movement and its underlying neural circuitry. This chapter focuses on the sensory integration required for goal-directed reaching and how that integration is implemented in the parietal cortex. We show that normative models developed for perceptual tasks and simple neural network models cannot, on their own, explain behavioral and physiological observations. These principles may nonetheless apply at a “local level” within each neuronal population. The link between local optimality and globally optimal behavior is then considered in the context of the broad network of sensorimotor areas in parietal cortex.
Modeling the psychophysics of sensory integration The principal hallmark of sensory integration should be the improvement of performance when multiple sensory signals are combined. In order to test this concept, we must choose a performance criterion by which to judge improvement. In the case of perception for action, the goal is often to estimate a spatial variable from the sensory input, for example, the location of the hand or an object in the world. In this case, the simplest and most commonly employed measure of performance is the variability of the estimate. It is not difficult to show that the minimum variance combination of two unbiased estimates of a variable x is given by the expression: ^ 1 1 1 x1 ^ x2 2 2 ^xinteg ¼ s þ ; s ¼ þ ; integ s21 s22 s21 s22 integ ð1Þ where ^ xi ; i ¼ 1; 2, are the unimodal estimates and si2 are their variances. In other words, the integrated estimate ^ xinteg is the weighted sum of
the two unimodal estimates, with weights inversely proportional to the respective variances. Importantly, the variance of the integrated estimate, sinteg2, is always less than either of the unimodal variances. While Eq. (1) assumes that the unimodal estimates ^xi are scalar and independent (given x), the solution is easily extended to correlated or multidimensional signals. Further, since the unimodal estimates are often well approximated by independent, normally distributed random variables, ^xinteg can also be viewed as the Maximum Likelihood (ML) integrated estimate (Ernst and Banks, 2002; Ghahramani et al., 1997). This model has been tested psychophysically by measuring performance variability with unimodal sensory cues and then predicting either variability or bias with bimodal cues. Numerous studies have reported ML-optimal or near-optimal sensory integration in human subjects performing perceptual tasks (e.g., Ernst and Banks, 2002; Ghahramani et al., 1997; Jacobs, 1999; Knill and Saunders, 2003; van Beers et al., 1999). Sensory integration during reach behavior Sensory integration is more complicated for movement planning than for a simple perceptual task. The problem is that movement planning and execution rely on a number of different computations, and estimates of the same spatial variable may be needed for several of these. For example, there is both psychophysical (Rossetti et al., 1995) and physiological (Batista et al., 1999; Buneo et al., 2002; Kakei et al., 1999, 2001) evidence for two separate stages of movement planning, as illustrated in Fig. 1. First, the movement vector is computed as the difference between the target location and the initial position of the hand. Next, the initial velocity along the planned movement vector must be converted into joint angle velocities (or other intrinsic variables such as muscle activations), which amounts to evaluating an inverse kinematic or
197 Movement vector planning
Inverse model evaluation
Execution
Target
Initial position
Actual movement direction
Planned movement vector
Planned joint angle changes
Fig. 1. Two separate computations required for reach planning. Adapted from Sober and Sabes (2005).
dynamic model. This evaluation also requires knowing the initial position of the arm. When planning a reaching movement, humans can often both see and feel the location of their hand. The ML model of sensory integration would seem to predict that the same weighting of vision and proprioception should be used for both of the computations illustrated in Fig. 1. However, we have previously shown that when reaching to visual targets, the relative weighting of these signals was quite different for the two computations: movement vector planning relied almost entirely on vision of the hand, and the inverse model evaluation relied more strongly on proprioception (Sober and Sabes, 2003). We hypothesized that the difference was due to the nature of the computations. Movement vector planning requires comparing the visual target location to the initial hand position. Since proprioceptive signals would first have to be transformed, this computation favors vision. Conversely, evaluation of the inverse model deals with intrinsic properties of the arm, favoring proprioception. Indeed, when subjects are asked to reach to a proprioceptive target (their other hand), the weighting of vision is significantly reduced in the movement vector calculation (Sober and Sabes, 2005). We hypothesized that these results are consistent with “local” ML integration, performed separately for each computation, if sensory transformations inject variability into the transformed signal. In order to make this hypothesis quantitative, we must understand the role of sensory
transformations during reach planning and their statistical properties. We developed and tested a model for these transformations by studying patterns of reach errors (McGuire and Sabes, 2009). Subjects made a series of interleaved reaches to visual targets, proprioceptive targets (the other hand, unseen), or bimodal targets (the other hand, visible), as illustrated in Fig. 2a. These reaches were made either with or without visual feedback of the hand prior to reach onset, during an enforced delay period after target presentation (after movement onset, feedback was extinguished in all trials). We took advantage of a bias in reaching that naturally occurs when subjects fixate a location distinct from the reach target. Specifically, when subjects reach to a visual target in the peripheral visual field, reaches tend to be biased further from the fixation point (Bock, 1993; Enright, 1995). This pattern of reach errors is illustrated in the left-hand panels of Fig. 2b: when reaching left of the fixation point, a leftward bias is observed, and similarly for the right. Thus, these errors follow a retinotopic pattern, that is, the bias curves shift with the fixation point. The bias pattern changes, but remains retinotopic, when reaching to bimodal targets (Fig. 2c) or proprioceptive targets (Fig. 2d). Most notably, the sign of the bias switches for proprioceptive reaches: subjects tend to reach closer to the point of fixation. Finally, the magnitude of these errors depends on whether visual feedback of the reaching hand is available prior to movement onset (compare the top and bottom panels of Fig. 2b–d; see also Beurze et al. (2007)).
198 (b)
Visual targets Fixation points
Unseen proprioceptive target (left fingertip)
No feedback Feedback Bias angle (°)
(a)
(c)
Visual targets
Bimodal targets
(d) Proprioceptive targets
2 0 2
2 0 2 20
0 Left fix
Reaching hand, shown with visual feedback of fingertip location
20 Right fix
(e)
20
0
20
Proprioception Target (t) Hand (h)
20
2 Retinotopic representation t ret, h ret
Transformation
Retinotopic movement vector MV ret t ret h ret
xret xbody g
Integrated movement vector
xbody xret g
Body-centered Representation t body, h body
Body-centered movement vector MV body t body h body
Standard deviation (°, mean subtracted)
Gaze (g)
0
Reach variability
(f) Vision Target (t) Hand (h)
20
Target angle (°)
1 0 1 2
Vis Bimod Prop No feedback
Vis Bimod Prop Feedback
Fig. 2. (a) Experimental setup. Left subpanel: Subjects sat in a simple virtual reality apparatus with a mirror reflecting images presented on a rear projection screen (Simani et al., 2007). View of both arms was blocked, but artificial feedback of either arm could be given in the form of a disk of light that moved with the fingertip. The right and left arms were separated by a thin table, allowing subject to reach to their left hand without tactile feedback. We were thus able to manipulate both the sensory modality of the target (visual, proprioceptive, or bimodal) and the presence or absence of visual feedback of the reaching hand. Right subpanel: For each target and feedback condition, reaches were made to an array of targets (displayed individually during the experiment) with the eyes fixated on one of two fixation points. (b–d) Reach biases. Average reach angular errors are plotted as a function of target and fixation location, separately for each trial condition (target modality and presence of visual feedback). Target modalities were randomly interleaved across two sessions, one with visual feedback and one without. Solid lines: average reach errors (with standard errors) across eight subjects for each trial condition. Dashed lines: model fits to the data. The color of the line indicates the gaze location. (d) Schematic of the Bayesian parallel representations model of reach planning. See text for details. (e) Reach variability. Average variability of reach angle plotted for each trial condition. Solid lines: average standard deviation of reach error across subjects for each trial condition. Dashed lines: model predictions. Adapted from McGuire and Sabes (2011).
While these bias patterns might seem arbitrary, they suggest an underlying mechanism. First, the difference in the sign of errors for visual and proprioceptive targets suggests that the bias arises in the transformation from a retinotopic (or eyecentered) representation to a body-centered representation. To see why, consider that in its simplified one-dimensional form, the transformation requires only adding or subtracting the gaze location (see the box labeled “Transformation”
in Fig. 2e). This might appear to be a trivial computation. However, the internal estimate of gaze location is itself an uncertain quantity. We argued that this estimate relies on current sensory signals (proprioception or efference copy) as well as on an internal prior that “expects” gaze to be coincident with the target. Thus, the estimate of gaze would be biased toward a retinally peripheral target. Since visual and proprioceptive information about target location travels in different directions
199
through this transformation, a biased estimate of gaze location results in oppositely signed errors for the two signals, as observed in Fig. 2b and d. Further, because the internal estimate of gaze location is uncertain, the transformation adds variability to the signal (see also Schlicht and Schrater, 2007), even if the addition or subtraction operation itself can be performed without error (not necessarily the case for neural computations, Shadlen and Newsome, 1994). One consequence of this variability is that access to visual feedback of the hand would improve the reliability of an eye-centered representation (upper pathway in Fig. 2e) more than it would improve the reliability of a body-centered representation (low pathway in Fig. 2e), since the latter receives a transformed, and thus more variable, version of the signal. Therefore, if the final movement plan were constructed from the optimal combination of an eye-centered and body-centered plan (rightmost box in Fig. 2e), the presence of visual feedback of the reaching hand should favor the eye-centered representation. This logic explains why the visual feedback of the reaching hand decreases the magnitude of the bias for visual targets (when the eye-centered space is unbiased; Fig. 2b) but increases the magnitude of the bias for proprioceptive targets (when the eyecentered space is biased; Fig. 2d). Together, these ideas form the Bayesian integration model of reach planning with “parallel representations,” illustrated in Fig. 2e. In this model, all sensory inputs related to a given spatial variable are combined with weights inversely proportional to their local variability (Eq. 1), and a movement vector is then computed. This computation occurs simultaneously in an eye-centered and a body-centered representation. The two resultant movement vectors have different uncertainties, depending on the availability and reliability of the sensory signals they receive in a given experimental condition. The final output of the network is itself a weighted sum of these two representations. We fit the four free parameters of the model (corresponding to values of sensory variability) to the reach error data
shown in solid lines in Fig. 2b–d. The model captures those error patterns (dashed lines in Fig. 2b–d) and predicts the error patterns from two similar studies described above (Beurze et al., 2007; Sober and Sabes, 2005). In addition, the model predicts the differences we observed in reach variability across experimental conditions (Fig. 2f). These results challenge the idea that movement planning should begin by mapping the relevant sensory signals into a single common reference frame (Batista et al., 1999; Buneo et al., 2002; Cohen et al., 2002). The model shows that the use of two parallel representations of the movement plan yields a less variable output in the face of variable and sometimes missing sensory signals and noisy internal transformations. It is not clear whether or how this model can be mapped onto the real neural circuits that underlie reach planning. For example, the two parallel representations could be implemented by a single neuronal population (Pouget et al., 2002; Xing and Andersen, 2000; Zipser and Andersen, 1988). Before addressing this issue, though, we consider the question of how single neurons or populations of neurons should integrate their afferent signals.
Modeling sensory integration in neural populations Stein and colleagues have studied multimodal responses in single neurons in the deep layers of cat superior colliculus and have found both enhancement and suppression of multimodal responses (Meredith and Stein, 1983, 1986; Stein and Stanford, 2008). Based on this work, they suggest that the definition of sensory integration at the level of the single unit is for the responses to be significantly enhanced or suppressed relative to the preferred unimodal stimulus (Stein et al., 2009). However, this definition is overly broad and includes computations that are not
200
typically thought of as integration. For example, Kadunce et al. (1997) showed that cross-modal suppressive effects in the superior colliculus often mimic those observed for paired within-modality stimuli. These effects are most likely due not to integration but rather to competition within the spatial map of the superior colliculus, similar to the process seen during saccade planning in primate superior colliculus (Dorris et al., 2007; Trappenberg et al., 2001). The criterion for signal integration should be the presence of a shared representation that offers improved performance (e.g., reduced variability) when multimodal inputs are available. Here, a “shared” representation is one that encodes all sensory inputs similarly. Using the notation of Eq. (1), the strongest form a shared representation is one in which neural activity is function only of xinteg and sinteg2, rather than being a function of the independent inputs, x1, s12 and x2, s22. The advantage of such a representation is that downstream areas need not know about which sensory signals were available in order to use the information. Ma et al. (2006) suggest a relatively simple approach to achieving such an integrated representation. They show that a population of neurons that simply adds the firing rates of independent input populations (or their linear transformations) effectively implements ML integration, at least when firing rates have Poisson-like distributions. This result can be understood intuitively for Poisson firing rates. The variance of the ML decode from each population is inversely proportional to its gain. Therefore, summing the inputs yields a representation with the summed gains, and thus with variance that matches the optimum defined in Eq. (1) above. Further, because addition preserves information about variability, this operation can be repeated hierarchically, a desirable feature for building more complex circuits like those required for sensory-guided movement. It remains unknown whether real neural circuits employ such a strategy, or even if they
combine their inputs in a statistically optimal manner. In practice, it can be difficult to quantitatively test the predictions of this and similar models. For example, strict additivity of the inputs is not to be expected in many situations, such as in the presence of inputs that are correlated or non-Poisson, or if activity levels are normalized within a given brain area (Ma et al., 2006; Ma and Pouget, 2008). These difficulties are compounded for recordings from single neurons. In this case, biases in the unimodal representations of space would lead to changes in firing rates across modalities even in the absence of integration-related changes in gain. Nonetheless, several hallmarks of optimal integration have been observed in the responses of bimodal (visual and vestibular) motion encoding neurons in macaque area MST: bimodal activity is well modeled as a weighted linear sum of unimodal responses; the visual weighting decreases when the visual stimulus is degraded; and the variability of the decoding improves in the bimodal condition (Morgan et al., 2008). Similarly, neurons in macaque Area 5 appear to integrate proprioceptive and visual cues of arm location (Graziano et al., 2000). In particular, the activity for a mismatched bimodal stimulus is between that observed for location-matched unimodal stimuli (weighting), and the activity for matched bimodal stimuli is greater than that observed for proprioception alone (variance reduction).
Sensory integration in the cortical circuits for reaching In the context of the neural circuits for reach planning, locally optimal integration of incoming signals could be sufficient to explain behavior, as illustrated in the parallel representations model of Fig. 2. Here we ask whether the principles in this model can be mapped onto the primate cortical reach network.
201
Volitional arm movements in primate involve a large network of brain areas with a rich pattern of interarea connectivity. Within this larger circuit, there is a subnetwork of areas, illustrated in Fig. 3, that appear to be responsible for the complex sensorimotor transformations required for goaldirected reaches under multisensory guidance. Visual information primarily enters this network via the parietal–occipital area, particularly Area V6 (Galletti et al., 1996; Shipp et al., 1998). Proprioceptive information primarily enters via Area 5, which receives direct projections from primary somatosensory cortex (Crammond and Kalaska, 1989; Kalaska et al., 1983; Pearson and Powell, 1985). These visual and proprioceptive signals converge on a group of parietal sensorimotor areas in or near the intraparietal sulcus (IPS): MDP and 7m (Ferraina et al., 1997a,b; Johnson et al., 1996), V6a (Galletti et al., 2001; Shipp et al., 1998), and MIP and VIP (Colby et al., 1993; Duhamel et al., 1998). The parietal reach region (PRR), characterized physiologically by Andersen and colleagues (Batista et al., 1999; Snyder et al., 1997), includes portions of MIP, V6a, and MDP (Snyder et al., 2000a). These parietal areas project forward to the dorsal premotor cortex (PMd) and, in some cases, the primary motor cortex (M1), and they all exhibit some degree of activity related to visual and proprioceptive movement cues, the pending movement plan (“set” or “delay” activity), and the ongoing movement kinematics or dynamics. While the network illustrated in Fig. 3 is clearly more complex than the simple computational
V2, MT,
Vision MST, etc.
PO (V6)
schematic of Fig. 2e, there is a suggestive parallel. While both Area 5 and MIP integrate multimodal signals and project extensively to the rest of the reach circuit, they differ in their anatomical proximity to their visual versus proprioceptive inputs: Area 5 is closer to somatosensory cortex, and MIP is closer to the visual inputs to reach circuit. Further, Area 5 uses more body- or hand-centered representations compared to the eye-centered representations reported in MIP (Batista et al., 1999; Buneo et al., 2002; Chang and Snyder, 2010; Colby and Duhamel, 1996; Ferraina et al., 2009; Kalaska, 1996; Lacquaniti et al., 1995; Marconi et al., 2001; Scott et al., 1997). Thus, these areas are potential candidates for the parallel representations predicted in the behavioral model. To test this possibility, we recorded from Area 5 and MIP (Fig. 4a) as macaque monkeys performed the same psychophysical task that was illustrated in Fig. 2a for human subjects (McGuire and Sabes, 2011). One of the questions we addressed in this study is whether there is evidence for parallel representations of the movement plan in body and eye-centered reference frames. We performed several different analyses to characterize neural reference frames; here we focus on the tuning-curve approach illustrated in Fig. 4b. Briefly, we fit a tuning curve to the neural responses for a range of targets with two different fixation points (illustrated schematically as the red and blue curves in Fig. 4b). Tuning was assumed to be a function of TdE, where T and E are the target and eye locations in absolute (or body-centered) space and d is a dimensionless quantity. If d ¼ 0,
MDP, 7m V6a, MIP
VIP
Proprioception
Area 3b
Areas 1,2
Fig. 3. Schematic illustration of the cortical reach circuit.
Area 5
PMd
M1
Motor
202 (c)
(a)
Fraction of recorded tuning curves VIS n = 95 BIMODAL n = 121 PROP n = 144
CS PCD
0.2
Are
Area 5
a5
0.1 10 mm IPS
P↔A
0
0
1
Schematic tuning curves VIS n = 195
(d)
(b) Body-centered
Intermediate
BIMODAL n = 185
Eye-centered
PROP n = 168
d0
d 0.5
MIP
Firing rate
0.2
d1
0.1
Left Right Fix Fix 0
0 Target location (with respect to body)
0
0
0
1 Shift (d)
Fig. 4. (a) Recording locations. Approximate location of neural recordings in Area 5 with respect to sulcal anatomy. Recordings in MIP were located in the same region of cortex, but at a deeper penetration depth. The boundary between Area 5 and MIP was set to a nominal value of 2000 mm below the dura. This value was chosen based on anatomical MR images and the stereotactic coordinates of the recording cylinder. (b) Schematic illustration of the tuning-curve shift, d. Each curve represents the firing rate for an idealized cell as a function of target for either the left (red) or right (blue) fixation points. Three idealized cells are illustrated, with shift values of d ¼ 0, 0.5, and 1. See text for more details. (c, d) Distribution of shift values estimated from neural recordings in Area 5 (c) and MIP (d). Each cell may be included up to three times, once for the delay, reaction time, and movement epoch tuning curves. The number of tuning curves varies across modalities because tuning curves were only included when the confidence limit on the best-fit value of d had a range of less than 1.5, a conservative value that excluded untuned cells. Adapted from McGuire and Sabes (2011).
firing rate depends on the body-centered location of the target (left panel of Fig. 4b), and if d ¼ 1, firing rate depends on the eye-centered location of the target (right panel of Fig. 4b). We found a large degree of heterogeneity in the shift values across cells, but there was no difference in the mean or distribution of shift values across target modality for either cortical area
(Fig. 4c and d), that is, these are shared (modality-invariant) representations. Although some evidence for other shared movement-related representations have been found in the parietal cortex (Cohen et al., 2002), many studies of multisensory areas in the parietal cortex and elsewhere have found that representations are determined, at least in part, by the representation
203
of the current sensory input (Avillac et al., 2005; Fetsch et al., 2007; Jay and Sparks, 1987; Mullette-Gillman et al., 2005; Stricanne et al., 1996). Shared representations such as those we have observed have the advantage that downstream areas do not need to know which sensory signals are available in order to use the representation. We also observed significant differences in the mean and distribution of shift values across cortical areas, with MIP exhibiting a more eye-centered representation (mean d ¼ 0.51), while Area 5 has a more body-centered representation (mean d ¼ 0.25). In a separate analysis, we showed that more MIP cells encode target location alone, compared to Area 5, where more cells encode both target and hand location (McGuire and Sabes, 2011). These inter-area differences parallel observations from Andersen and colleagues of eye-centered target coding in PRR (Batista et al., 1999) and eye-centered movement vector representation for Area 5 (Buneo et al., 2002). However, where those papers report consistent, eye-centered reference frames, we observed a great deal of heterogeneity in representations within each area, with most cells exhibiting “intermediate” shifts between 0 and 1. We think this discrepancy lies primarily in the analyses used: the shift analysis does not force a choice between alternative reference frames, but rather allows for a continuum of intermediate reference frames. When an approach very similar to ours was applied to recordings from a more posterior region of the IPS, a similar spread of shift values was obtained, although the mean shift value was somewhat closer to unity (Chang and Snyder, 2010). While we did not find the simple eye- and bodycentered representations that were built into the parallel representations model of Fig. 4d, these physiological results can nonetheless be interpreted in light of that model. We found that both Area 5 and MIP use modality-invariant representations of the movement plan, an important feature of the model. Further, there are multiple integrated representations of the movement plan within the superior parietal lobe, with an anterior to posterior gradient in the magnitude of
gaze-dependent shifts (Chang and Snyder, 2010; McGuire and Sabes, 2011). A statistically optimal combination of these representations, dynamically changing with the current sensory inputs, would likely provide a close match to the output of the model. The physiological recordings also revealed a great deal of heterogeneity in shift values, suggesting an alternate implementation of the model. Xing and Andersen (2000) have observed that a network with a broad distribution of reference-frame shifts can be used to compute multiple simultaneous readouts, each in a different reference frame. Indeed, a broad distribution of gazedependent tuning shifts has been observed within many parietal areas (Avillac et al., 2005; Chang and Snyder, 2010; Duhamel et al., 1998; Mullette-Gillman et al., 2005; Stricanne et al., 1996). Thus, parallel representations of movement planning could also be implemented within a single heterogeneous population of neurons.
From local to global optimality We have adopted a simple definition of sensory integration, namely, improved performance when multiple sensory modalities are available— whether in a behavioral task or with respect to the variability of neural representations. This definition leads naturally to criteria for optimal integration such as the minimum variance/ML model of Eq. (1), and a candidate mechanism for achieving such optimality was discussed above (Ma et al., 2006). In the context of a complex sensorimotor circuit, a mechanism such as this could be applied at the local level to integrate the afferent signals at each cortical area, independently across areas. However, these afferent signals will include the effects of the specific combination of upstream transformations, and so such a model would only appear to be optimal at the local level. It remains an open question as to how locally optimal (or near-optimal) integration could lead to globally optimal (or near-optimal) behavior.
204
The parietal network that underlies reaching is part of a larger region along the IPS that subserves a wide range of sensorimotor tasks (reviewed, e.g., in Andersen and Buneo, 2002; Burnod et al., 1999; Colby and Goldberg, 1999; Grefkes and Fink, 2005; Rizzolatti et al., 1997). These tasks make use of many sensory inputs, each naturally linked to a particular reference frame (e.g., visual signals originate in a retinotopic reference frame), as well as an array of kinematic feedback signals needed to transform from one reference frame to another. In this context, it seems logical to suggest a series of representations and transformations, for example, from eye-centered to hand-centered space, as illustrated in Fig. 5a. This schema offers a great degree of flexibility, since the “right” representation would be available for any given task. An attractive hypothesis is that a schema such as this could be mapped onto the series of sensorimotor representations that lie along the IPS, for example, from the retinotopic visual maps in Area V6 (Fattori et al., 2009; Galletti et al., 1996) to the hand-centered grasp-related activity in AIP. The pure reference-frame representations illustrated in the schema of Fig. 5a are not consistent with the evidence for heterogeneous “intermediate” representations. However, the general schema of a sequence of transformations and representations might still be correct, since the neural circuits implementing these transformations need not represent these variables in the reference frames of their inputs, as illustrated by several network models of reference-frame transformations (Blohm et al., 2009; Deneve et al., 2001; Salinas and Sejnowski, 2001; Xing and Andersen, 2000; Zipser and Andersen, 1988). The use of network models such as these could reconcile the schema of Fig. 5a with the physiological data (Pouget et al., 2002; Salinas and Sejnowski, 2001). While this schema is conceptually attractive, it has disadvantages. As described above, each transformation will inject variability into the circuit. This variability would accrue along the
sequence of transformations, a problem that could potentially be avoided by “direct” sensorimotor transformations such as those proposed by Buneo et al. (2002). Further, in order not to lose fidelity along this sequence, all intermediate representations require comparably sized neuronal populations, even representations that are rarely directly used for behavior. Ideally, one would be able to allocate more resources to a retinotopic representation, for example, than an elbow-centered representation. An alternative schema is to combine many sensory signals into each of a small number of representations; in the limit, a single complex representational network could be used (Fig. 5b). It has been shown that multiple reference frames can be read out from a single network of neurons when those neurons use “gain-field” representations, that is, when their responses are multiplicative in the various input signals (Salinas and Abbott, 1995; Salinas and Sejnowski, 2001; Xing and Andersen, 2000). More generally, nonlinear basis functions create general purpose representations that can be used to compute (at least approximately) a wide range of task-relevant variables (Pouget and Sejnowski, 1997; Pouget and Snyder, 2000). In particular, this approach would allow “direct” transformations from sensory to motor variables (Buneo et al., 2002) without the need for intervening sequences of transformations. However, this schema also has limitations. In order to represent all possible combinations of variables, the number of required neurons increases exponentially with the number of input variables (the “curse-of-dimensionality”). Indeed, computational models of such generic networks show a rapid increase in errors as the number of input variables grow (Salinas and Abbott, 1995). This limitation becomes prohibitive when the number of sensorimotor variables approaches a realistic value. A solution to this problem, illustrated in Fig. 5c, is to have a large number of networks, each with only a few inputs or encoding only a small subspace of the possible outputs. These representations
205
(a) Saccade
Gaze shift
Trunk movements
Eye-centered representation
Head-centered representation
Trunk-centered representation
Eye position
Vision
(b)
Saccade
Head position
Audition
Tactile inputs
Gaze shift
Trunk movements
Grasp
...
Tasks
Hand-centered representation Kinematic feedback for transformations
Wrist posture
Tactile inputs
Other sensory inputs
Grasp
Tasks
Tactile inputs
Kinematic feedback and other sensory inputs
Grasp
Tasks
High-dimensional integrated representation
Eye position Vision
(c)
Head position Audition
Saccade
Wrist posture
Tactile inputs
Gaze shift
Trunk movements
Low-dimensional integrated representations
...
Eye position Vision
Head position Audition
Tactile inputs
Wrist posture Tactile inputs
Kinematic feedback and other sensory inputs
Fig. 5. Three schematic models of the parietal representations of sensorimotor space. (a) A sequence of transformations that follows the kinematics of the body. Each behavior uses the representation that most closely matches the space of the task. (b) A single high-dimensional representation that integrates all of the relevant sensorimotor variables and subserves the downstream computations for all tasks. (c) A large collection of low-dimensional integrated representations with overlapping sensory inputs and a high degree of interconnectivity. Each white box represents a different representation of sensorimotor space. The nature of these representations is determined by their inputs, and their statistical properties (e.g., variability, gain) will depend on the sensory signals available at the time. The computations performed for any given task make use of several of these representations, with the relative weighting dynamically determined by their statistical properties.
206
would likely have complex or “intermediate” representations of sensorimotor space that would not directly map either to particular stages in the kinematic sequence (5A) or to the “right” reference frames for a set of tasks. Instead, the downstream circuits for behavior would draw upon several of these representations. This schema is consistent with the large continuum of representations seen along the IPS (reviewed, e.g., in Burnod et al., 1999), and the fact that the anatomical distinctions between nominal cortical areas in this region are unclear and remain a matter of debate (Cavada, 2001; Lewis and Van Essen, 2000). It is also consistent with the fact that there is a great deal of overlap in the pattern of cortical areas that are active during any given task, for example, saccade and reach activity have been observed in overlapping cortical areas (Snyder et al., 1997, 2000b) and grasp-related activity can be observed in nominally reach-related areas (Fattori et al., 2009). This suggests that brain areas around the IPS should not be thought of a set of task-specific domains (e.g., Andersen and Buneo, 2002; Colby and Goldberg, 1999; Grefkes and Fink, 2005), but rather as a palette of complex, sensorimotor representations. This picture suggests a mechanism by which locally optimal integration could yield globally optimal behavior, essentially a generalization of the parallel representations model of Fig. 4d. In both the parallel representations model and the schema of Fig. 5c, downstream motor circuits integrate overlapping information from multiple sensorimotor representations of space. For any specific instance of a behavior, the weighting of these representations should depend on their relative variability, perhaps determined by gain (Ma et al., 2006), and this variability would depend on the sensory and motor signals available at that time. If each of the representations in this palette contains a locally optimal mixture of its input signals, optimal weighting of the downstream projections from this palette could drive statistically efficient behavior.
Acknowledgments This work was supported by the National Eye Institute (R01 EY-015679) and the National Institute of Mental Health (P50 MH77970). I thank John Kalaska, Joseph Makin, and Matthew Fellows for reading and commenting on earlier drafts of this chapter.
Abbreviations ML MST MDP MIP VIP PMd M1 PRR IPS
maximum likelihood medial superior temporal area medial dorsal parietal area medial intraparietal area ventral intraparietal area dorsal premotor cortex primary motor cortex parietal reach region intraparietal sulcus
References Andersen, R. A., & Buneo, C. A. (2002). Intentional maps in posterior parietal cortex. Annual Review of Neuroscience, 25, 189–220. Avillac, M., Deneve, S., Olivier, E., Pouget, A., & Duhamel, J. R. (2005). Reference frames for representing visual and tactile locations in parietal cortex. Nature Neuroscience, 8, 941–949. Batista, A. P., Buneo, C. A., Snyder, L. H., & Andersen, R. A. (1999). Reach plans in eye-centered coordinates. Science, 285, 257–260. Beurze, S. M., de Lange, F. P., Toni, I., & Medendorp, W. P. (2007). Integration of target and effector information in the human brain during reach planning. Journal of Neurophysiology, 97, 188–199. Blohm, G., Keith, G. P., & Crawford, J. D. (2009). Decoding the cortical transformations for visually guided reaching in 3D space. Cerebral Cortex, 19(6), 1372–1393. Bock, O. (1993). Localization of objects in the peripheral visual field. Behavioural Brain Research, 56, 77–84. Buneo, C. A., Jarvis, M. R., Batista, A. P., & Andersen, R. A. (2002). Direct visuomotor transformations for reaching. Nature, 416, 632–636.
207 Burnod, Y., Baraduc, P., Battaglia-Mayer, A., Guigon, E., Koechlin, E., Ferraina, S., et al. (1999). Parieto-frontal coding of reaching: An integrated framework. Experimental Brain Research, 129, 325–346. Cavada, C. (2001). The visual parietal areas in the macaque monkey: Current structural knowledge and ignorance. Neuroimage, 14, S21–26. Chang, S. W., & Snyder, L. H. (2010). Idiosyncratic and systematic aspects of spatial representations in the macaque parietal cortex. Proceedings of the National Academy of Sciences of the United States of America, 107, 7951–7956. Cohen, Y. E., Batista, A. P., & Andersen, R. A. (2002). Comparison of neural activity preceding reaches to auditory and visual stimuli in the parietal reach region. Neuroreport, 13, 891–894. Colby, C. L., & Duhamel, J. R. (1996). Spatial representations for action in parietal cortex. Brain Research. Cognitive Brain Research, 5, 105–115. Colby, C. L., Duhamel, J. R., & Goldberg, M. E. (1993). Ventral intraparietal area of the macaque: Anatomic location and visual response properties. Journal of Neurophysiology, 69, 902–914. Colby, C. L., & Goldberg, M. E. (1999). Space and attention in parietal cortex. Annual Review of Neuroscience, 22, 319–349. Crammond, D. J., & Kalaska, J. F. (1989). Neuronal activity in primate parietal cortex area 5 varies with intended movement direction during an instructed-delay period. Experimental Brain Research, 76, 458–462. Deneve, S., Latham, P. E., & Pouget, A. (2001). Efficient computation and cue integration with noisy population codes. Nature Neuroscience, 4, 826–831. Dorris, M. C., Olivier, E., & Munoz, D. P. (2007). Competitive integration of visual and preparatory signals in the superior colliculus during saccadic programming. The Journal of Neuroscience, 27, 5053–5062. Duhamel, J. R., Colby, C. L., & Goldberg, M. E. (1998). Ventral intraparietal area of the macaque: Congruent visual and somatic response properties. Journal of Neurophysiology, 79, 126–136. Enright, J. T. (1995). The non-visual impact of eye orientation on eye-hand coordination. Vision Research, 35, 1611–1618. Ernst, M. O., & Banks, M. S. (2002). Humans integrate visual and haptic information in a statistically optimal fashion. Nature, 415, 429–433. Fattori, P., Pitzalis, S., & Galletti, C. (2009). The cortical visual area V6 in macaque and human brains. Journal of Physiology (Paris), 103, 88–97. Ferraina, S., Brunamonti, E., Giusti, M. A., Costa, S., Genovesio, A., & Caminiti, R. (2009). Reaching in depth: Hand position dominates over binocular eye position in the rostral superior parietal lobule. The Journal of Neuroscience, 29, 11461–11470.
Ferraina, S., Garasto, M. R., Battaglia-Mayer, A., Ferraresi, P., Johnson, P. B., Lacquaniti, F., et al. (1997a). Visual control of hand-reaching movement: Activity in parietal area 7m. The European Journal of Neuroscience, 9, 1090–1095. Ferraina, S., Johnson, P. B., Garasto, M. R., BattagliaMayer, A., Ercolani, L., Bianchi, L., et al. (1997b). Combination of hand and gaze signals during reaching: Activity in parietal area 7 m of the monkey. Journal of Neurophysiology, 77, 1034–1038. Fetsch, C. R., Wang, S., Gu, Y., Deangelis, G. C., & Angelaki, D. E. (2007). Spatial reference frames of visual, vestibular, and multimodal heading signals in the dorsal subdivision of the medial superior temporal area. The Journal of Neuroscience, 27, 700–712. Galletti, C., Fattori, P., Battaglini, P., Shipp, S., & Zeki, S. (1996). Functional demarcation of a border between areas V6 and V6A in the superior parietal gyrus of the macaque monkey. The European Journal of Neuroscience, 8, 30–52. Galletti, C., Gamberini, M., Kutz, D. F., Fattori, P., Luppino, G., & Matelli, M. (2001). The cortical connections of area V6: An occipito-parietal network processing visual information. The European Journal of Neuroscience, 13, 1572–1588. Ghahramani, Z., Wolpert, D. M., & Jordan, M. I. (1997). Computational models of sensorimotor integration. In P. G. Morasso & V. Sanguineti (Eds.), Self-organization, computational maps and motor control (pp. 117–147). Oxford: Elsevier. Graziano, M. S., Cooke, D. F., & Taylor, C. S. (2000). Coding the location of the arm by sight. Science, 290, 1782–1786. Grefkes, C., & Fink, G. R. (2005). The functional organization of the intraparietal sulcus in humans and monkeys. Journal of Anatomy, 207, 3–17. Jacobs, R. A. (1999). Optimal integration of texture and motion cues to depth. Vision Research, 39, 3621–3629. Jay, M. F., & Sparks, D. L. (1987). Sensorimotor integration in the primate superior colliculus. II. Coordinates of auditory signals. Journal of Neurophysiology, 57, 35–55. Johnson, P. B., Ferraina, S., Bianchi, L., & Caminiti, R. (1996). Cortical networks for visual reaching: Physiological and anatomical organization of frontal and parietal lobe arm regions. Cerebral Cortex, 6, 102–119. Kadunce, D. C., Vaughan, J. W., Wallace, M. T., Benedek, G., & Stein, B. E. (1997). Mechanisms of within- and crossmodality suppression in the superior colliculus. Journal of Neurophysiology, 78, 2834–2847. Kakei, S., Hoffman, D. S., & Strick, P. L. (1999). Muscle and movement representations in the primary motor cortex. Science, 285, 2136–2139. Kakei, S., Hoffman, D. S., & Strick, P. L. (2001). Direction of action is represented in the ventral premotor cortex. Nature Neuroscience, 4, 1020–1025.
208 Kalaska, J. F. (1996). Parietal cortex area 5 and visuomotor behavior. Canadian Journal of Physiology and Pharmacology, 74, 483–498. Kalaska, J. F., Caminiti, R., & Georgopoulos, A. P. (1983). Cortical mechanisms related to the direction of two-dimensional arm movements: Relations in parietal area 5 and comparison with motor cortex. Experimental Brain Research, 51, 247–260. Knill, D. C., & Saunders, J. A. (2003). Do humans optimally integrate stereo and texture information for judgments of surface slant? Vision Research, 43, 2539–2558. Lacquaniti, F., Guigon, E., Bianchi, L., Ferraina, S., & Caminiti, R. (1995). Representing spatial information for limb movement: Role of area 5 in the monkey. Cerebral Cortex, 5, 391–409. Lewis, J. W., & Van Essen, D. C. (2000). Mapping of architectonic subdivisions in the macaque monkey, with emphasis on parieto-occipital cortex. Journal of Comparative Neurology, 428, 79–111. Ma, W. J., Beck, J. M., Latham, P. E., & Pouget, A. (2006). Bayesian inference with probabilistic population codes. Nature Neuroscience, 9, 1432–1438. Ma, W. J., & Pouget, A. (2008). Linking neurons to behavior in multisensory perception: A computational review. Brain Research, 1242, 4–12. Marconi, B., Genovesio, A., Battaglia-Mayer, A., Ferraina, S., Squatrito, S., Molinari, M., et al. (2001). Eye-hand coordination during reaching. I. Anatomical relationships between parietal and frontal cortex. Cerebral Cortex, 11, 513–527. McGuire, L. M., & Sabes, P. N. (2009). Sensory transformations and the use of multiple reference frames for reach planning. Nature Neuroscience, 12, 1056–1061. McGuire, L. M. M., & Sabes, P. N. (2011). Heterogeneous representations in the superior parietal lobule are common across reaches to visual and proprioceptive targets. The Journal of Neuroscience, 31(18), 6661–6673. Meredith, M. A., & Stein, B. E. (1983). Interactions among converging sensory inputs in the superior colliculus. Science, 221, 389–391. Meredith, M. A., & Stein, B. E. (1986). Visual, auditory, and somatosensory convergence on cells in superior colliculus results in multisensory integration. Journal of Neurophysiology, 56, 640–662. Morgan, M. L., Deangelis, G. C., & Angelaki, D. E. (2008). Multisensory integration in macaque visual cortex depends on cue reliability. Neuron, 59, 662–673. Mullette-Gillman, O. A., Cohen, Y. E., & Groh, J. M. (2005). Eye-centered, head-centered, and complex coding of visual and auditory targets in the intraparietal sulcus. Journal of Neurophysiology, 94, 2331–2352. Pearson, R. C., & Powell, T. P. (1985). The projection of the primary somatic sensory cortex upon area 5 in the monkey. Brain Research, 356, 89–107.
Pouget, A., Deneve, S., & Duhamel, J. R. (2002). A computational perspective on the neural basis of multisensory spatial representations. Nature Reviews. Neuroscience, 3, 741–747. Pouget, A., & Sejnowski, T. J. (1997). Spatial transformations in the parietal cortex using basis functions. Journal of Cognitive Neuroscience, 9, 222–237. Pouget, A., & Snyder, L. H. (2000). Computational approaches to sensorimotor transformations. Nature Neuroscience, 3, 1192–1198. Rizzolatti, G., Fogassi, L., & Gallese, V. (1997). Parietal cortex: From sight to action. Current Opinion in Neurobiology, 7, 562–567. Rossetti, Y., Desmurget, M., & Prablanc, C. (1995). Vectorial coding of movement: Vision, proprioception, or both? Journal of Neurophysiology, 74, 457–463. Salinas, E., & Abbott, L. F. (1995). Transfer of coded information from sensory to motor networks. The Journal of Neuroscience, 15, 6461–6474. Salinas, E., & Sejnowski, T. J. (2001). Gain modulation in the central nervous system: Where behavior, neurophysiology, and computation meet. The Neuroscientist, 7, 430–440. Schlicht, E. J., & Schrater, P. R. (2007). Impact of coordinate transformation uncertainty on human sensorimotor control. Journal of Neurophysiology, 97, 4203–4214. Scott, S. H., Sergio, L. E., & Kalaska, J. F. (1997). Reaching movements with similar hand paths but different arm orientations. II. Activity of individual cells in dorsal premotor cortex and parietal area 5. Journal of Neurophysiology, 78, 2413–2426. Shadlen, M. N., & Newsome, W. T. (1994). Noise, neural codes and cortical organization. Current Opinion in Neurobiology, 4, 569–579. Shipp, S., Blanton, M., & Zeki, S. (1998). A visuosomatomotor pathway through superior parietal cortex in the macaque monkey: Cortical connections of areas V6 and V6A. The European Journal of Neuroscience, 10, 3171–3193. Simani, M. C., McGuire, L. M., & Sabes, P. N. (2007). Visualshift adaptation is composed of separable sensory and taskdependent effects. Journal of Neurophysiology, 98, 2827–2841. Snyder, L. H., Batista, A. P., & Andersen, R. A. (1997). Coding of intention in the posterior parietal cortex. Nature, 386, 167–170. Snyder, L. H., Batista, A. P., & Andersen, R. A. (2000a). Intention-related activity in the posterior parietal cortex: A review. Vision Research, 40, 1433–1441. Snyder, L. H., Batista, A. P., & Andersen, R. A. (2000b). Saccade-related activity in the parietal reach region. Journal of Neurophysiology, 83, 1099–1102. Sober, S. J., & Sabes, P. N. (2003). Multisensory integration during motor planning. The Journal of Neuroscience, 23, 6982–6992.
209 Sober, S. J., & Sabes, P. N. (2005). Flexible strategies for sensory integration during motor planning. Nature Neuroscience, 8, 490–497. Stein, B. E., & Stanford, T. R. (2008). Multisensory integration: Current issues from the perspective of the single neuron. Nature Reviews. Neuroscience, 9, 255–266. Stein, B. E., Stanford, T. R., Ramachandran, R., Perrault, T. J., Jr.& Rowland, B. A. (2009). Challenges in quantifying multisensory integration: Alternative criteria, models, and inverse effectiveness. Experimental Brain Research, 198, 113–126. Stricanne, B., Andersen, R. A., & Mazzoni, P. (1996). Eyecentered, head-centered, and intermediate coding of remembered sound locations in area LIP. Journal of Neurophysiology, 56, 2071–2076.
Trappenberg, T. P., Dorris, M. C., Munoz, D. P., & Klein, R. M. (2001). A model of saccade initiation based on the competitive integration of exogenous and endogenous signals in the superior colliculus. Journal of Cognitive Neuroscience, 13, 256–271. van Beers, R. J., Sittig, A. C., & Denier van der Gon, J. J. (1999). Integration of proprioceptive and visual positioninformation: An experimentally supported model. Journal of Neurophysiology, 81, 1355–1365. Xing, J., & Andersen, R. A. (2000). Models of the posterior parietal cortex which perform multimodal integration and represent space in several coordinate frames. Journal of Cognitive Neuroscience, 12, 601–614. Zipser, D., & Andersen, R. A. (1988). A back-propagation programmed network that simulates response properties of a subset of posterior parietal neurons. Nature, 331, 679–684.
A. M. Green, C. E. Chapman, J. F. Kalaska and F. Lepore (Eds.) Progress in Brain Research, Vol. 191 ISSN: 0079-6123 Copyright Ó 2011 Elsevier B.V. All rights reserved.
CHAPTER 14
Sensory rehabilitation in the plastic brain Olivier Collignon{,{,*, François Champoux}, Patrice Voss{ and Franco Lepore{ { {
Centre de Recherche en Neuropsychologie et Cognition (CERNEC), Université de Montréal, Montréal, Québec, Canada Centre de Recherche CHU Sainte-Justine, Université de Montréal, Montréal, Québec, Canada } Centre de Recherche Interdisciplinaire en Réadaptation du Montréal Métropolitain, Institut Raymond-Dewar, Montréal, Québec, Canada
Abstract: The purpose of this review is to consider new sensory rehabilitation avenues in the context of the brain's remarkable ability to reorganize itself following sensory deprivation. Here, deafness and blindness are taken as two illustrative models. Mainly, two promising rehabilitative strategies based on opposing theoretical principles will be considered: sensory substitution and neuroprostheses. Sensory substitution makes use of the remaining intact senses to provide blind or deaf individuals with coded information of the lost sensory system. This technique thus benefits from added neural resources in the processing of the remaining senses resulting from crossmodal plasticity, which is thought to be coupled with behavioral enhancements in the intact senses. On the other hand, neuroprostheses represent an invasive approach aimed at stimulating the deprived sensory system directly in order to restore, at least partially, its functioning. This technique therefore relies on the neuronal integrity of the brain areas normally dedicated to the deprived sense and is rather hindered by the compensatory reorganization observed in the deprived cortex. Here, we stress that our understanding of the neuroplastic changes that occur in sensory-deprived individuals may help guide the design and the implementation of such rehabilitative methods. Keywords: blindness; deafness; neuroplasticity; rehabilitation; sensory substitution; neuroprosthesis.
evolution. It is likely that the apparent regularity and homogeneity of cortical anatomy have prolonged this conception of an immutable brain. However, results acquired mainly in the past two decades have led to the recognition that the developing, and even adult, brain has a remarkable ability to remodel and restructure the different circuits within it, based on learning and experience. This concept, called
Introduction It has long been believed that the brain is hard-wired, in a predetermined manner mainly shaped by *Corresponding author. Tel.: þ1-514-343-6111x2667; Fax: þ1-514-343-5787 E-mail:
[email protected] DOI: 10.1016/B978-0-444-53752-2.00003-5
211
212
neuroplasticity, is opening up exciting new fields of research based on the brain's ability to constantly adapt itself to its environment throughout life. Recognizing the dynamic nature of cortical circuitry is important in understanding how the nervous system adapts after sensory deprivation. Pioneering studies of Wiesel and Hubel (1965, 1974) on the development of ocular dominance columns have compellingly demonstrated that alterations in visual experience can influence the normal development of the visual cortex. Other seminal experiments have also shown that cortical maps can change/expand with use; for example, the representation of the finger tips in the somatosensory cortex has been shown to expand after a period of intense stimulation (Kaas et al., 1983), as observed in proficient Braille blind readers (Pascual-Leone and Torres, 1993; Sterr et al., 1998). Similarly, the tonotopic map in the auditory cortex is larger in musicians (Pantev et al., 1998) and visually deprived individuals (Elbert et al., 2002). Aside from such intramodal plasticity, massive crossmodal changes have also been observed in sensory-deprived cortex (Bavelier and Neville, 2002; Pascual-Leone et al., 2005). Striking evidence that external inputs can determine the functional role of a sensory cortex has come from experiments on “rewired” animals. For instance, by making a series of brainstem lesions, researchers surgically rerouted visual input toward primary somatosensory or auditory areas (Frost and Metin, 1985; Frost et al., 2000; Roe et al., 1990; Sur et al., 1988). These experiments demonstrated that cells from the rewired regions shared some structural and functional similarities with cells recorded in the visual cortex of normally raised animals. Moreover, these authors demonstrated that these newly visual cells also mediated visually guided behavior (Frost et al., 2000; von Melchner et al., 2000). Taken together, these data suggest that primary cortical areas can change their functional specificity depending on which inputs they receive. Indeed, the observation that “visual” regions can be recruited for nonvisual processing
in blind subjects (Sadato et al., 1996; WanetDefalque et al., 1988) and that auditory regions can be recruited by nonauditory inputs in deaf subjects (Bavelier et al., 2001; Finney et al., 2001) has led to a change in how we think about the brain and its development in relation to experience. Importantly, these findings also demonstrate that these plastic changes are compensatory in nature because they appear to underlie improved abilities in the remaining senses of sensory-deprived individuals (Amedi et al., 2003; Bavelier et al., 2000, 2006; Collignon et al., 2006, 2009b; Gougoux et al., 2005). Overall, these results point to the important role of sensory experience in the development and the maintenance of sensory brain functions. This has major implications, given current developments in sensory rehabilitation technologies, whether they are of the invasive type or not (Veraart et al., 2004; see Fig. 1). Invasive interventions rely on the integrity of the deprived system. Plastic reorganization that occurs all along the sensory pathway after deprivation is therefore likely to interfere with the reacquisition of the initial function of the system (Merabet et al., 2005). Indeed, in addition to the technical and surgical challenge of sensory restoration, there exists a neuropsychological one: how will the restored sensory input be interpreted by the reorganized sensory cortex? In contrast, sensory substitution refers to the use of one sensory modality to supply information normally gathered from another sense (Bach-y-Rita and Kercel, 2003). In so doing, sensory substitution devices can take advantage of the crossmodal plasticity observed in deprived individuals whereby deafferented areas provide the neural basis for behavioral compensation reported in the preserved senses (Amedi et al., 2003; Gougoux et al., 2005). Indeed, studies on how the brain changes following sensory deprivation are not only central to our understanding of the development of brain function but are also crucial to the development of adequate and successful rehabilitation strategies in case of sensory alterations.
213 Sensory environment
Artificial organ
Artificial transduction
Preserved organ
Restored organ
Altered organ
Neurostimulator
Artificial organ
Primary sensory pathway and brain regions Simplification and coding
Simplification and coding Associative brain regions
Motor output Sensory substitution
Sensory deprived individual
Invasive neuroprostheses
Fig. 1. Model of rehabilitation procedures for sensory-deprived individuals. The middle section represents a sensory-deprived person for whom environmental information can be transmitted to the brain by means of a remaining modality after sensory substitution (left panel), surgical restoration of the defective organ, or by the use of an implanted neuroprosthesis stimulating the deficient sensory system (right panel). With sensory substitution, the environmental inputs usually gathered by the defective sense is simplified and coded in order to be manipulated in a preserved remaining modality. With neuroprostheses, the lacking sensory information is simplified and coded into electrical impulses to stimulate the fully or partly preserved part of the deficient sense.
Rehabilitation in blindness Early visual deprivation causes atrophy in the optic tracts and radiations as well as massive gray and white matter volume reduction in early visual areas (Noppeney et al., 2005; Pan et al., 2007; Park et al., 2009; Ptito et al., 2008b; Shu et al., 2009). Although increased cortical thickness of occipital cortex has also been reported in the blind (Jiang et al., 2009; Park et al., 2009), it is believed to reflect the reduced surface area of the primary and secondary visual cortices (Park et al., 2009). In addition to these structural changes, visual deprivation enables a new role for the visual cortex in that it becomes responsive to nonvisual inputs (Bavelier and Neville, 2002). Moreover, a growing number of studies show that the recruitment of the deafferented visual areas during nonvisual tasks is not simply an epiphenomenon. First, these changes are thought to underpin superior nonvisual abilities often
observed in blind individuals as several studies have shown positive correlations between nonvisual performance and occipital activity: the most efficient blind participants are the ones who recruit occipital regions the most (Amedi et al., 2003; Gougoux et al., 2005). Second, transient disruption of occipital activity induced by transcranial magnetic stimulation (TMS) disrupts nonvisual abilities, further demonstrating the functional role of occipital regions of congenitally blind subjects in nonvisual processing (Amedi et al., 2004; Cohen et al., 1997; Collignon et al., 2007, 2009a). Finally, some aspects of the functional architecture present in the occipital cortex of sighted subjects appear to be preserved in the blind (Collignon et al., 2009b, Dormal et al., 2011). For example, the “visual” dorsal stream appears to maintain its preferential coding for spatial processing (Collignon et al., 2007, 2011; Renier et al., 2010; Fig. 2), the ventral stream for the processing of the identity of the input
(a)
(b)
(c)
Auditory discrimination tasks
Sham rTMS
(d)
(e)
(f)
Intensity
Pitch
15
Localization
15
10
5
0
* Error rate (%)
Error rate (%)
Error rate (%)
15
Real rTMS
10
5
0
Sighted
10
5
0
Blind
Sighted
Blind
Sighted
Blind
PSVA tasks (g)
(h)
(i)
Figure Recognition
Exploration Time
100
95
*
90
*
90 85 80
85 time (s)
% correct
95
80 75 70
75
65
70
60 Sighted
Blind
Sighted
Blind
Fig. 2. Prosthesis substituting vision by audition (PSVA). (a) A head-worn video camera (fixed on glasses) allows online translation of visual patterns into sounds that are transmitted to the subject through headphones. (b) The artificial retina provided by the PSVA. The acquired image is divided into pixels according to a 2-resolution artificial retina scheme. The central part of the processed image or fovea has a four times higher resolution than the periphery. The coding scheme is based on a pixel–frequency association. Pixels in use are drawn with a bold border. Frequency is indicated in hertz in the lower part of the used pixels. A single sinusoidal tone is assigned to each pixel of the multiresolution image. The amplitude of each sine wave (the intensity of each sound) is modulated by the gray level of the corresponding pixel. The pattern moves on the grid according to the head movements of the subject, and the corresponding sounds of the activated pixels are transmitted to the subject in real time. (c) Examples of patterns used in the experiments. The second part of the figure denotes the average error rate in blind and sighted subjects after sham and real TMS targeting the dorsal occipital stream during auditory tasks involving discrimination of intensity (d), pitch (e), and spatial location (f). The data show a significant increase of the error rate after real rTMS only in the blind group and selectively for the sound location task. Also, the figure displays the average percentage of correct pattern recognition (g) and the mean exploration time (h) taken to recognize patterns with the PSVA. The data indicate a significant decrease of recognition score and a significant increase of exploration time after real compared to sham TMS in the blind group only. Panel (i) displays the projection of the site of TMS application. This area corresponds to the right dorsal extrastriate occipital cortex (BA 18). Adapted with permission from Collignon et al. (2007).
215
(Amedi et al., 2007; Gougoux et al., 2009), and hMTþ/V5 for processing movement (Bedny et al., 2010; Poirier et al., 2004; Ricciardi et al., 2007). Taken together, these structural and functional changes in “visual” areas of early-blind individuals are thought to induce permanent changes in visual capabilities (Maurer et al., 2005). For example, the ability to elicit phosphenes with application of TMS over the occipital cortex (a measure of visual cortex excitability) is dramatically reduced in congenitally blind individuals (Gothe et al., 2002). Sight restoration with surgery The study of adult sight-recovery patients after early-onset blindness, even if extremely rare, has served as an important testing ground for hypotheses about the role of experience in shaping the functional architecture of the brain. These studies have demonstrated that early visual deprivation permanently and deeply affects visual functions (Fine et al., 2003; Gregory, 2003; Levin et al., 2010). Probably the most famous case report concerns patient SB, studied by Richard Gregory (Gregory and Wallace, 1963). SB lost his sight at 10 months of age before regaining it at 52 years of age, by means of a corneal graft. Despite the fact that the visual world now mapped correctly on his retina, SB had severe problems interpreting what he saw. Perception of depth was notably problematic (i.e., Necker's cube appeared flat) and he was only able to recognize faces when they moved. SB continued to rely on audition and touch to interact with his environment and situations that he managed very well while blind, like crossing a street in traffic, suddenly became problematic for him because of the presence of concurrent confusing visual information. Shortly after implantation, he became clinically depressed, probably due to his change of status from a successful blind to an unsuccessful sighted person (Gregory and Wallace, 1963). Another fascinating case was documented more
recently in the literature, patient MM, who was blind since the age of 3 years and who had his sight restored at 43 years of age, thanks to stem cell transplant (Fine et al., 2003). MM also had considerable difficulty perceiving depth and perceiving the specific details of objects, including faces. Even 7 years after the intervention, MM still had poor spatial resolution and limited visual abilities that did not allow him to rely on his vision in day-to-day activities (Levin et al., 2010). Imaging studies of MM showed extensive cortical reorganization, even after implantation, which may play a role in his visual difficulties (Fine et al., 2003; Levin et al., 2010; Saenz et al., 2008; Fig. 3). This is hypothesized to be due to an absence of mature cells coding for “fine” details because these cells were still not tuned at 3 years of age when MM lost his sight (Levin et al., 2010). In contrast to visual acuity and form or face perception, visual motion ability appeared relatively preserved after vision restoration in both SB and MM, with robust and specific brain activations for visual motion stimuli having been observed in subject MM (Fine et al., 2003; Levin et al., 2010; Sacks, 1995; Saenz et al., 2008). This is thought to be due to the fact that motion processing develops very early in infancy compared to form processing and might therefore have been more established and robust, allowing its preservation despite many years of visual deprivation (Fine et al., 2003). It was also shown that robust and specific crossmodal auditory motion responses coexist with regained visual motion responses in area hMTþ/V5 after sight restoration in subject MM (Saenz et al., 2008). However, it was not ascertained if the presence of such crossmodal auditory motion responses competes with or improves visual motion perception after recovery, nor whether the interaction between these two senses is enhanced or decreased due to interference (see our related discussion in the cochlear implant (CI) section below). This question is of major importance because the challenge for MM is to use the strong nonvisual skills he developed
216 (1.a)
(1.b) 10 deg 5 deg
Psychophysics
(3)
FMRI
(5.a)
Auditory motion MT + overlapping MT + non overlapping
5
0.2
2
0.5
Projected amplitude (% signal change)
0.1
Threshold (% contrast)
Sensitivity (% contrast)
0.6 10
1
1
(2)
0.4 0.3 0.2 0.1 0 –0.1
3 0.25 1 2 Spatial frequency (c.p.d.) MM + 5 m MM + 11 m MM + 17 m MM + 21 m
Sighted controls
0.5
MM
0.25 1 2 3 Spatial frequency (c.p.d.) V1 MT+
Control
MM Control
(4)
MM
OcP
AB
OcP
(5.b) MM Sighted controls Monocular Blind
% fMRI signal change
Radial (mm2/msec)
1.5
1
Posterior 0.5 1.25
Medial 1.5 1.75 2 2.25 Longitudinal (mm2/msec)
1
0.4 0 0.4 Correlation
1
Visual motion Auditory motion (ILD)
** ** 1
**
** **
**
** **
0.5 0 –0.5 c1 c2 c3 c4 c5 c6
MM MS
Fig. 3. Patchwork of different studies carried out with MM, an early-blind person who recovered sight at 43 years. Altogether, the results show major alteration in visual processing in this subject. (1.a) MM's sensitivity as a function of spatial frequency measured psychophysically 5–21 months after surgery. (1.b) Neural responses as a function of spatial frequency measured using fMRI in MT þ (dashed line) and V1 (solid line). (2) Comparison of radial and longitudinal diffusivities in the optic tracts and optic radiations (a) Three-dimensional rendering of the optic tract fibers (blue) shown superimposed on axial and coronal slices of MM's brain. The optic tracts connect the optic chiasm and the LGN (white sphere). Scatter plot of the radial and longitudinal diffusivities for the average of the right and left optic tracts. Data are from MM (gray star), 10 normal controls (black open circles), two seeing monocular subjects (black asterisks), and one blind subject (black closed circle). The 2 standard deviation covariance ellipsoid (dashed) is shown. (3) Visual field eccentricity representations in medial-ventral and dorsal-lateral cortex visual field eccentricity maps in lateral-occipital surface of MM's left (left panel) and right (right panel) hemispheres. Several extrastriate regions respond unusually to foveal stimuli. The right hemisphere shows some regions and a color map defining the visual field eccentricity representations.(4) Left hemisphere activation in response to faces versus objects with red–orange regions that responded more to faces and green–blue regions that responded more to objects. A control subject (AB) showed a typical pattern of activation, with large contiguous regions that responded more either to faces or objects near the fusiform gyrus (FuG) and lingual gyrus (LiG). In contrast, MM showed little activity to objects, and almost no activity to faces. (5.a) Surface maps of auditory and visual motion responses in MT for MM and sighted controls. Yellow regions responded more to moving versus stationary auditory white noise. Green and blue regions show MT location as determined by a visual MT localizer scans run in the same subjects (green, MT overlapped by auditory ILD motion responses; blue, MT not overlapped by auditory ILD motion responses). Note the nearcomplete overlap (very little blue) in subject MM indicating colocalization of MT for auditory motion processing. Adapted with permission from Fine et al. (2003; parts 1 and 4), Levin et al. (2010; parts 2 and 3), and Saenz et al. (2008; part 5).
as a proficient blind subject (sensory compensation in the remaining senses) in conjunction with his rudimentary vision in order to improve his use of visual functions. Indeed, knowledge of how visual and auditory responses interact in sight-recovery patients is important for optimizing patients’ use of their restored vision (Saenz et al., 2008).
The study of children treated for congenital bilateral cataracts after varying periods of visual deprivation presents the opportunity to examine the fundamental role of visual inputs for the normal development of specific aspects of vision. Particular studies on this topic have shed light on the fact that different visual abilities have various sensitive periods during which the absence
217
of visual inputs permanently impairs the investigated process. For example, even when treated for congenital bilateral cataracts before the first 6 months of age, permanent deficits in sensitivity to global motion have been shown to develop (Ellemberg et al., 2002; Lewis and Maurer, 2005), as well as for holistic face processing (Le Grand et al., 2001, 2004). However, the loss of sight after 6 months of age preserves the global detection of motion even if the period of blindness is extended as shown in patients MM and SB (Fine et al., 2003; Gregory and Wallace, 1963) but still can dramatically impair acuity, peripheral light sensitivity, and object and face processing (Fine et al., 2003; Levin et al., 2010; Lewis and Maurer, 2005; Gregory and Wallace, 1963). Strikingly, in some visual domains, visual input is necessary throughout the period of normal development and even after the age when performance reaches adult levels (Maurer et al., 2005). For instance, a short period of visual deprivation beginning any time before the age of 10 years causes permanent deficits in letter visual acuity, which normally reaches adult levels by the age of 6 years (Lewis and Maurer, 2005). Similarly, short periods of deprivation beginning even in early adolescence cause permanent deficits in peripheral light sensitivity, which normally reaches adult functional levels by 7 years of age (Bowering et al., 1993). It thus appears that visual input is necessary not only for the development but also for the consolidation of some visual connections (Lewis and Maurer, 2005). Regarding multisensory integration abilities, recent studies conducted in bilateral congenital cataract patients treated within the first two years of life demonstrated that visual input in early infancy is also a prerequisite for the normal development of multisensory functions (Putzar et al., 2007, 2010). Even if some studies demonstrated that the human brain retains an impressive capacity for visual learning well into late childhood (Ostrovsky et al., 2006, 2009), an important point raised by these studies in sightrestored patients is that early intervention is often a good predictor of visual abilities in adults. In the
particular case of congenital blindness, sight restoration in adults may be less miraculous than intuitively expected, probably because of the deterioration of visual tracts and massive crossmodal plasticity observed in the visual cortex of these persons (Noppeney, 2007). Sensory substitution in the blind The fact that the crossmodal recruitment of visually deafferented occipital areas effectively contributes to the processing of nonvisual inputs offers a real opportunity for rehabilitation via sensory substitution. Indeed, this fact has been intuitively exploited in numerous rehabilitation programs aimed at promoting nonvisual skills. Since it was discovered that the enrichment of the environment is an effective means of dramatically enhancing crossmodal plasticity associated with blindness (Piche et al., 2004), and because such reorganization mechanisms are thought to underlie enhanced perceptual skills in the blind (Amedi et al., 2003; Gougoux et al., 2005), orientation and mobility programs assume that they can help develop enhanced skills in the remaining senses of blind subjects though rehabilitation. These rehabilitation programs rely on the concept of sensory substitution, which refers to the use of one sensory modality to supply information normally gathered from another sense (Bach-y-Rita et al., 1969). The use of the long-cane as an extension of the body (Serino et al., 2007), the development of refined tactile discrimination in order to fluently read Braille dots (Van Boven et al., 2000; Wong et al., 2011), or the use of the reverberation of sounds to locate obstacles and discriminate object size (Dufour et al., 2005; Rice, 1967; Rice and Feinstein, 1965; Strelow and Brabyn, 1982) are excellent examples of such abilities that appear “supranormal” for a naïve sighted person but which are mastered by blind individuals due to a combination of extensive training programs and neuroplastic mechanisms. The Braille reading system is probably the best
218
example of these effects and massive involvement of the occipital cortex has been demonstrated in blind individuals when reading (Buchel, 1998; Burton et al., 2002; Sadato et al., 1996, 1998). Moreover, it has been shown that TMS over the occipital cortex of early-blind subjects disrupts Braille reading and even induces tactile sensations on the tip of the reading fingers in experienced users (Cohen et al., 1997; Kupers et al., 2007; Ptito et al., 2008a). Such findings demonstrate the functional involvement of the reorganized occipital cortex of blind subjects in Braille reading. This notion is even further supported by the reported case study of an expert blind Braille reader who lost her ability (Braille alexia) following an ischemic stroke which caused bilateral lesions to her occipital cortex (Hamilton et al., 2000). Aside from these classical rehabilitative programs, researchers have also considered providing blind people with new sensory-motor interactions with their environment in order to lower the impact of visual deprivation. Bach-y-Rita can arguably be seen as a visionary in the field since he had the idea in 1969 to design the first sensory substitution devices for the blind by using the preserved sense of touch to supply information usually gathered from vision (Bach-y-Rita et al., 1969). Since this seminal work, and partly due to subsequent technological improvements, several laboratories have been engaged in developing and testing new sensory substitution prosthesis (Bach-y-Rita et al., 1998; Capelle et al., 1998; Cronly-Dillon et al., 1999; Kaczmarek et al., 1985; Meijer, 1992). All these systems are designed to make use of the residual intact senses, mainly audition or touch, to provide blind people with a sample of the visual world that has been coded into another modality via specific algorithms that can be learned through practice (Veraart et al., 2004). These systems have proven their efficiency for the recognition of quite complex two-dimensional shapes (Arno et al., 1999, 2001b), to localize objects (Proulx et al., 2008; Renier and De Volder, 2010) or to navigate in a “virtual” environment
(Segond et al., 2005) and were found to massively and crossmodally recruit the occipital cortex of blind subjects (Amedi et al., 2007; De Volder et al., 1999; Kupers et al., 2010; Merabet et al., 2009; Poirier et al., 2007; Ptito et al., 2005). In our group, we investigated one such system, a prosthesis for substitution of vision by audition (PSVA) (Capelle et al., 1998). Early-blind participants were found to be more accurate when using the PSVA (Arno et al., 2001b) and their occipital cortex was more strongly activated than in the sighted in a pattern recognition task (Arno et al., 2001a). We also demonstrated that TMS interfered with the use of the PSVA when applied over the right dorsal extrastriate cortex of blind participants, probably due to the spatial cognitive components associated with the use of the prosthesis (Collignon et al., 2007). By contrast, TMS targeting the same cortical area had no effect on performance in sighted subjects (Fig. 2). As stated previously, we postulate that occipital regions are recruited in a compensatory crossmodal manner that may account for the superior abilities seen when using the prosthesis. The sensory substitution devices, therefore, constitute interesting noninvasive techniques, in great part because their working principles follow the natural tendency of the brain to reorganize itself in favor of the remaining sensory modalities. That being said, their principal drawback is that they are currently mainly dedicated to fundamental research on crossmodal reorganization; in their present form, there are no realistic opportunities for their introduction into the blind community. This is generally related to the poor ergonomic quality of such human–machine interfaces. In addition, the coding scheme may appear quite difficult, and the visual information gathered by the camera is generally too complex to be entirely recorded in the substitutive modality without creating a “noisy” percept. Indeed, laboratory settings where such systems are tested are extremely impoverished in order to avoid an excessive sensory and cognitive load when using such devices. These experimental situations are
219
usually composed of few target elements having a high figure-ground contrast (i.e., white shape on a black background). In the case of auditory devices, the technology appropriates a sensory channel that blind people already use in a skilful way for their daily-life activities. Modern tactile devices have mainly used the tongue to deliver the substituted information. This body part has been preferred because its sensitivity, spatial acuity, and discrimination abilities are better than other parts of the body (Bach-y-Rita et al., 1998). However, this choice probably adds aesthetic and hygienic problems, which may impact on the willingness of the blind community to introduce the system as a standard aid. Moreover, in order to become a real option for the blind in guiding their navigation, such systems should be complementary and thus provide new information to existing aids like the guide-dog and the white cane. Consequently, it appears evident that more consideration is needed in the design of more ergonometric sensory substitution systems for visual rehabilitation purposes. However, because sensory substitution greatly benefit from the crossmodal changes that occur in the brain of blind individuals they constitute a promising solution especially for early-blind individuals for whom surgical intervention is not possible, particularly if introduced in early infancy when the plasticity of the brain is the highest. Neuroprostheses in the blind Visual prosthetic implants aim to electrically stimulate the remaining functional parts of the previously fully developed visual system in order to restore some visual-like perception, mainly by inducing the perception of patterned spots of light called phosphenes (Merabet et al., 2005; Zrenner, 2002). Such implants would connect a digital camera to a signal processor that would convert visual information into patterned electrical signals (Fig. 1). Several approaches are currently under investigation and involve subretinal (Pardue
et al., 2006a,b; Zrenner et al., 1999), epiretinal (Humayun et al., 2003; Rizzo et al., 2003a,b), optic nerve (Veraart et al., 1998, 2003), or occipital (Schiller and Tehovnik, 2008; Schmidt et al., 1996; Tehovnik et al., 2005) stimulation. Aside from the major issues of electrical safety and biocompatibility of the material (Veraart et al., 2004), knowledge about the selectivity and diffusivity of the stimulation is an essential problem in evaluating the behavioral effects of the stimulated area itself. As a result, researchers are currently trying to combine microstimulation of neural tissue with fMRI in order to provide the unique opportunity to visualize the networks underlying electrostimulation-induced perceptions (Logothetis et al., 2010). In contrast to sensory substitution systems, the visual prostheses do not take advantage of the natural reorganization of the cortex of the blind since such invasive approaches attempt to stimulate the deficient sensory system directly. As such, these prostheses are mainly dedicated to blindness acquired at a later age since the development of the visual system and previous visual experience would be a prerequisite to trigger and interpret the visual percept induced by the stimulation of neural tissues. For example, one study demonstrated that the ability to elicit phosphenes with application of TMS over the occipital area is dramatically reduced in subjects with an early onset of visual deafferentation, especially in those without history of visual experience (Gothe et al., 2002). Indeed, the structural (deterioration of visual tracks) and functional (crossmodal plasticity) changes following early visual deprivation might hamper the reacquisition of the original visual function of a given structure via the prosthetic implant. There are reasons to believe, however, that such devices might work with late-blind individuals since far less alterations in the visual tracks and areas (Jiang et al., 2009; Noppeney et al., 2005; Park et al., 2009) and less-crossmodal recruitment of occipital regions by nonvisual stimuli (Burton et al., 2003; Cohen et al., 1999; Voss et al., 2008) have been observed in subjects
220
who developed late-onset blindness. Moreover, studies of sustained blindfolding in sighted subjects suggest that the crossmodal recruitment of occipital cortex that appears after visual deprivation later in life may be more reversible after the reintroduction of vision (Merabet et al., 2008; Pascual-Leone et al., 2005). In fact, the mechanisms underlying crossmodal occipital recruitment in early- and late-blind individuals may differ considerably (Collignon et al., 2009b). Early deprivation could favor the maintenance of intermodal connections between cortical areas that are normally pruned in infancy, thus preventing the strengthening of typical visual cortical networks. In late blindness, however, these extrinsic connections would not escape the normal developmental synaptic pruning due to the presence of stabilizing visual input. Indeed, crossmodal recruitment of occipital regions observed in late blindness may reflect the strengthening, probably via Hebbian mechanisms1 (Hebb, 1949), of existing intermodal connections also present in sighted subjects. In line with such an assumption, an elegant study combining PET-scan and TMS showed that the application of TMS over the primary somatosensory cortex induced significant activation of the primary visual cortex only in an early-blind group but not in late-blind or sighted subjects (Wittenberg et al., 2004). These results are consistent with the hypothesis of reinforced corticocortical connections between primary sensory cortices in early- but not in late-blind subjects (Collignon et al., 2009b). These results place late-blind individuals as the candidate of choice for visual prosthetic implantation, especially because blindness acquired later in life may prevent the development of all the compensatory mechanisms observed in the early blind; this is also true because in the absence of
1 “When the axon of cell A excites cell B and repeatedly or persistently takes part in firing it, some growth process or metabolic change takes place in one or both cells so that A's efficiency as one of the cells firing B is increased.”
enhanced abilities in the remaining senses, the late blind may encounter greater difficulty in coping with the handicap (Wan et al., 2010).
Rehabilitation in deafness While crossmodal plasticity has been less extensively studied in deaf than in blind individuals, research in deaf subjects again leads to the conclusion that crossmodal reorganization occurs, such that cortical territories from the unused auditory modality can be recruited by other senses, in particular vision (Bavelier et al., 2006). Sensory substitution in the deaf These functional changes in the network dedicated to visual processing in the deaf appear to be accompanied by behavioral enhancements in visual attention and visual localization in peripheral visual space (Bavelier et al., 2000; Bosworth and Dobkins, 2002; Neville, 1990; Neville and Lawson, 1987a,b; Proksch and Bavelier, 2002; Rettenbach et al., 1999). Along with these lowlevel processing enhancements (i.e., devoid of phonetics), extensive visual-to-auditory reorganization has also been demonstrated with the presentation of visual stimuli activating the auditory cortex of deaf individuals. Indeed, activation of primary, secondary, and association auditory regions has been observed in early-deaf subjects during the observation of moving dot patterns (Armstrong et al., 2002; Finney et al., 2001) or moving sinusoidal luminance gratings (Finney et al., 2003). Crossmodal changes have also been related to cognitive functions. In normally hearing individuals, speech comprehension is achieved in a multisensory mode that combines auditory and visual (e.g., movement of the lips) speech information. To improve speech recognition or discrimination capabilities, this multisensory process is substituted to favor more exclusively the visual strategies in profoundly
221
deaf individuals. These communication strategies consist mainly of lipreading (Kaiser et al., 2003; Tyler et al., 1997) and sign language reading capabilities (Brozinsky and Bavelier, 2004; Neville et al., 1997; Proksch and Bavelier, 2002). Again, activity in traditionally considered auditory regions has been reported in the deaf during the observation of visual lip motion in the left planum temporale and during the visual presentation of sign language in the superior temporal gyrus and association auditory cortex (Hirano et al., 2000; MacSweeney et al., 2002; Nishimura et al., 1999; Petitto et al., 2000; Sadato et al., 2005). As in the literature on blind subjects, it is believed that the crossmodal plasticity observed in deaf subjects directly leads to a behavioral advantage and improved communication strategies (Bavelier et al., 2006). In those individuals who are trying to achieve some recovery of hearing function, however, such extensive reorganization may represent a challenge that may, in some case, hinder their rehabilitation. Cochlear implant While the visual takeover of the normally auditory cortices represents an impressive cerebral ability to adapt to changes in environment, it begs an important question relative to the recovery of the hearing function. Indeed, once responsive to a new input modality, can the auditory cortices respond to their original auditory input? This question bears special importance given that profound deafness can sometimes be reversed by auditory stimulation via a cochlear implant (CI) (Ponton et al., 1996). Put simply, the device replaces normal cochlear function by converting auditory signals into electrical impulses delivered to the auditory nerve (see Mens, 2007 for a more detailed description). Over the past decade, advances in engineering and surgical implantation techniques have begun to make the CI a standard part of the treatment for hearing loss (Clark, 2006; Fallon et al., 2008). Such success has
allowed researchers to ascertain the consequences of crossmodal plasticity in the deaf population on the success rate of CIs. In deaf individuals, activity in auditory cortical regions is increased following cochlear implantation (Lee et al., 2001; Naito et al., 1995; Wong et al., 1999), as soon as the implant is turned on (Giraud et al., 2001). In their longitudinal electrophysiological investigation, Pantev et al. (2006) showed that the cortical activity in auditory regions had normal component configurations and localizations, confirming that the input from the CI stimulation may be transmitted adequately to auditory structures as soon as the implant is made active in postlingually deaf individuals. The authors also showed that brain activity increased progressively over several months following implantation (Pantev et al., 2006). However, the general outcome of the hearing proficiency following implantation is still highly unpredictable (Green et al., 2007). It has been argued that the level of crossmodal plasticity occurring as a consequence of early deprivation can predict the performance with an auditory prosthesis, with less reorganization leading to greater proficiency with the implant and vice versa (Giraud and Lee, 2007). For instance, it was shown that speech perception performance was positively associated with preoperative activity in frontoparietal networks and negatively associated with activity in occipito-temporal networks (Lee et al., 2005), even when factoring out the confounding effect of age of implantation (Lee et al., 2007). Indeed, the hindering effect of preoperative activity in temporal areas might be a sign that auditory areas may have been taken over by the visual modality, suggesting that crossmodal recruitment can serve as a predictor of the outcome of implantation. Similarly, a recent study compared cortical evoked potentials involved in the processing of visual stimuli between implanted (at least 1 year post-op) and hearing subjects (Doucet et al., 2006). After evaluation of speech perception abilities of the implanted subjects, they were subsequently divided into two groups based on their
222
performance. The results showed that implanted individuals with broader and more anterior scalp distributions (i.e., showing signs of visual processing in the temporal cortices) in response to visual stimuli were those who performed more poorly in the speech perception task and vice versa. In fact, several factors interact and influence crossmodal reorganization in deaf individuals, which in turn impacts auditory perception following implantation. The most influential factors are most likely the duration of deafness, the deafness onset, the time of implantation, and the communication strategy used before implantation. (i) Duration of deafness. Straightforward correlations have been reported between postimplantation auditory-word recognition performance, cortical activity in response to auditory stimulation, and the duration of deafness. Indeed, it appears that implanted deaf individuals who had a longer period of deprivation show less cortical activity in response to auditory stimulation and poorer auditory performance (Lee et al., 2001). The results of this neuroimaging study suggest that a long duration of deafness might lead the higher visual cognitive functions to invade the underutilized areas of the auditory cortex. However, in a retrospective case review, Green et al. (2007) showed that the duration of deprivation only accounted for 9% of the variability in implant outcome, which is substantially less than first thought. In fact, Lee et al. (2001) had already suggested that other factors, such as the onset of deafness or the preimplantation communication strategies, could also have a dramatic impact on auditory perception following implantation. (ii) Onset of deafness. It is in fact commonly acknowledged that postlingually deafened candidates perform better following cochlear implantation in adulthood in all auditory tasks compared to prelingually deaf individuals implanted in later life (Giraud
et al., 2001). Supporting this behavioral evidence, imaging data also suggest more extensive plastic changes in the early-deafened individuals. Indeed, auditory stimuli have been shown to activate both the primary and secondary auditory cortices in postlingually deafened individuals, whereas they merely activate the primary auditory cortex in the prelingually deafened ones following implantation (Naito et al., 1997). Also illustrative of the importance of the age of onset of deafness, Sadato et al. (2004) demonstrated that both early- and late-onset deaf groups showed similar activation of the planum temporale in a visual sentence comprehension task whereas early-deaf subjects showed more prominent activation in the middle superior temporal sulcus (STS), a region thought to be important for the processing of vocalizations (Belin et al., 2000). (iii) Time of implantation. Several studies have shown that if implanted before the age of 2, implanted children can acquire spoken language in a comparable time frame to normal hearing children (Hammes et al., 2002; Waltzman and Cohen, 1998). However, this time window for the recovery of auditory function following deprivation is generally limited to the first few years of life, with the chances of recovery rapidly decreasing afterward (Kral et al., 2005). (iv) Communication strategy before implantation. Hirano et al. (2000) have suggested that crossmodal plasticity may be influenced by the communication strategies (i.e., familiarity with lipreading or sign language ability) used before implantation. Indeed, the authors showed that patients trained to communicate with visual modes of communication are more prone to extensive crossmodal changes compared to individuals trained in a more exclusive auditory mode (i.e., with conventional auditory amplification strategies based on the residual hearing). This last rehabilitation technique seems to prevent visual
223
information from invading the relatively unused cortical regions (Hirano et al., 2000). However, it is worth noting here that the use of this technique in patients with very little or no residual hearing may have a dramatic impact on the communication capabilities of these persons. Although difficult to assess, it is commonly acknowledged that these features (duration of deafness, onset of deafness, time of implantation, and communication strategy before implantation) might also interact in determining the degree to which crossmodal changes might occur, and so, in defining the level of proficiency reached by each participant following cochlear implantation. Multisensory interactions in CI users Since the world around us is made up of events that stimulate several senses simultaneously, it begs the question of how the regained auditory modality might interact with other sensory information during multisensory perception in CI users, especially with regard to speech perception. The integration of congruent cues. Greater visual activity during speech recognition tasks has been reported in deaf individuals with a CI (Giraud et al., 2001). Some evidence even suggests that such visual activity increases progressively with the use of the auditory device (Desai et al., 2008). Indeed, Giraud et al. (2001) suggested that cochlear implantation might result in a mutual reinforcement between vision and hearing. In accordance with this belief of reciprocal enhancement, there seems to be a consensus surrounding the notion that accessing simultaneous visual and auditory information, when both cues are related, is beneficial in CI users (Bergeson and Pisoni, 2004; Geers, 2004; Kaiser et al., 2003; Moody-Antonio et al., 2005; Tyler et al., 1997). Some have even argued that CI users might be better at integrating congruent auditory and visual information when compared to normally hearing individuals (Rouger et al., 2007).
The fusion of incongruent cues. The ability to fuse incongruent audiovisual information has also been studied recently. Schorr et al. (2005) used McGurk-like stimuli, where incongruent lip movements can induce the misperception of spoken syllables (McGurk and MacDonald, 1976), to investigate the ability to integrate incongruent multisensory cues in children with a CI, as a function of experience with spoken language (Schorr et al., 2005). In children aged two and a half years or younger, the authors found normal-like results in the audiovisual task. In contrast, the fusion capability in children implanted later in life was significantly reduced. This is consistent with the notion that an extended duration of deafness might be detrimental to the use of a CI. In addition, typical McGurk-like effects have recently been showed in postlingually deafened candidates (Rouger et al., 2007; Tremblay et al., 2010), in accordance with the idea that crossmodal changes depend of the onset of sensory deprivation. The segregation of incongruent cues. In our laboratory, we investigated the ability of CI users to segregate conflicting auditory and visual inputs (Champoux et al., 2009; see Fig. 4). An auditory speech recognition task was used in the presence of three different incongruent visual stimuli (color-shift, random-dot motion, and lip movement). We showed that the presentation of visual stimuli significantly impairs auditory-word recognition in nonproficient CI users (individuals with poor performance in the speech task without any concurrent visual presentation) while not affecting the performance of proficient CI users and normal hearing subjects. Moreover, this effect was not specific to the presence of linguistic cues (lip movement condition) but was also present during the random-dot motion stimuli. These results are consistent with the notion of extensive changes for the motion-processing dorsal pathway in the deaf (Armstrong et al., 2002) and with our idea that the level of plastic changes consequent to deafferentation might be a crucial factor for auditory rehabilitation through the use of a CI (Doucet et al., 2006). Most
224 Audiovisual interaction in cochlear implant users None
Color-shift
Dot motion
Lip motion
(a) 250 ms
(d)
Auditory stimulus 15 kHz 10 kHz 5 kHz
(b) 500 ms
(c)
500 ms 250 ms
(f)
100
Decrease of performance (%)
Decrease of performance (%)
(e)
Control-paired
75
Proficient CI users
50
25
0
Color-shift
Dot motion
Lip motion
*
100 Control-paired
75
Non-proficient CI users
50
*
25
0
Color-shift
Dot motion
Lip motion
Fig. 4. Audiovisual interaction in CI users. In the top panel is the illustration of the experimental procedure. Each condition began (a) and ended (c) in a static neutral position. In all audiovisual conditions (b), auditory stimuli (d) were simultaneously presented with a visual stimulus change (color, movement, or video sequence). In the bottom panel are plotted the decreases in performance (%) for each audiovisual condition for both proficient (e) and nonproficient (f) CI users. Adapted with permission from Champoux et al. (2009).
important, these data suggest that although visual signals can facilitate speech perception in CI users in congruent audiovisual conditions, they might also hinder speech discrimination performance in some CI users when audiovisual inputs need to be segregated.
Conclusion The immaturity of the human brain at birth is a valuable trait. Delaying the maturation and growth of brain circuits allows initial confrontations with the environment to shape the
225
developing neural architecture in order to create the most adapted circuitry to cope with the external world (Meltzoff et al., 2009). Over the first few years of life, the brain grows rapidly, with each neuron having 2500 synapses at birth and going to 15,000 synapses per neuron after 2–3 years (Gopnik et al., 1999). As we age, experience will drive a process called synaptic pruning, which eliminates or strengthens connections based on the frequency of their use. Indeed, in the same way a gardener would prune a tree in order to give it a desired shape, ineffective connections are pruned in order to adapt the brain to its environment. Even if experience-dependent plasticity appears to be far more pronounced in children, synaptic connection efficiency changes based on experience are also present at more advanced ages. As discussed at length in this chapter, sensory deprivation at early and, to a lesser extent, later ages will induce plastic changes in the structural and functional architecture of sensory cortices. Any severe sensory deafferentation precipitates unexpected sensory access to the affected cortex by the remaining senses. Such crossmodal plasticity is thought to be intrinsically linked to behavioral compensation mechanisms observed in sensory-deprived individuals (Amedi et al., 2003; Gougoux et al., 2005). Indeed, we have argued that rehabilitation based on sensory substitution systems, among which the two most well known are probably the Braille reading system for the blind and the sign language system for the deaf, spontaneously benefit from the natural tendency of the sensory-deprived brain to reorganize itself to optimize the processing of nonvisual inputs. In contrast, rehabilitation techniques aimed at restoring the deprived sense, like neuroprostheses, are based on an opposite principle of rehabilitation and rely on the integrity of the original function of sensory-deprived cortex. In both cases, we strongly believe that a better understanding of the mechanisms underlying experience-dependent crossmodal plasticity is a necessary prerequisite to properly develop new rehabilitation avenues. The task is obviously not
an easy one because the full impact of sensory deprivation is always the result of a complex interaction between the specific etiology, the age of onset, the length of the deprivation, as well as the strategy that has been put in place in order to cope with the handicap. However, some lessons can be learned from the studies described above. For instance, if an invasive intervention for restoring the deprived sense is chosen in the case of congenital or early childhood deprivation, the “the earlier, the better” adage holds true based on the principle that it is easier to build than to rebuild, meaning that when neural circuitry has reached maturity, the possibility of rewiring it by the introduction of a novel input is more limited. The rapid development of neuroimaging tools over the past few decades has allowed us to probe the brain's functioning and anatomy in a noninvasive manner and thus may serve as a standard procedure in order to evaluate the suitability of specific rehabilitation procedures in the future (Merabet et al., 2005). For example, the observation of massive crossmodal recruitment of the deafferented cortex could alert us that the restoration of the deprived function with new rehabilitative interventions may be more problematic than first thought (Gregory and Wallace, 1963). This is reminiscent of a quote from the philosopher Jean-Jacques Rousseau: “With progress, we know what we gain but not what we lose.” We again stress that a better basic comprehension of the underlying mechanisms of crossmodal plasticity will help us better understand and predict the outcome of sensory restoration based on increasingly complex biotechnologies.
Acknowledgments This research was supported in part by the Canada Research Chair Program (F. L.), the Canadian Institutes of Health Research (P. V. and F. L.), and the Natural Sciences and Engineering Research Council of Canada (F. L.).
226
References Amedi, A., Floel, A., Knecht, S., Zohary, E., & Cohen, L. G. (2004). Transcranial magnetic stimulation of the occipital pole interferes with verbal processing in blind subjects. Nature Neuroscience. Amedi, A., Raz, N., Pianka, P., Malach, R., & Zohary, E. (2003). Early “visual” cortex activation correlates with superior verbal memory performance in the blind. Nature Neuroscience, 6, 758–766. Amedi, A., Stern, W. M., Camprodon, J. A., Bermpohl, F., Merabet, L., Rotman, S., et al. (2007). Shape conveyed by visual-to-auditory sensory substitution activates the lateral occipital complex. Nature Neuroscience, 10, 687–689. Armstrong, B. A., Neville, H. J., Hillyard, S. A., & Mitchell, T. V. (2002). Auditory deprivation affects processing of motion, but not color. Brain Research. Cognitive Brain Research, 14, 422–434. Arno, P., Capelle, C., Wanet-Defalque, M. C., CatalanAhumada, M., & Veraart, C. (1999). Auditory coding of visual patterns for the blind. Perception, 28, 1013–1029. Arno, P., De Volder, A. G., Vanlierde, A., WanetDefalque, M. C., Streel, E., Robert, A., et al. (2001). Occipital activation by pattern recognition in the early blind using auditory substitution for vision. Neuroimage, 13, 632–645. Arno, P., Vanlierde, A., Streel, E., Wanet-Defalque, M. C., Sanabria-Bohorquez, S. M., & Veraart, C. (2001). Auditory substitution of vision: Pattern recognition by blind. Applied Cognitive Psychology, 15, 509–519. Bach-y-Rita, P., Kaczmarek, K. A., Tyler, M. E., & GarciaLara, J. (1998). Form perception with a 49-point electrotactile stimulus array on the tongue: A technical note. Journal of Rehabilitation Research and Development, 35, 427–430. Bach-y-Rita, P., & Kercel, S. (2003). Sensory substitution and the human-machine interface. Trends in Cognitive Sciences, 7, 541–546. Bavelier, D., Brozinsky, C., Tomann, A., Mitchell, T., Neville, H., & Liu, G. (2001). Impact of early deafness and early exposure to sign language on the cerebral organization for motion processing. The Journal of Neuroscience, 21, 8931–8942. Bavelier, D., Dye, M. W., & Hauser, P. C. (2006). Do deaf individuals see better? Trends in Cognitive Sciences, 10, 512–518. Bavelier, D., & Neville, H. J. (2002). Cross-modal plasticity: Where and how? Nature Reviews. Neuroscience, 3, 443–452. Bavelier, D., Tomann, A., Hutton, C., Mitchell, T., Corina, D., Liu, G., et al. (2000). Visual attention to the periphery is enhanced in congenitally deaf individuals. The Journal of Neuroscience, 20, RC93. Bedny, M., Konkle, T., Pelphrey, K., Saxe, R., & PascualLeone, A. (2010). Sensitive period for a multimodal
response in human visual motion area MT/MST. Current Biology, 20, 1900–1906. Belin, P., Zatorre, R. J., Lafaille, P., Ahad, P., & Pike, B. (2000). Voice-selective areas in human auditory cortex. Nature, 403, 309–312. Bergeson, T. R., & Pisoni, D. B. (2004). Audiovisual speech perception in deaf adults and children following cochlear implantation. In G. Calvert, C. Sence & B. E. Stein (Eds.), Handbook of multisensory processes (pp. 749–772). Cambridge: MIT Press. Bosworth, R. G., & Dobkins, K. R. (2002). The effects of spatial attention on motion processing in deaf signers, hearing signers, and hearing nonsigners. Brain and Cognition, 49, 152–169. Bowering, E. R., Maurer, D., Lewis, T. L., & Brent, H. P. (1993). Sensitivity in the nasal and temporal hemifields in children treated for cataract. Investigative Ophthalmology and Visual Science, 34, 3501–3509. Brozinsky, C. J., & Bavelier, D. (2004). Motion velocity thresholds in deaf signers: Changes in lateralization but not in overall sensitivity. Brain Research. Cognitive Brain Research, 21, 1–10. Buchel, C. (1998). Functional neuroimaging studies of Braille reading: Cross-modal reorganization and its implications. Brain, 121(Pt. 7), 1193–1194. Burton, H., Diamond, J. B., & McDermott, K. B. (2003). Dissociating cortical regions activated by semantic and phonological tasks: A FMRI study in blind and sighted people. Journal of Neurophysiology, 90, 1965–1982. Burton, H., Snyder, A. Z., Conturo, T. E., Akbudak, E., Ollinger, J. M., & Raichle, M. E. (2002). Adaptive changes in early and late blind: A fMRI study of Braille reading. Journal of Neurophysiology, 87, 589–607. Capelle, C., Trullemans, C., Arno, P., & Veraart, C. (1998). A real-time experimental prototype for enhancement of vision rehabilitation using auditory substitution. IEEE Transactions on Biomedical Engineering, 45, 1279–1293. Champoux, F., Lepore, F., Gagne, J. P., & Theoret, H. (2009). Visual stimuli can impair auditory processing in cochlear implant users. Neuropsychologia, 47, 17–22. Clark, G. M. (2006). The multiple-channel cochlear implant: The interface between sound and the central nervous system for hearing, speech, and language in deaf people—A personal perspective. Philosophical Transactions of the Royal Society of London. Series B: Biological Sciences, 361, 791–810. Cohen, L. G., Celnik, P., Pascual-Leone, A., Corwell, B., Falz, L., Dambrosia, J., et al. (1997). Functional relevance of cross-modal plasticity in blind humans. Nature, 389, 180–183. Cohen, L. G., Weeks, R. A., Sadato, N., Celnik, P., Ishii, K., & Hallett, M. (1999). Period of susceptibility for cross-modal plasticity in the blind. Annals of Neurology, 45, 451–460.
227 Collignon, O., Davare, M., Olivier, E., & De Volder, A. G. (2009). Reorganisation of the right occipito-parietal stream for auditory spatial processing in early blind humans. A transcranial magnetic stimulation study. Brain Topography, 21, 232–240. Collignon, O., Lassonde, M., Lepore, F., Bastien, D., & Veraart, C. (2007). Functional cerebral reorganization for auditory spatial processing and auditory substitution of vision in early blind subjects. Cerebral Cortex, 17, 457–465. Collignon, O., Renier, L., Bruyer, R., Tranduy, D., & Veraart, C. (2006). Improved selective and divided spatial attention in early blind subjects. Brain Research, 1075, 175–182. Collignon, O., Vandewalle, G., Voss, P., Albouy, G., Charbonneau, G., Lassonde, M., & Lepore, F. (2011). Functional specialization for auditory-spatial processing in the occipital cortex of congenitally blind humans. Proceedings of the National Academy of Sciences, 108, 4435–4440. Collignon, O., Voss, P., Lassonde, M., & Lepore, F. (2009). Cross-modal plasticity for the spatial processing of sounds in visually deprived subjects. Experimental Brain Research, 192, 343–358. Cronly-Dillon, J., Persaud, K., & Gregory, R. P. (1999). The perception of visual images encoded in musical form: A study in cross-modality information transfer. Proceedings of Biological Sciences, 266, 2427–2433. De Volder, A. G., Catalan-Ahumada, M., Robert, A., Bol, A., Labar, D., Coppens, A., et al. (1999). Changes in occipital cortex activity in early blind humans using a sensory substitution device. Brain Research, 826, 128–134. Desai, S., Stickney, G., & Zeng, F. G. (2008). Auditory-visual speech perception in normal-hearing and cochlear-implant listeners. The Journal of the Acoustical Society of America, 123, 428–440. Dormal, G., & Collignon O. (2011). Functional selectivity in sensory deprived cortices. Journal of Neurophysiology. In Press. Doucet, M. E., Bergeron, F., Lassonde, M., Ferron, P., & Lepore, F. (2006). Cross-modal reorganization and speech perception in cochlear implant users. Brain, 129, 3376–3383. Dufour, A., Despres, O., & Candas, V. (2005). Enhanced sensitivity to echo cues in blind subjects. Experimental Brain Research, 165, 515–519. Elbert, T., Sterr, A., Rockstroh, B., Pantev, C., Muller, M. M., & Taub, E. (2002). Expansion of the tonotopic area in the auditory cortex of the blind. The Journal of Neuroscience, 22, 9941–9944. Ellemberg, D., Lewis, T. L., Maurer, D., Brar, S., & Brent, H. P. (2002). Better perception of global motion after monocular than after binocular deprivation. Vision Research, 42, 169–179. Fallon, J. B., Irvine, D. R., & Shepherd, R. K. (2008). Cochlear implants and brain plasticity. Hearing Research, 238, 110–117.
Fine, I., Wade, A. R., Brewer, A. A., May, M. G., Goodman, D. F., Boynton, G. M., et al. (2003). Long-term deprivation affects visual perception and cortex. Nature Neuroscience, 6, 915–916. Finney, E. M., Clementz, B. A., Hickok, G., & Dobkins, K. R. (2003). Visual stimuli activate auditory cortex in deaf subjects: Evidence from MEG. Neuroreport, 14, 1425–1427. Finney, E. M., Fine, I., & Dobkins, K. R. (2001). Visual stimuli activate auditory cortex in the deaf. Nature Neuroscience, 4, 1171–1173. Frost, D. O., Boire, D., Gingras, G., & Ptito, M. (2000). Surgically created neural pathways mediate visual pattern discrimination. Proceedings of the National Academy of Sciences of the United States of America, 97, 11068–11073. Frost, D. O., & Metin, C. (1985). Induction of functional retinal projections to the somatosensory system. Nature, 317, 162–164. Geers, A. E. (2004). Speech, language, and reading skills after early cochlear implantation. Archives of Otolaryngology, 130, 634–638. Giraud, A. L., & Lee, H. J. (2007). Predicting cochlear implant outcome from brain organisation in the deaf. Restorative Neurology and Neuroscience, 25, 381–390. Giraud, A. L., Price, C. J., Graham, J. M., & Frackowiak, R. S. (2001). Functional plasticity of language-related brain areas after cochlear implantation. Brain, 124, 1307–1316. Gopnik, A., Meltzoff, A., & Kuhl, P. (1999). The scientist in the crib: What early learning tells us about the mind. New York, NY: HarperCollins Publishers. Gothe, J., Brandt, S. A., Irlbacher, K., Roricht, S., Sabel, B. A., & Meyer, B. U. (2002). Changes in visual cortex excitability in blind subjects as demonstrated by transcranial magnetic stimulation. Brain, 125, 479–490. Gougoux, F., Belin, P., Voss, P., Lepore, F., Lassonde, M., & Zatorre, R. J. (2009). Voice perception in blind persons: A functional magnetic resonance imaging study. Neuropsychologia, 47, 2967–2974. Gougoux, F., Zatorre, R. J., Lassonde, M., Voss, P., & Lepore, F. (2005). A functional neuroimaging study of sound localization: Visual cortex activity predicts performance in early-blind individuals. PLoS Biology, 3, e27. Green, K. M., Bhatt, Y. M., Mawman, D. J., O'Driscoll, M. P., Saeed, S. R., Ramsden, R. T., et al. (2007). Predictors of audiological outcome following cochlear implantation in adults. Cochlear Implants International, 8, 1–11. Gregory, R. L. (2003). Seeing after blindness. Nature Neuroscience, 6, 909–910. Gregory, R. L., & Wallace, J. (1963). Recovery from early blindness: A case study. Experimental Psychology Society Monograph, 2, Heffers, Cambridge. (Reprinted in: Gregory, R. L. (1974). Concepts and mechanisms of perception. Duckworth, London.).
228 Hamilton, R., Keenan, J. P., Catala, M., & Pascual-Leone, A. (2000). Alexia for Braille following bilateral occipital stroke in an early blind woman. Neuroreport, 11, 237–240. Hammes, D. M., Novak, M. A., Rotz, L. A., Willis, M., Edmondson, D. M., & Thomas, J. F. (2002). Early identification and cochlear implantation: Critical factors for spoken language development. The Annals of Otology, Rhinology and Laryngology. Supplement, 189, 74–78. Hebb, D. O. (1949). The organization of behavior. New York: John Wiley. Hirano, S., Naito, Y., Kojima, H., Honjo, I., Inoue, M., Shoji, K., et al. (2000). Functional differentiation of the auditory association area in prelingually deaf subjects. Auris, Nasus, Larynx, 27, 303–310. Humayun, M. S., Weiland, J. D., Fujii, G. Y., Greenberg, R., Williamson, R., Little, J., et al. (2003). Visual perception in a blind subject with a chronic microelectronic retinal prosthesis. Vision Research, 43, 2573–2581. Jiang, J., Zhu, W., Shi, F., Liu, Y., Li, J., Qin, W., et al. (2009). Thick visual cortex in the early blind. The Journal of Neuroscience, 29, 2205–2211. Kaas, J. H., Merzenich, M. M., & Killackey, H. P. (1983). The reorganization of somatosensory cortex following peripheral nerve damage in adult and developing mammals. Annual Review of Neuroscience, 6, 325–356. Kaczmarek, K., Rita, P., Tompkins, W. J., & Webster, J. G. (1985). A tactile vision-substitution system for the blind: Computer-controlled partial image sequencing. IEEE Transactions on Biomedical Engineering, 32, 602–608. Kaiser, A. R., Kirk, K. I., Lachs, L., & Pisoni, D. B. (2003). Talker and lexical effects on audiovisual word recognition by adults with cochlear implants. Journal of Speech, Language, and Hearing Research, 46, 390–404. Kral, A., Tillein, J., Heid, S., Hartmann, R., & Klinke, R. (2005). Postnatal cortical development in congenital auditory deprivation. Cerebral Cortex, 15, 552–562. Kupers, R., Chebat, D. R., Madsen, K. H., Paulson, O. B., & Ptito, M. (2010). Neural correlates of virtual route recognition in congenital blindness. Proceedings of the National Academy of Sciences of the United States of America, 107, 12716–12721. Kupers, R., Pappens, M., de Noordhout, A. M., Schoenen, J., Ptito, M., & Fumal, A. (2007). rTMS of the occipital cortex abolishes Braille reading and repetition priming in blind subjects. Neurology, 68, 691–693. Le Grand, R., Mondloch, C. J., Maurer, D., & Brent, H. P. (2001). Neuroperception. Early visual experience and face processing. Nature, 410, 890. Le Grand, R., Mondloch, C. J., Maurer, D., & Brent, H. P. (2004). Impairment in holistic face processing following early visual deprivation. Psychological Science, 15, 762–768. Lee, H. J., Giraud, A. L., Kang, E., Oh, S. H., Kang, H., Kim, C. S., et al. (2007). Cortical activity at rest predicts
cochlear implantation outcome. Cerebral Cortex, 17, 909–917. Lee, H. J., Kang, E., Oh, S. H., Kang, H., Lee, D. S., Lee, M. C., et al. (2005). Preoperative differences of cerebral metabolism relate to the outcome of cochlear implants in congenitally deaf children. Hearing Research, 203, 2–9. Lee, D. S., Lee, J. S., Oh, S. H., Kim, S. K., Kim, J. W., Chung, J. K., et al. (2001). Cross-modal plasticity and cochlear implants. Nature, 409, 149–150. Levin, N., Dumoulin, S. O., Winawer, J., Dougherty, R. F., & Wandell, B. A. (2010). Cortical maps and white matter tracts following long period of visual deprivation and retinal image restoration. Neuron, 65, 21–31. Lewis, T. L., & Maurer, D. (2005). Multiple sensitive periods in human visual development: Evidence from visually deprived children. Developmental Psychobiology, 46, 163–183. Logothetis, N. K., Augath, M., Murayama, Y., Rauch, A., Sultan, F., Goense, J., et al. (2010). The effects of electrical microstimulation on cortical signal propagation. Nature Neuroscience, 13, 1283–1291. MacSweeney, M., Calvert, G. A., Campbell, R., McGuire, P. K., David, A. S., Williams, S. C., et al. (2002). Speechreading circuits in people born deaf. Neuropsychologia, 40, 801–807. Maurer, D., Lewis, T. L., & Mondloch, C. J. (2005). Missing sights: Consequences for visual cognitive development. Trends in Cognitive Sciences, 9, 144–151. McGurk, H., & MacDonald, J. (1976). Hearing lips and seeing voices. Nature, 264, 746–748. Meijer, P. B. (1992). An experimental system for auditory image representations. IEEE Transactions on Biomedical Engineering, 39, 112–121. Meltzoff, A. N., Kuhl, P. K., Movellan, J., & Sejnowski, T. J. (2009). Foundations for a new science of learning. Science, 325, 284–288. Mens, L. H. (2007). Advances in cochlear implant telemetry: Evoked neural responses, electrical field imaging, and technical integrity. Trends in Amplification, 11, 143–159. Merabet, L. B., Battelli, L., Obretenova, S., Maguire, S., Meijer, P., & Pascual-Leone, A. (2009). Functional recruitment of visual cortex for sound encoded object identification in the blind. Neuroreport, 20, 132–138. Merabet, L. B., Hamilton, R., Schlaug, G., Swisher, J. D., Kiriakopoulos, E. T., Pitskel, N. B., et al. (2008). Rapid and reversible recruitment of early visual cortex for touch. PLoS ONE, 3, e3046. Merabet, L. B., Rizzo, J. F., Amedi, A., Somers, D. C., & Pascual-Leone, A. (2005). What blindness can tell us about seeing again: Merging neuroplasticity and neuroprostheses. Nature Reviews Neuroscience, 6, 71–77. Moody-Antonio, S., Takayanagi, S., Masuda, A., Auer, E. T., Jr.Fisher, L., & Bernstein, L. E. (2005). Improved speech
229 perception in adult congenitally deafened cochlear implant recipients. Otology and Neurotology, 26, 649–654. Naito, Y., Hirano, S., Honjo, I., Okazawa, H., Ishizu, K., Takahashi, H., et al. (1997). Sound-induced activation of auditory cortices in cochlear implant users with post- and pre-lingual deafness demonstrated by positron emission tomography. Acta Otolaryngologica, 117, 490–496. Naito, Y., Okazawa, H., Honjo, I., Takahashi, H., Kawano, M., Ishizu, K., et al. (1995). Cortical activation during sound stimulation in cochlear implant users demonstrated by positron emission tomography. The Annals of Otology, Rhinology and Laryngology. Supplement, 166, 60–64. Neville, H. J. (1990). Intermodal competition and compensation in development. Evidence from studies of the visual system in congenitally deaf adults. Annals of the New York Academy of Sciences, 608, 71–87 discussion, 87–91. Neville, H. J., Coffey, S. A., Lawson, D. S., Fischer, A., Emmorey, K., & Bellugi, U. (1997). Neural systems mediating American sign language: Effects of sensory experience and age of acquisition. Brain and Language, 57, 285–308. Neville, H. J., & Lawson, D. (1987a). Attention to central and peripheral visual space in a movement detection task: An event-related potential and behavioral study. II. Congenitally deaf adults. Brain Research, 405, 268–283. Neville, H. J., & Lawson, D. (1987b). Attention to central and peripheral visual space in a movement detection task. III. Separate effects of auditory deprivation and acquisition of a visual language. Brain Research, 405, 284–294. Nishimura, H., Hashikawa, K., Doi, K., Iwaki, T., Watanabe, Y., Kusuoka, H., et al. (1999). Sign language “heard” in the auditory cortex. Nature, 397, 116. Noppeney, U. (2007). The effects of visual deprivation on functional and structural organization of the human brain. Neuroscience and Biobehavioral Reviews, 31, 1169–1180. Noppeney, U., Friston, K. J., Ashburner, J., Frackowiak, R., & Price, C. J. (2005). Early visual deprivation induces structural plasticity in gray and white matter. Current Biology, 15, R488–R490. Ostrovsky, Y., Andalman, A., & Sinha, P. (2006). Vision following extended congenital blindness. Psychological Science, 17, 1009–1014. Ostrovsky, Y., Meyers, E., Ganesh, S., Mathur, U., & Sinha, P. (2009). Visual parsing after recovery from blindness. Psychological Science, 20, 1484–1491. Pan, W. J., Wu, G., Li, C. X., Lin, F., Sun, J., & Lei, H. (2007). Progressive atrophy in the optic pathway and visual cortex of early blind Chinese adults: A voxel-based morphometry magnetic resonance imaging study. Neuroimage, 37, 212–220. Pantev, C., Dinnesen, A., Ross, B., Wollbrink, A., & Knief, A. (2006). Dynamics of auditory plasticity after cochlear implantation: A longitudinal study. Cerebral Cortex, 16, 31–36.
Pantev, C., Oostenveld, R., Engelien, A., Ross, B., Roberts, L. E., & Hoke, M. (1998). Increased auditory cortical representation in musicians. Nature, 392, 811–814. Pardue, M. T., Ball, S. L., Phillips, M. J., Faulkner, A. E., Walker, T. A., Chow, A. Y., et al. (2006). Status of the feline retina 5 years after subretinal implantation. Journal of Rehabilitation Research and Development, 43, 723–732. Pardue, M. T., Phillips, M. J., Hanzlicek, B., Yin, H., Chow, A. Y., & Ball, S. L. (2006). Neuroprotection of photoreceptors in the RCS rat after implantation of a subretinal implant in the superior or inferior retina. Advances in Experimental Medicine and Biology, 572, 321–326. Park, H. J., Lee, J. D., Kim, E. Y., Park, B., Oh, M. K., Lee, S., et al. (2009). Morphological alterations in the congenital blind based on the analysis of cortical thickness and surface area. Neuroimage, 47, 98–106. Pascual-Leone, A., Amedi, A., Fregni, F., & Merabet, L. B. (2005). The plastic human brain cortex. Annual Review of Neuroscience, 28, 377–401. Pascual-Leone, A., & Torres, F. (1993). Plasticity of the sensorimotor cortex representation of the reading finger in Braille readers. Brain, 116(Pt. 1), 39–52. Petitto, L. A., Zatorre, R. J., Gauna, K., Nikelski, E. J., Dostie, D., & Evans, A. C. (2000). Speech-like cerebral activity in profoundly deaf people processing signed languages: Implications for the neural basis of human language. Proceedings of the National Academy of Sciences of the United States of America, 97, 13961–13966. Piche, M., Robert, S., Miceli, D., & Bronchti, G. (2004). Environmental enrichment enhances auditory takeover of the occipital cortex in anophthalmic mice. The European Journal of Neuroscience, 20, 3463–3472. Poirier, C., Collignon, O., Scheiber, C., & De Volder, A. G. (2004). Auditory motion processing in early blind subjects. Cognitive Processing, 5(4), 254–256. Poirier, C., De Volder, A. G., & Scheiber, C. (2007). What neuroimaging tells us about sensory substitution. Neuroscience and Biobehavioral Reviews, 31, 1064–1070. Ponton, C. W., Don, M., Eggermont, J. J., Waring, M. D., Kwong, B., & Masuda, A. (1996). Auditory system plasticity in children after long periods of complete deafness. Neuroreport, 8, 61–65. Proksch, J., & Bavelier, D. (2002). Changes in the spatial distribution of visual attention after early deafness. Journal of Cognitive Neuroscience, 14, 687–701. Proulx, M. J., Stoerig, P., Ludowig, E., & Knoll, I. (2008). Seeing “where” through the ears: Effects of learning-by-doing and long-term sensory deprivation on localization based on image-to-sound substitution. PLoS ONE, 3, e1840. Ptito, M., Fumal, A., de Noordhout, A. M., Schoenen, J., Gjedde, A., & Kupers, R. (2008). TMS of the occipital
230 cortex induces tactile sensations in the fingers of blind Braille readers. Experimental Brain Research, 184, 193–200. Ptito, M., Moesgaard, S. M., Gjedde, A., & Kupers, R. (2005). Cross-modal plasticity revealed by electrotactile stimulation of the tongue in the congenitally blind. Brain, 128, 606–614. Ptito, M., Schneider, F. C., Paulson, O. B., & Kupers, R. (2008). Alterations of the visual pathways in congenital blindness. Experimental Brain Research, 187, 41–49. Putzar, L., Goerendt, I., Lange, K., Rosler, F., & Roder, B. (2007). Early visual deprivation impairs multisensory interactions in humans. Nature Neuroscience, 10, 1243–1245. Putzar, L., Hötting, K., & Röder, B. (2010). Early visual deprivation affects the development of face recognition and audio-visual speech perception. Restorative Neurology and Neuroscience, 28, 251–257. Renier, L., Anurova, I., De Volder, A. G., Carlson, S., VanMeter, J., & Rauschecker, J. P. (2010). Preserved functional specialization for spatial processing in the middle occipital gyrus of the early blind. Neuron, 68, 138–148. Renier, L., & De Volder, A. G. (2010). Vision substitution and depth perception: Early blind subjects experience visual perspective through their ears. Disability and Rehabilitation: Assistive Technology, 5, 175–183. Rettenbach, R., Diller, G., & Sireteanu, R. (1999). Do deaf people see better? Texture segmentation and visual search compensate in adult but not in juvenile subjects. Journal of Cognitive Neuroscience, 11, 560–583. Ricciardi, E., Vanello, N., Sani, L., Gentili, C., Scilingo, E. P., Landini, L., et al. (2007). The effect of visual experience on the development of functional architecture in hMTþ. Cerebral Cortex, 17, 2933–2939. Rice, C. E. (1967). Human echo perception. Science, 155, 656–664. Rice, C. E., & Feinstein, S. H. (1965). Sonar system of the blind: Size discrimination. Science, 148, 1107–1108. Rita, P., Collins, C. C., Saunders, F. A., White, B., & Scadden, L. (1969). Vision substitution by tactile image projection. Nature, 221, 963–964. Rizzo, J. F., 3rd., Wyatt, J., Loewenstein, J., Kelly, S., & Shire, D. (2003a). Methods and perceptual thresholds for short-term electrical stimulation of human retina with microelectrode arrays. Investigative Ophthalmology and Visual Science, 44, 5355–5361. Rizzo, J. F., 3rd., Wyatt, J., Loewenstein, J., Kelly, S., & Shire, D. (2003b). Perceptual efficacy of electrical stimulation of human retina with a microelectrode array during short-term surgical trials. Investigative Ophthalmology and Visual Science, 44, 5362–5369. Roe, A. W., Pallas, S. L., Hahm, J. O., & Sur, M. (1990). A map of visual space induced in primary auditory cortex. Science, 250, 818–820.
Rouger, J., Lagleyre, S., Fraysse, B., Deneve, S., Deguine, O., & Barone, P. (2007). Evidence that cochlear-implanted deaf patients are better multisensory integrators. Proceedings of the National Academy of Sciences of the United States of America, 104, 7295–7300. Sacks, O. (1995). An anthropolgist on Mars. New York: Knopf. Sadato, N., Okada, T., Honda, M., Matsuki, K., Yoshida, M., Kashikura, K., et al. (2005). Cross-modal integration and plastic changes revealed by lip movement, random-dot motion and sign languages in the hearing and deaf. Cerebral Cortex, 15, 1113–1122. Sadato, N., Pascual-Leone, A., Grafman, J., Deiber, M. P., Ibanez, V., & Hallett, M. (1998). Neural networks for Braille reading by the blind. Brain, 121(Pt 7), 1213–1229. Sadato, N., Pascual-Leone, A., Grafman, J., Ibanez, V., Deiber, M. P., Dold, G., et al. (1996). Activation of the primary visual cortex by Braille reading in blind subjects. Nature, 380, 526–528. Sadato, N., Yamada, H., Okada, T., Yoshida, M., Hasegawa, T., Matsuki, K., et al. (2004). Age-dependent plasticity in the superior temporal sulcus in deaf humans: A functional MRI study. BMC Neuroscience, 5, 56. Saenz, M., Lewis, L. B., Huth, A. G., Fine, I., & Koch, C. (2008). Visual motion area MTþ/V5 responds to auditory motion in human sight-recovery subjects. The Journal of Neuroscience, 28, 5141–5148. Schiller, P. H., & Tehovnik, E. J. (2008). Visual prosthesis. Perception, 37, 1529–1559. Schmidt, E. M., Bak, M. J., Hambrecht, F. T., Kufta, C. V., O'Rourke, D. K., & Vallabhanath, P. (1996). Feasibility of a visual prosthesis for the blind based on intracortical microstimulation of the visual cortex. Brain, 119(Pt 2), 507–522. Schorr, E. A., Fox, N. A., van, W. V., & Knudsen, E. I. (2005). Auditory-visual fusion in speech perception in children with cochlear implants. Proceedings of the National Academy of Sciences of the United States of America, 102, 18748–18750. Segond, H., Weiss, D., & Sampaio, E. (2005). Human spatial navigation via a visuo-tactile sensory substitution system. Perception, 34, 1231–1249. Serino, A., Bassolino, M., Farne, A., & Ladavas, E. (2007). Extended multisensory space in blind cane users. Psychological Science, 18, 642–648. Shu, N., Li, J., Li, K., Yu, C., & Jiang, T. (2009). Abnormal diffusion of cerebral white matter in early blindness. Human Brain Mapping, 30, 220–227. Sterr, A., Muller, M. M., Elbert, T., Rockstroh, B., Pantev, C., & Taub, E. (1998). Perceptual correlates of changes in cortical representation of fingers in blind multifinger Braille readers. The Journal of Neuroscience, 18, 4417–4423. Strelow, E. R., & Brabyn, J. A. (1982). Locomotion of the blind controlled by natural sound cues. Perception, 11, 635–640.
231 Sur, M., Garraghty, P. E., & Roe, A. W. (1988). Experimentally induced visual projections into auditory thalamus and cortex SUR1988. Science, 242, 1437–1441. Tehovnik, E. J., Slocum, W. M., Carvey, C. E., & Schiller, P. H. (2005). Phosphene induction and the generation of saccadic eye movements by striate cortex. Journal of Neurophysiology, 93, 1–19. Tremblay, C., Champoux, F., Lepore, F., & Theoret, H. (2010). Audiovisual fusion and cochlear implant proficiency. Restorative Neurology and Neuroscience, 28, 283–291. Tyler, R. S., Fryauf-Bertschy, H., Kelsay, D. M., Gantz, B. J., Woodworth, G. P., & Parkinson, A. (1997). Speech perception by prelingually deaf children using cochlear implants. Otolaryngology—Head and Neck Surgery, 117, 180–187. Van Boven, R. W., Hamilton, R. H., Kauffman, T., Keenan, J. P., & Pascual-Leone, A. (2000). Tactile spatial resolution in blind braille readers. Neurology, 54, 2230–2236. Veraart, C., Duret, F., Brelen, M., Oozeer, M., & Delbeke, J. (2004). Vision rehabilitation in the case of blindness. Expert Review of Medical Devices, 1, 139–153. Veraart, C., Raftopoulos, C., Mortimer, J. T., Delbeke, J., Pins, D., Michaux, G., et al. (1998). Visual sensations produced by optic nerve stimulation using an implanted selfsizing spiral cuff electrode. Brain Research, 813, 181–186. Veraart, C., Wanet-Defalque, M. C., Gerard, B., Vanlierde, A., & Delbeke, J. (2003). Pattern recognition with the optic nerve visual prosthesis. Artificial Organs, 27, 996–1004. von Melchner, L., Pallas, S. L., & Sur, M. (2000). Visual behaviour mediated by retinal projections directed to the auditory pathway. Nature, 404, 871–876. Voss, P., Gougoux, F., Zatorre, R. J., Lassonde, M., & Lepore, F. (2008). Differential occipital responses in earlyand late-blind individuals during a sound-source discrimination task. Neuroimage, 40, 746–758.
Waltzman, S. B., & Cohen, N. L. (1998). Cochlear implantation in children younger than 2 years old. The American Journal of Otology, 19, 158–162. Wan, C. Y., Wood, A. G., Reutens, D. C., & Wilson, S. J. (2010). Early but not late-blindness leads to enhanced auditory perception. Neuropsychologia, 48, 344–348. Wanet-Defalque, M. C., Veraart, C., De Volder, A., Metz, R., Michel, C., Dooms, G., et al. (1988). High metabolic activity in the visual cortex of early blind human subjects. Brain Research, 446, 369–373. Wiesel, T. N., & Hubel, D. H. (1965). Extent of recovery from the effects of visual deprivation in kittens. Journal of Neurophysiology, 28, 1060–1072. Wiesel, T. N., & Hubel, D. H. (1974). Ordered arrangement of orientation columns in monkeys lacking visual experience. The Journal of Comparative Neurology, 158, 307–318. Wittenberg, G. F., Werhahn, K. J., Wassermann, E. M., Herscovitch, P., & Cohen, L. G. (2004). Functional connectivity between somatosensory and visual cortex in early blind humans. The European Journal of Neuroscience, 20, 1923–1927. Wong, M., Gnanakumaran, V., & Goldreich, D. (2011). Tactile spatial acuity enhancement in blindness: Evidence for experience-dependent MechanismsTactile spatial acuity enhancement in blindness: Evidence for experience-dependent mechanisms. The Journal of Neuroscience, 31, 7028–7037. Wong, D., Miyamoto, R. T., Pisoni, D. B., Sehgal, M., & Hutchins, G. D. (1999). PET imaging of cochlear-implant and normal-hearing subjects listening to speech and nonspeech. Hearing Research, 132, 34–42. Zrenner, E. (2002). Will retinal implants restore vision? Science, 295, 1022–1025. Zrenner, E., Stett, A., Weiss, S., Aramant, R. B., Guenther, E., Kohler, K., et al. (1999). Can subretinal microphotodiodes successfully replace degenerated photoreceptors? Vision Research, 39, 2555–2567.
A. M. Green, C. E. Chapman, J. F. Kalaska and F. Lepore (Eds.) Progress in Brain Research, Vol. 191 ISSN: 0079-6123 Copyright Ó 2011 Elsevier B.V. All rights reserved.
CHAPTER 15
Crossmodal plasticity in sensory loss Johannes Frasnelli{,*, Olivier Collignon{,}, Patrice Voss{ and Franco Lepore{ {
{
Département de Psychologie, Centre de Recherche en Neuropsychologie et Cognition, Université de Montréal, Montréal, Québec, Canada International Laboratory for Brain, Music and Sound Research, Université de Montréal, Montréal, Québec, Canada } Centre de Recherche CHU Sainte-Justine, Université de Montréal, Montréal, Québec, Canada
Abstract: In this review, we describe crossmodal plasticity following sensory loss in three parts, with each section focusing on one sensory system. We summarize a wide range of studies showing that sensory loss may lead, depending of the affected sensory system, to functional changes in other, primarily not affected senses, which range from heightened to lowered abilities. In the first part, the effects of blindness on mainly audition and touch are described. The latest findings on brain reorganization in blindness are reported, with a particular emphasis on imaging studies illustrating how nonvisual inputs recruit the visually deafferented occipital cortex. The second part covers crossmodal processing in deafness, with a special focus on the effects of deafness on visual processing. In the last portion of this review, we present the effects that the loss of a chemical sense have on the sensitivity of the other chemical senses, that is, smell, taste, and trigeminal chemosensation. We outline how the convergence of the chemical senses to the same central processing areas may lead to the observed reduction in sensitivity of the primarily not affected senses. Altogether, the studies reviewed herein illustrate the fascinating plasticity of the brain when coping with sensory deprivation. Keywords: blindness; deafness; anosmia; crossmodal plasticity.
senses during their lifetime. Still, persons with sensory loss are often able to live independently and can achieve an impressive degree of accomplishments. In fact, there is a plethora of reports (though often anecdotic) of persons with a sensory loss demonstrating extraordinary abilities with one or several of their remaining senses, with the large number of successful blind musicians being the most prominent example. Going back several decades, Diderot, in his “Lettre
Introduction While most humans can rely on several sensory systems to appropriately interact with the environment, some individuals are born without one or more senses while others may lose one or more *Corresponding author. Tel.: þ1-514-343-6111x0705; Fax: þ1-514-343-5787 E-mail:
[email protected] DOI: 10.1016/B978-0-444-53752-2.00002-3
233
234
sur les aveugles” (Diderot, 1749), reported the famous case of a blind mathematician who could recognize fake from real money coins just by touching them. Similarly, William James explained blind individuals’ remarkable ability to navigate through their environment without colliding with obstacles as resulting from a form of “facial perception” (James, 1890). At first glance, such performance may seem somewhat “supranormal.” However, over the past decades, we have acquired extensive knowledge on compensatory and adaptive changes in primarily unaffected senses occurring after sensory loss and have a better understanding as to how and why they occur. The substantial literature on such compensatory mechanisms that are observed in the blind has often attributed these enhancements to some form of “crossmodal plasticity.” Crossmodal plasticity generally refers to the adaptive reorganization of neurons to integrate the function of a new sensory modality following the loss of another. In fact, such crossmodal plasticity appears to at least partly explain many extraordinary abilities observed in persons with sensory loss. In the following sections, we provide an overview of crossmodal plastic changes that follow sensory loss. We specifically focus on three major topics, that is, blindness, deafness, and loss of chemical senses and how these states affect the other sensory systems.
Blindness Behavioral reorganization in blindness It has long been debated whether blind individuals have perceptual advantages or disadvantages in processing information received via the intact modalities. The fundamental question has been whether the lack of vision disrupts the proper development of nonvisual skills or if, in contrast, blindness enables above-normal performance in the preserved modalities. Even if
several studies support the notion that vision may be required to adequately calibrate other sensory modalities (Axelrod, 1959; Lewald, 2002; Zwiers et al., 2001), a substantial number of recent experiments have demonstrated that blind people are able to compensate for their lack of vision through efficient use of their remaining senses. In studies exploring sharpened nonvisual skills in blind people, spatial processing has been extensively investigated (Collignon et al., 2009c). This observation is probably due to the predominant role of vision in this cognitive ability and the importance for blind people to efficiently extract spatial information from the remaining senses in order to properly and safely navigate in their environment. In a seminal study, Lessard et al. (1998) investigated the auditory localization abilities of early blind individuals under binaural and monaural listening conditions. They first demonstrated that blind subjects can localize binaurally presented sounds as well as sighted individuals, suggesting that vision is not necessary for the construction of a three-dimensional auditory map of space. Moreover, half of the blind subjects significantly outperformed the sighted ones when they had to localize the sounds with one ear occluded (monaural localization). This finding strongly suggests that some blind individuals can use subtle spatial cues (i.e., spectral cues) more efficiently than sighted controls. Another consistent finding is that blind individuals typically outperform sighted ones in binaural localization tasks when the sound sources are located in more peripheral positions as opposed to when they are presented centrally (Roder et al., 1999; Simon et al., 2002, Voss et al., 2004). In recent experiments, we investigated the ability of blind participants to sharply focus their attention and quickly react to auditory or tactile spatial targets (Collignon and De Volder, 2009; Collignon et al., 2006). These studies demonstrated that blind subjects reacted faster than sighted controls to non visual spatial targets in selective and divided attention tasks further extending the
235
view that blind individuals are able to compensate their lack of vision by developing capacities in their remaining senses that exceed those of sighted individuals. The studies described above examined spatial hearing in near space, a region where auditory representations can be calibrated through sensory-motor feedback in blind subjects, such as touching the source of the sound or through the use of a cane, for example. In a later study, we evaluated sound localization in far space, a region of space where sensori-motor feedback could not contribute to the calibration of auditory spatial maps. We showed not only that blind individuals properly mapped their auditory distant space, but actually outperformed their sighted counterparts under specific conditions (Voss et al., 2004). Moreover, we examined whether late-onset blind subjects can manifest sensory compensation, since only a few studies have investigated this point. We thus carried out the task in late-blind subjects and showed that this group could also develop above-normal spatial abilities (Voss et al., 2004), as confirmed in another study (Fieger et al., 2006). However, a recent experiment showed that early but not late-blind participants showed better performance than that of sighted participants on a range of auditory perception tasks (Wan et al., 2010). Interestingly, in the above-mentioned studies, the superiority of early- and late-blind subjects was only present when sounds were presented in the periphery, where more subtle (e.g., spectral) auditory cues have to be exploited to efficiently resolve the task (Fieger et al., 2006; Roder et al., 1999; Simon et al., 2002; Voss et al., 2004). Similarly, when behavioral compensations are observed for the processing of visuospatial stimuli in deaf subjects, they also mainly concern inputs originating in the peripheral visual field (Bavelier et al., 2000; Neville and Lawson, 1987). These compensations observed specifically for peripheral stimuli may be related to the fact that differences in performance may emerge preferentially in conditions where the task is difficult
(i.e., the sighted subjects are not performing at near perfect levels). Recent studies have also pointed out that visual deprivation during early development results in important qualitative changes in nonvisual spatial perception (Eimer, 2004). Other experiments with blind people have suggested that the default localization of touch and proprioception in external space is in fact dependent on early visual experience (Hotting and Roder, 2009; Roder et al., 2004, 2008). For example, Roder et al. (2004) asked participants to judge the temporal order in which two tactile stimuli were delivered to their left and right hands. As expected, they found that temporal order judgments of sighted participants were less accurate with crossed than with uncrossed hands, which would result from the conflict between external and somatotopic spatial codes. By contrast, a congenitally blind group was completely unaffected by crossing the hands. Thus, it seems that sighted persons always use a visually defined reference frame to localize tactile events in external space (Kitazawa, 2002), and are impaired by conflicting external and somatotopic spatial information. By contrast, congenitally blind subjects do not use external spatial coordinates and thus remain unaffected by this conflict. Moreover, the fact that there is no need, in the case of early blindness, to make a correspondence between a nonvisual frame of reference and a visual one would contribute to a faster processing of nonvisual spatial information (Roder et al., 2004). This explanation was supported by an electroencephalographic study showing that the detection of deviant tactile stimuli at the hand induced event-related potentials that varied in crossed when compared to uncrossed postural conditions in sighted subjects, whereas changing the posture of the hand had no influence on the early blind subjects’ brain activity (Roder et al., 2008). In a recent study, we extended this finding by demonstrating that the use of an anatomically anchored reference system for touch and proprioception in subjects visually deprived since birth
236
impaired their ability to integrate audio-tactile information across postural changes (Collignon et al., 2009a). Altogether, these results thus demonstrate that the default remapping of touch/proprioception into external coordinates is acquired during early development as a consequence of visual input. It is, however, important to note that compensatory mechanisms following visual deprivation could extend beyond the auditory spatial domain. For example, enhanced performance in blind participants was also observed in auditory tasks involving pitch (Gougoux et al., 2004; Wan et al., 2010), echoes (Rice and Feinstein, 1965; Rice et al., 1965), or verbal (Amedi et al., 2003) discrimination. The tactile modality has also been studied in blind individuals and is especially interesting given its importance in Braille reading. Compared to sighted controls, blind subjects showed superior abilities in some tactile tasks, such as a haptic angle discrimination task (Alary et al., 2008) and a texture discrimination task, but exhibited similar grating orientation thresholds and vibrotactile frequency discrimination thresholds as the sighted subjects (Alary et al., 2009). A carefully designed study demonstrated that when age and sex of the two groups were carefully matched, the average blind subject had the acuity of an average sighted person of the same gender but 23 years younger (Goldreich and Kanics, 2003). A recent study by Wong and collaborators (2011) observed this heightened tactile acuity in blind subjects to depend on braille readings skills suggesting the sensory compensation to be a direct consequence of the practice of the blind subjects with the braille system. With regard to the chemical senses, several studies suggest that blind subjects outperform sighted subjects in difficult higher-order olfactory tasks, such as free odor identification and odor labeling (Murphy and Cain, 1986; Rosenbluth et al., 2000; Wakefield et al., 2004), but not in simpler and more basic olfactory tasks such as odor threshold or odor discrimination (Diekmann et al., 1994; Schwenn et al., 2002; Smith et al., 1993; Wakefield et al., 2004).
Brain reorganization in blindness Researchers have hypothesized for a long time that brain reorganization could underlie the changes in behavior observed in blind individuals. In particular, it was postulated that the functioning of visual structures changed dramatically following visual deprivation, and increasing evidence points now to the extensive colonization of the occipital cortex (OC)—traditionally considered as visual—by nonvisual inputs in blind individuals (Collignon et al., 2009c). In pioneering studies using positron emission tomography (PET), Veraart and collaborators demonstrated elevated metabolic activity in OC of early blind individuals at rest, which was at about the same level as in sighted subjects involved in a visual task (Veraart et al., 1990; Wanet-Defalque et al., 1988). Following the advent of more powerful neuroimaging techniques, a plethora of studies have demonstrated task-dependent activations of the OC during auditory (Kujala et al., 1997; Roder et al., 1999; Weeks et al., 2000), olfactory (Kupers et al., 2011) and tactile (Buchel et al., 1998; Burton et al., 2004; Gizewski et al., 2003) processing in early blind subjects. It is, however, possible that these results simply reflect an association between stimulus presentation and cortical activation, without there being any functional involvement of occipital areas in nonvisual processing. Transcranial magnetic stimulation (TMS), which induces a focal and transient disruption of the proper functioning of a targeted area, has been used to demonstrate the necessity of the OC of the blind for Braille reading (Cohen et al., 1997; Kupers et al., 2007) and verbal (Amedi et al., 2004) processing. We also demonstrated that TMS applied over the right dorsal extrastriate cortex interfered with the use of a prosthesis substituting vision by audition and with the localization of sounds in blind subjects (Collignon et al., 2007). By contrast, TMS targeting the same cortical area had no effect on any auditory performance in sighted subjects and did not interfere with pitch and intensity discriminations in the blind. The demonstration that transient perturbation of OC with TMS selectively
237
disrupted specific auditory processing in the blind compared to sighted subjects illustrates that this “visual” area is functionally linked to the neural network that underlies this auditory ability. We thus concluded that early visual deprivation leads to functional cerebral reorganization such that the right dorsal visual stream is recruited for the spatial processing of sounds, a result which is in clear agreement with previous neuroimaging studies on nonvisual space processing in this population (Arno et al., 2001; Poirier et al., 2006; Ricciardi et al., 2007; Vanlierde et al., 2003; Weeks et al., 2000). In a recent fMRI study we compared brain activity of congenitally blind and sighted participants processing either the spatial or the pitch properties of sounds carrying information in both domains (the same sounds were used in both tasks), using an adaptive procedure specifically designed to adjust for performance level. In addition to showing a substantial recruitment of the occipital cortex for sound processing in the blind, we also demonstrated that auditory-spatial processing mainly recruited regions of the dorsal occipital stream. Moreover, functional connectivity analyses revealed that these reorganized occipital regions are part of an extensive brain network including regions known to underlie audio-visual spatial abilities in sighted subjects (Collignon et al., 2011). It is worth noting that dorsal occipital regions have previously been shown to be involved in visuospatial processing in sighted subjects (Haxby et al., 1991). The similarity in the activation foci between visuospatial processing in the sighted and auditory spatial processing in the blind suggests that these areas may retain their functional and neuronal coding ability, which would enable them to process input from a different sensory modality. These results suggest that spatial processing in the blind maps onto specialized subregions of the OC known to be involved in the spatial processing of visual input in sighted people (Haxby et al., 1991). Interestingly, a recent study reported activation of a subregion of the lateraloccipital complex normally responsive to visual and tactile object-related processing when blind subjects extracted shape information from visualto-auditory sensory substitution soundscapes
(Amedi et al., 2007; see also Pietrini et al., 2004 for ventral activations in tactile shape recognition in the blind). In a similar manner, mental imagery of object shape recruited more ventral occipital areas (De Volder et al., 2001), whereas mental imagery of object position recruited more dorsal occipital regions (Vanlierde et al., 2003) in the blind. It thus appears that a functional dissociation between a ventral “what?” stream for the processing of object shape and a dorsal “where?” stream for the processing of space may also exist for nonvisual stimuli processed in the OC of blind subjects (Collignon et al., 2009c; Dormal and Collignon, 2011). In order to further understand whether occipital activity levels leads to differences in behavioral performance, several studies correlated individual levels of occipital activity in blind participants with performance in nonvisual tasks. In a study conducted in early blind individuals using a speaker array that permitted pseudo-free-field presentations of sounds during PET scanning, Gougoux and collaborators (Gougoux et al., 2005) observed that during monaural sound localization (one ear plugged), the degree of activation of several foci in the striate and extrastriate cortex correlated with sound localization accuracy (Fig. 1). This result not only confirms an enhanced recruitment of occipital regions in auditory spatial processing in blind subjects but also suggests that such restructuring of the auditory circuit may underlie their superior abilities. The above-mentioned studies undoubtedly demonstrate the presence of crossmodal plasticity in blind individuals, as cortical territories normally involved in visual processing are recruited for nonvisual functions. Still, questions remain about the nature of the mechanisms mediating such massive reorganizations. Top-down processing from associative cortices, feed-forward connections between primary sensory regions, or subcortical reorganizations are putative pathways that could explain how nonvisual inputs enter occipital areas of visually deprived subjects (Bavelier and Neville, 2002; Pascual-Leone et al., 2005). In order to further understand such
238
Percent CBF change
20 15
r = –0.81
10 5 0 –5
10
20
30
40
50
60
70
80
–10 –15
Y = –67
–20 20
Percent CBF change
15
r = –0.77
10 5 0 –5
10
20
30
40
50
60
70
80
–10 –15 –20
Y = –64
20
r = –0.68
Percent CBF change
15
Y = –83
10 5 0 –5
10
20
30
40
50
60
70
80
–10 –15 –20
Mean absolute error score (∞)
Fig. 1. Data of a correlational analysis between performance (mean absolute error) in a pointing task to monaurally presented sounds and cerebral blood flow (as measured by PET) in a group of blind subjects. The column of brain images illustrates regions in the ventral extrastriate (top), in the dorsal extrastriate (middle), and striate (bottom) cortices that correlate with monaural sound location performance in early blind subjects. Arrows point to the regions of interest. The scattergram shows the individual values extracted from each of these regions; closed circles indicate blind subjects; open circles indicate sighted controls; regression lines were fitted to data from blind subjects. Y coordinates refer to standardized stereotaxic space. With permission from Gougoux et al. (2005).
mechanisms, we used event-related TMS to disclose the time course of the spatial processing of sounds in the dorsolateral “where” stream of blind and sighted individuals (Collignon et al.,
2008, 2009b). To address this issue, we induced a virtual lesion of either the right intraparietal sulcus (rIPS) or the right dorsal extrastriate occipital cortex (rOC) at different delays in blind and
239
sighted subjects performing a sound lateralization task. We observed that TMS applied over rIPS 100–150 ms after sound onset disrupted the spatial processing of sound in sighted subjects but surprisingly had no influence on the task performance in blind individuals at any timing. In contrast, TMS applied over rOC 50 ms after sound onset disrupted the spatial processing of sounds in blind and in sighted participants. These studies suggest an early contribution of rOC in the spatial processing of sound in blind but also, to some extent, in sighted participants and also point to a lesser involvement of rIPS in this ability in blind participants. Given the very short latency of the disruptive effect of TMS applied over rOC on auditory spatial processing and considering the absence of rIPS contribution to this function in the blind, we suggested that sounds may reach the OC in blind subjects either via subcortical connections (Piche et al., 2007) or direct “feedforward” afferent projections arising from the auditory cortex (Falchier et al., 2002). However, further studies are needed to better understand how these mechanisms combine together and the influence of age of onset of blindness on the installation of such mechanisms.
Deafness The previous section provided evidence as to why the study of blind individuals constitutes an excellent model of the adaptability of the human brain, and how its plastic properties can in turn influence behavior and often improve sensory and cognitive abilities in these individuals. While crossmodal plasticity has been less extensively studied in the deaf, with the advent of small and efficient cochlear implants, it will become more and more important to understand crossmodal plasticity in deafness in order to comprehend the brain's ability to reverse the changes that followed sensory loss. Here, we will briefly review some of the main findings in the literature regarding crossmodal processing and plasticity in the deaf.
Behavioral reorganization in deafness Deaf individuals must rely more heavily on their remaining senses to carry out their everyday activities. The fine input they receive from the outside world is essentially limited to the binocular visual field, whereas precious information obtained from the auditory system can capture precepts from all directions in space covering 360 along any axis. Given this loss of information, do deaf individuals compensate for their deficit via heightened visual abilities? In other words, do they “see better” than hearing individuals? While some of the earlier studies produced very conflicting results, recent findings suggesting improved visual skills in the deaf tend to be more homogenous, in part because the individuals studied were themselves more homogenous as groups than in the past (see Bavelier et al., 2006). In recent studies, these groups were generally composed exclusively of deaf native signers, a subsample of the deaf population known to not suffer from comorbidity confounds related to language and communication deficits often associated with deafness (Meier, 1991). The heightened visual abilities in deaf native signers do not appear to be widespread, however, but rather seem limited to specific areas of visual cognition. For instance, basic sensory thresholds, such as contrast sensitivity (Finney and Dobkins, 2001), motion velocity (Brozinsky and Bavelier, 2004), motion sensitivity (Bosworth and Dobkins, 1999), brightness discrimination (Bross, 1979), and temporal resolution (Nava et al., 2008; Poizner and Tallal, 1987), do not appear to be enhanced in deaf individuals. Enhanced visual skills have rather revealed themselves in more complex tasks, where visual attention and/or processing of the peripheral visual field are manipulated (Bavelier et al., 2001; Dye et al., 2007; Loke and Song, 1991; Neville and Lawson, 1987; Neville et al., 1983; Proksch and Bavelier, 2002; Sladen et al., 2005; Stevens and Neville, 2006). It has thus been proposed that the loss of hearing leads to changes in higher-level attentional processing, with a redistribution of attentional resources to the periphery (see Bavelier
240
et al., 2006). However, this hypothesis has been challenged by the results of a recent study showing faster reactivity to visual events in the deaf compared to hearing individuals, regardless of spatial location (both peripheral and central; Bottari et al., 2010). Moreover, while hearing subjects were substantially slower for peripheral targets (in relation to central ones), deaf subjects were equally efficient across all spatial locations, suggesting functional enhancements for the peripheral visual field that cannot be explained by different attentional gradients alone.
Brain reorganization in deafness When considering the above-highlighted changes in visual processing, it naturally follows to ask whether we can observe an associated neuronal substrate to these improvements. There is now a substantial body of work looking at compensatory changes in the brain following early auditory deprivation; several studies have focused their attention on the middle temporal (MT) and middle superior temporal (MST) areas known to be not only involved in visual motion processing but also known to be heavily modulated by attentional processes. Consistent with the behavioral data, neuroimaging has revealed that differences in MT/MST between deaf and hearing individuals in response to motion stimuli only emerge when they are attended to in the peripheral field (Bavelier et al., 2001; Fine et al., 2005). However, one could argue that given the substantial role of motion in sign language, this difference could be due to the acquisition of this visuospatial language rather than to auditory deprivation per se. Bavelier et al. (2001) addressed this issue by including a second control group, one composed of hearing native signers, and showed that only early deafness and not early exposure to sign language lead to an increase of MT/MST activation. Other notable areas of interest are the auditory cortices that are deprived of their normal input following deafness. Early animal studies showed
that neurons in the primary auditory cortex could reorganize themselves to process visual information in the absence of auditory input (Pallas et al., 1990; Roe et al., 1992). More recently, several groups have shown BOLD changes in the auditory cortex of deaf individuals in response to visual motion (Finney and Dobkins, 2001; Finney et al., 2003; Sadato et al., 2004; Shibata, 2007). We have also recently investigated BOLD signal changes in both deaf and sighted individuals using global motion and forms defined by motion stimuli previously validated in healthy hearing individuals (see Vachon et al., 2009). Our preliminary results with deaf individuals are consistent with the current literature and show the involvement of higher-order auditory areas in the processing of the stimuli, most notably the right supratemporal gyrus (P. Vachon et al., unpublished). Similarly, several other groups have shown recruitment of the auditory cortex by visually presented sign language in deaf subjects (Nishimura et al., 1999; Petitto et al., 2000), and importantly, it was also shown that this crossmodal recruitment is not a by-product of signing, but rather of being auditorily deafferented (Fine et al., 2005). There are several potential ways in which crossmodal reorganization could lead to the observed functional changes in the deaf. First, anatomical support for visual processing in the auditory cortex comes from animal studies showing direct connections between both primary cortices (Falchier et al., 2002; Rockland and Ojima, 2003). However, corresponding pathways have yet to be identified in humans. Other anatomical findings have focused on the auditory cortex and the superior temporal gyrus, where morphometry and diffusion tensor imaging studies have shown a reduction in white matter as well as reduced diffusion anisotropy within remaining white matter in deaf individuals compared to hearing individuals (Emmorey et al., 2003; Kim et al., 2009; Shibata, 2007). While finding no differences within the auditory cortices, Penhune et al. (2003) did reveal an increase in gray matter density within the left motor
241
hand area, possibly related to more active use of the dominant hand in sign language. Finally, an important point worth discussing is the impact of the age of onset of deafness on crossmodal processing and plasticity. While studies with blind individuals have clearly shown the age of acquisition of blindness to modulate the observed plastic changes, only one study, to our knowledge, has specifically attempted to address this important issue in the deaf (Sadato et al., 2004). Both early and late-onset deaf groups showed similar activation of the planum temporale, but differed with respect to the activation in the middle superior temporal sulcus (STS), which was more prominent in the early deaf. Given that the middle STS corresponds to the main voice sensitive area, the authors argued that exposure to voices had hindered the region's ability to ultimately process sign language in the late deaf.
Anosmia, ageusia, loss of trigeminal chemosensation The chemical senses, that is, smell, taste, and the chemosensory trigeminal system, have obtained considerably less attention when compared to vision or audition. As opposed to physical senses, such as vision, audition, and touch, they allow us to experience our chemical environment via the interaction of substances with sensory organs, mostly, but not exclusively (Lindemann, 1996), via ligand–receptor interactions (Alimohammadi and Silver, 2000; Buck and Axel, 1991). Together, the three chemical senses constitute the main components of flavor perception (Small et al., 1997b). In the following paragraph, we will briefly outline the physiology of the chemical senses, in order to better understand the adaptive changes that occur when one of these senses is impaired or lost. Gustation, better known as the sense of taste, allows us to perceive five distinct taste qualities. In addition to the four classical ones (bitterness, sourness, saltiness, and sweetness; Lindemann,
2000), a fifth taste quality, umami, allows for the perception of the savory aspects of protein-rich food (Chaudhari et al., 2000). Taste receptors are located mostly on the tongue, although elsewhere in the oral cavity as well. In contrast to the sense of taste, the sense of smell allows us to perceive a virtually unlimited number of different odors. Volatile substances reach the olfactory receptor neurons, which are located in the upper portions of the nasal cavity, either orthonasally via the nostrils (while sniffing) or retronasally via the nasopharynx (Burdach et al., 1984). The latter is of utmost importance when perceiving the olfactory components of flavors from the oral cavity (Frasnelli et al., 2005). The chemosensory trigeminal system, finally, allows for the perception of burning, cooling, stinging, and other sensations originating from chemical substances (Laska et al., 1997). Here, trigeminal stimuli interact with receptors and free nerve endings of the trigeminal nerve throughout the oral and the nasal cavities. Since the chemical senses are perceptually interconnected so tightly (Small et al., 1997b), some have put forward the idea of a unique flavor sense (Auvray and Spence, 2008). In fact, a major complaint of individuals who lose one of their chemical senses relates to their reduced ability to appreciate foods.
Behavioral reorganization in chemosensory loss Olfactory dysfunctions can be categorized into quantitative dysfunctions (reduced sense of smell—hyposmia; loss of sense of smell—anosmia) and qualitative dysfunctions (altered perception of existing odors—parosmia; perception of inexistent odors—phantosmia; Leopold, 2002). These are relatively common conditions as up to 5% and 15% of the population are thought to exhibit anosmia and hyposmia, respectively (Bramerson et al., 2004; Landis and Hummel, 2006; Landis et al., 2004). Next to the physiological age related decline of olfactory function, the major etiologies of olfactory dysfunction are sinunasal diseases (polyps,
242
chronic rhino-sinusitis), viral infections (persisting dysfunction after upper respiratory tract infection), traumatic brain injury, neurodegenerative diseases (Parkinson's and Alzheimer's disease, etc.), and others. Up to 1% of the anosmic individuals exhibit congenital anosmia (Kallmann's syndrome, isolated congenital anosmia; Temmel et al., 2002). There are several reports on crossmodal effects of olfactory dysfunctions, mainly on other chemosensory systems. There is an established detrimental effect of olfactory dysfunction on trigeminal perception. When compared to controls, individuals with reduced olfactory function can perceive trigeminal stimuli only at higher concentrations (Frasnelli et al., 2010; Gudziol et al., 2001) and perceive suprathreshold stimuli as less intense (Frasnelli et al., 2007a). This reduced trigeminal sensitivity is, however, restricted to chemosensory trigeminal fibers (Frasnelli et al., 2006). A specific method to test trigeminal sensitivity is the odor lateralization task. In this test, subjects have to determine which of their two nostrils had been stimulated by an odorant in a monorhinal stimulation paradigm. We are only able to do so if the odorant also stimulates the trigeminal system (Kobal et al., 1989). Anosmic individuals have been shown to perform worse than healthy controls in the odor localization task (Hummel et al., 2003). With regard to effects of olfactory dysfunction on taste perception, it is important to note that most of the individuals suffering from an olfactory dysfunction complain about a taste disturbance (Deems et al., 1991). This is because they mainly experience the reduced retronasal olfactory sensation during flavor perception (Deems et al., 1991). This phenomenon can be very impressive as some persons with olfactory dysfunction do not believe their olfactory system to be disturbed at all. However, when referring specifically to gustation, that is, the perception of the five taste qualities, effects of olfactory loss on gustation are more debated. Some studies have reported that, in analogy to trigeminal function, gustatory function is also reduced in individuals with olfactory dysfunction (Gudziol et al., 2007; Landis
et al., 2010), while a recent report failed to confirm this finding (Stinton et al., 2010). As opposed to the commonly observed olfactory dysfunctions, a loss of trigeminal chemosensation is a very rare condition. In a case report, olfactory function was assessed in a woman who suffered from unilateral loss of trigeminal function on the left side resulting from a meningeoma. She also exhibited reduced olfactory function, as assessed with a behavioral test and the measurement of olfactory event-related potentials, but only ipsilaterally to the affected side. Her gustatory function was, however, similar on both sides of the tongue (Husner et al., 2006). While patients seeking help with a medical specialist often complain about a qualitatively altered taste perception (dysgeusia), a complete loss of gustatory sensation (ageusia) is a very rare condition (Deems et al., 1991). No reports of crossmodal effects of loss of gustatory function are known. In summary, a dysfunction or loss of one of the chemical senses is a relatively common finding. Olfaction is by far the most affected sensory system. However, no compensatory mechanisms appear to take place, where another (chemical) sense becomes more sensitive. Rather, the loss of a chemical sense (which in most cases is the loss of olfactory function) is usually accompanied by a reduced sensitivity in the other chemical senses. This is in sharp contrast to blindness and deafness, as described above. A possible explanation for this may be the tight connection of the different chemical senses, an expression of which is the perception of flavor. As stated above, some researchers have in fact put forward the idea of a unique “flavor sense,” consisting of inputs of all different contributing sensory channels (Auvray and Spence, 2008). The loss of one sense would therefore lead to a breakdown of the whole flavor system. There is indeed also evidence from imaging studies for such a flavor sense. The chemical senses share important central processing areas. For example, it has been shown that the orbitofrontal cortex (OFC) and its different subdivisions are activated by olfactory (e.g., Gottfried and Zald,
243
2005; Savic and Gulyas, 2000; Zatorre et al., 1992), gustatory (e.g., Hummel et al., 2007; Small et al., 1997a, 2003; Veldhuizen et al., 2007), and trigeminal (e.g., Albrecht et al., 2010; Boyle et al., 2007b) stimulation. Similarly, the insula is activated following olfactory (e.g., Bengtsson et al., 2001; Cerf-Ducastel and Murphy, 2003; Savic and Gulyas, 2000), gustatory (e.g., Small et al., 1999, 2003; Veldhuizen et al., 2007), and trigeminal (e.g., Albrecht et al., 2010; Boyle et al., 2007b; Iannilli et al., 2008) stimulation. More importantly, combined stimuli consisting of mixtures of gustatory, olfactory, and/or trigeminal stimuli have been shown to activate “chemosensory” brain regions to a higher degree than their single constituents. In their seminal paper, Small and collaborators (1997b) showed that the administration of matching gustatory and olfactory stimuli together evoked different changes in cerebral blood flow in the insula, the opercula, and the OFC than the administration of both kinds of stimuli on their own. Similarly, using the trigeminal stimulus CO2 together with the pure olfactory stimulus phenyl ethanol, we showed that a mixture of both activated chemosensory centers (left OFC) and integration areas (left STS, rIPS) to a higher degree than the mathematical sum of the single components (Boyle et al., 2007a). Cerf-Ducastel et al. (2001) finally showed that both gustatory and lingual trigeminal stimuli showed a striking overlap in their activation of the insula as well as the rolandic, frontal, and temporal opercula. Again, these studies support the existence of a cerebral network for flavor consisting mainly of the OFC as well as the insula and surrounding cortex.
Brain reorganization in chemosensory loss Unfortunately, only few reports are available on changes in brain activations due to chemosensory loss. In accordance with the behavioral findings, anosmic and hyposmic individuals exhibit smaller trigeminal event-related potentials (Frasnelli
et al., 2007a; Hummel et al., 1996). Similarly, following trigeminal stimulation with the trigeminal stimulus carbon dioxide, persons suffering from anosmia were described to exhibit smaller activations in “chemosensory” brain regions when compared to controls with a normal sense of smell. The anosmia group, however, exhibited larger responses in other regions in the frontal and temporal lobe, which usually are not involved in chemosensory perception (Iannilli et al., 2007). However, there appears to be a dissociation between peripheral and central levels of trigeminal processing. When the negative mucosal potential (NMP)—a measure of peripheral responsiveness—is assessed, individuals with anosmia or hyposmia exhibit larger responses than healthy controls, which is in striking contrast to the findings in central responses (Frasnelli et al., 2007a,b). Thus, a model of mixed sensory adaptation/compensation in the interaction between the olfactory and the trigeminal system has been put forward. In normal functioning systems, peripheral trigeminal responsiveness is constantly inhibited; consequently, the periphery of the trigeminal system is functionally downregulated. On central levels, trigeminal input is increased by olfactory costimulation resulting in larger signals. In olfactory loss, however, a release of peripheral inhibition occurs, resulting in increased peripheral susceptibility. However, there is no olfactory costimulation to be integrated, resulting in relatively smaller central signals (Frasnelli et al., 2007a,b; Fig. 2). These data therefore suggest the mechanisms in chemosensory loss to be different from other sensory systems. A first difference is that the chemical senses converge, at least partly, to the same processing areas. Second, sensory loss leads to a reduction in sensitivity in the other senses as well, in addition to the loss in the primarily affected sense. More studies are needed to confirm a causal connection between these consistent observations and to deepen our understanding of crossmodal effects of a loss in the chemical senses.
244 (a) -5
(b1)
(b2)
mV
-15
1s
-200
mV
150
1s
Fig. 2. Effects of loss of olfactory function on the trigeminal chemosensory system. (A) Grand means of trigeminal event-related potentials (central measure; top) and negative mucosal potential (NMP; peripheral measure; bottom) following stimuli of 60% (v/v) CO2 in subjects with acquired anosmia (black) and controls (gray). The black horizontal bars indicate the onset and duration of the CO2 stimulus. (B) Model of the interaction between olfactory (gray arrows) and trigeminal (black arrows) systems. (B1) Normal conditions. Peripheral responsiveness is decreased due to constant activation of intrabulbar trigeminal collaterals and consequent functional downregulation in the periphery of the trigeminal system. Functional integration of olfactory and trigeminal processes leads to augmented cortical signal. (B2) Olfactory loss. Increased NMP due to top downregulation; decreased event-related potential due to missing olfactory augmentation. With permission from Frasnelli et al. (2007b).
Conclusion
References
Loss of a sensory system has vast consequences for the affected person and his interactions with environment. Here, we have outlined how sensory loss leads to changes in primarily unaffected sensory systems. This crossmodal plasticity shows in a fascinating way how the brain copes with sensory deprivation. Only the proper understanding of the mechanisms of crossmodal plasticity will allow us to develop tools to help persons with sensory loss to better experience the world with the unaffected senses and thus enable them to live more independently.
Alary, F., Duquette, M., Goldstein, R., Elaine Chapman, C., Voss, P., La Buissonniere-Ariza, V., et al. (2009). Tactile acuity in the blind: A closer look reveals superiority over the sighted in some but not all cutaneous tasks. Neuropsychologia, 47, 2037–2043. Alary, F., Goldstein, R., Duquette, M., Chapman, C. E., Voss, P., & Lepore, F. (2008). Tactile acuity in the blind: A psychophysical study using a two-dimensional angle discrimination task. Experimental Brain Research, 187, 587–594. Albrecht, J., Kopietz, R., Frasnelli, J., Wiesmann, M., Hummel, T., & Lundstrom, J. N. (2010). The neuronal correlates of intranasal trigeminal function—An ALE metaanalysis of human functional brain imaging data. Brain Research Reviews, 62, 183–196.
245 Alimohammadi, H., & Silver, W. L. (2000). Evidence for nicotinic acetylcholine receptors on nasal trigeminal nerve endings of the rat. Chemical Senses, 25, 61–66. Amedi, A., Floel, A., Knecht, S., Zohary, E., & Cohen, L. G. (2004). Transcranial magnetic stimulation of the occipital pole interferes with verbal processing in blind subjects. Nature Neuroscience, 7, 1266–1270. Amedi, A., Raz, N., Pianka, P., Malach, R., & Zohary, E. (2003). Early ‘visual’ cortex activation correlates with superior verbal memory performance in the blind. Nature Neuroscience, 6, 758–766. Amedi, A., Stern, W. M., Camprodon, J. A., Bermpohl, F., Merabet, L., Rotman, S., et al. (2007). Shape conveyed by visual-to-auditory sensory substitution activates the lateral occipital complex. Nature Neuroscience, 10, 687–689. Arno, P., De Volder, A. G., Vanlierde, A., WanetDefalque, M. C., Streel, E., Robert, A., et al. (2001). Occipital activation by pattern recognition in the early blind using auditory substitution for vision. Neuroimage, 13, 632–645. Auvray, M., & Spence, C. (2008). The multisensory perception of flavor. Consciousness and Cognition, 17, 1016–1031. Axelrod, S. (Ed.), (1959). Effect of early blindness: Performance of blind and sighted children on tactile and auditory tasks. (Research Series No. 7) New York: American Foundation for the Blind. Bavelier, D., Brozinsky, C., Tomann, A., Mitchell, T., Neville, H., & Liu, G. (2001). Impact of early deafness and early exposure to sign language on the cerebral organization for motion processing. The Journal of Neuroscience, 21, 8931–8942. Bavelier, D., Dye, M. W. G., & Hauser, P. C. (2006). Do deaf individuals see better? Trends in Cognitive Sciences, 10, 512–518. Bavelier, D., & Neville, H. J. (2002). Cross-modal plasticity: Where and how? Nature Reviews. Neuroscience, 3, 443–452. Bavelier, D., Tomann, A., Hutton, C., Mitchell, T., Corina, D., Liu, G., et al. (2000). Visual attention to the periphery is enhanced in congenitally deaf individuals. The Journal of Neuroscience, 20, RC93. Bengtsson, S., Berglund, H., Gulyas, B., Cohen, E., & Savic, I. (2001). Brain activation during odor perception in males and females. Neuroreport, 12, 2027–2033. Bosworth, R. G., & Dobkins, K. R. (1999). Left-hemisphere dominance for motion processing in deaf signers. Psychological Science, 10, 256–262. Bottari, D., Nava, E., Ley, P., & Pavani, F. (2010). Enhanced reactivity to visual stimuli in deaf individuals. Restorative Neurology and Neuroscience, 28, 167–179. Boyle, J. A., Frasnelli, J., Gerber, J., Heinke, M., & Hummel, T. (2007). Cross-modal integration of intranasal stimuli: A functional magnetic resonance imaging study. Neuroscience, 149, 223–231.
Boyle, J. A., Heinke, M., Gerber, J., Frasnelli, J., & Hummel, T. (2007). Cerebral activation to intranasal chemosensory trigeminal stimulation. Chemical Senses, 32, 343–353. Bramerson, A., Johansson, L., Ek, L., Nordin, S., & Bende, M. (2004). Prevalence of olfactory dysfunction: The skovde population-based study. The Laryngoscope, 114, 733–737. Bross, M. (1979). Residual sensory capacities of the deaf—Signal-detection analysis of a visual-discrimination task. Perceptual and Motor Skills, 48, 187–194. Brozinsky, C. J., & Bavelier, D. (2004). Motion velocity thresholds in deaf signers: Changes in lateralization but not in overall sensitivity. Cognitive Brain Research, 21, 1–10. Buchel, C., Price, C., Frackowiak, R. S. J., & Friston, K. (1998). Different activation patterns in the visual cortex of late and congenitally blind subjects. Brain, 121, 409–419. Buck, L., & Axel, R. (1991). A novel multigene family may encode odorant receptors: A molecular basis for odor recognition. Cell, 65, 175–187. Burdach, K. J., Kroeze, J. H., & Koster, E. P. (1984). Nasal, retronasal, and gustatory perception: An experimental comparison. Perception & Psychophysics, 36, 205–208. Burton, H., Sinclair, R. J., & McLaren, D. G. (2004). Cortical activity to vibrotactile stimulation: An fMRI study in blind and sighted individuals. Human Brain Mapping, 23, 210–228. Cerf-Ducastel, B., & Murphy, C. (2003). FMRI brain activation in response to odors is reduced in primary olfactory areas of elderly subjects. Brain Research, 986, 39–53. Cerf-Ducastel, B., Van de Moortele, P. F., MacLeod, P., Le Bihan, D., & Faurion, A. (2001). Interaction of gustatory and lingual somatosensory perceptions at the cortical level in the human: A functional magnetic resonance imaging study. Chemical Senses, 26, 371–383. Chaudhari, N., Landin, M. A., & Roper, S. D. (2000). A metabotropic glutamate receptor variant functions as a taste receptor. Nature Neuroscience, 3, 113–119. Cohen, L. G., Celnik, P., PascualLeone, A., Corwell, B., Faiz, L., Dambrosia, J., et al. (1997). Functional relevance of crossmodal plasticity in blind humans. Nature, 389, 180–183. Collignon, O., Charbonneau, G., Lassonde, M., & Lepore, F. (2009). Early visual deprivation alters multisensory processing in peripersonal space. Neuropsychologia, 47, 3236–3243. Collignon, O., Davare, M., De Volder, A. G., Poirier, C., Olivier, E., & Veraart, C. (2008). Time-course of posterior parietal and occipital cortex contribution to sound localization. Journal of Cognitive Neuroscience, 20, 1454–1463. Collignon, O., Davare, M., Olivier, E., & De Volder, A. G. (2009). Reorganisation of the right occipito-parietal stream for auditory spatial processing in early blind humans. A transcranial magnetic stimulation study. Brain Topography, 21, 232–240.
246 Collignon, O., & De Volder, A. G. (2009). Further evidence that congenitally blind participants react faster to auditory and tactile spatial targets. Canadian Journal of Experimental Psychology, 63, 287–293. Collignon, O., Lassonde, M., Lepore, F., Bastien, D., & Veraart, C. (2007). Functional cerebral reorganization for auditory spatial processing and auditory substitution of vision in early blind subjects. Cerebral Cortex, 17, 457–465. Collignon, O., Renier, L., Bruyer, R., Tranduy, D., & Veraart, C. (2006). Improved selective and divided spatial attention in early blind subjects. Brain Research, 1075, 175–182. Collignon, O., Vandewalle, G., Voss, P., Albouy, G., Charbonneau, G., Lassonde, M., & Lepore, F. (2011). Functional specialization for auditory-spatial processing in the occipital cortex of congenitally blind humans. Proceedings of the National Academy of Sciences, 108, 4435–4440. Collignon, O., Voss, P., Lassonde, M., & Lepore, F. (2009). Cross-modal plasticity for the spatial processing of sounds in visually deprived subjects. Experimental Brain Research, 192, 343–358. De Volder, A. G., Toyama, H., Kimura, Y., Kiyosawa, M., Nakano, H., Vanlierde, A., et al. (2001). Auditory triggered mental imagery of shape involves visual association areas in early blind humans. Neuroimage, 14, 129–139. Deems, D. A., Doty, R. L., Settle, R. G., Moore-Gillon, V., Shaman, P., Mester, A. F., et al. (1991). Smell and taste disorders, a study of 750 patients from the University of Pennsylvania Smell and Taste Center. Archives of Otolaryngology—Head & Neck Surgery, 117, 519–528. Diderot, D., 1749. Lettre sur les aveugles à l'usage de ceux qui voient (London). Diekmann, H., Walger, M., & von Wedel, H. (1994). Sense of smell in deaf and blind patients. HNO, 42, 264–269. Dormal, G., & Collignon, O. (2011). Functional selectivity in sensory deprived cortices. Journal of Neurophysiology, doi:10.1152/jn.00109.2011. Dye, M. W. G., Baril, D. E., & Bavelier, D. (2007). Which aspects of visual attention are changed by deafness? The case of the Attentional Network Test. Neuropsychologia, 45, 1801–1811. Eimer, M. (2004). Multisensory integration: How visual experience shapes spatial perception. Current Biology, 14, R115–R117. Emmorey, K., Allen, J. S., Bruss, J., Schenker, N., & Damasio, H. (2003). A morphornetric analysis of auditory brain regions in congenitally deaf adults. Proceedings of the National Academy of Sciences of the United States of America, 100, 10049–10054. Falchier, A., Clavagnier, S., Barone, P., & Kennedy, H. (2002). Anatomical evidence of multimodal integration in primate striate cortex. The Journal of Neuroscience, 22, 5749–5759.
Fieger, A., Roder, B., Teder-Salejarvi, W., Hillyard, S. A., & Neville, H. J. (2006). Auditory spatial tuning in late-onset blindness in humans. Journal of Cognitive Neuroscience, 18, 149–157. Fine, I., Finney, E. M., Boynton, G. M., & Dobkins, K. R. (2005). Comparing the effects of auditory deprivation and sign language within the auditory and visual cortex. Journal of Cognitive Neuroscience, 17, 1621–1637. Finney, E. M., Clementz, B. A., Hickok, G., & Dobkins, D. R. (2003). Visual stimuli activate auditory cortex in deaf subjects: Evidence from MEG. Neuroreport, 14, 1425–1427. Finney, E. M., & Dobkins, K. R. (2001). Visual contrast sensitivity in deaf versus hearing populations: Exploring the perceptual consequences of auditory deprivation and experience with a visual language. Cognitive Brain Research, 11, 171–183. Frasnelli, J., Schuster, B., & Hummel, T. (2007a). Interactions between olfaction and the trigeminal system: What can be learned from olfactory loss. Cerebral Cortex, 17, 2268–2275. Frasnelli, J., Schuster, B., & Hummel, T. (2007b). Subjects with congenital anosmia have larger peripheral but similar central trigeminal responses. Cerebral Cortex, 17, 370–377. Frasnelli, J., Schuster, B., & Hummel, T. (2010). Olfactory dysfunction affects thresholds to trigeminal chemosensory sensations. Neuroscience Letters, 468, 259–263. Frasnelli, J., Schuster, B., Zahnert, T., & Hummel, T. (2006). Chemosensory specific reduction of trigeminal sensitivity in subjects with olfactory dysfunction. Neuroscience, 142, 541–546. Frasnelli, J., van Ruth, S., Kriukova, I., & Hummel, T. (2005). Intranasal concentrations of orally administered flavors. Chemical Senses, 30, 575–582. Gizewski, E. R., Gasser, T., de Greiff, A., Boehm, A., & Forsting, M. (2003). Cross-modal plasticity for sensory and motor activation patterns in blind subjects. Neuroimage, 19, 968–975. Goldreich, D., & Kanics, I. M. (2003). Tactile acuity is enhanced in blindness. The Journal of Neuroscience, 23, 3439–3445. Gottfried, J. A., & Zald, D. H. (2005). On the scent of human olfactory orbitofrontal cortex: Meta-analysis and comparison to non-human primates. Brain Research Reviews, 50, 287–304. Gougoux, F., Lepore, F., Lassonde, M., Voss, P., Zatorre, R. J., & Belin, P. (2004). Neuropsychology: Pitch discrimination in the early blind. Nature, 430, 309. Gougoux, F., Zatorre, R. J., Lassonde, M., Voss, P., & Lepore, F. (2005). A functional neuroimaging study of sound localization: Visual cortex activity predicts performance in early-blind individuals. PLoS Biology, 3, 324–333. Gudziol, H., Rahneberg, K., & Burkert, S. (2007). Anosmiker schmecken schlechter als Gesunde. Laryngo-rhino-otologie, 86, 640–643.
247 Gudziol, H., Schubert, M., & Hummel, T. (2001). Decreased trigeminal sensitivity in anosmia. ORL; Journal of Otorhinolaryngology and its Related Specialties, 63, 72–75. Haxby, J. V., Grady, C. L., Horwitz, B., Ungerleider, L. G., Mishkin, M., Carson, R. E., et al. (1991). Dissociation of object and spatial visual processing pathways in human extrastriate cortex. Proceedings of the National Academy of Sciences of the United States of America, 88, 1621–1625. Hotting, K., & Roder, B. (2009). Auditory and auditory-tactile processing in congenitally blind humans. Hearing Research, 258, 165–174. Hummel, T., Barz, S., Lotsch, J., Roscher, S., Kettenmann, B., & Kobal, G. (1996). Loss of olfactory function leads to a decrease of trigeminal sensitivity. Chemical Senses, 21, 75–79. Hummel, C., Frasnelli, J., Gerber, J., & Hummel, T. (2007). Cerebral processing of gustatory stimuli in patients with taste loss. Behavioural Brain Research, 185, 59–64. Hummel, T., Futschik, T., Frasnelli, J., & Huttenbrink, K. B. (2003). Effects of olfactory function, age, and gender on trigeminally mediated sensations: A study based on the lateralization of chemosensory stimuli. Toxicology Letters, 140–141, 273–280. Husner, A., Frasnelli, J., Welge-Lussen, A., Reiss, G., Zahnert, T., & Hummel, T. (2006). Loss of trigeminal sensitivity reduces olfactory function. The Laryngoscope, 116, 1520–1522. Iannilli, E., Del Gratta, C., Gerber, J. C., Romani, G. L., & Hummel, T. (2008). Trigeminal activation using chemical, electrical, and mechanical stimuli. Pain, 139, 376–388. Iannilli, E., Gerber, J., Frasnelli, J., & Hummel, T. (2007). Intranasal trigeminal function in subjects with and without an intact sense of smell. Brain Research, 1139, 235–244. James, W. (1890). Principles of psychology (Vol. 1). New York: Henry Holt and Company. Kim, D. J., Park, S. Y., Kim, J., Lee, D. H., & Park, H. J. (2009). Alterations of white matter diffusion anisotropy in early deafness. Neuroreport, 20, 1032–1036. Kitazawa, S. (2002). Where conscious sensation takes place. Consciousness and Cognition, 11, 475–477. Kobal, G., Van Toller, S., & Hummel, T. (1989). Is there directional smelling? Experientia, 45, 130–132. Kujala, T., Alho, K., Huotilainen, M., Ilmoniemi, R. J., Lehtokoski, A., Leinonen, A., et al. (1997). Electrophysiological evidence for cross-modal plasticity in humans with earlyand late-onset blindness. Psychophysiology, 34, 213–216. Kupers, R., Beaulieu-Lefebvre, M., Schneider, F. C., Kassuba, T., Paulson, O. B., Siebner, H. R., & Ptito, M. (2011). Neural correlates of olfactory processing in congenital blindness. Neuropsychologia, doi:10.1016/j. neuropsychologia.2011.03.033. Kupers, R., Pappens, M., de Noordhout, A. M., Schoenen, J., Ptito, M., & Fumal, A. (2007). rTMS of the occipital cortex
abolishes Braille reading and repetition priming in blind subjects. Neurology, 68, 691–693. Landis, B. N., & Hummel, T. (2006). New evidence for high occurrence of olfactory dysfunctions within the population. The American Journal of Medicine, 119, 91–92. Landis, B. N., Konnerth, C. G., & Hummel, T. (2004). A study on the frequency of olfactory dysfunction. The Laryngoscope, 114, 1764–1769. Landis, B. N., Scheibe, M., Weber, C., Berger, R., Bramerson, A., Bende, M., et al. (2010). Chemosensory interaction: Acquired olfactory impairment is associated with decreased taste function. Journal of Neurology, 257, 1303–1308. Laska, M., Distel, H., & Hudson, R. (1997). Trigeminal perception of odorant quality in congenitally anosmic subjects. Chemical Senses, 22, 447–456. Leopold, D. (2002). Distortion of olfactory perception: Diagnosis and treatment. Chemical Senses, 27, 611–615. Lessard, N., Pare, M., Lepore, F., & Lassonde, W. (1998). Early-blind human subjects localize sound sources better than sighted subjects. Nature, 395, 278–280. Lewald, J. (2002). Vertical sound localization in blind humans. Neuropsychologia, 40, 1868–1872. Lindemann, B. (1996). Taste reception. Physiological Reviews, 76, 719–766. Lindemann, B. (2000). A taste for umami. Nature Neuroscience, 3, 99–100. Loke, W. H., & Song, S. R. (1991). Central and peripheral visual processing in hearing and nonhearing individuals. Bulletin of the Psychonomic Society, 29, 437–440. Meier, R. P. (1991). Language-acquisition by deaf-children. American Scientist, 79, 60–70. Murphy, C., & Cain, W. S. (1986). Odor identification: The blind are better. Physiology & Behavior, 37, 177–180. Nava, E., Bottari, D., Zampini, M., & Pavani, F. (2008). Visual temporal order judgment in profoundly deaf individuals. Experimental Brain Research, 190, 179–188. Neville, H. J., & Lawson, D. (1987). Attention to central and peripheral visual space in a movement detection task: An event-related potential and behavioral study. II. Congenitally deaf adults. Brain Research, 405, 268–283. Neville, H. J., Schmidt, A., & Kutas, M. (1983). Altered visual evoked-potentials in congenitally deaf adults. Brain Research, 266, 127–132. Nishimura, H., Hashikawa, K., Doi, K., Iwaki, T., Watanabe, Y., Kusuoka, H., et al. (1999). Sign language ‘heard’ in the auditory cortex. Nature, 397, 116. Pallas, S. L., Roe, A. W., & Sur, M. (1990). Visual projections induced into the auditory pathway of ferrets. 1. Novel inputs to primary auditory-cortex (Ai) from the Lp pulvinar complex and the topography of the MGN-AI projection. The Journal of Comparative Neurology, 298, 50–68.
248 Pascual-Leone, A., Amedi, A., Fregni, F., & Merabet, L. B. (2005). The plastic human brain cortex. Annual Review of Neuroscience, 28, 377–401. Penhune, V. B., Cismaru, R., Dorsaint-Pierre, R., Petitto, L. A., & Zatorre, R. J. (2003). The morphometry of auditory cortex in the congenitally deaf measured using MRI. Neuroimage, 20, 1215–1225. Petitto, L. A., Zatorre, R. J., Gauna, K., Nikelski, E. J., Dostie, D., & Evans, A. C. (2000). Speech-like cerebral activity in profoundly deaf people processing signed languages: Implications for the neural basis of human language. Proceedings of the National Academy of Sciences of the United States of America, 97, 13961–13966. Piche, M., Chabot, N., Bronchti, G., Miceli, D., Lepore, F., & Guillemot, J. P. (2007). Auditory responses in the visual cortex of neonatally enucleated rats. Neuroscience, 145, 1144–1156. Pietrini, P., Furey, M. L., Ricciardi, E., Gobbini, M. I., Wu, W. H. C., Cohen, L., et al. (2004). Beyond sensory images: Object-based representation in the human ventral pathway. Proceedings of the National Academy of Sciences of the United States of America, 101, 5658–5663. Poirier, C., Collignon, O., Scheiber, C., Renier, L., Vanlierde, A., Tranduy, D., et al. (2006). Auditory motion perception activates visual motion areas in early blind subjects. Neuroimage, 31, 279–285. Poizner, H., & Tallal, P. (1987). Temporal processing in deaf signers. Brain and Language, 30, 52–62. Proksch, J., & Bavelier, D. (2002). Changes in the spatial distribution of visual attention after early deafness. Journal of Cognitive Neuroscience, 14, 687–701. Ricciardi, E., Vanello, N., Sani, L., Gentili, C., Scilingo, E. P., Landini, L., et al. (2007). The effect of visual experience on the development of functional architecture in hMT. Cerebral Cortex, 17, 2933–2939. Rice, C. E., & Feinstein, S. H. (1965). Sonar system of the blind: Size discrimination. Science, 148, 1107–1108. Rice, C. E., Feinstein, S. H., & Schusterman, R. J. (1965). Echo-detection ability of the blind: Size and distance factors. Journal of Experimental Psychology, 70, 246–255. Rockland, K. S., & Ojima, H. (2003). Multisensory convergence in calcarine visual areas in macaque monkey. International Journal of Psychophysiology, 50, 19–26. Roder, B., Focker, J., Hotting, K., & Spence, C. (2008). Spatial coordinate systems for tactile spatial attention depend on developmental vision: Evidence from event-related potentials in sighted and congenitally blind adult humans. The European Journal of Neuroscience, 28, 475–483. Roder, B., Rosler, F., & Spence, C. (2004). Early vision impairs tactile perception in the blind. Current Biology, 14, 121–124. Roder, B., Teder-Salejarvi, W., Sterr, A., Rosler, F., Hillyard, S. A., & Neville, H. J. (1999). Improved auditory spatial tuning in blind humans. Nature, 400, 162–166.
Roe, A. W., Pallas, S. L., Kwon, Y. H., & Sur, M. (1992). Visual projections routed to the auditory pathway in ferrets: Receptive fields of visual neurons in primary auditory cortex. The Journal of Neuroscience, 12, 3651–3664. Rosenbluth, R., Grossman, E. S., & Kaitz, M. (2000). Performance of early-blind and sighted children on olfactory tasks. Perception, 29, 101–110. Sadato, N., Yamada, H., Okada, T., Yoshida, M., Hasegawa, T., Matsuki, K., et al. (2004). Age-dependent plasticity in the superior temporal sulcus in deaf humans: A functional MRI study. BMC Neuroscience, 5, 56. Savic, I., & Gulyas, B. (2000). PET shows that odors are processed both ipsilaterally and contralaterally to the stimulated nostril. Neuroreport, 11, 2861–2866. Schwenn, O., Hundorf, I., Moll, B., Pitz, S., & Mann, W. J. (2002). Do blind persons have a better sense of smell than normal sighted people? Klinische Monatsblätter für Augenheilkunde, 219, 649–654. Shibata, D. K. (2007). Differences in brain structure in deaf persons on MR imaging studied with voxel-based morphometry. American Journal of Neuroradiology, 28, 243–249. Simon, H. J., Divenyi, P. L., & Lotze, A. (2002). Lateralization of narrow-band noise by blind and sighted listeners. Perception, 31, 855–873. Sladen, D. P., Tharpe, A. M., Daniel, A., & Grantham, D. W. (2005). Visual attention in deaf and normal hearing adults: Effects of stimulus compatibility. Journal of Speech, Language, and Hearing Research, 48, 1529–1537. Small, D. M., Gregory, M. D., Mak, Y. E., Gitelman, D., Mesulam, M. M., & Parrish, T. (2003). Dissociation of neural representation of intensity and affective valuation in human gustation. Neuron, 39, 701–711. Small, D. M., Jones-Gotman, M., Zatorre, R. J., Petrides, M., & Evans, A. C. (1997a). A role for the right anterior temporal lobe in taste quality recognition. The Journal of Neuroscience, 17, 5136–5142. Small, D. M., Jones-Gotman, M., Zatorre, R. J., Petrides, M., & Evans, A. C. (1997b). Flavor processing: More than the sum of its parts. Neuroreport, 8, 3913–3917. Small, D. M., Zald, D. H., Jones-Gotman, M., Zatorre, R. J., Pardo, J. V., Frey, S., et al. (1999). Human cortical gustatory areas: A review of functional neuroimaging data. Neuroreport, 10, 7–14. Smith, R. S., Doty, R. L., Burlingame, G. K., & McKeown, D. A. (1993). Smell and taste function in the visually impaired. Perception & Psychophysics, 54, 649–655. Stevens, C., & Neville, H. (2006). Neuroplasticity as a doubleedged sword: Deaf enhancements and dyslexic deficits in motion processing. Journal of Cognitive Neuroscience, 18, 701–714. Stinton, N., Atif, M. A., Barkat, N., & Doty, R. L. (2010). Influence of smell loss on taste function. Behavioral Neuroscience, 124, 256–264.
249 Temmel, A. F., Quint, C., Schickinger-Fischer, B., Klimek, L., Stoller, E., & Hummel, T. (2002). Characteristics of olfactory disorders in relation to major causes of olfactory loss. Archives of Otolaryngology—Head & Neck Surgery, 128, 635–641. Vachon, P., Voss, P., Lassonde, M., Leroux, J. M., Mensour, B., Beaudoin, G., et al. (2009). Global motion stimuli and form-from-motion stimuli: Common characteristics and differential activation patterns. The International Journal of Neuroscience, 119, 1584–1601. Vanlierde, A., De Volder, A. G., Wanet-Defalque, M. C., & Veraart, C. (2003). Occipito-parietal cortex activation during visuo-spatial imagery in early blind humans. Neuroimage, 19, 698–709. Veldhuizen, M. G., Bender, G., Constable, R. T., & Small, D. M. (2007). Trying to detect taste in a tasteless solution: Modulation of early gustatory cortex by attention to taste. Chemical Senses, 32, 569–581. Veraart, C., Devolder, A. G., Wanetdefalque, M. C., Bol, A., Michel, C., & Goffinet, A. M. (1990). Glucose-utilization in human visual-cortex is abnormally elevated in blindness of early onset but decreased in blindness of late onset. Brain Research, 510, 115–121. Voss, P., Lassonde, M., Gougoux, F., Fortin, M., Guillemot, J. P., & Lepore, F. (2004). Early- and late-onset blind individuals show supra-normal auditory abilities in far-space. Current Biology, 14, 1734–1738.
Wakefield, C. E., Homewood, J., & Taylor, A. J. (2004). Cognitive compensations for blindness in children: An investigation using odour naming. Perception, 33, 429–442. Wan, C. Y., Wood, A. G., Reutens, D. C., & Wilson, S. J. (2010). Early but not late-blindness leads to enhanced auditory perception. Neuropsychologia, 48, 344–348. Wanet-Defalque, M. C., Veraart, C., De Volder, A., Metz, R., Michel, C., Dooms, G., et al. (1988). High metabolic activity in the visual cortex of early blind human subjects. Brain Research, 446, 369–373. Weeks, R., Horwitz, B., Aziz-Sultan, A., Tian, B., Wessinger, C. M., Cohen, L. G., et al. (2000). A positron emission tomographic study of auditory localization in the congenitally blind. The Journal of Neuroscience, 20, 2664–2672. Wong, M., Gnanakumaran, V., & Goldreich, D. (2011). Tactile Spatial Acuity Enhancement in Blindness: Evidence for Experience-Dependent MechanismsTactile Spatial Acuity Enhancement in Blindness: Evidence for ExperienceDependent Mechanisms. The Journal of Neuroscience, 31, 7028–7037. Zatorre, R. J., Jones-Gotman, M., Evans, A. C., & Meyer, E. (1992). Functional localization and lateralization of human olfactory cortex. Nature, 360, 339–340. Zwiers, M. P., Van Opstal, A. J., & Cruysberg, J. R. (2001). A spatial hearing deficit in early-blind humans. The Journal of Neuroscience, 21, RC142, 1–5.
A. M. Green, C. E. Chapman, J. F. Kalaska and F. Lepore (Eds.) Progress in Brain Research, Vol. 191 ISSN: 0079-6123 Copyright Ó 2011 Elsevier B.V. All rights reserved.
CHAPTER 16
Adaptive crossmodal plasticity in deaf auditory cortex: areal and laminar contributions to supranormal vision in the deaf Stephen G. Lomber{,{,*, M. Alex Meredith} and Andrej Kral} {
Department of Physiology and Pharmacology, Centre for Brain and Mind, The University of Western Ontario, London, Ontario, Canada { Department of Psychology, Centre for Brain and Mind, The University of Western Ontario, London, Ontario, Canada } Department of Anatomy and Neurobiology, School of Medicine, Virginia Commonwealth University, Richmond, Virginia, USA } Department of Experimental Otology, Institute of Audioneurotechnology, Medical University Hannover, Hannover, Germany
Abstract: This chapter is a summary of three interdigitated investigations to identify the neural substrate underlying supranormal vision in the congenitally deaf. In the first study, we tested both congenitally deaf and hearing cats on a battery of visual psychophysical tasks to identify those visual functions that are enhanced in the congenitally deaf. From this investigation, we found that congenitally deaf, compared to hearing, cats have superior visual localization in the peripheral field and lower visual movement detection thresholds. In the second study, we examined the role of “deaf” auditory cortex in mediating the supranormal visual abilities by reversibly deactivating specific cortical loci with cooling. We identified that in deaf cats, reversible deactivation of a region of cortex typically identified as the posterior auditory field (PAF) in hearing cats selectively eliminated superior visual localization abilities. It was also found that deactivation of the dorsal zone (DZ) of “auditory” cortex eliminated the superior visual motion detection abilities of deaf cats. In the third study, graded cooling was applied to deaf PAF and deaf DZ to examine the laminar contributions to the superior visual abilities of the deaf. Graded cooling of deaf PAF revealed that deactivation of the superficial layers alone does not cause significant visual localization deficits. Profound deficits were identified only when cooling extended through all six layers of deaf PAF. In contrast, graded cooling of deaf DZ showed that deactivation of only the superficial layers was required to elicit increased visual motion detection *Corresponding author. Tel.: þ1-519-663-5777x24110; Fax: þ1-519-663-3193 E-mail:
[email protected] DOI: 10.1016/B978-0-444-53752-2.00001-1
251
252
thresholds. Collectively, these three studies show that the superficial layers of deaf DZ mediate the enhanced visual motion detection of the deaf, while the full thickness of deaf PAF must be deactivated in order to eliminate the superior visual localization abilities of the congenitally deaf. Taken together, this combination of experimental approaches has demonstrated a causal link between the crossmodal reorganization of auditory cortex and enhanced visual abilities of the deaf, as well as identified the cortical regions responsible for adaptive supranormal vision. Keywords: reversible deactivation; posterior auditory field; dorsal zone; congenital deafness; cortical plasticity.
Introduction A remarkable feature of the brain is its ability to respond to change. Among other functions, this neuroplastic process endows a complex nervous system with the facility to adapt itself to its environment but, at the same time, also makes it susceptible to impoverished sensory or developmental experiences. For example, the expansion of somatosensory maps following limb amputation often results in spurious perceptual events known as “phantom limb pain” (e.g., Ramachandran and Hirstein, 1998) or untreated amblyopia results in the profound loss of visual acuity (reviewed by Webber and Wood, 2005). Neither of these neuroplastic effects have adaptive significance. However, there is a clear adaptive benefit when the inputs from another, intact modality substitute for those that have been lost (Collignon et al., 2009; Merabet and PascualLeone, 2010). Adaptive crossmodal plasticity can not only provide a form of partial compensation by one modality for another (e.g., auditory spatial localization in the blind) but also enhance perceptual performance within the remaining sensory modalities (but see Brozinsky and Bavelier, 2004; Finney and Dobkins, 2001). Numerous reports document improvement over intact subjects in auditory and somatosensory tasks in blind individuals (D'Anguilli and Waraich, 2002; Grant et al., 2000; Lewald, 2007; Sathian, 2000, 2005; Weeks et al., 2000), as well as enhanced performance in visual and tactile behaviors in
the deaf (Bavelier et al., 2000; Levanen and Hamdof, 2001). Although research has endeavored to identify the brain structures responsible for the behavioral enhancements resulting from adaptive crossmodal plasticity, it has been noted by many of these same studies that the specific neurological substrate for the effect is largely unknown (Doucet et al., 2006; Lambertz et al., 2005; Lee et al., 2003). Furthermore, the scant but growing literature on this topic seems to be fractionated into sides: one which asserts that crossmodal plasticity results in the wholesale reorganization of all of the affected regions, while the other indicates that crossmodal plasticity occurs only at selective regions therein (see review of Bavelier and Neville, 2002). Given that compensatory crossmodal plasticity appears not to affect brainstem structures (Langers et al., 2005, but see Shore et al., 2009), the suggestion that this phenomenon requires the cerebral cortex is supported by numerous studies (Rauschecker, 1995, 2002). Many of these investigations indicate that entire cortical representations vacated by the damaged sensory modality are completely replaced by inputs from the remaining systems (Bavelier and Neville, 2002). For example, imaging studies of crossmodal plasticity in early-deaf individuals have reported visual activation of auditory cortex partially including its core, or primary levels (Finney et al., 2001; Lambertz et al., 2005), and Braille reading or tactile tasks activated visual cortices in blind subjects
253
(Levanen and Hamdof, 2001; Sathian, 2000, 2005). Accordingly, these observations logically led to the general assumption that all cortical areas possess the ability for crossmodal plasticity. Indeed, the potential for such wholesale reorganization is supported by results from studies using a series of neonatal lesions in experimental animals (Roe et al., 1990; Sur et al., 1990). However, support for such global effects is not universal, and several studies (Nishimura et al., 1999; Weeks et al., 2000) specifically noted that primary auditory cortex was not crossmodally reorganized in their early-deaf subjects. Also, these observations favoring selective reorganization have been corroborated more directly by electrophysiological recordings from primary auditory cortices of congenitally deaf cats, which found no evidence of crossmodal plasticity (Kral et al., 2003). Therefore, while a clear and increasing effort has been directed toward investigating the neural bases for adaptive crossmodal plasticity, knowledge of the underlying brain circuitry remains virtually unexplored. A modest number of studies have been directed toward revealing behavioral/perceptual effects of crossmodal plasticity. The most notable of these efforts is the work of Rauschecker and colleagues, who used visual deprivation to examine the effect of crossmodal compensatory plasticity in cortex. These now classic studies revealed that, in cats visually deprived from birth, the extent of the auditory field of the anterior ectosylvian sulcus (FAES) was greatly expanded (Rauschecker and Korte, 1993), its constituent neurons were more sharply spatially tuned (Korte and Rauschecker, 1993), and the behavioral localization of auditory stimuli was enhanced (Rauschecker and Kniepert, 1994). However, this ground-breaking work has not been furthered since the original series of reports and few, if any, other investigators have incorporated this model of crossmodal plasticity in their studies. In contrast, several labs have produced a highly engineered model of crossmodal plasticity through a strategic series of neonatal lesions in
hamsters (Metin and Frost, 1989) and in ferrets (Pallas et al., 1999; Roe et al., 1990; Sur et al., 1990). However, such a model is as contrived as it is ingenious and, as such, it bears little semblance to naturally occurring neurological phenomena, such as blindness or deafness. Most profound examples of crossmodal plasticity result from loss of function in the peripheral sensory receptors or nerves, whereas central lesions that result in sensory loss generally are not available for reorganization because much of the affected area is essentially dead. However, a major effort has been directed toward understanding other forms of crossmodal effects, including plasticity (but not adaptive plasticity) involved in the visual calibration of auditory brainstem responses in barn owls (Gutfreund et al., 2002; Knudsen and Knudsen, 1989) and ferrets (King, 2002; King and Parsons, 1999). However, outside of these important efforts, the knowledge of cortical crossmodal reorganization is meager and a robust, repeatable, and more naturally occurring model of adaptive crossmodal plasticity has yet to be developed.
Congenitally deaf cat: a model for adaptive crossmodal plasticity Like the visual system, auditory development passes through a sensitive period in which circuits and connections are established and then refined by experience (Knudsen, 2004; Kral et al., 2000). During this period, the functional maturation of auditory processing and perception is critically dependent on adequate auditory experience. Cats appear to progress through a critical phase at 2–3 months old, and complete their auditory maturation by 6 months (Kral et al., 2005). A similar, but more prolonged sensitive period seems to apply to humans (up to 13 years; Doucet et al., 2006), as evidenced by congenitally deaf subjects who receive cochlear implants in early childhood and develop complete language competence. In contrast, those who do not receive such treatment
254
until later in life generally do not develop sophisticated language skills. The specific defects in the auditory system that underlie such persistent deficits remain to be identified. Some practitioners using imaging or EEG techniques have asserted that such deficits are the result of crossmodal plasticity that subsumes the nonfunctional parts of the auditory system into other sensory modes (Doucet et al., 2006; Finney et al., 2001; Lee et al., 2001). In contrast, studies done in congenitally deaf animals using single cell recording techniques have failed to show any crossmodal activation of primary auditory cortex (Kral et al., 2003) and that auditory nerve stimulation maintained access to primary auditory cortex even in congenitally deaf adults (Kral et al., 2002, 2005). Field A1 is functionally well characterized in congenitally deaf cats, with extensive deficits in spatiotemporal activity profiles as well as feature representation (Kral et al., 2009, Tillein et al., 2010) and corticocortical connectivity (reviewed in Kral and Eggermont, 2007). Chronic electrostimulation with a cochlear implant is known to show a sensitive period in cortical plasticity (reviewed in Kral et al., 2006). Thus, this model has been successful in demonstrating neurophysiological substrates of functional deficits after cochlear implantation. Ironically, despite the intense scrutiny that AI has received in these studies, with perhaps the exception of Sadato et al. (1996) in the visual cortex, virtually none of the crossmodally reorganized non-primary areas have been specifically identified. Although non-primary areas are “expected” to be reorganized, it is unclear whether these are similarly affected (and to the same degree). Therefore, the crucial debate in this regard is not only if deafness induces crossmodal plasticity, but where such plasticity occurs. To that end, we initiated a series of experiments to examine adaptive crossmodal plasticity in the congenitally deaf cat. The cat is an appealing model system to use for these types of investigations on cerebral networks in auditory
cortex. It is a simplified and tractable version of the more complex networks present in monkeys and humans. Cats are ideal because (1) they can quickly be trained to perform complex auditory tasks; (2) unlike the monkey, the majority of the auditory areas are easily approachable because they are exposed on the surfaces of gyri, rather than being buried in the depths of a sulcus; (3) each area is small enough so that it may be cooled by a single cryoloop (Lomber et al., 1999); and (4) they develop to maturity relatively quickly (over the course of months rather than years). Adult congenitally deaf cats show a Scheibe type of dysplasia in the organ of Corti with no hair cells present, although the spiral ganglion and cochlear bony structure are preserved (Heid et al., 1998). Preservation of the spiral ganglion cells is a major advantage when compared to pharmacologically deafened animals. The central auditory system of the congenitally deaf cat nonetheless shows expected deprivation-induced changes (Heid et al., 1998; Kral et al., 2006) although the central visual system appears normal in structure and function (Guillery et al., 1981; Levick et al., 1980). In the present study, deafness was confirmed by a standard screening method using auditory brainstem responses. In the first study, mature congenitally deaf cats and age-matched hearing cats were trained on a battery of seven visual psychophysical tests to identify those visual functions that are enhanced in the congenitally deaf. In the second study, we examined the role of “deaf” auditory cortex in mediating the superior visual abilities by reversibly deactivating specific cortical loci with cooling. This investigation revealed whether individual areas or collections of areas in deaf auditory cortex were the neural substrates for the superior visual functions. In the third study, graded cooling was applied to the areas identified in the second study to examine the laminar contributions to the superior visual abilities of the deaf. Overall, this combination of experimental approaches has demonstrated a causal link between the crossmodal reorganization of auditory cortex and enhanced
255
visual abilities of the deaf as well as identified the cortical regions responsible for supranormal vision.
(a)
Study 1: supranormal visual abilities of congenitally deaf cats In the first study, the performance of adult hearing (n ¼ 3) and congenitally deaf cats (n ¼ 3) was compared on a battery of seven visual psychophysical tasks. For specific details on the tasks, see Lomber et al. (2010). The cats’ ability to detect and localize flashed visual stimuli was assessed in a visual orienting arena (Fig. 1a) as we have done previously (Lomber and Payne, 2001; Malhotra et al., 2004). The six other tasks were conducted in a two-alternative forced-choice apparatus (Fig. 1b). To determine psychophysical thresholds, a standard staircase procedure was used, with three consecutive correct responses resulting in a decrease in the difference between the two stimuli, while each incorrect response resulted in an increase in the difference between the two comparison stimuli. Statistical significance was assessed using an analysis of variance and follow-up t-tests (p < 0.01). In the first task, we tested visual localization by placing the animals in an arena and examining their ability to accurately localize, by orienting and approaching, the illumination of red lightemitting diodes (LEDs) that were placed at 15 intervals across 180 of azimuth (Fig. 1a). In hearing controls, performance was excellent throughout the central 90 of the visual field (45 to the left and right), but accurate localization declined across the most peripheral targets tested (60–90 ; Fig. 2a). In contrast, visual localization performance of deaf cats was maintained at higher levels throughout the most peripheral visual field (Fig. 2a). Performance of the deaf cats was significantly better for the 60 , 75 , and 90 positions (p < 0.01), while there was no significant difference across the central 90 of the visual field (Fig. 2b). This result was consistent for both
speaker LED food reward
0°
A
30°
60°
90°
B
(b)
Fig. 1. (a) Orienting arena used for visual localization task. A loudspeaker (top circle) and a light-emitting diode (LED, black dot) were located above a food reward locus (lower circle) at each of 13 regularly spaced (15 ) intervals (for sake of clarity, only 30 intervals are labeled). (A) The animal was first required to fixate on the central (0 ) LED. (B) It then had to orient to, and approach, a secondary acoustic (100 ms broad-band noise) or visual (illumination of an LED) stimulus to receive a food reward. Adapted from Lomber et al. (2007). (b) Two-alternative forced-choice (2AFC) apparatus used for visual discrimination training and testing. The testing apparatus was a 52 29 41 cm Plexiglas box with a 14 cm diameter opening at one end. This opening lead to a smaller Plexiglas enclosure into which the animal placed its head. This chamber contained two hinged transparent response keys which the cat could depress with its nose to register a response. The stimuli could be viewed through the response keys. The monitors were located 28 cm from the cat's eyes (thus 1 cm on the screen was 2 visual degrees). Beneath the response keys was the food reward terminal that dispensed a puree of beef liver and ground pork when the animal made a correct response.
(a)
Visual localization
(b)
Visual localization Hearing 100
0° 30°
Deaf 30°
60°
60°
Percent correct
80
* 60
*
*
40 20
90° 100%
50%
0%
90° 100%
50%
Percent correct
0
0°
15°
30°
45°
60°
75°
90°
Eccentricity
Movement detection
(d)
1
*
0
(f)
(e)
10
Hearing
Threshold (cycles deg−1)
Threshold (deg s−1)
2
Grating acuity
Orientation discrimination
8 6 4 2 0
Deaf
(g)
Vernier acuity
2
Threshold (min arc)
(c)
Hearing
0
Deaf
Direction of motion
1
(h)
Hearing
Deaf
Velocity discrimination
5
4
4
3 2
Weber fraction (DV/V)
5
Threshold (degs)
Threshold (degs)
2
3 2
1
1
0
0
1 Hearing
0 Hearing
Deaf
Hearing
Deaf
Deaf 248
16
32
64
Stimulus velocity (deg s−1)
Fig. 2. Performance of hearing and deaf cats on the battery of seven visual psychophysical tasks. (a) Polar plot of the visual localization responses of hearing cats (light gray bars) and the superior performance of deaf cats (dark gray bars). The two concentric semicircles represent 50% and 100% correct response levels and the length of each colored line corresponds to the percentage of correct responses at each location tested. For both the hearing and deaf cats, data represent mean performance for 200 stimulus presentations at each peripheral target location and 400 stimulus presentations for the central target. (b) Histograms of combined data from left and right hemifields showing mean s.e. performance for the hearing (light gray) and deaf (dark gray) cats at each of the tested positions in the visual localization task. For both hearing and deaf cats, data represent mean performance for 400 stimulus presentations at each peripheral target location and 800 stimulus presentations for the central target (0 ). (c–g) Mean threshold s.e. for the hearing and deaf cats on the movement detection (c), grating acuity (d), Vernier acuity (e), orientation (f), and direction of motion (g), discriminations. (h) Performance of the hearing and deaf cats on the velocity discrimination task. Data are presented as Weber fractions for six different stimulus velocities. Asterisks indicate significant differences (p < 0.01) between the hearing and deaf conditions. Sample stimuli are shown for each task. Figure adapted from Lomber et al. (2010).
257
binocular and monocular testing. Overall, the superior visual localization abilities of deaf cats correspond well with findings from prelingually deaf human subjects (Bavelier et al., 2006). Six additional visual tests were all conducted in a two-alternative forced-choice apparatus using standard staircase procedures to determine psychophysical thresholds (Fig. 1b). In hearing cats, movement detection thresholds agreed with earlier reports (Pasternak and Merigan, 1980) and were identified to be 1.3 0.4 s 1 (Fig. 2c). In contrast, movement detection thresholds for the deaf cats were significantly lower (0.5 0.2 s 1; Fig. 2c). For the remaining five tests of visual function (grating acuity, Vernier acuity, orientation discrimination, direction of motion discrimination, and velocity discrimination), performance of the deaf cats was not significantly different from hearing controls (Fig. 2d–h). Overall, in the first study, we found that congenitally deaf, compared to hearing, cats have supranormal visual abilities, specifically, superior visual localization in the peripheral field and lower visual movement detection thresholds. Study 2: contributions of “deaf” auditory cortex to supranormal visual localization and detection In the second study, portions of auditory cortex (Fig. 3a) were collectively and individually deactivated to determine if specific cortical areas mediated the enhanced visual functions. In both the deaf and hearing cats, individual cooling loops (Lomber et al., 1999) were bilaterally placed over the posterior auditory field (PAF), the dorsal zone of auditory cortex (area DZ), and primary auditory field (A1) because of their involvement in auditory localization in hearing cats (Malhotra and Lomber, 2007; Malhotra et al., 2008; Fig. 3b). An additional control cooling loop was placed over the anterior auditory field (AAF) because of its involvement in pattern, but not spatial, processing (Lomber and Malhotra, 2008).
(a) DZ
ss
dPE AAF
A1 PAF
FAES
iPE
A2
aes
IN T
vPE
D
pes
VAF
VPAF
A
P V
(b) ss DZ
A1 AAF PAF 3 mm
aes
pes
Fig. 3. Cortical areas examined in deaf auditory cortex. (a) Illustration of the left hemisphere of the cat cerebrum (adapted from Reinoso-Suárez, 1961) showing all auditory areas (lateral view) compiled from Reale and Imig (1980), de Ribaupierre (1997), and Tian and Rauschecker (1998). For abbreviations, see List. Areas examined are highlighted in gray. The areal borders shown in this figure are based on a compilation of electrophysiological mapping and cytoarchitectonic studies. (b) Cooling loops in contact with areas AAF, DZ, A1, and PAF of the left hemisphere of a congenitally deaf cat at the time of implantation. Left is anterior. The areal borders presented in this figure are based on the postmortem analysis of SMI-32 processed tissue from the brain shown in this photo. For abbreviations, see List. Figure adapted from Lomber et al. (2010).
Reversible cooling deactivation The cooling method to reversibly deactivate neural tissue is an exciting, potent, and appropriate technique for examining cerebral contributions to behavior and has a number of highly beneficial
258
and practical features (Lomber, 1999). (1) Limited regions of the cerebral cortex can be selectively and reversibly deactivated in a controlled and reproducible way. Baseline and experimental measures can be made within minutes of each other (Lomber et al., 1996). (2) Repeated coolings over months or years produce stable, reversible deficits, with little evidence of attenuation or neural compensations (Lomber et al., 1994, 1999). (3) Repeated cooling induces neither local nor distant degenerations that might compromise conclusions (Yang et al., 2006). (4) Compared to traditional ablation studies, fewer animals are needed because within-animal-comparisons and double dissociations are possible, permitting large volumes of high-quality data to be acquired from each animal (Lomber and Malhotra, 2008; Lomber et al., 1996). (5) Finally, as the major effect of cooling is to block synaptic transmission, activity in fibers of passage is not compromised (Bénita and Condé, 1972; Jasper et al., 1970). Overall, the technique induces localized hypothermia in a restricted region of the brain. The locus of the deactivation is kept small by the constant perfusion of warm blood into, and around, the cooled region. The cooling disrupts calcium channel function in the presynaptic terminal and disrupts normal neurotransmitter release (reviewed by Brooks, 1983). We have verified that the surgical procedure to implant cryoloops, their presence in contact with the cerebrum, and their operation disrupts neither the normal structural nor functional integrity of cortex (Lomber et al., 1999; Yang et al., 2006). In every instance, cell and myelin stains are rich, and the cyto- and myelo-architecture of the region are characteristic of the region investigated, with no signs of pathology, as might be revealed by a marked pale staining of neurons or gliosis or light staining of cytochrome oxidase (Lomber and Payne, 1996). However, the lack of damage to the cortex means that it is not possible to use traditional histological techniques to determine the region that was deactivated. In the second study, cortical temperatures surrounding the
cooling loops were measured using multiple microthermocouples (150 mm in diameter; Omega Engineering, Stamford, CT) to determine the region of deactivation (Carrasco and Lomber, 2009). Across the cortical surface, 300–400 thermal measurements were taken from positions 500 mm below the pial surface. From these measurements, thermal cortical maps from cooling each individual cryoloop were constructed (Fig. 4). Depth of the cooling deactivation was also measured at four different coronal levels to provide an assessment of cooling spread in the Z-dimension. This information is provided in the third study.
Cortical loci investigated We used reversible cooling deactivation (Lomber et al., 1999) to examine the contributions of PAF, DZ, A1, and AAF to determine if specific cortical areas mediated the enhanced visual functions. The extent of the cooling deactivations (Fig. 4) was determined from direct cortical temperature recordings that were matched with adjacent sections processed for SMI-32 that permitted the delineation of the different areas of auditory cortex (Mellott et al., 2010) as we have done previously (Lomber and Malhotra, 2008). The positions of these four loci, as well as how they relate to the cortical maps of other investigators, are described below. Cooling loops were placed on PAF (Phillips and Orman, 1984; Reale and Imig, 1980), located caudal and ventral to A1. Loops were 6 mm long and extended from the anterior one-third of the dorsal-posterior ectosylvian gyrus to the fundus of the posterior ectosylvian sulcus (pes). A heat shielding compound was applied to the anterior side of the PAF loops to keep the cooling deactivations localized to the posterior bank of the pes. All deactivations extended down the posterior bank of the pes to the fundus and did not include the anterior bank. Therefore, the deactivated region included all of area PAF or area P (Fig. 4a; Imig et al., 1982; Phillips and Orman, 1984). For all DZ cooling loops, the dorsal
259
edge of the middle ectosylvian gyrus along the lip of the middle suprasylvian sulcus (mss) was deactivated (Fig. 4b). The region of deactivation included the dorsal-most portion of the lateral bank of the mss. However, the cooling did not appear to directly affect either the anterolateral (ALLS) or posterolateral (PLLS) lateral suprasylvian visual areas (Palmer et al., 1978). For each loop, the deactivated region included the totality of the regions previously described as the DZ (Middlebrooks and Zook, 1983) and the suprasylvian fringe (Beneyto et al., 1998; Niimi and Matsuoka, 1979; Paula-Barbosa et al., 1975; Rose, 1949; Woolsey, 1960). For all A1 cryoloops, the central region of the middle ectosylvian gyrus between the dorsal tips of the anterior and pes was deactivated (Fig. 4c). The deactivations were from stereotaxic coronal levels A1–A12. The deactivated region did not include the dorsal-most aspect of the middle ectosylvian gyrus, along the lateral lip of the mss (Fig. 4c). For each loop, the deactivated region included the ventral 2/3's of the classically defined area A1 (Reale and Imig, 1980). The AAF (Knight, 1977; Phillips and Irvine, 1982; Reale and Imig, 1980) cryoloops were 7 mm long and were located on the crown of the anterior suprasylvian gyrus between A10 and A17. All deactivations included the dorsal half of the lateral bank of the anterior suprasylvian sulcus and the dorsal half of the medial bank of the AES. Therefore, the deactivations included all of area AAF or area A (Fig. 4d), as defined by Knight (1977) and Reale and Imig (1980). Visual localization in the peripheral field
Fig. 4. Representative cooling deactivation reconstructions for the four cortical loci examined in the left hemisphere of a deaf cat. Black regions indicate deactivation extent as plotted from direct temperature measurements. The areal borders were determined by using SMI-32 staining criteria as we have done previously (Lomber and Malhotra, 2008). (a) Deactivation reconstruction showing a lateral (left is anterior) view of the left hemisphere with three horizontal sections in
For the visual localization task, the first step was to determine if auditory cortex could be mediating the enhanced visual performance of the deaf cats. the vicinity of the cooling locus. (b–d) Reconstructions showing a lateral (left is anterior) and dorsal (top is anterior) view of the left hemisphere with three coronal sections in the vicinity of the deactivation locus. For abbreviations, see List.
260
Therefore, we simultaneously deactivated all four areas (PAF, DZ, A1, and AAF) bilaterally, which resulted in a significant reduction in visual localization performance restricted to the most peripheral positions (60 , 75 , and 90 positions; Fig. 5a and b). Although the animals often failed to accurately or precisely localize the stimulus in the far periphery, they were not blind to the onset of the stimulus as the illumination of any LED always triggered a response. Therefore, the nature of the deficit was one of localization and not detection. Errors made during bilateral deactivation of all four areas were almost always undershoots of 30–60 (97.8% of all errors). Rarely (4.3% of all errors) were errors made to the incorrect hemifield. These results demonstrated that auditory cortex does have a role in mediating the enhanced visual localization performance of the congenitally deaf cats. In order to ascertain if the enhanced localization skills could be further localized to specific cortical loci, each of the four auditory areas was individually bilaterally deactivated. In the deaf cats, bilateral deactivation of PAF significantly reduced localization performance to the most peripheral targets (60 , 75 , and 90 positions, p < 0.01) while leaving localization performance for the 0 , 15 , 30 , and 45 targets unchanged (Fig. 5c). The reduction in visual localization at the most peripheral locations resulted in performance that was not different from deactivating all four areas simultaneously (Fig. 5b). Moreover, the localization performance of the deaf cats during bilateral cooling of PAF was not different from hearing cats (Fig. 5g). Neither bilateral nor unilateral deactivation of DZ, A1, or AAF modified visual localization performance (Fig. 5d–f). Unilateral deactivation of PAF resulted in reduced visual localization to the same peripheral positions; however, the deficit was specific to the contralateral hemifield (Lomber et al., 2010). Consequently, the neural basis for the enhanced visual localization skills of the deaf cats can be ascribed to PAF. This is an intriguing finding because, in hearing cats, PAF is normally involved in the accurate localization of acoustic stimuli (Fig. 6; Lomber
and Malhotra, 2008; Malhotra and Lomber, 2007). Bilateral deactivation of PAF in hearing cats results in profound acoustic localization deficits across the frontal field (Fig. 6). Therefore, the present results demonstrate that in deafness, PAF maintains a role in localization, albeit visual rather than acoustic. These results demonstrate that crossmodal plasticity can substitute one sensory modality for another while maintaining the functional repertoire of the reorganized region.
Visual motion detection For the supranormal visual motion detection abilities identified in the congenitally deaf cats, a similar experimental approach was taken to ascertain if “deaf” auditory cortex played a role in the enhanced motion detection. To determine if auditory cortex could be mediating the enhanced motion detection performance of deaf cats, we simultaneously deactivated all four areas (PAF, DZ, A1, and AAF). Bilateral deactivation of all four areas significantly increased motion discrimination thresholds from 0.44 0.19 to 1.39 0.35 s 1 (Fig. 7a). This finding established that auditory cortex does have a role in mediating the enhanced motion detection performance of the deaf cats. Next, to determine if a specific auditory region could be mediating the enhanced visual motion detection skills of deaf cats, areas PAF, DZ, A1, and AAF were individually bilaterally cooled. Bilateral deactivation of DZ significantly increased the motion detection thresholds from 0.40 0.15 to 1.46 0.4 s 1 (Fig. 9c). This increase resulted in performance that was not different from deactivating all four areas simultaneously (Fig. 7c). Moreover, the increase in threshold resulted in performance that was not different from performance of the hearing cats (Fig. 7f). There was no evidence of any functional lateralization, as unilateral deactivation of either left or right DZ did not alter performance (Lomber et al., 2010). Neither bilateral (Fig. 7b, d, and e) nor unilateral (Lomber et al.,
Fig. 5. Visual localization task data from deaf cats during bilateral reversible deactivation of PAF, DZ, A1, and AAF. (a) Polar plot of the visual localization responses of deaf cats while cortex was warm (dark gray) and active and during simultaneous cooling deactivation of PAF, DZ, A1, and AAF (black). (b–f) Histogram of combined data from the left and right hemifields showing mean s.e. performance for deaf cats while cortex was warm (dark gray) and active and while it was cooled (black) and deactivated. Asterisks indicate a significant difference (p < 0.01) between the warm and cool conditions. (b) Data from the simultaneous deactivation of PAF, DZ, A1, and AAF. (c–f) Data from individual area deactivations. (g) Visual localization data comparing performance at each position for hearing cats (light gray), deaf cats while PAF was warm (dark gray), and deaf cats while PAF was cooled (black). Asterisks indicate a significant difference (p < 0.01) from the hearing and deaf PAF cool conditions. Figure adapted from Lomber et al. (2010).
262
Fig. 6. Orienting responses to an acoustic stimulus during deactivation of PAF. Lateral view icons of the cat brain indicate the presence and position of a cryoloop (gray shading), and its operational status (black indicates loop was on and cortex was deactivated). For conventions, see Fig. 2. (a) Control data collected: (i) prior to PAF cryoloop implantation, (ii) after PAF cryoloop implantation and prior to cooling in each testing session, and (iii) shortly after termination of cooling. (b). Deactivation data collected: (iv) during cooling of left PAF, (v) during bilateral cooling of PAF, and (vi) during cooling of right PAF. Note that unilateral deactivation of PAF caused sound localization deficits in the contralateral field with no impairments in the ipsilateral hemifield. Bilateral deactivation of PAF resulted in bilateral sound localization deficits. Data summarized from seven animals. Figure adapted from Malhotra and Lomber (2007).
2010) deactivation of PAF, A1, or AAF resulted in any change in motion detection thresholds. These results demonstrate that DZ cortex mediates the superior visual motion detection thresholds of deaf cats. DZ has neuronal properties that are distinct
from A1 (He et al., 1997; Stecker et al., 2005) and is involved in sound source localization (Malhotra et al., 2008) and duration coding (Stecker et al., 2005). Here, we show DZs involvement in visual motion detection in deaf cats. A role for DZ in
263
Deactivation of auditory cortex in hearing cats does not alter visual function As we have demonstrated that “deaf” auditory cortex is the neural substrate for the enhanced visual abilities of the deaf, it was essential to also demonstrate that the auditory cortex of hearing cats does not contribute to visual function. Therefore, for the group of hearing cats, we both simultaneously and individually deactivated the four auditory areas on each of the seven visual tasks. Overall, neither simultaneous nor individual deactivation of the four auditory regions altered the ability of the hearing cats to perform any of the seven visual tasks (Lomber et al., 2010). These results demonstrate that in the presence of functional hearing, the auditory cortex does not contribute to any of the visual tasks examined. Therefore, deficits in visual function identified during bilateral deactivation of PAF or DZ in the deaf cats must be caused by underlying crossmodal adaptive plasticity in each area.
Study 3: laminar contributions to supranormal vision in the deaf Fig. 7. Motion detection thresholds for the deaf cats before and after cooling deactivation and during bilateral reversible deactivation. (a–e) Histograms showing mean s.e. motion detection thresholds for deaf cats while cortex was warm (dark gray) and active and while it was cooled (black) and deactivated. Asterisks indicate a significant difference (p < 0.01) between the warm and cool conditions. (a) Motion detection thresholds from deaf cats during bilateral reversible deactivation of PAF, DZ, A1, and AAF. (b–e) Data from individual area deactivations. (f) Motion detection thresholds to compare performance of hearing cats (light gray), deaf cats while DZ was warm (dark gray), and deaf cats while DZ was cooled (black). Asterisks indicate a significant difference (p < 0.01) from the hearing and deaf DZ cool conditions. Figure adapted from Lomber et al. (2010).
acoustic motion processing has yet to be investigated. Overall, in the second study, we were able to ascribe superior visual localization functions to PAF (Fig. 5g) and the superior motion detection abilities to DZ (Fig. 7f) in the same animals.
As we have demonstrated that individual areas of deaf auditory cortex contribute to supranormal visual localization in the periphery or visual motion detection, we next sought to determine if these functions could then be further localized in the laminar domain (Lomber and Payne, 2000; Lomber et al., 2007). Our approach was to apply lesser or greater levels of cooling to PAF or DZ to deactivate the cortical thickness in a graded, yet consistent, way, the more-superficial layers alone or in combination with the deep layers (Lomber and Payne, 2000; Lomber et al., 2007). With PAF cryoloop temperatures between 10 and 38 C, deaf cats are proficient at accurately reporting the location of a peripheral visual stimulus (Fig. 8a). Cooling to progressively lower temperatures (< 10 C) first initiated and then maximized an impairment in peripheral visual
264
localization, which was reduced to performance levels of hearing animals, at a cryoloop temperature of 3 1 C (Fig. 8a). Similarly, cooling of the DZ loop to progressively lower temperatures resulted in a rise in visual detection threshold (Fig. 8b). Visual motion detection threshold began to rise at cryoloop temperatures of 14 C and continued to rise to performance levels no different from hearing animals, until a temperature of 8 C were reached (Fig. 8b). However, the initiation temperature for the change in performance (14 C) and the temperature producing a maximal deficit (8 C) were both lower in all three deaf cats examined than the respective temperatures identified on the visual localization task for the same animals during PAF cooling. The different temperatures for initiation and maximum deficit for the two cortical areas can potentially be explained by changes in the laminar extent of cooling to disrupt visual localization in PAF rather than visual motion detection in DZ. As 20 C is the critical temperature below which neurons are silenced by blockade of synaptic transmission from afferent fibers (Bénita and Condé, 1972; Jasper et al., 1970; Lomber et al., 1999), we used arrays of microthermocouples to measure temperatures at more than 300 sites below each of the cryoloops (PAF and DZ) to ascertain the position of the 20 C thermocline. The positions of the temperature measurements were reconstructed using microlesions and depth measurements to determine the temperature profiles in the deaf cats from which the recordings were made. For each of the cooling loop locations (PAF and DZ), data were collected from each of the three deaf cats. A compilation of data from multiple tracks with a DZ cryoloop sequentially cooled to two different temperatures (8 C and 3 C) is presented in Fig. 9. Cortex between the 20 C thermocline and the cryoloop (gray field) has temperatures of < 20 C and is deactivated by the cooling, whereas locations more distal from the cryoloop than the 20 C thermocline have temperatures > 20 C and remain active (Fig. 10). Similar laminar deactivations were also determined for PAF (Fig. 11) cooling loops.
Fig. 8. Graphic representation of performance levels of deaf cats on the visual localization task (a) and the motion detection task (b) as a function of PAF or DZ cryoloop temperature, respectively. Each graph shows mean s.e. performance for blocks of trials collected at different cryoloop temperatures. (a) Black diamonds and lines represent mean performance of deaf cats performing the visual orienting task (mean performance across the three peripheral-most positions (60 , 75 , and 90 )) during bilateral cooling of PAF. (b) Black circles and lines represent mean performance of deaf cats performing visual motion detection task during bilateral cooling of DZ. Note that for the motion detection task (b) that thresholds begin to increase at cryoloop temperatures below 16 C and reaches a maximum deficit at 8 C. In contrast, visual localization performance (a) begins to fall at cryoloop temperatures below 10 C and reaches a maximum deficit at 2 C.
265 (a) 0 mss
8 ⬚C
Cryoloop
3 ⬚C
I 200
II
5 mm
mss 35 37 25 20 23 17 17 24 26
37
29
37
34
37 37
35
34
35
36
37
37
37
37
III
800 1000
IV
1200
V
38
VI
38 38
38
600
1400
37
37 38
Cryoloop = 8 ⬚C
32 33 34
23
34
35
Cortical Depth (mm)
400
1600
38
wm D
38
Fig. 10. Summary of deactivation depth when a cryoloop was cooled to 8 and 3 C. Temperatures 140%. Thus, the major effect of the additional cooling was to push the 20 C thermocline across the deep layers of cortex with minor lateral surface movement. The most parsimonious interpretation of the differences in extents of deactivations is that
266
pes 5 mm
(a)
38 38 38 38 37 37 37 36
38 38 38 38 37 37 37 36 36 36 36
36 36
(b)
38 38 38 38 38 37 37 37 36 36
Cryoloop = 8 ⬚C
35 30 26 18 14 12 13 14 13 13 12
36 32 29 26 24 23 24 25 24 24 18 16 12
34 30 24 17 12 11 12 12 14 13 12 10
32 22 19 18 15 16 14 15 15 15 14 13 10
35 34 32 31 31 30 31 30 27 19 15
37 37 36 36 36 36 35 36 33 33 32 31
pes 38 38 38 38 37 37 37 37 36 36 36
Cryoloop = 3 ⬚C 36 33 30 27 33 23 22 22 21 19 18 17 15 12
32 30 30 28 26 24 23 23 22
M A
P L
2 mm Fig. 11. Temperature measurements recorded from identical sites in the posterior ectosylvian suclus (pes) when a PAF cooling loop was cooled to 8 C (a) and 3 C (b). Horizontal line on the lateral view of the left cerebrum (top right) shows the position of the horizontal section shown in (a) and (b). Gray region indicates the depth of cortex that was at, or below, 20 C as estimated from these measurements. Note that temperatures remain high in the anterior bank of the posterior ectosylvian sulcus due to the application of a heat shielding compound to the anterior surface of the cooling loop. For abbreviations, see List.
motion detection processing in deaf DZ is critically dependent upon the superficial cortical layers and that visual localization processing in deaf PAF is critically dependent upon the deep cortical layers. A critical component in acceptance of this
interpretation is the recognition that deep layer neurons remain active when upper layer neurons are silenced. Control physiological measures made in other cats verify deep layer activity in the absence of upper layer activity, and confirm the results of others in the visual system of intact cats that deep layer neurons remain active in the absence of activity in the superficial layers (Ferster et al., 1996; Schwark et al., 1986; Weyand et al., 1986, 1991). In the deaf cats, we observed deactivation of PAF eliminates supranormal visual localization abilities. We further observed that it is necessary to cool both the superficial and deep layers of PAF in order to completely eliminate the visual localization sensory enhancements. These results are interesting for two reasons. First, in hearing cats, PAF is normally involved in the accurate localization of acoustic stimuli (Fig. 6; Lomber and Malhotra, 2008; Malhotra and Lomber, 2007). This suggests that in deafness, PAF maintains a role in localization, albeit visual rather than acoustic. This is consistent with the hypothesis that the behavioral role of a crossmodally reorganized area is related to its role in hearing/ sighted individuals (Lomber et al., 2010; Meredith et al., 2011). Second, in hearing cats, in order to eliminate accurate acoustic localization, it is only necessary to deactivate the superficial layers of PAF (Lomber et al., 2007). Therefore, only the superficial layers of PAF need to be deactivated to disrupt acoustic localization in hearing animals, while both the superficial and deep layers of PAF must be deactivated in order to disrupt the supranormal visual localization abilities of congenitally deaf cats. Taken together, it will be interesting to examine possible differences in the input and output circuitry of the superficial and deep layers of PAF in congenitally deaf cats compared to hearing animals. Identification of the circuitry underlying crossmodal plasticity is essential toward providing a substrate on which the phenomenon can be studied and manipulated to reveal the fundamental principles governing its organization, function, and potential for therapeutic intervention.
267
Significance Collectively, these results provide new and comprehensive insight into the crossmodal effects induced by congenital deafness to a level that is essentially unobtainable through other methods. In addition, these observations form the basis for a robust and repeatable model of adaptive crossmodal plasticity that will be used to uncover the basic principles that characterize this phenomenon as well as better understand its relation to neuroplastic processes as a whole. By characterizing the regions of auditory cortex that are susceptible to crossmodal plasticity following deafness, we may be able to reveal the roles of intrinsic constraints and environmental input in determining cortical functional specificity. Such information will be critical for predicting and evaluating the success of sensory implants in humans (Kral and O'Donoghue, 2010; Rauschecker and Shannon, 2002; Zrenner, 2002). Specifically, crossmodal reorganization in deprived auditory cortex, like that identified in the present investigations, may hinder the ability of auditory cortex to process new auditory input provided by a cochlear implant (Bavelier and Neville, 2002; Kral and Eggermont, 2007). Studies suggest that deaf subjects, in whom crossmodal plasticity was the most extensive, were the least likely to benefit from cochlear prosthetics (Lee et al., 2001). Therefore, further investigations are necessary in order to more closely examine the link between crossmodal plasticity in deprived auditory cortex and the functional outcomes of cochlear prosthetics. Ultimately, future experiments could use this model of crossmodal plasticity to empirically assess potential windows for therapeutic interventions.
Acknowledgments We thank Amee McMillan for preparing all the figures and for help with the preparation of the chapter. We gratefully acknowledge the support of the Canadian Institutes of Health Research (CAN),
Deutsche Forschungsgemeinschaft (GER), and the National Institutes of Health (USA).
Abbreviations A AAF aes AI or A1 AII or A2 D dPE DZ FAES IN iPE L M mss P pes PAF ss T V VAF VPAF vPE
anterior anterior auditory field anterior ectosylvian sulcus primary auditory cortex second auditory cortex dorsal dorsal-posterior ectosylvian area dorsal zone of auditory cortex auditory field of the anterior ectosylvian sulcus insular region intermediate posterior ectosylvian area lateral medial middle suprasylvian sulcus posterior posterior ectosylvian sulcus posterior auditory field suprasylvian sulcus temporal region ventral ventral auditory field ventral posterior auditory field ventral posterior ectosylvian area
References Bavelier, D., Dye, M. W. G., & Hauser, P. C. (2006). Do deaf individuals see better? Trends in Cognitive Science, 10, 512–518. Bavelier, D., & Neville, H. (2002). Brain plasticity: Where and how? Nature Neuroscience, 3, 443–452. Bavelier, D., Tomann, A., Hutton, C., Mitchell, T., Corina, D., Liu, G., et al. (2000). Visual attention to the periphery is enhanced in congenitally deaf individuals. The Journal of Neuroscience, 20, 1–6. Beneyto, M., Winer, J. A., Larue, D. T., & Prieto, J. J. (1998). Auditory connections and neurochemistry of the sagulum. The Journal of Comparative Neurology, 401, 329–351.
268 Bénita, M., & Condé, H. (1972). Effects of local cooling upon conduction and synaptic transmission. Brain Research, 36, 133–151. Brooks, V. B. (1983). Study of brain function by local, reversible cooling. Reviews of Physiology Biochemistry and Pharmacology, 95, 1–109. Brozinsky, C. J., & Bavelier, D. (2004). Motion velocity thresholds in deaf signers: Changes in lateralization but not overall sensitivity. Cognitive Brain Research, 21, 1–10. Carrasco, A., & Lomber, S. G. (2009). Evidence for hierarchical processing in cat auditory cortex: Nonreciprocal influence of primary auditory cortex on the posterior auditory field. The Journal of Neuroscience, 29, 14323–14333. Collignon, O., Voss, P., Lassonde, M., & Lepore, F. (2009). Cross-modal plasticity for the spatial processing of sounds in visually deprived subjects. Experimental Brain Research, 192, 343–358. D'Anguilli, A., & Waraich, P. (2002). Enhanced tactile encoding and memory recognition in congenital blindness. International Journal of Rehabilitation Research, 25, 143–145. de Ribaupierre, F. (1997). Acoustical information processing in the auditory thalamus and cerebral cortex. In G. Ehret & R. Romand (Eds.), The central auditory system (pp. 317–397). New York: Oxford University Press. Doucet, M. E., Bergeron, F., Lassonde, M., Ferron, P., & Lepore, F. (2006). Cross-modal reorganization and speech perception in cochlear implant users. Brain, 129, 3376–3383. Ferster, D., Chung, S., & Wheat, H. (1996). Orientation selectivity of thalamic input to simple cells of cat visual cortex. Nature, 380, 249–252. Finney, E. M., & Dobkins, K. R. (2001). Visual contrast sensitivity in deaf versus hearing populations: Exploring the perceptual consequences of auditory deprivation and experience with a visual language. Cognitive Brain Research, 11, 171–183. Finney, E. M., Fine, I., & Dobkins, K. R. (2001). Visual stimuli activate auditory cortex in the deaf. Nature Neuroscience, 12, 1171–1173. Grant, A. C., Thiagarajah, M. C., & Sathian, K. (2000). Tactile perception in blind Braille readers: A psychophysical study of acuity and hyperacuity using gratings and dot patterns. Perception & Psychophysics, 62, 301–312. Guillery, R. W., Hickey, T. L., & Spear, P. D. (1981). Do blue-eyed white cats have normal or abnormal retinofugal pathways? Investigative Ophthalmology & Visual Science, 21, 27–33. Gutfreund, Y., Zheng, W., & Knudsen, E. I. (2002). Gated visual input to the central auditory system. Science, 297, 1556–1559. He, J., Hashikawa, T., Ojima, H., & Kinouchi, Y. (1997). Temporal integration and duration tuning in the dorsal zone of cat auditory cortex. The Journal of Neuroscience, 17, 2615–2625. Heid, S., Hartmann, R., & Klinke, R. (1998). A model for prelingual deafness, the congenitally deaf white cat–population statistics and degenerative changes. Hearing Research, 115, 101–112.
Imig, T. J., Reale, R. A., & Brugge, J. F. (1982). The auditory cortex: Patterns of corticocortical projections related to physiological maps in the cat. In C. N. Woolsey (Ed.), Cortical sensory organization: Vol. 3: Multiple auditory areas. New Jersey: Humana Press. Jasper, H., Shacter, D. G., & Montplaisir, J. (1970). The effect of local cooling upon spontaneous and evoked electrical activity of cerebral cortex. Canadian Journal of Physiology and Pharmacology, 48, 640–652. King, A. J. (2002). Neural plasticity: How the eye tells the brain about sound location. Current Biology, 12, R393–R395. King, A. J., & Parsons, C. H. (1999). Improved auditory spatial acuity in visually deprived ferrets. The European Journal of Neuroscience, 11, 3945–3956. Knight, P. L. (1977). Representation of the cochlea within the anterior auditory field (AAF) of the cat. Brain Research, 130, 447–467. Knudsen, E. I. (2004). Sensitive periods in the development of the brain and behavior. Journal of Cognitive Neuroscience, 16, 1412–1425. Knudsen, E. I., & Knudsen, P. F. (1989). Vision calibrates sound localization in developing barn owls. The Journal of Neuroscience, 9, 3306–3313. Korte, M., & Rauschecker, J. P. (1993). Auditory tuning of cortical neurons is sharpened in cats with early blindness. Journal of Neurophysiology, 70, 1717–1721. Kral, A., & Eggermont, J. J. (2007). What's to lose and what's to learn: Development under auditory deprivation, cochlear implants and limits of cortical plasticity. Brain Research Reviews, 56, 259–269. Kral, A., Hartmann, R., Tillein, J., Heid, S., & Klinke, R. (2000). Congenital auditory deprivation reduces synaptic activity within the auditory cortex in a layer-specific manner. Cerebral Cortex, 10, 714–726. Kral, A., Hartmann, R., Tillein, J., Heid, S., & Klinke, R. (2002). Hearing after congenital deafness: Central auditory plasticity and sensory deprivation. Cerebral Cortex, 12, 797–807. Kral, A., & O'Donoghue, G. M. (2010). Profound deafness in childhood. The New England Journal of Medicine, 363, 1438–1450. Kral, A., Schroder, J. H., Klinke, R., & Engel, A. K. (2003). Absence of cross-modal reorganization in the primary auditory cortex of congenitally deaf cats. Experimental Brain Research, 153, 605–613. Kral, A., Tillein, J., Heid, J., Hartmann, R., & Klinke, R. (2005). Postnatal cortical development in congenital auditory deprivation. Cerebral Cortex, 15, 552–562. Kral, A., Tillein, J., Heid, S., Klinke, R., & Hartmann, R. (2006). Cochlear implants: Cortical plasticity in congenital deprivation. Progress in Brain Research, 157, 283–313. Kral, A., Tillein, J., Hubka, P., Schiemann, D., Heid, S., Hartmann, R., et al. (2009). Spatiotemporal patterns of cortical activity with bilateral cochlear implants in congenital deafness. The Journal of Neuroscience, 29, 811–827.
269 Lambertz, N., Gizewski, E. R., Greiff, A., & Forsting, M. (2005). Cross-modal plasticity in deaf subjects dependent on extent of hearing loss. Cognitive Brain Research, 25, 884–890. Langers, D. R., van Dijk, P., & Backes, W. H. (2005). Lateralization, connectivity and plasticity in the human central auditory system. Neuroimage, 28, 490–499. Lee, D. S., Lee, J. S., Oh, S. H., Kim, S. K., Kim, J.-W., Chung, J. K., et al. (2001). Cross-modal plasticity and cochlear implants. Nature, 409, 149–150. Lee, J. S., Lee, D. S., Oh, S. H., Kim, C. S., Kim, J.-W., Hwang, Ch., et al. (2003). PET evidence of neuroplasticity in auditory auditory cortex of postlingual deafness. Journal of Nuclear Medicine, 44, 1435–1439. Levanen, S., & Hamdof, D. (2001). Feeling vibrations: Enhanced tactile sensitivity in congenitally deaf humans. Neuroscience Letters, 301, 75–77. Levick, W. R., Thibos, L. N., & Morstyn, R. (1980). Retinal ganglion cells and optic decussation of white cats. Vision Research, 20, 1001–1006. Lewald, J. (2007). More accurate sound localization induced by short-term light deprivation. Neuropsychologia, 5, 1215–1222. Lomber, S. G. (1999). The advantages and limitations of permanent or reversible deactivation techniques in the assessment of neural function. Journal of Neuroscience Methods, 86, 109–117. Lomber, S. G., Cornwell, P., Sun, J.-S., MacNeil, M. A., & Payne, B. R. (1994). Reversible inactivation of visual processing operations in middle suprasylvian cortex of the behaving cat. Proceedings of the National Academy of Sciences of the United States of America, 91, 2999–3003. Lomber, S. G., & Malhotra, S. (2008). Double dissociation of “what” and “where” processing in auditory cortex. Nature Neuroscience, 11, 609–616. Lomber, S. G., Malhotra, S., & Hall, A. J. (2007). Functional specialization in non-primary auditory cortex of the cat: Areal and laminar contributions to sound localization. Hearing Research, 229, 31–45. Lomber, S. G., Meredith, M. A., & Kral, A. (2010). Crossmodal plasticity in specific auditory cortices underlies visual compensations in the deaf. Nature Neuroscience, 13, 1421–1427. Lomber, S. G., & Payne, B. R. (1996). Removal of two halves restores the whole: Reversal of visual hemineglect during bilateral cortical or collicular inactivation in the cat. Visual Neuroscience, 13, 1143–1156. Lomber, S. G., & Payne, B. R. (2000). Translaminar differentiation of visually-guided behaviors revealed by restricted cerebral cooling deactivation. Cerebral Cortex, 10, 1066–1077. Lomber, S. G., & Payne, B. R. (2001). Task-specific reversal of visual hemineglect following bilateral reversible deactivation of posterior parietal cortex: A comparison with deactivation of the superior colliculus. Visual Neuroscience, 18, 487–499.
Lomber, S. G., Payne, B. R., Cornwell, P., & Long, K. D. (1996). Perceptual and cognitive visual functions of parietal and temporal cortices in the cat. Cerebral Cortex, 6, 673–695. Lomber, S. G., Payne, B. R., & Horel, J. A. (1999). The cryoloop: An adaptable reversible cooling deactivation method for behavioral or electrophysiological assessment of neural function. Journal of Neuroscience Methods, 86, 179–194. Malhotra, S., Hall, A. J., & Lomber, S. G. (2004). Cortical control of sound localization in the cat: Unilateral cooling deactivation of 19 cerebral areas. Journal of Neurophysiology, 92, 1625–1643. Malhotra, S., & Lomber, S. G. (2007). Sound localization during homotopic and heterotopic bilateral cooling deactivation of primary and non-primary auditory cortical areas in the cat. Journal of Neurophysiology, 97, 26–43. Malhotra, S., Stecker, G. C., Middlebrooks, J. C., & Lomber, S. G. (2008). Sound localization deficits during reversible deactivation of primary auditory cortex and/or the dorsal zone. Journal of Neurophysiology, 99, 1628–1642. Mellott, J. G., Van der Gucht, E., Lee, C. C., Carrasco, A., Winer, J. A., & Lomber, S. G. (2010). Areas of the cat auditory cortex as defined by neurofilament proteins expressing SMI-32. Hearing Research, 267, 119–136. Merabet, L. B., & Pascual-Leone, A. (2010). Neural reorganization following sensory loss: The opportunity of change. Nature Reviews. Neuroscience, 11, 44–52. Meredith, M. A., Kryklywy, J., McMillan, A. J., Malhotra, S., Lum-Tai, R., & Lomber, S. G. (2011). Crossmodal reorganization in the early-deaf switches sensory, but not behavioral roles of auditory cortex. Proceedings of the National Academy of Sciences of the United States of America, 108, 8856–8861. Metin, C., & Frost, D. O. (1989). Visual responses of neurons in somatosensory cortex of hamsters with experimentally induced retinal projections to somatosensory thalamus. Proceedings of the National Academy of Sciences of the United States of America, 86, 357–361. Middlebrooks, J. C., & Zook, J. M. (1983). Intrinsic organization of the cat's medial geniculate body identified by projections to binaural response-specific bands in the primary auditory cortex. The Journal of Neuroscience, 1, 203–224. Niimi, K., & Matsuoka, H. (1979). Thalamocortical organization of the auditory system in the cat studied by retrograde and axonal transport of horseradish peroxidase. Advances in Anatomy, Embryology and Cell Biology, 57, 1–56. Nishimura, H., Hashikawa, K., Doi, K., Ixaki, T., Watanabe, Y., Kusuoka, H., et al. (1999). Sign language ‘heard’ in the auditory cortex. Nature, 397, 16. Pallas, S. L., Littman, T., & Moore, D. R. (1999). Cross-modal reorganization of callosal connectivity without altering thalamocortical projections. Proceedings of the National Academy of Sciences of the United States of America, 96, 8751–8756.
270 Palmer, L. A., Rosenquist, A. C., & Tusa, R. J. (1978). The retinotopic organization of lateral suprasylvian visual areas in the cat. The Journal of Comparative Neurology, 177, 237–256. Pasternak, T., & Merigan, W. H. (1980). Movement detection by cats: Invariance with direction and target configuration. Journal of Comparative and Physiological Psychology, 94, 943–952. Paula-Barbosa, M. M., Feyo, P. B., & Sousa-Pinto, A. (1975). The association connexions of the suprasylvian fringe (SF) and other areas of the cat auditory cortex. Experimental Brain Research, 23, 535–554. Phillips, D. P., & Irvine, D. R. (1982). Properties of single neurons in the anterior auditory field (AAF) of cat cerebral cortex. Brain Research, 248, 237–244. Phillips, D. P., & Orman, S. S. (1984). Responses of single neurons in posterior field of cat auditory cortex to tonal stimulation. Journal of Neurophysiology, 51, 147–163. Ramachandran, V. S., & Hirstein, W. (1998). The perception of phantom limbs: The D.O. Hebb lecture. Brain, 9, 1603–1630. Rauschecker, J. P. (1995). Compensatory plasticity and sensory substitution in the cerebral cortex. Trends in Neuroscience, 18, 36–43. Rauschecker, J. P. (2002). Cortical map plasticity in animals and humans. Progress in Brain Research, 138, 73–88. Rauschecker, J. P., & Kniepert, U. (1994). Auditory localization behaviour in visually deprived cats. The European Journal of Neuroscience, 6, 149–160. Rauschecker, J. P., & Korte, M. (1993). Auditory compensation for early blindness in cat cerebral cortex. The Journal of Neuroscience, 13, 4538–4548. Rauschecker, J. P., & Shannon, R. V. (2002). Sensing sound to the brain. Science, 295, 1025–1029. Reale, R. A., & Imig, T. J. (1980). Tonotopic organization in auditory cortex of the cat. The Journal of Comparative Neurology, 192, 265–291. Reinoso-Suárez, F. (1961). Topographical atlas of the cat brain for experimental-physiological research [in German]. Merck, Darmstad: Federal Republic of Germany. Roe, A. W., Pallas, S. L., Hahm, J.-O., & Sur, M. (1990). A map of visual space induced in primary auditory cortex. Science, 250, 818–820. Rose, J. E. (1949). The cellular structure of the auditory region of the cat. The Journal of Comparative Neurology, 91, 409–440. Sadato, N., Pascual-Leone, A., Grafman, J., Ibanez, V., Deiber, M. P., Dold, G., et al. (1996). Activation of the primary visual cortex by Braille reading in blind subjects. Nature, 380, 526–528. Sathian, K. (2000). Practice makes perfect: Sharper tactile perception in the blind. Neurology, 54, 2203–2204.
Sathian, K. (2005). Visual cortical activity during tactile perception in the sighted and the visually deprived. Developmental Psychobiology, 46, 279–286. Schwark, H. D., Malpeli, J. G., Weyand, T. G., & Lee, C. (1986). Cat area 17. II. Response properties of infragranular layer neurons in the absence of supragranular layer activity. Journal of Neurophysiology, 56, 1074–1087. Shore, S. E., Koehler, S., Oldakowski, M., Hughes, L. F., & Syed, S. (2009). Dorsal cochlear nucleus responses to somatosensory stimulation are enhanced after noise-induced hearing loss. The European Journal of Neuroscience, 27, 155–168. Stecker, G. C., Harrington, I. A., Macpherson, E. A., & Middlebrooks, J. C. (2005). Spatial sensitivity in the dorsal zone (area DZ) of cat auditory cortex. Journal of Neurophysiology, 94, 1267–1280. Sur, M., Pallas, S. L., & Roe, A. W. (1990). Cross-modal plasticity in cortical development: Differentiation and specification of sensory neocortex. Trends in Neuroscience, 13, 227–233. Tian, B., & Rauschecker, J. P. (1998). Processing of frequencymodulated sounds in the cat's posterior auditory field. Journal of Neurophysiology, 79, 2629–2642. Tillein, J., Hubka, P., Syed, E., Hartmann, R., Engel, A. K., & Kral, A. (2010). Cortical represetnation of onteraural time difference in congenital deafness. Cerebral Cortex, 20, 492–506. Webber, A. L., & Wood, J. (2005). Amblyopia: Prevalence, natural history, functional effects, and treatment. Clinical & Experimental Optometry, 88, 365–375. Weeks, R., Horwitz, B., Aziz-Sultan, A., Tian, B., Wessinger, M., Cohen, L., et al. (2000). A positron emission tomographic study of auditory localization in the congenitally blind. The Journal of Neuroscience, 20, 2664–2672. Weyand, T. G., Malpeli, J. G., & Lee, C. (1991). Area 18 corticotectal cells: Response properties and identification of sustaining geniculate inputs. Journal of Neurophysiology, 65, 1078–1088. Weyand, T. G., Malpeli, J. G., Lee, C., & Schwark, H. D. (1986). Cat area 17. IV. Two types of corticotectal cells defined by controlling geniculate inputs. Journal of Neurophysiology, 56, 1102–1108. Woolsey, C. N. (1960). Organization of cortical auditory system: A review and synthesis. In G. L. Rasmussen & W. F. Windle (Eds.), Neural mechanisms of the auditory and vestibular systems (pp. 165–180). Springfield: C.C. Thomas. Yang, X. F., Kennedy, B. R., Lomber, S. G., Schmidt, R. E., & Rothman, S. M. (2006). Cooling produces minimal neuropathology in neocortex and hippocampus. Neurobiology of Disease, 23, 637–643. Zrenner, E. (2002). Will retinal implants restore vision? Science, 295, 1022–1025.
Subject Index
Adaptation, brain plasticity definition, 177–178 exaptation, 180–183 experience-dependent plasticity, 179 Adaptive crossmodal plasticity behavioral/perceptual effects, 253 congenitally deaf cat cortical loci, 258–259 deactivation, auditory cortex, 263 laminar contributions, 263–266 reversible cooling deactivation, 257–258 supranormal visual abilities, 255–257 visual localization, peripheral field, 259–260 visual motion detection, 260–263 phantom limb pain, 252 significance, 267
occipital activity, 237 right dorsal extrastriate occipital cortex (rOC), 238–239 right intraparietal sulcus (rIPS), 238–239 transcranial magnetic stimulation (TMS), 236 sensory rehabilitation auditory discrimination tasks, 214 neuroprostheses, 219–220 prosthesis substituting vision by audition (PSVA), 214 sensory substitution, 217–219 sight restoration, 215–217 visual deprivation, 213 Brain plasticity adaptation definition, 177–178 exaptation, 180–183 experience-dependent plasticity, 179 environmental manipulations, 191 evolutionary process, 177 maladaptation cochlear implant, 189–190 congenitally deprived sensory modalities comparison, 186 cortical reorganization rewiring, 188–189 phantom limb pain, 185–186 tinnitus, 186–188 natural selection, 178 structural and functional change, 178 Brain reorganization blindness correlational analysis, 238 occipital activity, 237
Behavioral reorganization blindness external and somatotopic spatial codes, 235 peripheral visual field, 235 spatial processing, 234 tactile modality, 236 chemosensory loss, 241–243 deafness, 239–240 Bilateral deactivation, 260 Blindness behavioral reorganization external and somatotopic spatial codes, 235 peripheral visual field, 235 spatial processing, 234 tactile modality, 236 brain reorganization correlational analysis, 238 271
272
Brain reorganization (Continued) right dorsal extrastriate occipital cortex (rOC), 238–239 right intraparietal sulcus (rIPS), 238–239 transcranial magnetic stimulation (TMS), 236 chemosensory loss, 243–244 deafness, 240–241
Deafness crossmodal plasticity behavioral reorganization, 239–240 brain reorganization, 240–241 sensory rehabilitation cochlear implant (CI), 221–224 sensory substitution, 220–221
Chemosensory loss behavioral reorganization, 241–243 brain reorganization, 243–244 Cochlear implant (CI) audiovisual interaction, 224 auditory input, 221 influential factors, 222 maladaptation, 189–190 multisensory interactions, 223 occipito-temporal networks, 221 Congenitally deaf cat, crossmodal plasticity cortical loci, 258–259 deactivation, auditory cortex, 263 laminar contributions accurate acoustic localization, 266 deactivations, 264 PAF cryoloop temperatures, 263–264 performance levels, 264 reversible cooling deactivation, 257–258 supranormal visual abilities, 255–257 visual localization, peripheral field, 259–260 visual motion detection bilateral deactivation, 260 motion detection thresholds, 263 orienting responses, 262 Crossmodal plasticity, sensory loss. See also Adaptive crossmodal plasticity blindness behavioral reorganization, 234–236 brain reorganization, 236–239 chemosensory loss behavioral reorganization, 241–243 brain reorganization, 243–244 deafness behavioral reorganization, 239–240 brain reorganization, 240–241
Euclidean space definition, 48 distance, 49 metric properties, 50 Haptic discrimination task, 20 Human–machine interfaces (HMIs) clinical perspective average normalized movement errors, 59 distribution, motor variance, 61–62 low-dimensional geometrical structure, 58 movement trajectories, 61 principal component analysis (PCA), 60 2D task-spaces, 63 dual-learning problem, 55–58 features, 63 intrinsic geometry, central nervous system, 49 inverse geometrical model complementary subspaces, 50 glove-signal space, 51 hand-to-cursor mapping experiment, 53 minimum Euclidean norm, 52 task-space, 55 metric properties, Euclidean space, 50 motor learning, 46–47 ordinary space electromyographic (EMG) signals, 47 Euclidean properties, 48–49 Human sensorimotor control. See Sensorimotor control Inverse geometrical model, HMIs complementary subspaces, 50 glove-signal space, 51
273
hand-to-cursor mapping experiment, 53 minimum Euclidean norm, 52 task-space, 55 Locomotor adaptation adaptation and deadaptation rates, 68–69 advantages, 74 anatomical circuits, 69 baseline behavior, 72 conscious correction and distraction effects, 69 errors, 71 individual subject data, 71 late childhood adaptive abilities, 67–68 learning paradigm, 66–67 pattern transfer, 69–70 proprioceptive signals, 73 spatial and temporal strategies, 66 split-belt treadmill paradigm, 68 temporal control, 71 walking rehabilitation, 65–66 Maladaptation cochlear implant, 189–190 congenitally deprived sensory modalities comparison, 186 cortical reorganization rewiring, 188–189 phantom limb pain, 185–186 tinnitus, 186–188 Motion detection thresholds, 263 Motor adaptation and proprioceptive recalibration comparison, 97–98 methodology, 92–94 motor learning conditions, 95–97 visual feedback, 92 visuomotor adaptation, 94–95 Motor learning functional connections, 32 hypothesis, 31 limb movement study, 40–41 plasticity, 31–32 somatosensory perception, upper limb force field learning, 32–33 mean movement curvature, 36
parameter estimation by sequential testing (PEST), 32 perceptual boundary, 34–35 speech sounds experiment, 37 histogram, 40 perceptual classification, 40–41 perceptual psychometric functions, 38–39 sagittal plane view, 38 statistical tests, 40 Multisensory integration adulthood, 158–161 changes experienced, 153–158 development, 150–153 mature superior colliculus, 146–147 objectives, 146 principles, 147–148 SC model, 148–150 semantic issues, 147 senses, 145 simple heuristics, 148 underlying computation, 147 Multisensory object recognition multisensory cortical processing, 170–171 size-dependence, 168–169 structural and surface properties integration, 169–170 view-dependence, 166–168 visual imagery, 171–172 visuo-haptic model, 172–173 Naturalistic approaches, sensorimotor control animal psychology, 4–5 arm movements, 13–15 ethology, object manipulation, 15–16 eye movements, 8–10 hand movements, 11–13 haptic discrimination task, 20 human behavior, 7–8 human cognitive ethology, 5–6 object manipulation task, 20–22 physical objects, familiar dynamics, 16–17 simulated objects, 17–20 visual system, 6–7 Neuroplasticity, 211–212. See also Brain plasticity
274
Parameter estimation by sequential testing (PEST), 32 Perception and action, singing absolute and relative accuracy, 105 deficits, 109–111 feedback deficits, 113–114 limitations, 106 motor control deficits, 112–113 neural bases, 114–115 non-Western societies, 104 pitch errors, 104 poor singing, 106–109 sensorimotor translation deficits, 111–112 tuning, 106 types, 105 Phantom limb pain, 252 Proprioceptive recalibration. See Motor adaptation and proprioceptive recalibration Rat auditory cortex animal model, 120–121 cortical sensitive periods succession A1 frequency tuning, 122 critical periods, 123 progressive crystallization, 124 subcortical processing center, 121 history, 120 local regulation, CP plasticity, 124–125 perceptual and neurological specialization, 120 stimulus selectivity age-related changes, 128 moderate-level noise exposure, 127 plastic rewiring, 126 Reversible cooling deactivation, 257–258 Right dorsal extrastriate occipital cortex (rOC), 238–239 Right intraparietal sulcus (rIPS), 238–239 Sensorimotor control definition, 3 naturalistic approaches animal psychology, 4–5 arm movements, 13–15 ethology, object manipulation, 15–16 eye movements, 8–10
hand movements, 11–13 haptic discrimination task, 20 human behavior, 7–8 human cognitive ethology, 5–6 object manipulation task, 20–22 physical objects, familiar dynamics, 16–17 simulated objects, 17–20 visual system, 6–7 transformations, 3–4 Sensory integration cortical circuits neural reference frames, 201 recording locations, 202 shift analysis, 203 volitional arm movements, 201 local to global optimality complex sensorimotor circuit, 203 parietal representations, 205 pure reference-frame representations, 204 variability, 206 neural populations modeling, 199–200 optimal integration, 196 psychophysics modeling, 196 reach behavior, 196–199 Sensory motor remapping. See Human–machine interfaces (HMIs) Sensory rehabilitation blindness auditory discrimination tasks, 214 neuroprostheses, 219–220 prosthesis substituting vision by audition (PSVA), 214 sensory substitution, 217–219 sight restoration, 215–217 visual deprivation, 213 deafness cochlear implant (CI), 221–224 sensory substitution, 220–221 neuroplasticity, 211–212 Singing, perception and action absolute and relative accuracy, 105 deficits, 109–111 feedback deficits, 113–114 limitations, 106 motor control deficits, 112–113
275
neural bases, 114–115 non-Western societies, 104 pitch errors, 104 poor singing, 106–109 sensorimotor translation deficits, 111–112 tuning, 106 types, 105 Sleep, cognitive function aging neurodegenerative diseases, 76 healthy aging, changes circadian regulation, 79 cognition, 80–81 cross-sectional and longitudinal studies, 81–82 homeostatic regulation, 79–80 sleep architecture, 77–78 sleep-dependent consolidation studies, 82–85 sleep deprivation and restriction studies, 81 sleep-related neuroendocrine, 80 total sleep time (TST), 76 Somatosensory perception, motor learning force field learning, 32–33 mean movement curvature, 36 parameter estimation by sequential testing (PEST), 32 perceptual boundary, 34–35 Speech sounds, motor learning experiment, 37
histogram, 40 perceptual classification, 40–41 perceptual psychometric functions, 38–39 sagittal plane view, 38 statistical tests, 40 Tinnitus abnormal activity, 187 animal and human studies, 188 causes, 187 frequency, 187 objective/subjective, 186 phantom sounds, 187 Transcranial magnetic stimulation (TMS), 236 Visual orientation cues, gravity direction, 134–135 enhancement cognitive demands, 137 compression, 138 motion, 138 polarized cues, 138 gravity direction estimation, 136 perceived orientation, 136–137 perceptual tasks, 135 Walking adaptation. See Locomotor adaptation
Other volumes in PROGRESS IN BRAIN RESEARCH Volume 149: Cortical Function: A View from the Thalamus, by V.A. Casagrande, R.W. Guillery and S.M. Sherman (Eds.) – 2005 ISBN 0-444-51679-4. Volume 150: The Boundaries of Consciousness: Neurobiology and Neuropathology, by Steven Laureys (Ed.) – 2005, ISBN 0-444-51851-7. Volume 151: Neuroanatomy of the Oculomotor System, by J.A. Büttner-Ennever (Ed.) – 2006, ISBN 0-444-51696-4. Volume 152: Autonomic Dysfunction after Spinal Cord Injury, by L.C. Weaver and C. Polosa (Eds.) – 2006, ISBN 0-444-51925-4. Volume 153: Hypothalamic Integration of Energy Metabolism, by A. Kalsbeek, E. Fliers, M.A. Hofman, D.F. Swaab, E.J.W. Van Someren and R.M. Buijs (Eds.) – 2006, ISBN 978-0-444-52261-0. Volume 154: Visual Perception, Part 1, Fundamentals of Vision: Low and Mid-Level Processes in Perception, by S. Martinez-Conde, S.L. Macknik, L.M. Martinez, J.M. Alonso and P.U. Tse (Eds.) – 2006, ISBN 978-0-444-52966-4. Volume 155: Visual Perception, Part 2, Fundamentals of Awareness, Multi-Sensory Integration and High-Order Perception, by S. Martinez-Conde, S.L. Macknik, L.M. Martinez, J.M. Alonso and P.U. Tse (Eds.) – 2006, ISBN 978-0-444-51927-6. Volume 156: Understanding Emotions, by S. Anders, G. Ende, M. Junghofer, J. Kissler and D. Wildgruber (Eds.) – 2006, ISBN 978-0-444-52182-8. Volume 157: Reprogramming of the Brain, by A.R. M ller (Ed.) – 2006, ISBN 978-0-444-51602-2. Volume 158: Functional Genomics and Proteomics in the Clinical Neurosciences, by S.E. Hemby and S. Bahn (Eds.) – 2006, ISBN 978-0-444-51853-8. Volume 159: Event-Related Dynamics of Brain Oscillations, by C. Neuper and W. Klimesch (Eds.) – 2006, ISBN 978-0-444-52183-5. Volume 160: GABA and the Basal Ganglia: From Molecules to Systems, by J.M. Tepper, E.D. Abercrombie and J.P. Bolam (Eds.) – 2007, ISBN 978-0-444-52184-2. Volume 161: Neurotrauma: New Insights into Pathology and Treatment, by J.T. Weber and A.I.R. Maas (Eds.) – 2007, ISBN 978-0-444-53017-2. Volume 162: Neurobiology of Hyperthermia, by H.S. Sharma (Ed.) – 2007, ISBN 978-0-444-51926-9. Volume 163: The Dentate Gyrus: A Comprehensive Guide to Structure, Function, and Clinical Implications, by H.E. Scharfman (Ed.) – 2007, ISBN 978-0-444-53015-8. Volume 164: From Action to Cognition, by C. von Hofsten and K. Rosander (Eds.) – 2007, ISBN 978-0-444-53016-5. Volume 165: Computational Neuroscience: Theoretical Insights into Brain Function, by P. Cisek, T. Drew and J.F. Kalaska (Eds.) – 2007, ISBN 978-0-444-52823-0. Volume 166: Tinnitus: Pathophysiology and Treatment, by B. Langguth, G. Hajak, T. Kleinjung, A. Cacace and A.R. M ller (Eds.) – 2007, ISBN 978-0-444-53167-4. Volume 167: Stress Hormones and Post Traumatic Stress Disorder: Basic Studies and Clinical Perspectives, by E.R. de Kloet, M.S. Oitzl and E. Vermetten (Eds.) – 2008, ISBN 978-0-444-53140-7. Volume 168: Models of Brain and Mind: Physical, Computational and Psychological Approaches, by R. Banerjee and B.K. Chakrabarti (Eds.) – 2008, ISBN 978-0-444-53050-9. Volume 169: Essence of Memory, by W.S. Sossin, J.-C. Lacaille, V.F. Castellucci and S. Belleville (Eds.) – 2008, ISBN 978-0-444-53164-3. Volume 170: Advances in Vasopressin and Oxytocin – From Genes to Behaviour to Disease, by I.D. Neumann and R. Landgraf (Eds.) – 2008, ISBN 978-0-444-53201-5. Volume 171: Using Eye Movements as an Experimental Probe of Brain Function—A Symposium in Honor of Jean BüttnerEnnever, by Christopher Kennard and R. John Leigh (Eds.) – 2008, ISBN 978-0-444-53163-6. Volume 172: Serotonin–Dopamine Interaction: Experimental Evidence and Therapeutic Relevance, by Giuseppe Di Giovanni, Vincenzo Di Matteo and Ennio Esposito (Eds.) – 2008, ISBN 978-0-444-53235-0. Volume 173: Glaucoma: An Open Window to Neurodegeneration and Neuroprotection, by Carlo Nucci, Neville N. Osborne, Giacinto Bagetta and Luciano Cerulli (Eds.) – 2008, ISBN 978-0-444-53256-5. Volume 174: Mind and Motion: The Bidirectional Link Between Thought and Action, by Markus Raab, Joseph G. Johnson and Hauke R. Heekeren (Eds.) – 2009, 978-0-444-53356-2. Volume 175: Neurotherapy: Progress in Restorative Neuroscience and Neurology — Proceedings of the 25th International Summer School of Brain Research, held at the Royal Netherlands Academy of Arts and Sciences, Amsterdam, The Netherlands, August 25–28, 2008, by J. Verhaagen, E.M. Hol, I. Huitinga, J. Wijnholds, A.A. Bergen, G.J. Boer and D.F. Swaab (Eds.) –2009, ISBN 978-0-12-374511-8. Volume 176: Attention, by Narayanan Srinivasan (Ed.) – 2009, ISBN 978-0-444-53426-2. Volume 177: Coma Science: Clinical and Ethical Implications, by Steven Laureys, Nicholas D. Schiff and Adrian M. Owen (Eds.) – 2009, 978-0-444-53432-3. Volume 178: Cultural Neuroscience: Cultural Influences On Brain Function, by Joan Y. Chiao (Ed.) – 2009, 978-0-444-53361-6. Volume 179: Genetic models of schizophrenia, by Akira Sawa (Ed.) – 2009, 978-0-444-53430-9. Volume 180: Nanoneuroscience and Nanoneuropharmacology, by Hari Shanker Sharma (Ed.) – 2009, 978-0-444-53431-6.
278
Other volumes in PROGRESS IN BRAIN RESEARCH
Volume 181: Neuroendocrinology: The Normal Neuroendocrine System, by Luciano Martini, George P. Chrousos, Fernand Labrie, Karel Pacak and Donald W. Pfaff (Eds.) – 2010, 978-0-444-53617-4. Volume 182: Neuroendocrinology: Pathological Situations and Diseases, by Luciano Martini, George P. Chrousos, Fernand Labrie, Karel Pacak and Donald W. Pfaff (Eds.) – 2010, 978-0-444-53616-7. Volume 183: Recent Advances in Parkinson's Disease: Basic Research, by Anders Björklund and M. Angela Cenci (Eds.) – 2010, 978-0-444-53614-3. Volume 184: Recent Advances in Parkinson's Disease: Translational and Clinical Research, by Anders Björklund and M. Angela Cenci (Eds.) – 2010, 978-0-444-53750-8. Volume 185: Human Sleep and Cognition, by Gerard A. Kerkhof and Hans P.A. Van Dongen (Eds.) – 2010, 978-0-444-53702-7. Volume 186: Sex Differences in the Human Brain, their Underpinnings and Implications, by Ivanka Savic (Ed.) – 2010, 978-0-44453630-3. Volume 187: Breathe, Walk and Chew: The Neural Challenge: Part I, by Jean-Pierre Gossard, Réjean Dubuc and Arlette Kolta (Eds.) – 2010, 978-0-444-53613-6. Volume 188: Breathe, Walk and Chew; The Neural Challenge: Part II, by Jean-Pierre Gossard, Réjean Dubuc and Arlette Kolta (Eds.) – 2011, 978-0-444-53825-3. Volume 189: Gene Expression to Neurobiology and Behaviour: Human Brain Development and Developmental Disorders by Oliver Braddick, Janette Atkinson and Giorgio M. Innocenti (Eds.) – 2011, 978-0-444-53884-0. Volume 190: Human Sleep and Cognition Part II: Clinical and Applied Research, by Hans P.A. Van Dongen and Gerard A. Kerkhof (Eds.) – 2011, 978-0-444-53817-8.
