Common Mechanisms in Perception and Action (Attention and Performance (Oxford)) (v. 19)

Common Mechanisms in Perception and Action: Attention and Performance XIX Wolfgang Prinz Bernhard Hommel Editors OXFOR...

Author: Wolfgang Prinz | Bernhard Hommel

21 downloads 771 Views 34MB Size Report

This content was uploaded by our users and we assume good faith they have the permission to share this book. If you own the copyright to this book and it is wrongfully on our website, we offer a simple DMCA procedure to remove your content from our site. Start by pressing the button below!
Report copyright / DMCA form

DOWNLOAD PDF

Common Mechanisms in Perception and Action: Attention and Performance XIX

Wolfgang Prinz Bernhard Hommel Editors

OXFORD UNIVERSITY PRESS

aapa01.fm Page i Wednesday, December 5, 2001 9:14 AM

Common Mechanisms in Perception and Action

aapa01.fm Page ii Wednesday, December 5, 2001 9:14 AM

Attention and Performance Attention and Performance XIV: Synergies in Experimental Psychology, Arti1cial Intelligence, and Cognitive Neuroscience. Edited by David E. Meyer and Sylvan Kornblum, 1992 Attention and Performance XV: Conscious and Nonconscious Information Processing. Edited by Carlo Umiltà and Morris Moscovitch, 1994 Attention and Performance XVI: Information Integration in Perception and Action. Edited by Toshio Inui and James L. McClelland, 1996 Attention and Performance XVII: Cognitive Regulation of Performance: Interaction of Theory and Application. Edited by Daniel Gopher and Asher Koriat, 1998 Attention and Performance XVIII: Control of Cognitive Processes. Edited by Stephen Monsell and Jon Driver, 2000 Attention and Performance XIX: Common Mechanisms in Perception and Action. Edited by Wolfgang Prinz and Bernhard Hommel, 2002

aapa01.fm Page iii Wednesday, December 5, 2001 9:14 AM

Common Mechanisms in Perception and Action Attention and Performance XIX

edited by Wolfgang Prinz and Bernhard Hommel

This book is based on the papers presented at the Nineteenth International Symposium on Attention and Performance held at Kloster Irsee, Germany, July 16–22, 2000

aapa01.fm Page iv Wednesday, December 5, 2001 9:14 AM

Great Clarendon Street, Oxford OX2 6DP Oxford University Press is a department of the University of Oxford. It furthers the University’s objective of excellence in research, scholarship, and education by publishing worldwide in Oxford New York Auckland Bangkok Buenos Aires Cape Town Chennai Dar es Salaam Delhi Hong Kong Istanbul Karachi Kolkata Kuala Lumpur Madrid Melbourne Mexico City Mumbai Nairobi São Paulo Shanghai Singapore Taipei Tokyo Toronto and an associated company in Berlin Oxford is a registered trade mark of Oxford University Press in the UK and in certain other countries Published in the United States by Oxford University Press Inc., New York © The International Association for the Study of Attention and Performance, 2002 The moral rights of the author have been asserted Database right Oxford University Press (maker) First published 2002 All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, without the prior permission in writing of Oxford University Press, or as expressly permitted by law, or under terms agreed with the appropriate reprographics rights organization. Enquiries concerning reproduction outside the scope of the above should be sent to the Rights Department, Oxford University Press, at the address above You must not circulate this book in any other binding or cover and you must impose this same condition on any acquirer A catalogue record for this title is available from the British Library Library of Congress Cataloging in Publication Data (Data available) ISBN 0 19 851069 1 10

9

8

7

6

5

4

3

2

1

Typeset by Integra Software Services Pvt. Ltd., Pondicherry, India 605 005 www.integra-india.com Printed in Great Britain on acid-free paper by T.J. International Ltd, Padstow, Cornwall

aapa01.fm Page v Wednesday, December 5, 2001 9:14 AM

Contents Acknowledgements The Attention and Performance Symposia Authors and Participants Group Photo

ix xi xiii xxii

Editors’ introduction

1

1 Common mechanisms in perception and action Introductory remarks Wolfgang Prinz and Bernhard Hommel

3

Association lecture

7

2 Sequential effects of dimensional overlap: findings and issues Sylvan Kornblum and Gregory Stevens

9

I

Space perception and spatially oriented action

55

3 Perception and action: what, how, when, and why Introduction to Section I Glyn W. Humphreys

57

4 Several ‘vision for action’ systems: a guide to dissociating and integrating dorsal and ventral functions (tutorial) Yves Rossetti and Laure Pisella

62

5 Attention and visually guided behavior in distinct systems Bruce Bridgeman 6 How the brain represents the body: insights from neurophysiology and psychology Michael S.A. Graziano and Matthew M. Botvinick

120

136

7 Action planning affects spatial localization Jerome Scott Jordan, Sonja Stork, Lothar Knuf, Dirk Kerzel, and Jochen Müsseler

158

8 The perception and representation of human locomotion John J. Rieser and Herbert L. Pick, Jr.

177

II

195

Timing in perception and action

9 Perspectives on the timing of events and actions Introduction to Section II Jeff Summers

197

aapa01.fm Page vi Wednesday, December 5, 2001 9:14 AM

vi

Contents

10 Movement timing: a tutorial Alan M. Wing and Peter J. Beek

202

11 Timing mechanisms in sensorimotor synchronization Gisa Aschersleben, Prisca Stenneken, Jonathan Cole, and Wolfgang Prinz

227

12 The embodiment of musical structure: effects of musical context on sensorimotor synchronization with complex timing patterns Bruno H. Repp

245

13 Action, binding, and awareness Patrick Haggard, Gisa Aschersleben, Jörg Gehrke, and Wolfgang Prinz

266

III

287

Action perception and imitation

14 Processing mechanisms and neural structures involved in the recognition and production of actions Introduction to Section III Raffaella Ida Rumiati 15 Action perception and imitation: a tutorial Harold Bekkering and Andreas Wohlschläger 16 Observing a human or a robotic hand grasping an object: differential motor priming effects Umberto Castiello, Dean Lusher, Morena Mari, Martin Edwards, and Glyn W. Humphreys

289 294

315

17 Action representation and the inferior parietal lobule Vittorio Gallese, Luciano Fadiga, Leonardo Fogassi, and Giacomo Rizzolatti

334

18 Coding of visible and hidden actions Tjeerd Jellema and David I. Perrett

356

19 The visual analysis of bodily motion Maggie Shiffrar and Jeannine Pinto

381

IV

401

Content-specific interactions between perception and action

20 Content-specific interactions between perception and action Introduction to Section IV Martin Eimer 21 Motor competence in the perception of dynamic events: a tutorial Paolo Viviani 22 Eliminating, magnifying, and reversing spatial compatibility effects with mixed location-relevant and irrelevant trials Robert W. Proctor and Kim-Phuong L. Vu 23 Does stimulus-driven response activation underlie the Simon effect? Fernando Valle-Inclán, Steven A. Hackley, and Carmen de Labra 24 Activation and suppression in conflict tasks: empirical clarification through distributional analyses K. Richard Ridderinkhof

403 406

443 474

494

aapa01.fm Page vii Wednesday, December 5, 2001 9:14 AM

Contents

25 Response-evoked interference in visual encoding Jochen Müsseler and Peter Wühr

520

26 Interaction between feature binding in perception and action Gijsbert Stoet and Bernhard Hommel

538

V

Coordination and integration in perception and action

553

27 Coordination and integration in perception and action Introduction to Section V Robert Ward

555

28 From perception to action: making the connection—a tutorial Pierre Jolicœur, Michael Tombu, Chris Oriet, and Biljana Stevanovski

558

29 The dimensional-action system: a distinct visual system Asher Cohen and Uri Feintuch

587

30 Selection-for-perception and selection-for-spatial-motor-action are coupled by visual attention: a review of recent findings and new evidence from stimulus-driven saccade control Werner X. Schneider and Heiner Deubel

609

31 Response features in the coordination of perception and action Gordon D. Logan and N. Jane Zbrodoff

628

32 Effect anticipation in action planning Michael Ziessler and Dieter Nattkemper

645

33 The representational nature of sequence learning: evidence for goal-based codes Eliot Hazeltine

673

Author index Subject index

691 709

vii

aapa01.fm Page viii Wednesday, December 5, 2001 9:14 AM

This page intentionally left blank

aapa01.fm Page ix Wednesday, December 5, 2001 9:14 AM

Acknowledgements We gratefully acknowledge the 1nancial support for the Symposium received from the Max Planck Society and the Deutsche Forschungsgemeinschaft. The Symposium took place at Kloster Irsee, located in the scenic Alpine foothills of the Swabian part of Bavaria, Germany. We thank the director of Kloster Irsee, Dr Rainer Jehl and his staff for their friendly reception, the ef1cient handling of organizational matters and their helpfulness. Further, we owe thanks to Dr Horst Gundlach and his assistant Christian Paulitsch of the Institute for the History of Psychology of the University of Passau, Germany, for the small but outstanding instrument exhibition Short Time Measurement in Early Experimental Psychology. The participants thoroughly enjoyed the unusual exhibition. Above all, we owe our very special thanks to Marina von Bernhardi, a staff member of the Max Planck Institute for Psychological Research in Munich, who was in charge of the entire, extensive organization—mainly prior to and during the symposium. She did a marvelous job in making everything work very smoothly and pleasantly. There was warm praise for her efforts from the participants, to which we add our special appreciation. Competent support in technical and administrative matters was given by the Munich institute’s staff, especially Max Schreder and Klaus Hereth, as well as Peter Schönfelder. Dirk Kerzel diligently created and constantly updated the A&P site containing manuscript versions. Last not least we are particularly grateful to Heide John of the Max Planck Institute, the secretarial of1ce in charge of handling a great deal of the manuscripts, communication between authors, reviewers, editors, and the 1nal compilation of the entire manuscript, tasks that even in—or because of—the electronic age need careful attention. We express to her our gratitude for doing that job with utmost diligence and patience. All chapters based on papers at the symposium were anonymously reviewed by two other participants and went through an extensive revision process. We are indebted to the reviewers. Most of all we would like to express our gratitude to Stephen Monsell, secretary of the Association. He proved to be the Keeper of the Seal of the Association, providing us with excerpts from the bylaws or information about its rules and traditions where applicable and with advice whenever issues beyond bylaws, rules, and traditions came up. WP and BH Munich and Leiden, March 2001

aapa01.fm Page x Wednesday, December 5, 2001 9:14 AM


aapa01.fm Page xi Wednesday, December 5, 2001 9:14 AM

The Attention and Performance Symposia Since the 1rst was held in The Netherlands in 1966, the Attention and Performance Symposia have become an established and highly successful institution. They are now held every two years, in a different country. The original purpose remains: to promote communication among researchers in experimental cognitive psychology and cognate areas working at the frontiers of research on ‘attention, performance, and information processing’. The format is an invited workshop-style meeting, with plenty of time for papers and discussion, leading to the publication of an edited volume of the proceedings. The International Association for the Study of Attention and Performance exists solely to run the meetings and publish the volume. Its Executive Committee selects the organizers of the next meeting, and develops the program in collaboration with them, with advice on potential participants from an Advisory Council of up to 100 members. Participation is by invitation only, and the rules of the Association1 are constructed to ensure participation from a wide range of countries, with a high proportion of young researchers, and a substantial injection of new participants from meeting to meeting. Held usually in a relatively isolated location, each meeting has four and a half days of papers presented by a maximum of 26 speakers, plus an invited Association Lecture from a leading 1gure in the 1eld. There is a maximum of 65 participants (incl. the current members of the executive committee and the organizers). There are no parallel sessions, and all participants commit themselves to attending all the sessions. There is thus time for substantial papers followed by extended discussion, both organized and informal, and opportunities for issues and ideas introduced at one point in the meeting to be returned to and developed later. Speakers are encouraged to be provocative and speculative, and participants who do not present formal papers are encouraged to contribute actively to discussion in various ways; for example, as formal discussants, by presenting a poster, or as contributors to scheduled discussion sessions. This intensive workshop atmosphere has been one of the major strengths and attractions of these meetings. Manuscript versions of the papers are refereed anonymously by other participants and external referees and published in a high-quality volume edited by the organizers, with a publication lag similar to many journals. Unlike many edited volumes, the Attention and Performance series reaches a wide audience and has considerable prestige. Although not a journal, it is listed in journal citation indices with the top dozen journals in experimental psychology. According to the Constitution, ‘Papers presented at meetings are expected to describe work not previously published, and to represent a substantial contribution . . .’ Over the years, contributors have been willing to publish original experimental and theoretical research of high quality in the volume, and this tradition continues. A&P review papers have also been much cited. The series has attracted widespread praise in terms such as ‘unfailingly presented the best work in the 1eld’ (S. Kosslyn, Harvard), ‘most distinguished series in the 1eld of cognitive psychology’ (C. Bundesen, Copenhagen), ‘held in high esteem throughout the 1eld because of its attention to rigor, quality and scope . . . indispensable to anyone

aapa01.fm Page xii Wednesday, December 5, 2001 9:14 AM

xii

The Attention and Performance Symposia

who is serious about understanding the current state of the science’ (M. Jordan, MIT), ‘the books are an up-to-the-minute tutorial on topics fundamental to understanding mental processes’ (M. Posner, Oregon). In the early days of the Symposium, when the scienti1c analysis of attention and performance was in its infancy, thematic coherence could be generated merely by gathering together the most active researchers in the 1eld. More recently, experimental psychology has rami1ed, ‘cognitive science’ has been born, and converging approaches to the issues we study have developed in neuroscience. Participation has therefore become interdisciplinary, with neuroscientists, neuropsychologists and computational modelers joining the experimental psychologists. Each meeting now focuses on a restricted theme under the general heading of ‘attention and performance’. Recent themes include Synergies in Experimental Psychology, Arti1cial Intelligence and Cognitive Neuroscience (USA, 1990), Conscious and Unconscious Processes (Italy, 1992), Integration of Information (Japan, 1994), Cognitive Regulation of Performance: Interaction of Theory and Application (Israel, 1996), and Control of Cognitive Processes (UK, 1998).

1

For more information about the Association and previous symposia, visit the webpage http://go.to/A&P

aapa01.fm Page xiii Wednesday, December 5, 2001 9:14 AM

Authors and Participants Alan Allport Dept. of Experimental Psychology University of Oxford South Parks Road Oxford OX1 3UD, UK [email protected]

Bruce Bridgeman Psychology, Social Sciences 2 University of California Santa Cruz, CA 95064 USA [email protected]

Gisa Aschersleben Max Planck Institute for Psychological Research Amalienstr. 33 D-80799 München, Germany [email protected]

Umberto Castiello Department of Psychology Royal Holloway The University of London Egham, Surrey TW20 0EX, UK [email protected]

Peter Beek Faculty of Human Movement Sciences Vrije Universiteit Van der Boechorststraat 9 NL-1081 BT Amsterdam [email protected]

Asher Cohen Department of Psychology The Hebrew University Mount Scopus Jerusalem 91905 Israel [email protected]

Harold Bekkering Experimental and Work Psychology University of Groningen Grote Kruisstraat 2/1 NL-9712 TS Groningen [email protected] Matthew M. Botvinick Center for the Neural Basis of Cognition 115 Mellon Institute 4400 Fifth Avenue Pittsburgh, PA 15232 USA [email protected]

Jonathan Cole Dept. of Clinical Neurophysiology Poole Hospital Long2eet Road Poole, BH15 2JB UK [email protected] Laila Craighero Istituto di Fisiologia Umana Università di Parma Via Volturno, 39 I-43100 Parma, Italy [email protected]

aapa01.fm Page xiv Wednesday, December 5, 2001 9:14 AM

xiv

Authors and Participants

Shai Danziger School of Psychology University of Wales Bangor, Gwynedd LL57 2DG, UK [email protected] Jan De Houwer Department of Psychology University of Southampton High1eld Southampton SO17 1BJ, UK [email protected] Roberto Dell’Acqua Department of Psychology University of Padova 8, Via Venezia I-35131 Padova, Italy [email protected] Heiner Deubel Experimental Psychology Ludwig-Maximilians-University Leopoldstr. 13 D-80802 München, Germany [email protected] John Duncan MRC Cognition and Brain Sciences Unit 15 Chaucer Road Cambridge CB2 2EF, UK [email protected]

Luciano Fadiga Instituto di Fisiologia Umana Università di Ferrara Via Fossato di Mortara 17/19, I-44100 Ferrara Italy [email protected] Uri Feintuch Department of Psychology The Hebrew University Mount Scopus Jerusalem 91905 Israel [email protected] Leonardo Fogassi Istituto di Fisiologia Umana Università di Parma Via Volturno 39 I-43100 Parma Italy [email protected] Liz Franz Department of Psychology University of Otago Box 56 Dunedin New Zealand [email protected]

Martin Edwards Behavioral Brain Sciences Centre School of Psychology University of Birmingham Birmingham B15 2TT, UK [email protected]

Luis Fuentes Dept. of Psychology Universidad de Almería E-04120 Almería Spain [email protected]

Martin Eimer Department of Psychology Birkbeck College University of London Malet Street London WC1E 7HX, UK [email protected]

Vittorio Gallese Istituto di Fisiologia Umana Università di Parma Via Volturno, 39 I-43100 Parma Italy [email protected]

aapa01.fm Page xv Wednesday, December 5, 2001 9:14 AM


Jörg Gehrke Max Planck Institute for Psychological Research Amalienstr. 33 D-80799 München, Germany Daniel Gopher Industrial Engineering and Management Technion Haifa, 32000 Israel [email protected] Michael Graziano Psychology Department Princeton University 1-E-14 Green Hall Princeton, NJ 08544 USA [email protected] Steven Hackley University of Missouri Department of Psychological Sciences 210 McAlester Hall Columbia, MO 65211 USA [email protected] Patrick Haggard Institute of Cognitive Neuroscience Department of Psychology University College London 17 Queen Square London WC1N 3AR, UK [email protected] Eliot Hazeltine NASA Ames Research Center MS 262-4 Moffett Field, CA 94035 USA [email protected]

Bernhard Hommel Section of Experimental and Theoretical Psychology University of Leiden P.O. Box 9555 NL-2300 RB Leiden [email protected] Glyn Humphreys Behavioural Brain Sciences Centre School of Psychology University of Birmingham Birmingham B15 2TT, UK [email protected] Tjeerd Jellema Helmholtz Research Institute Utrecht University Heidelberglaan 2 NL-3584 CS Utrecht [email protected] Pierre Jolicœur Department of Psychology University of Waterloo Waterloo, Ontario N2L 3G1 Canada [email protected] Jerome Scott Jordan Dept. of Psychology Illinois State University Campus Box 4620 Normal, IL 61790-4620 USA [email protected] Nancy Kanwisher NE20-454 MIT Dept. of Brain and Cognitive Sciences 77 Mass Ave. Cambridge, MA 02138 USA [email protected]

xv

aapa01.fm Page xvi Wednesday, December 5, 2001 9:14 AM

xvi


Mitsuo Kawato Dept. 3, ATR HIP Labs Hikaridai 2-2 Seikacyo, Sorakugun Kyoto 619-0288 Japan [email protected]

Carmen de Labra Departamento de Psicología Universidad de La Coruña Elviña E-15071 La Coruña Spain [email protected]

Dirk Kerzel Max Planck Institute for Psychological Research Amalienstr. 33 D-80799 München, Germany [email protected]

Susan Lederman Dept. of Psychology Queen’s University Kingston, Ontario Canada K7L 3N6 [email protected]

Lothar Knuf Max Planck Institute for Psychological Research Amalienstr. 33 D-80799 München, Germany [email protected]

Hartmut Leuthold Department of Psychology 58 Hillhead Street Glasgow G12 8QB Scotland UK [email protected]

Sylvan Kornblum Mental Health Research Institute The University of Michigan 205 Zina Pitcher Place Ann Arbor, MI 48109-0720 USA [email protected]

Gordon Logan Department of Psychology Vanderbilt University Nashville, TN 37240 USA [email protected]

Ralf Krampe Max Planck Institute for Human Development Center for Lifespan Psychology Lentze-Allee 94 D-14195 Berlin, Germany [email protected]

Dean Lusher Department of Psychology The University of Melbourne Parkville 3052 Victoria, Australia [email protected]

Wilfried Kunde Lehrstuhl für Psychologie III Universität Würzburg Röntgenring 11 D-97070 Würzburg, Germany [email protected]

Morena Mari Autism Unit Maggiore Hospital Bologna Italy [email protected]

aapa01.fm Page xvii Wednesday, December 5, 2001 9:14 AM


Nachshon Meiran Department of Behavioural Sciences Ben-Gurion University of the Negev Beer-Sheva, 84105 Israel [email protected]

David Perrett School of Psychology St Andrews University St Andrews, Fife, KY 16 9JU Scotland, UK [email protected]

Stephen Monsell University of Exeter School of Psychology Washington Singer Laboratories Exeter EX4 4QG, UK [email protected]

Herbert L. Pick jr. Institute of Child Development 55 East River Road University of Minnesota Minneapolis, MN 55455 USA

Jochen Müsseler Max Planck Institute for Psychological Research Amalienstr. 33 D-80799 München, Germany [email protected] Dieter Nattkemper Institut für Psychologie Humboldt Universität zu Berlin Oranienburger Str. 18 D-10178 Berlin, Germany Dieter.nattkemper@psychologie. hu-berlin.de Kevin O’Regan Laboratoire de Psychologie Expérimentale Centre Universitaire de Boulogne 71, ave. E. Vaillant F-92774 Boulogne-Billancourt Cdx. [email protected] Chris Oriet Department of Psychology University of Waterloo Waterloo, Ontario N2L 3G1 Canada Giuseppe di Pellegrino Centre for Cognitive Neuroscience University of Wales Bangor LL57 2AS UK [email protected]

Jeannine Pinto Department of Psychology Pardee Hall Lafayette College Easton, PA 18042 USA [email protected] Laure Pisella Neuropsychologie Cognitive Unité 534—Espace et Action INSERM—Institut National de la Santé et de la Recherche Médicale 16 avenue Lépine F-69676 Bron France [email protected] Mary C. Potter Department of Brain and Cognitive Sciences Massachussetts Institute of Technology Cambridge, MA 02139 USA [email protected] Wolfgang Prinz Max Planck Institute for Psychological Research Amalienstr. 33 D-80799 München Germany [email protected]

xvii

aapa01.fm Page xviii Wednesday, December 5, 2001 9:14 AM

xviii


Robert Proctor Psychological Sciences Purdue University 1364 Psychology Building West Lafayette, IN 47907-1364 USA [email protected]

David A. Rosenbaum Department of Psychology 642 Moore Building Pennsylvania State University University Park, PA 16802-3104 USA [email protected]

Bruno M. Repp Haskins Laboratories 270 Crown Street New Haven, CT 06511-6695 USA [email protected]

Yves Rossetti Neuropsychologie Cognitive Unité 534—Espace et Action INSERM—Institut National de la Santé et de la Recherche Médicale 16 avenue Lépine F-69676 Bron, France [email protected]

Richard Ridderinkhof Department of Psychology University of Amsterdam Roeterstraat 15 1018 WB Amsterdam The Netherlands [email protected] M. Jane Riddoch Behavioural Brain Sciences Centre School of Psychology University of Birmingham Birmingham, B15 2TT, UK [email protected] John Rieser Psychology and Human Development Box 512, Peabody Vanderbilt University Nashville, TN 37203 USA [email protected] Giacomo Rizzolatti Instituto di Fisiologia Umana Università di Parma Via Volturno 39 I-43100 Parma Italy [email protected]

Raffaella Rumiati Cognitive Neuroscience Sector Scuola Internazionale Superiore di Studi Avanzati Via Beirut, n. 2-4 I-34014 Trieste, Italy [email protected] Werner X. Schneider Experimental Psychology Ludwig-Maximilians-University Leopoldstr. 13 D-80802 München, Germany [email protected] Maggie Shiffrar Department of Psychology Rutgers University 101 Warren Street Newark, NJ 07102, USA [email protected] Jeroen Smeets Vakgroep Fysiologie Erasmus University of Rotterdam P.O. Box 1738 NL-3000 DR Rotterdam [email protected]

aapa01.fm Page xix Wednesday, December 5, 2001 9:14 AM


Michael Spivey Department of Psychology Cornell University Ithaca, NY 14853 USA [email protected] Prisca Stenneken Max Planck Institute for Psychological Research Amalienstr. 33 D-80799 Munich Germany [email protected]

Jeff Summers School of Psychology University of Tasmania GPO Box 252-30 Hobart, Tasmania 7001 Australia [email protected] Michael Tombu Department of Psychology University of Waterloo Waterloo, Ontario N2L 3G1 Canada

Biljana Stevanovski Department of Psychology University of Waterloo Waterloo, Ontario N2L 3G1 Canada

Carlo Umiltà Dept. of General Psychology University of Padova Via Venezia, 8 I-35131 Padova, Italy [email protected]

Gregory Stevens Department of Psychology University of California – Los Angeles 1285 Franz Mall Los Angeles, CA 90095-1563 USA [email protected]

Fernando Valle-Inclán Department of Psychology University of La Coruña Campus Elviña E-15071 La Coruña Spain [email protected]

Gijsbert Stoet Washington University School of Medicine 660 South Euclid Avenue Saint Louis, MO 63110 USA [email protected]

Paolo Viviani Faculty of Psychol. and Educational Sciences University of Geneva 40, Boulevard du Pont d’Arve CH-1205 Geneva Switzerland [email protected] and Faculty of Psychology UHSR University 58, via Olgettina I-20132 Milan Italy [email protected]

Sonja Stork Max Planck Institute for Psychological Research Amalienstr. 33 D-80799 München Germany [email protected]

xix

aapa01.fm Page xx Wednesday, December 5, 2001 9:14 AM

xx


Kim-Phuong Vu Psychological Sciences Purdue University 1364 Psychology Building West Lafayette, IN 47907-1364, USA [email protected] Robert Ward Centre for Cognitive Neuroscience University of Wales Bangor LL57 2AS, UK [email protected] Alan Wing Behavioural Brain Sciences Centre School of Psychology University of Birmingham Birmingham, B15 2TT, UK [email protected] Andreas Wohlschläger Max Planck Institute for Psychological Research Amalienstr. 33 D-80799 München, Germany [email protected]

Peter Wühr Max Planck Institute for Psychological Research Amalienstr. 33 D-80799 München Germany [email protected]

N. Jane Zbrodoff Department of Psychology Vanderbilt University Nashville, TN 37240 USA [email protected]

Michael Zießler University of Sunderland Business School Dept. of Psychology St. Peter’s Campus Sunderland SR6 0DD UK [email protected]

aapa01.fm Page xxi Wednesday, December 5, 2001 9:14 AM


aapa01.fm Page xxii Wednesday, December 5, 2001 9:14 AM

5

2 1

3

8 7

4 6

19

14 17 9 16 13 10

31 30 29

20

39 38

21

15

22

40

33

28 27

11

52

32

34

45

42

53 51

41

37

46

58

47 44

59

54

50

26

12 23

55

35

24

36

25

43 48

61

49

60 56 57

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

Hartmut Leuthold Wolfgang Prinz Patrick Haggard Alan Allport Gijsbert Stoet Mitsuo Kawato Michael Zießler John Duncan Ralf Krampe Elizabeth Franz Raffaella Rumiati Laila Craighero David Perrett Asher Cohen Shai Danziger Mary C. Potter

17 18 19 20 21 22. 23 24 25 26 27 28 29 30 31 32

Kevin O’Regan Stephen Monsell Robert Proctor Umberto Castiello Jochen Müsseler Gordon Logan Nachshon Meiran Luis Fuentes Paolo Viviani Carlo Umiltà Robert Ward Bruno Repp Jeff Summers Peter Beek J. Scott Jordan Martin Eimer

33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

Fernando del’ Valle-Inclán Vittorio Gallese Daniel Gopher Sylvan Kornblum Bruce Bridgeman Eliot Hazeltine Michael Graziano John Rieser Roberto Dell’Acqua Werner X. Schneider Susan Lederman Andreas Wohlschläger Jeroen Smeets Glyn Humphreys Pierre Jolicœur Nancy Kanwisher

49 50 51 52 53 54 55 56 57 58 59 60 61

Yves Rossetti Gregory Stevens M. Jane Riddoch Harold Bekkering Bernhard Hommel Maggie Shiffrar Michael Spivey David Rosenbaum Gisa Aschersleben Jan De Houwer Wilfried Kunde Alan Wing Richard Ridderinkhof

aapa01.fm Page xxiii Wednesday, December 5, 2001 9:14 AM

18

aapa01.fm Page xxiv Wednesday, December 5, 2001 9:14 AM


aapc01.fm Page 1 Wednesday, December 5, 2001 9:17 AM

Editors’ introduction




1

Common mechanisms in perception and action Introductory remarks Wolfgang Prinz and Bernhard Hommel

The contributions to this volume discuss a classical theme in human-performance research, and they do so under some new perspectives that have emerged in recent years. The classical theme refers to the interplay between perception and action—a theme that is, and has ever been, one of the core issues in the 1eld of attention and performance. For instance, in the classical work inspired by linear stage theory, notions like stimulus–response translation and/or response selection have been introduced to account for putative operations underlying the transition from stimulus- to response-related processing, and a number of factors affecting these operations have been identi1ed. Yet, despite their supposed central function, the study of translation mechanisms has always played a somewhat marginal role. Instead, research has tended to emphasize stimulus- over response-related processing, and there has been rather little interest in response-related processing mechanisms and the way they are linked to those dealing with stimulus information. In recent years, some new perspectives have emerged, suggesting both structural diversity and functional coherence in the interplay between perception and action. On the one face there is now substantial evidence from a large variety of neuroscience studies supporting diversity in the sense that interaction between perception and action may be going on in parallel in a number of pathways, and a variety of maps or modules for special computational purposes may be involved. On the other face there is also much evidence supporting a substantial degree of functional coherence within these modules—in the sense of questioning the classical separation between sensory, or stimulusrelated processing and motor, or response-related processing and calling for more overlap and integration between the two. Surprising interactions between perception and action have been observed in a number of both behavioral and neuroscience studies indicating that input and output may draw on tightly coupled, or perhaps even identical representations. At the same time, new theoretical frameworks and models have been proposed to meet the challenges inherent in these observations and account for these interactions, for example, in terms of shared mechanisms that draw on common representational resources. The aim of this volume is to gather these various approaches in an attempt to focus on structural and functional aspects of the architecture mediating between stimulus- and response-related processing, with an emphasis on both diversity due to its modular organization and coherence due to common mechanisms within modules.


4

Common mechanisms in perception and action

The chapters in this volume are based on oral contributions to: ‘Attention and Performance, XIX: Common Mechanisms in Perception and Action’, a symposium held on behalf of the International Association for the Study of Attention and Performance (IASAP) at Kloster Irsee, Bavaria, Germany from July 16 to 22, 2000. At every Attention and Performance symposium it is customary to honor an eminent researcher’s distinguished contribution to the 1eld by an invitation to give the Association Lecture. For this symposium, IASAP’s Executive Committee invited Sylvan Kornblum to deliver the Association Lecture. In his Lecture, Kornblum presents evidence on interactions between stimulus–response compatibility and sequential order in choice-reaction tasks and discusses their implications for his Dimensional Overlap model. The Association Lecture is followed by 1ve sections devoted to different domains and forms of interactions between perception and action. The 1rst two sections are concerned with the classical domains of space and time, where issues related to the interplay between perception and action have long been topical. The third section deals with action perception and imitation, which has recently attracted converging attention in developmental and cognitive psychology, neurophysiology as well as clinical neuropsychology. The last two sections then address various forms of interaction between perception and action, partly going from input to output, partly taking the reverse perspective, and partly dealing with their integration. Since each section comes with an introductory overview of its own, we can here be very brief in sketching the varieties of the themes discussed in the sections and the underlying theoretical issues that link them together. Section I considers Space perception and spatially oriented action. Spatially adapted behavior requires reliable, high-precision alignment of perceptual space and action space. Traditional theories in this domain have therefore tended to invoke underlying representational structures that act as a common representational basis for both space perception and spatially oriented action. Over the past two decades this view has become increasingly challenged by clinical and experimental evidence suggesting parallel pathways and multiple maps in the brain as well as related dissociations between perception and action in behavioral performance. The contributions to this section present new evidence on the dialectic relationship between the diversity of pathways and maps and the functional unity of perception and action in space perception and spatially adapted behavior. Section II considers Timing in perception and action. Time is a dimension underlying both actorindependent events in the environment and actions, that is, actor-generated body movements. Like in the spatial domain, adaptive behavior requires high-precision alignment of the timing of actions to the timing of events, suggesting a common representational basis for events and actions and shared mechanisms for their timing. Moreover, since any representational operation and any neural activity carrying such operation is, in itself too, extended in time, the dimension of time has often been considered special in the sense that the representation of time is isomorphically grounded in the time of representation. Accordingly, the contributions to this section address mechanisms of timing and sequencing in perception and action as well as relationships between the representation of time and the timing of representational operations. Section III considers Action perception and imitation. Issues of action perception and imitation have recently become topical in neurophysiology, brain imaging, human development, and human performance. Studies from these 1elds differ considerably with respect to scope and aims, ranging from single-cell-based mechanisms involved in afferent and efferent processing to high-level mechanisms subserving the construction of mental selves. However, they do converge in suggesting close couplings, and even a considerable degree of equivalence between perceiving and producing actions. Correspondingly, processing theories in this 1eld tend to invoke shared representational


Common mechanisms in perception and action: introductory remarks

structures for information from different modalities (e.g. vision and proprioception) or for representations of more abstract features shared by perceived and produced actions. The contributions to this section discuss new 1ndings from various approaches to action perception and imitation and assess their implications for understanding the underlying representational structures and processing mechanisms. Section IV considers Content-specific interactions between perception and action. Research on stimulus–response compatibility plays an increasingly important role in providing insights into both the processes and cognitive structures underlying the relationship between perception and action planning; into how stimulus and response codes are formed; how they speak to each other; and how their interactions change in the course of practice. Progress has also been made in tapping into the temporal dynamics of stimulus and response coding, and of the interactions between those codes, especially by applying increasingly sophisticated data-analysis techniques and by including psychophysiological measurements. Interestingly, recent investigations have revealed that stimulus– response relations do not only affect action planning, but perception as well. In fact, planning an action sometimes facilitates, sometimes interferes with, and sometimes even changes the perception of stimulus events, depending on the speci1c relation between planned action and perceived stimulus. The contributions to this section sketch the emerging picture of perception and action as the outcome of a dynamic interplay between content-speci1c codes, rather than a unidirectional 2ow of information from stimulus processing to motor execution. Section V considers Coordination and integration in perception and action. Perceptual and action-related structures and processes are tightly coupled and coordinated, and in several cases they share cognitive resources. However, resource sharing creates all sorts of capacity bottlenecks and binding problems—problems that are revealing with respect to how stimulus and response information is organized, integrated, and coordinated. Aspects of stimulus–response (or response–effect) coordination and integration have gained attention only recently, but both empirical evidence and theoretical insights are growing steadily. Accordingly, the contributions to this section provide a colorful but nicely converging overview of basic principles governing the integration of perception and action, and of actions and action goals, in and across stimulus and response processing, manual action and eye movements, and perception–action sequences.

5




Association lecture




2 Sequential effects of dimensional overlap: findings and issues Sylvan Kornblum and Gregory Stevens

Abstract. We begin this chapter by outlining some of the basic principles of the dimensional overlap (DO) model (Kornblum, Hasbroucq, and Osman 1990; Kornblum, Stevens, Whipple, and Requin 1999), spelling out how these principles generate a taxonomy of tasks, and showing how, based on these principles, the structure of four of these tasks can be represented by a common processing architecture, and performance with them accounted for. We then consider the effects of stimulus and response repetitions in choice reaction time (RT) tasks and the in2uence that DO has on this repetition effect. We report data from four experiments that demonstrate this in2uence with a prime-probe, trial pair procedure in which the relevant or irrelevant stimuli in either or both trials of the pair have DO and, in the case of relevant DO, repeat either physically or conceptually. The DO model is able to account for the results by postulating that the information requirements on repeated trials are less than on non-repeated trials. We call this the Information Reduction Hypothesis. When the relevant stimuli overlap, the repetition effects are accounted for by a reduction in either the stimulus and/or the response thresholds. When the irrelevant stimuli overlap, the repetition effects are accounted for by a reduction in the time needed to distinguish between relevant and irrelevant stimuli. Thus, depending on whether the relevant or irrelevant stimulus dimension has DO, one or the other of two parameters in the DO model is modi1ed, contingent on the occurrence of a repetition. Simulations, based on this implementation of the hypothesis in the DO model, 1t the experimental results well.

2.1 Introduction Thirty years ago, at the fourth International Symposium on Attention and Performance, one of us presented a tutorial on sequential effects in choice reaction time (RT) (Kornblum 1973). Ten years ago we published the initial version of the dimensional overlap (DO) model in which we addressed, what we viewed as, some of the basic issues in stimulus–stimulus (S–S) and stimulus–response (S–R) compatibility (Kornblum, Hasbroucq, and Osman 1990). In this chapter we would like to bring these two problem areas together theoretically and empirically. As will be evident, even though this effort has resulted in modest successes it has also uncovered some interesting problems that remain to be solved. This is roughly how the chapter is organized:

• We start with a brief description of the computational version of the DO model (Kornblum, Stevens, Whipple, and Requin 1999);

• this is followed by a set of experiments in which we look at basic sequential effects in tasks with and without DO between relevant stimuli and responses;

• we then present the DO model’s account of those results;


10


• this is followed by a second set of experiments in which we take a further look at sequential effects in tasks with and without DO between the relevant and irrelevant stimuli, and between the irrelevant stimuli and the responses;

• we then present the DO model’s account for those results; • we end with a summary and conclusions.

2.2 The dimensional overlap model 2.2.1 Representational component From the very outset, we have always made a sharp distinction between the representational and the processing parts of the model (see Kornblum et al. 1990). The representational component of a theory spells out how the phenomena to be explained are to be described and abstracted; the processing component speci1es a set of possible mechanisms that might account for these observations.1 At the heart of the representational component of the DO model is the notion of dimensional overlap (DO). This is de1ned as the degree to which stimulus and/or response sets are perceptually, conceptually, or structurally similar. Dimensional overlap is, therefore, an attribute of the mental representations of sets, and patterns of DO de1ne certain task properties. We have used these dimensional relationships as the basis of a taxonomy which, up to now, has identi1ed eight unique types of compatibility tasks (see Kornblum et al. 1999, for the most recent version of this taxonomy). In a poster shown at this meeting, Stevens (Stevens and Kornblum 2000) has extended this representational aspect of the model to include response effects and ends up with a taxonomy of over a dozen tasks. He also presents the results of simulations that demonstrate the critical role that DO and the patterns of dimensional relationships play in the functional interpretation of response effects. In this chapter, we shall focus on just four of these tasks. A task in which the set of relevant stimuli, or features, does not have DO with either the set of responses or with the set of irrelevant stimuli, or features, we call a Type 1 task. This is the basic choice RT task in which the relevant stimuli could, for example, be color patches presented in different shapes that are irrelevant, and the responses are key presses. In the context of S–R compatibility, this is a neutral task for which, in principle, any stimulus–response pairing is as good as any other pairing (see Fig. 2.1). When the DO is between the set of relevant stimuli and the set of responses, we call it a Type 2 task (e.g. Fitts and Seeger 1953). In the literature this is often referred to as a straightforward ‘stimulus–response compatibility’ (SRC) task. Depending on the S–R mapping rule, the individual stimuli in such tasks either do or do not match the responses; we call this S–R relation ‘stimulus–response (S–R) congruence’ (see Fig. 2.1). When the overlap is between the set of irrelevant stimuli and the set of responses, we call it a Type 3 task (see Kornblum and Lee 1995). When the overlapping dimension is spatial the literature refers to it as a ‘Simon task’ (see Simon 1990). We often refer to Type 3 tasks as ‘Simon-like’ when the irrelevant dimension is non-spatial. Because of the pattern of overlap, individual irrelevant stimuli are either consistent or inconsistent with the responses; we call this property ‘stimulus–response (S–R) consistency’ (see Fig. 2.1). When the overlap is between the set of relevant and irrelevant stimuli, it is a Type 4 task (see Keele 1967; Kornblum 1994). When the overlapping dimension is color, the literature often refers to it as a ‘Stroop’ task. This, we believe is an error that leads to confusion. ‘Stroop-like’ task, which is also


Sequential effects of dimensional overlap: findings and issues

Fig. 2.1 Dimensional relationships between relevant stimuli, irrelevant stimuli and responses that characterize 1ve of the eight tasks in the current DO taxonomy. Whenever any two aspects of a task have dimensional overlap they are joined by a line indicating the nature (S–S or S–R) and value (+/–) of the consistency or congruence relationship between them. Horizontal and vertical striations in the stimulus rectangles depict blue and green color patches respectively.

11


12


Table 2.1 Five of the eight task types in the DO taxonomy with indications in columns 2, 3, and 4 of the locus of overlap2 Task type

Overlapping relevant stimulus and response

Overlapping irrelevant stimulus and response

Overlapping irrelevant and relevant stimulus

#1 Neutral #2 SRC #3 Simon #4 Stroop-like #8 Stroop

No Yes No No Yes

No No Yes No Yes

No No No Yes Yes

often used, seems more accurate. The important criterion is that the irrelevant stimulus dimension overlap with the relevant stimulus dimension, and that this be the only overlap in the task. Because of the pattern of overlap, irrelevant stimuli on particular trials are either consistent or inconsistent with the relevant stimuli; we call this property ‘stimulus–stimulus (S–S) consistency’ (see Fig. 2.1). When DO is between the sets of irrelevant and relevant stimuli as well as the set of responses, and the dimension is the same, we call it a Type 8 task. In the literature, when that dimension is color, it is usually referred to as a ‘Stroop’ task—correctly this time (see McLeod 1991; Stroop 1935). Because of the pattern of overlap, the mapping instructions can be either congruent or incongruent; moreover, when the mapping is congruent, the individual irrelevant stimuli are consistent or inconsistent with both the relevant stimuli and the responses, which leads to a serious confounding. We have shown that these factors can be unconfounded by using incongruent S–R mapping with Stroop tasks (Zhang and Kornblum 1998; but see also Stevens and Kornblum 2001). Because we will be using the DO terminology throughout this article we have summarized it in Table 2.1. If these taxonomic classes have any functional signi1cance at all, then one would expect all tasks in the same taxonomic category to show the same pattern of effects regardless of the particular stimuli or responses used—and this, for the most part, has been veri1ed by the results of many studies in the literature (for a review, see Kornblum 1992). Based purely on this representational scheme, the DO model asserts that RT is generally faster for consistent than for inconsistent conditions, and the RT for congruent mapping is faster than for incongruent mapping. Differences in the magnitude of these effects occur between tasks, of course; most of these may attributed to differences in the degree of DO between sets.

2.2.2 The processing component The processing part of the model is where we have been proposing, what seemed to us, plausible sets of mechanisms that might underlie the compatibility effects observed in the family of tasks encompassed by the representational part of the model. Ten years ago, the model started out as a boxology. However, this was recently replaced by a connectionist architecture (Kornblum et al. 1999) where processing takes place in a system of interconnected modules, arranged in two layers: a stimulus layer and a response layer (see Fig. 2.2). Each stimulus and response module represents a dimension, or class of features. Within each module are individual units that represent the individual features of the stimulus or response. The activation of a unit within a module, therefore, represents activation of a feature along that dimension.



Fig. 2.2 The three generic processing modules of a task (relevant stimuli, irrelevant stimuli, and responses) and the possible positive connections between them, according to the DO model. Negative connections are not shown (but see text).

2.2.2.1 Architecture and connectivity The connections between modules are of two types: automatic and controlled. Automatic lines, which have also been called Long Term Memory (LTM) connections (Barber and O’Leary 1997), connect modules that represent overlapping dimensions. These could both be stimulus dimensions, or one could be a response and the other a stimulus dimension—relevant or irrelevant. Controlled lines, which have also been called Short Term Memory (STM) connections (Barber and O’Leary 1997), are speci1ed by the task instructions instead of by the DO. They connect each unit in the relevant stimulus module with the correct unit in the response module. The strength of the signal sent over the automatic lines is a function of the level of stimulus activation, weighted by the degree of DO between the pair of connected modules. Because the activation level of the stimulus unit changes over time, the signal sent over the automatic lines changes over time as well, and is thus continuous. In contrast, the signal that is sent over the controlled lines is all or none, and may be said to represent a binary decision (for details see Kornblum et al. 1999). These simple architectural principles can be used to represent each of the tasks that we have described thus far (these are all illustrated in the ‘architecture’ column of Fig. 2.3). The 1rst task is a Type 1, neutral, task in which the relevant stimuli are color patches mapped onto left and right keypress responses. Because there is no DO in this task, there are no automatic connections. Controlled lines connect the relevant stimulus units to their assigned response units (see Fig. 2.3). Next is a Type 2 task, in which color stimuli are mapped onto color-name responses. As is true in Type 1 tasks, the controlled lines connect the relevant stimulus and response units in accordance with the task instructions. However, because of the dimensional overlap between the set of relevant stimuli and the set of responses, automatic lines also connect the relevant stimulus units to the response units. Whenever two modules represent overlapping dimensions, positive automatic lines connect corresponding units, and negative automatic lines connect non-corresponding units. Only the positive connections are shown in the 1gure. When the mapping instructions are congruent, both the automatic and the controlled lines connect each stimulus unit to its matching response unit (see

13


14


Fig. 2.3—Type 2, congruent mapping). In effect, then, each correct response unit receives two positive inputs: one from the controlled line, the other from the automatic line. When the mapping instructions are incongruent (see Fig. 2.3—Type 2, incongruent mapping), each correct response unit receives one positive input from the controlled line, and one negative input from the automatic line. As a result, the total net input to the correct response unit is less than in the congruent case. The same general rules apply to Type 3, Simon-like, tasks. Here, controlled lines connect the relevant stimulus units to their assigned response units; and, because the DO is between the irrelevant stimuli and the responses, positive automatic lines connect the irrelevant stimulus units to their corresponding response units, with negative connections between non-corresponding units, not shown here (see Fig. 2.3). Similarly for Type 4, Stroop-like, tasks: controlled lines connect the relevant stimulus and response units; with positive automatic connections between the irrelevant stimulus and their corresponding relevant stimulus units, and negative connections between non-corresponding irrelevant and relevant stimulus units (see Fig. 2.3). To get a clearer picture of how this architecture and pattern of connectivity works in processing information we need to spend a brief moment on the details of activation in individual units.

2.2.2.2 Activation and information 2ow According to the model, inputs to both the relevant and irrelevant stimulus units start at the same value—say 1. The input to the relevant unit remains at one. The input to the irrelevant unit starts decaying at a 1xed rate shortly after onset. The time (identi1ed by the parameter τ in Kornblum et al. 1999) between when these two inputs begin and when their values start to diverge is the time the system takes to distinguish between the relevant and the irrelevant input.3 Whether one believes that attention remains focused on the relevant input and is withdrawn from the irrelevant input after this distinction is made, or that the irrelevant input just gradually decays away, is not a question that we deal with in this paper. Suf1ce it to say that this decrease in the irrelevant stimulus input is a critical property of the model that enables it to account for the time-course and distributional properties of reaction times in S–S and S–R consistency tasks (see Kornblum et al. 1999). Given these two sources of input, the activation levels in the relevant and irrelevant stimulus units change over time according to a gradual, time-averaging activation function. Given a constant input, as in the case of the relevant stimulus unit, activation gradually increases and asymptotically approaches the input level. With a decreasing input, as in the case of the irrelevant stimulus unit, activation is an inverted U-shaped function of time (see Kornblum et al. 1999 for the details). Now let us examine the actual 2ow of information over time for each of the tasks that we have listed (these are illustrated in the ‘activation’ column in Fig. 2.3). The three fundamental steps to keep in mind are:

• the stimulus is presented; • the input is turned on; and • activation accumulates. Consider a Type 1, neutral, task 1rst. The information 2ow in this task can be thought of as a baseline, or generic activation 2ow, as speci1ed by the model (see Fig. 2.3). It represents the simplest instance of the three basic steps: the stimulus is presented causing the input to the relevant stimulus unit to turn on which, in turn, causes activation in the relevant stimulus unit to start accumulating



Fig. 2.3 Processing architecture and activation patterns for 2-choice tasks (used for illustrative purposes only) of Types 1, 2, 3, and 4 showing the congruent/incongruent cases for Type 2, and the consistent/inconsistent cases for Types 3 and 4 tasks. Whenever two modules represent overlapping dimensions, positive automatic lines connect corresponding units, and negative automatic lines connect non-corresponding units. Only the positive connections are shown in this 1gure. The rectangles in the architecture column represent modules, the circles represent features. Shaded circles indicate activated feature units. Horizontal and vertical striations in the stimulus rectangles depict blue and green color patches, respectively. The vertical dotted line in the activation column marks the combined duration of the stimulus and response units for the neutral, Type 1 task; this is included for purposes of comparison. (*) This curve depicts the decaying, relevant stimulus activation value after it has reached threshold. (**) This curve depicts the level of activation for the irrelevant stimulus (see also Fig. 2.4).

15


16


until it reaches threshold. When this threshold has been reached, it indicates that the stimulus has been fully identi1ed. At this point, and not before, the controlled line sends a discrete ‘on’ signal (equal 1) from the relevant stimulus to the correct response unit. Because this input is 1, activation in the response unit accumulates in exactly the same fashion as it did in the relevant stimulus unit. Once it reaches its threshold, the response is considered fully selected, and the overt response is initiated. Because of the existence of a threshold in both the relevant stimulus and the response units, and because the controlled line sends a discrete ‘on’ signal from the stimulus to the correct response unit, reaction times can be partitioned into two distinct, stage-like, intervals: a stimulus identi1cation time, and a response selection time. This discrete characteristic was present in the boxology (Kornblum et al. 1990), and is retained in the PDP version of the model. The total time taken from the moment the relevant stimulus is presented to when the response unit reaches threshold, is de1ned in the model as the RT, and is what the model simulates as the RT. As will be apparent, the activation patterns in all the other tasks in the taxonomy are modi1cations of this basic pattern. Consider a Type 2 task next. Processing in the stimulus module is, of course, identical to what it was for the Type 1 task. Stimulus is presented, input is turned on, activation accumulates, and threshold is reached in the relevant stimulus unit (see Fig. 2.3). Once the stimulus activation reaches this threshold, it is no longer needed and starts decaying back to zero. This decay was utterly inconsequential in Type 1 tasks, because the only signal being sent from the stimulus to the response unit was the ‘on’ signal on the controlled line. That same ‘on’ signal is now being sent along the controlled line in the Type 2 task as well. However, because of the DO between the relevant stimulus set and the response set, an automatic signal is also being sent from the stimulus unit to the response unit. The strength of that signal, you will recall, is proportional to the amount of activation in the stimulus unit, so that as activation level changes, so does the strength of that signal. When the mapping instructions are congruent, the positive automatic signal goes to the same response unit that is getting the ‘on’ signal via the controlled line. Thus, even though activation in the stimulus unit is decaying, the total positive input to the response unit is high, and activation in that unit accumulates quickly. When mapping is incongruent, stimulus processing remains the same. However, the response unit that is getting the ‘on’ signal via the controlled line is now connected to the stimulus unit by a negative automatic line. As a result, the activation in the stimulus unit is subtracted from the total input to the response unit, making its activation accumulate more slowly. This, of course, increases the total reaction time which, when compared to the RT with congruent mapping, is the mapping effect or, as it is known in the literature, ‘the SR compatibility effect’. We now come to tasks in which there is dimensional overlap between the irrelevant stimulus dimension and some other dimension of the task, either the response or the relevant stimulus (see also Kornblum et al. 1999, Fig. 7). Type 3 tasks are those in which the irrelevant stimulus dimension overlaps with the response. The stimulus identi1cation stage in these tasks is no different from what it is in the two tasks that we just 1nished discussing (Types 1 and 2). However, simultaneously with the presentation of the relevant stimulus, we are now also presenting an irrelevant stimulus. The basic three-step activation process is still in place, but is now modi1ed to take this new fact into account: the stimulus is presented; inputs to both relevant and irrelevant stimulus units are turned on, and activation accumulates in both the relevant and irrelevant stimulus units. Because there is no DO between the relevant and irrelevant stimuli, the irrelevant stimulus has no in2uence on processing in the relevant stimulus unit. So, when relevant stimulus activation reaches threshold, the controlled line sends its discrete ‘on’ signal to the correct response unit, just as it did in the Type 1 neutral task.



Fig. 2.4 A: Input to the relevant and irrelevant stimulus units shown as a function of time. The irrelevant input starts decreasing after a duration (τ). B: Activation functions in the relevant and irrelevant stimulus units shown as a function of time.

However, there is DO between the irrelevant stimuli and the responses, which means that the irrelevant stimulus units have automatic connections to the response units. On S–R consistent trials, these automatic connections with the correct response units are positive, which means that activation in the correct response unit gets a boost, and accumulates more rapidly, thus reaching threshold sooner (see Fig. 2.3). On S–R inconsistent trials, these automatic connections between the irrelevant stimuli and the correct responses are negative. The net effect of this is to slow the rate of accumulation of activation for the correct response, thus causing it to reach threshold more slowly. It is this processing difference between S–R consistent and S–R inconsistent trials that, according to our model, generates the S–R consistency, or Simon, effect. Exactly the same argument holds for Type 4 tasks when the overlap is between the relevant and irrelevant stimuli: the stimulus is presented; inputs to both relevant and irrelevant stimulus units are turned on; activation accumulates in both the relevant and irrelevant stimulus units; but now because the DO is between the irrelevant and relevant stimuli, they have automatic connections between them, so that the irrelevant stimulus does in2uence processing in the relevant stimulus unit. Evidence in support of this assumption has recently been reported by Stevens (2000). On S–S consistent trials the input of the irrelevant stimulus to the corresponding relevant stimulus unit is positive, which means that activation in that unit accumulates faster than it would without this added input, thus reaching threshold sooner. On S–S inconsistent trials, instead of providing positive input to the relevant stimulus, the input of the irrelevant stimulus is negative, thus slowing the rate of accumulation of activation for the relevant stimulus. Once activation reaches threshold in the relevant stimulus unit an on signal is sent to the correct response along the controlled line, just as in the Type 1 neutral task. To summarize: 1. Activation and information 2ow in the DO model consists of three basic steps: (a) a stimulus is presented; (b) input is turned on; (c) activation accumulates in the relevant, and possibly irrelevant, stimulus units.

17


18


2. When activation reaches the stimulus identi1cation threshold: (a) the controlled line sends a discrete signal to the correct response unit; and (b) the relevant stimulus activation level starts decaying back to zero. 3. This process is repeated in the response unit until the response threshold is reached. 4. The strength of the automatic signal being sent from a stimulus unit to either a response or to another stimulus unit is a function of the level of stimulus activation weighted by the level of DO between the sets of relevant stimuli and responses, or the two sets of stimuli. 5. This means that when there is either S–R or S–S overlap, activation in the irrelevant stimulus unit produces either facilitation (in the consistent case) or interference (in the inconsistent case). There are many technical details of the model that we have not presented here that an interested reader may 1nd elsewhere (Kornblum et al. 1999). We now turn to the second theme of our tale: sequential effects.

2.3 Sequential effects 2.3.1 Introduction In his classic book on reaction times Luce observed that: ‘ . . . sequential effects . . . have a major impact on . . . response times . . . any model or experiment that ignores this or fails to predict it surely is incomplete and likely wrong, as well.’ (Luce 1986, p. 405). Because of our current work on the DO model, together with our past interest in sequential effects, it should come as no surprise that we should have been doubly attentive to Luce’s admonition. In this second portion of our chapter, therefore, we shall be looking into the DO model to see whether it is suf1ciently complete, at least in principle, to account for sequential effects. This goal is doubly appealing, for to be able to demonstrate this would: (1) extend and provide further validation of the DO model, and (2) account for some of the sequential effects that up to now have proven dif1cult to explain. The DO model as it stands, like most (non-learning) models of human performance, assumes that successive trials are independent. That is, every trial starts fresh, unaffected by the history of earlier trials in the block. Empirically, we have known for a long time that this is patently false and that sequential effects of all sorts permeate the data. By ‘sequential effects’ we mean that: ‘If a subset of trials can be selected from a series of consecutive trials on the basis of a particular relationship that each of these selected trials bear to their predecessor(s) in the series, and the data for that subset differ signi1cantly from the rest of the trials, then these data may be said to exhibit sequential effects’ (Kornblum 1973, p. 260). De1ned in this way, the term ‘sequential effects’ covers many different phenomena including stimulus and/or response repetitions and non-repetitions (of 1rst and higher orders), task switching, set or einstellung effects, etc. The sequential effects that we consider in this chapter are 1rst-order stimulus and/or response repetitions and non-repetitions, in which relevant and irrelevant stimuli do or do not have DO. As will be evident, task switching effects are also present in our data; however, because of space limitations these are not discussed. Readers interested in pursuing them should consult recent reviews (e.g. Allport, Styles, and Hsieh 1994; Monsell and Driver 2000).



2.3.2 Summary of earlier empirical 1ndings Sequential effects in RT tasks were 1rst reported by Hyman (1953). Shortly thereafter, Bertelson and his colleagues published a series of in2uential studies in which they described important properties of this new phenomenon (e.g. Bertelson 1961, 1963, 1965; Bertelson and Renkin 1966; Bertelson and Tysseyre 1966). Studies by other investigators followed that veri1ed and extended many of Bertelson’s original 1ndings, and added new observations as well. Some of these are brie2y summarized below:4 1. Given equiprobable stimuli and responses, the RT for repetitions is faster than for non-repetitions (e.g. Bertelson 1961). 2. Given equiprobable stimuli and responses, the size of the repetition effect (where ‘repetition effect’ is de1ned as the difference in RT between non-repetitions and repetitions) is greater for incompatible than for compatible tasks (Bertelson 1963). This is principally due to the increase in RT with incompatible tasks being greater for non-repetitions than for repetitions (see Kirby 1980 and Kornblum 1973 for reviews). 3. Given equiprobable stimuli and responses, the RT for repetitions and non repetitions is inversely related to the probability of these transitions. In the case of two-choice tasks this often results in the RT for non-repetitions (often called ‘alternations’ in two-choice) being faster than for repetitions (e.g. Hyman 1953; Kornblum 1969; Williams 1966). 4. The magnitude of the repetition effect increases as the number (k) of equiprobable stimuli and responses increases. This is due primarily to the fact that increasing (k) increases the RT for non-repetitions much more than for repetitions (even though the probability of non-repetitions increases with (k)—see Kornblum 1969, 1973 for more detail). 5. Repetition effects extend beyond immediate, 1rst order repetitions and non-repetitions up to about fourth order (e.g. see Remington 1969, 1971). 6. The probability of error is usually higher for non-repetitions than for repetitions (e.g. Falmagne 1965; for a review see Luce 1986). 7. The response to stimulus interval (RSI) has extensive, albeit dif1cult to systematize, effects on the magnitude the repetition effect (see Kirby 1980; Kornblum 1973; Luce 1986; Soetens 1998). This list is not intended to be exhaustive. However, it includes the principal 1ndings that investigators in the area regard as having been reasonably well established.

2.3.3 Summary of earlier accounts As the empirical 1ndings accumulated, various proposals were made to account for different aspects of the data. Most of these explanations fall into one of two major lines of argument 1rst formulated by Bertelson (1961). He suggested that sequential effects might need to be accounted for by two different types of mechanisms: the 1rst, based on the subjects’ ‘expectation of’, hence ‘preparation for’, certain events; the second, an otherwise unspeci1ed ‘residual effect’ generated

19


20


by one trial that facilitated repetitions on the next trial. Both mechanisms, according to Bertelson (1961), were sensitive to changes in RSI: the effects of expectation increased with RSI, whereas the residual effects decreased with RSI. As these conjectures were elaborated, expectation came to be viewed as a controlled, or strategic component, while residual effects were viewed as an automatic part of the process. This dichotomy has held up fairly well, supported in part by the data of Soetens and his colleagues (Soetens 1998; Soetens, Boer, and Hueting 1985) whose work has focused on substantiating and spelling out the conditions under which one or the other component would be evident. Further support has also comes from ERP (event related potentials) data (e.g. Leuthold and Sommer 1993; Squires, Wickens, Squires, and Donchin 1976). Some have characterized the automatic component in terms of activation, or sensory stimulation produced by one stimulus that leaves a trace so that if the next stimulus is the same it gets a boost by being superimposed on the traces of the 1rst (e.g. Vervaeck and Boer 1980). Others have characterized it in terms of repeated stimuli being able to bypass some of the processing stages (e.g. Bertelson 1965). Others still, speak of the stimulus (or response) on one trial priming the occurrence of the same stimulus (or response) on the next trial. None of these conjectures is spelled out in suf1cient detail to be tested, however. The most detailed model of sequential effects was constructed by Falmagne and his colleagues (Falmagne 1965; Falmagne and Theios 1969; Falmagne, Cohen, and Dwivedi 1975). Falmagne bases his model on the notion of preparation and treats preparation in the conceptual framework of a memory search. According to the model, the relative position of an item in a memory stack determines the probability with which a subject is prepared, or not prepared, for that item: the higher in the stack, the more prepared and the shorter the RT. Quantitative predictions of the model are well supported by their data. The DO model, and the extensions made to it to accommodate the sequential data, do not 1t easily into either camp, as we shall see.

2.4 Overview of the experiments 2.4.1 Objectives Our first objective, and the issue of greatest concern and interest to us in this chapter, was to examine the interaction of the repetition effect with SRC (Bertelson 1963; Kirby 1976; Schvaneveldt and Chase 1969) wherein the increase of RT with incompatible tasks is greater for non-repetitions than for repetitions. Bertelson (1963) originally accounted for this result in terms of a processing short cut that favors repetitions. He suggested that the 1rst thing a subject does when presented with a stimulus is check to see if it matches the stimulus on the previous trial. If the match is con1rmed, stimulus processing is bypassed and the response made on the previous trial is retrieved from memory and made again on this trial. If there is no match, processing proceeds until the correct response is identi1ed and executed. Because, by assumption, this processing is more complex, hence more time consuming, for incompatible than for compatible tasks, incompatibility will increase the RT for non-repetition more than for repetitions. On the surface this reasoning seems straightforward. However, nowhere is the underlying processing structure made explicit. The particular subprocesses that are being short-circuited on the one hand, and those that increase in complexity on the other, therefore, remain vague and dif1cult to identify. Our 1rst experiment is explicitly designed to address this issue.



Our second objective was to examine and compare DO and sequential effects in terms of their underlying stimulus coding processes. In particular, DO, whether based on physical or conceptual similarity, produces robust compatibility effects. Some of the best known examples of the effects of conceptual similarity are the Stroop (Type 8) and Stroop-like tasks in which the presence of color words interferes with the processing of physical colors even though these two aspects of the stimulus are physically very different. Sequential effects have been reported for the repetition of physically identical stimuli (Bertelson 1961), the repetition of categories (Marcel and Forrin 1974), and the repetition of S–R mapping rules (Shaffer 1965). However, the effects of repeating conceptually similar but physically different stimuli are not known. Because of the functional signi1cance of conceptual similarity in the DO model, it was important to learn what role, if any, this type of similarity plays in the production of sequential effects. This question was addressed in Experiment 1. In the ‘same carrier’ condition, the stimuli on the prime and probe trials were physically identical; i.e. they were either both color patches, color words, digits, etc. In the ‘different carrier’ condition the stimuli on the prime and probe trials were conceptually similar but physically different; e.g. if the stimulus on the prime was a color patch, then the stimulus on the probe was a color word, etc. Our third objective, and concern, not unrelated to the 1rst, is the question of the locus of the repetition effect. As is true of many issues in this area, this question was 1rst broached by Bertelson (1963) using a task in which he mapped different pairs of stimuli onto each of two responses. This generated three types of transitions: ‘identical’, in which both the stimulus and the response of one trial were repeated on the next trial; ‘equivalent’, in which only the response of the preceding trial was repeated on the next trial; and ‘different’, in which neither the stimulus nor the response of one trial was repeated on the next. The logic was simple: the effect of stimulus repetition was obtained by subtracting identical from equivalent RTs, and the effect of response repetition was obtained by subtracting equivalent from different RTs. Based on this procedure Bertelson concluded that the principal component of sequential effects was the repetition of the response. Pashler and Bayliss (1991), using a three-response task with the same basic paradigm, reached the same conclusion. According to Soetens (1998), however, whether one attributes sequential effects to stimulus or to response repetitions depends on RSI: response processes appear to be responsible at long RSIs, and stimulus processes at short RSIs. The basic logic of this many-to-one procedure is brought into question by Smith (1968), who reported the results of an experiment in which the equivalent RT, instead of lying between the identical and different RTs, was actually longer than the different RT. Rabbitt (1968) also reported that the relative position of the equivalent RT, between identical and different, changed with training. Overall, therefore, the locus of the repetition effect appears to be an important open question. Furthermore, based on the results of our 1rst experiment we shall conclude that both stimulus and response repetitions play a critical role in the repetition effect. Our second experiment will follow up these results and address the locus question using a one-to-one rather than a many-to-one procedure. Our fourth objective was to examine the effects of repeating and not repeating irrelevant stimuli in task Types 1, 3, and 4. We know of no other work that explicitly addresses this question in tasks with more than two choices. Experiments 3 and 4 do so.

2.4.2 General procedures 1. All experiments use four-choice tasks. The basic experimental unit is the trial pair. The 1rst trial in such pairs is called the prime, the second trial the probe. The stimulus transition probabilities

21


22


between primes and probes were randomized and balanced. The time interval between primes and probes (700 ms and 1500 ms) was blocked. The data that are reported for the two shortest RPIs are from the probe trials; the data for RSI = 3000 ms are from prime trials. 2. It is clear from the literature that the effects of RSI are capricious and problematic (see e.g. Kirby 1980; and Kornblum 1973, Table 3). Yet they cannot be ignored. In order to present a more complete empirical picture of the phenomena that we are investigating we included RSI in the design of our experiments (700 ms and 1500 ms within pairs for all four experiments, and 3000 ms between balanced pairs for Experiments 1 and 2). However, because of space limitations, we analyze the details and discuss the data for the shortest RSI only. One general observation that can be made is that the longer the RSI the slower the overall RT. The individual results of RSI for particular experiments are presented in Appendices B–D. 3. Errors are reported in Appendix A for all experiments.

2.5 Experiment 1 In this 1rst experiment we were interested in examining the interaction between sequential effects and stimulus–response compatibility. Our experimental prime–probe pairs consisted of either Type 1 or Type 2 pairs.

2.5.1 Methods 2.5.1.1 Time line A trial (whether it was a prime or probe) began with a warning signal. Seven hundred milliseconds later, the stimulus was presented and was terminated by the response. The prime-to-probe interval was either 700 ms or 1500 ms and was constant for a block. The time interval between prime–probe pairs was always three seconds. 2.5.1.2 Stimuli and responses The stimuli were presented on a CRT screen, and consisted of either four color words (RED, BLUE, GREEN, and YELLOW) or four rectangular color patches (red, blue, green, and yellow). The responses were verbal and consisted of either four color names (‘red’, ‘blue’, ‘green’, and ‘yellow’) or four-digit names (‘two’, ‘four’, ‘six’, and ‘eight’). When the responses were color names it was a Type 2 task for which the mapping was either congruent (e.g. RED → ‘red’), or incongruent (RED → ‘blue’). When the responses were digit names it was a Type 1 task, for which the mapping was neutral (e.g. RED → ‘two’). 2.5.1.3 Same/different carrier The stimuli on the prime and probe trials were either color words or color patches. In our illustration (see Fig. 2.5) we use color patches as the prime stimuli (however, note that the experiment included another set of prime–probe pairs in which the prime stimuli are color words). Following this illustration, a Type 1 prime with a color patch stimulus was followed by a Type 1 probe with either a color patch or a color word as the stimulus. In the same carrier condition, if the probe stimulus was a color patch, the nature of the probe stimulus remained what it was on the prime (color patch–color patch); this is true whether it was a repetition or a non-repetition. In the different carrier condition if the probe stimulus was a color-word, the nature of the probe stimulus changed from what it was on



Fig. 2.5 The different prime–probe pairs for Type 1 and Type 2 tasks in Experiment 1 for which the prime stimuli were color patches. (There is another set of prime–probe pairs (not shown) for which the prime stimuli were color words; same/different carrier transitions with these prime stimuli are simply the reverse of what is shown on this 1gure.) The particular colors used are for illustrative purposes only; horizontal striations indicate the color blue, diagonal striations the color red. Same carrier transitions (see text for explanation) are indicated by a dotted line, different carriers by a solid line. Whether the probe was a repetition or a non-repetition is indicated on the right. In Type 2 tasks the prime could be either congruent or incongruent, as shown. Similarly, the probe, in addition to being a congruent repetition or non-repetition, could also be an incongruent non-repetition; these are marked as ‘congruent’ or ‘incongruent’, respectively, on the right.

the prime (color patch–color word); this is true whether it was a repetition or a non-repetition. This same/different carrier designation was, of course, reversed when the prime stimulus was a color word. The Type 2 task had exactly the same properties. A prime with a color patch stimulus and congruent or incongruent mapping was followed by a probe with either a color patch or a color word as the stimulus. Same and different carrier conditions were de1ned in precisely the same manner as they were for the Type 1 tasks.

2.5.1.4 Conditions, blocks, and procedure There were three groups of six subjects each: congruent mapping (Type 2), incongruent mapping (Type 2), and neutral (Type 1). The incongruent mapping group was further divided into three subgroups, each with a different S–R mapping. The neutral group also included three subgroups each

23


24


Fig. 2.6

Results of Experiment 1 for RSI = 700 ms.

with its own S–R mapping. Each subject in each group was run on six experimental blocks of 32 trials each at one RSI, followed by six more experimental blocks at the other RSI. This order was balanced. At the start of each series of trials each subject was run on one practice block. The prime–probe transition frequencies were balanced within 64 prime–probe pairs presented in two sub-blocks of 32 pairs. Mapping (which included DO, i.e. task type), carrier, and repetition were factorially combined and constituted the three principal independent variables of the experiment.

2.5.1.5 Subjects Eighteen University of Michigan students participated in this experiment. They were all right handed, native English speakers with self reported normal hearing and vision, and tested normal color vision. They were volunteers and were paid for their participation.

2.5.2 Results The principal results that we report for this and all the other experiments are for the shortest RSI (700 ms). (For the results of different RSIs see Appendices B–D.)

2.5.2.1 Same carrier We start with the results for the same carrier condition. In Type 1 tasks the RT for repetitions is 100 ms faster than for non-repetitions [F(1, 3) = 94.72, p < 0.0023]—no surprises. In Type 2 tasks a number of things should be noted: 1. The overall RT with congruent mapping is 335 ms faster than with incongruent mapping [F(1, 10) = 82.95, p < 0.0001], and the RT for the neutral mapping is almost exactly half way in between: neutral vs. congruent [F(1, 10) = 76.62, p < 0.0001], neutral vs. incongruent [F(1, 10) = 12.96, p < 0.0049].



Fig. 2.7 Activation pattern, according to the Residual Activation Hypothesis, for a probe trial in a Type 1 task in which neither (upper panel) or both (lower panel) the stimulus and the response repeat. 2. There is also a highly signi1cant interaction between mapping, and repetition. When the S–R mapping is congruent, there is a signi1cant 21 ms difference between repetitions and non repetitions [F(1, 5) = 12.11, p < 0.0176]; when it is incongruent this difference is 235 ms [F(1, 3) = 22.63, p < 0.0176]; and when it is neutral it is in between, at 100 ms [F(1, 3) = 94.72, p < 0.0023]. These results generally replicate earlier 1ndings in the literature.

2.5.2.2 Different carrier We turn to the results with different carrier next. Recall that in the different carrier condition when the prime stimulus is a color patch the probe stimulus is a color word, and vice versa. The basic results that we obtained with the same carrier condition replicate: the overall RT for congruent mapping is 382 ms faster than for incongruent mapping [F(1, 10) = 124.04, p < 0.0001], with the neutral mapping condition falling between the two: neutral vs. congruent, [F(1, 10) = 53.47, p < 0.0001]; neutral vs. incongruent, [F(1, 10) = 22.88, p < 0.0007]. As was also true in the same carrier condition there is a highly signi1cant interaction between mapping, and repetition. When the S–R mapping is congruent the difference between repetitions and non repetitions is not statistically signi1cant (1 ms) [F(1, 5) = 0.13 p < 0.7315]. When it is incongruent, it is 132 ms [F(1, 3) = 9.48, p < 0.0542], and when it is neutral it is in between at 55 ms [F(1, 3) = 12.06, p < 0.0403]. Note that the mapping effect for repetitions with different carriers (264 ms) is larger than with same carriers (155 ms). This generates the highly signi1cant triple interaction between carrier, mapping, and repetition [F(1, 10) = 20.35, p < 0.001]. 2.5.3 Discussion: the Information Reduction Hypothesis The model must now show that it can account for the following: (1) the effects of repetition; (2) the interaction between repetition and S–R mapping; and (3) the interaction between repetition, S–R mapping, and carrier.

25


26


Fig. 2.8 Activation pattern, according to the Information Reduction Hypothesis, for a probe trial in a Type 1 task in which neither (upper panel) or both (lower panel) the stimulus and the response repeat. Note that repetitions, instead of having a higher starting point, as in the Residual Activation Hypothesis (Fig. 2.7), have a lower threshold. See text for the implications of this difference.

Common to all the ‘automatic’ accounts of the repetition effect, as we have seen, is the notion that having performed a certain action, residual traces (e.g. memory, perceptual, or response traces) are left that automatically facilitate the processing of subsequently repeated stimuli or responses. One easy way of implementing this view in our model would be to represent this trace as residual activation left over from the previous trial that has not yet decayed all the way to zero by the time the current trial begins (see Fig. 2.7). Let us call this the ‘Residual Activation Hypothesis’.5 Given the head-start provided by the residual activation, activation levels in the stimulus and response units would reach threshold earlier than they otherwise might. This would reduce overall processing time, hence overall RT—thus producing a repetition effect. Simple?—Yes; correct?—Unfortunately, no. This scheme can be shown to account for the effects of repetitions, the effects of mapping, and their interaction. However, it cannot account for the interaction with carrier which is one of the striking aspects of our results. We will now argue that what we are calling the ‘Information Reduction Hypothesis’, can. It is a property of the DO model that each individual stimulus and response unit has its own threshold. According to the information reduction hypothesis, whenever a stimulus is identi1ed or a response made or selected, the amount of information required to identify that stimulus or select that response again is temporarily lowered. This is implemented as a decrease in the stimulus or response threshold associated with the appropriate unit (see Fig. 2.8). If the same stimulus is presented again, or the same response is selected on the next trial, activation has a shorter way to go before it reaches this lower threshold, and the processing durations of the stimulus or response units are consequently reduced.



Fig. 2.9 Activation patterns, according to the Information Reduction Hypothesis, for probe trials in Type 1 and Type 2 tasks for same carrier condition where, by hypothesis, both or neither the stimulus and the response repeat. As illustrated here, for the non-repetitions, activation on probe trials is identical to activation on primes. Note that not only is the overall RT for repetitions faster than for non-repetition (the repetition effect), but the mapping effect is smaller as well. At 1rst glance these two hypotheses seem equivalent. Exactly the same reduction in processing time is achieved by postulating residual activation as is by lowering thresholds. However, these two hypotheses have profoundly different consequences further down the line (see * in Fig. 2.3, Type 2 tasks).

2.5.3.1 Same carrier We start with the results from the same carrier condition. According to the information reduction hypothesis, performance in the three different mapping conditions is determined as follows: In Type 1 tasks, when the mapping is neutral, the only factor in2uencing the RT is whether the stimulus or response repeat or not. When neither repeat, the thresholds of the stimulus and response units are identical on probe and prime trials. The RTs on probe trials are, therefore, the same as on primes. When both the stimulus and the response repeat, the thresholds of both the stimulus and the response units are lower on probe trials than they were on prime trials. The RTs on probe trials are, therefore, faster than on primes. This is the baseline, the basic repetition effect (see Fig. 2.9, Type 1). In Type 2 tasks, when the mapping is either congruent or incongruent two factors come into play: the 1rst is whether the stimulus or response repeat or not, and we just saw how this factor affects RT just by itself in the Type 1 task where the mapping is neutral; the second is the facilitation and interference produced by the congruent and incongruent mappings. When neither stimulus nor response repeat, the thresholds of the stimulus and response units are identical on probe and prime trials—just like in Type 1 tasks. Irrespective of mapping, the RTs on probes are the same as on primes. When both the stimulus and response repeat, the thresholds of both the stimulus and response units are lower on probe trials than they were on primes. The effect of this lowered threshold in the stimulus unit is identical for the congruent, incongruent, and neutral mappings (see Fig. 2.9). However,

27


28


because of the DO between the relevant stimulus and the response, this lower stimulus threshold affects the input to the response unit. In particular, the lower threshold reduces the level to which stimulus activation rises, hence from which it starts to decay (see * in Fig. 2.3, Type 2 tasks). Therefore, the automatic input to the response unit is less when the stimulus repeats than when it does not repeat. In addition, because the rate at which activation accumulates in the response unit is faster for congruent than for incongruent mapping, lowering the response selection threshold has differential effects for congruent and incongruent mapping: it decreases the RT for both, however, it decreases the RT for incongruent mapping more than for congruent mapping. The observed interaction between repetition and S–R mapping is, therefore, the result of lowering both the stimulus and the response threshold.

2.5.3.2 Different carrier Now consider the different carrier condition. This is when the prime stimulus is a color word and the probe stimulus is a color patch, or vice versa. We shall look at the repetition effect 1rst, and the mapping effect next. We’ve already seen that in Type 1 tasks, when the mapping is neutral, the only factor in2uencing RT is whether or not the stimulus or the response repeat. When neither repeat, according to the model, the RTs for same (682 ms) and different (690 ms) carriers ought to be identical—which they are; this 8 ms difference in the data is not statistically signi1cant [F(1, 5) = 1.63, p < 0.2583]. The Type 2 results are not as clean. When the mapping is congruent and neither stimulus nor response repeat, the RT for same carrier (484 ms) is faster than for different carrier (490 ms), and this 6 ms difference is statistically signi1cant [F(1, 5) = 11.47, p < 0.0195]. The same is true for the incongruent mapping. The RT for non-repetitions with the same carrier (853 ms) is faster than with the different carrier (885 ms); and this 32 ms difference is also statistically signi1cant [F(1, 5) = 6.75, p < 0.0484]. It seemed reasonable to us to attribute these small differences to the cost of switching between carriers (i.e. from color patch to word and vice versa). For consider, these differences are for total nonrepetitions, which means that neither the stimulus nor the response on the prime are repeated on the probe. Therefore, the most plausible contrast between the same and different carrier conditions that might account for this difference in RT is that repetition of the carrier itself had an effect. This interpretation of the data is consistent with the fact that same carrier RTs were faster than different carrier RTs. Consider the repetitions next. Recall that repetitions with different carriers were 49 ms slower than with same carrier. This probably re2ects the difference between the physical and the possible conceptual repetition, or total non-repetition of the stimulus. However, we had no way of assessing whether conceptual repetition contributed in any way to the repetition effect in this experiment. In our modeling, therefore, we treated repetition trials in different carrier conditions as pure response repetitions (see Fig. 2.10). Thus, when only the response repeats, as it does by hypothesis, the threshold of the response unit is lower on the probe than on the prime and RTs on probe trials are, therefore, faster than on primes, but not as fast as with probes in the same carrier condition where both the stimulus and response are repeated. We go to the mapping effects next. When the mapping is either congruent or incongruent, we again have two factors coming into play: repetition and mapping. On non-repetition trials, when neither stimulus nor response are repeated, probe RTs are the same as on prime trials by the same argument that we made in the case of same carrier. On repetition trials, when only the response is repeated, only the threshold for the



Fig. 2.10 Activation patterns for probe trials in Type 1 and Type 2 tasks for different carrier condition where, by hypothesis, the stimulus does not repeat but the response may. As was true for the same carrier condition, the non-repetition probe trials are identical to the primes. The overall RT for repetitions is faster than for non-repetitions. However, the repetition effect is smaller here than it was for the same carrier condition (Fig. 2.9). This is because here only the response repeats. Note also, that as was true for the same carrier condition, the mapping effect is smaller for repetitions than for non-repetitions, but this reduction is not as large here as it was for the same carrier condition. response unit is lower on the probe than on the prime trial. This means that stimulus activation will start its decaying process from a higher level in the different carrier than in the same carrier condition producing a larger mapping effect thus generating the triple interaction between repetition, mapping, and carrier. Based on these principles we used the model to simulate the results for Experiment 1. As can be seen on Fig. 2.11 they appear to match the empirical data reasonably well. Thus, based on our results and model, repetitions of both stimulus and response play an important role in the production of the repetition effect. However, as we have seen, previous studies that have explicitly addressed the question of locus (e.g. Bertelson 1963) concluded that the bulk of the repetition effect lies with the repetition of the response. Our next experiment addresses this issue directly.

2.6 Experiment 2 Here we examine the locus of the repetition effect. Because of our interest in the interaction of S–R compatibility with the repetition effect, our primary focus will be on Type 2 tasks. However, we shall also be looking at the effects of repetitions and non-repetitions for Type 1 and Type 2 tasks, each preceded by the other. Consider a Type 2 probe—for example, one where the stimulus is drawn from a set of color patches and the response from a set of color names. Now consider the Type 1 prime preceding this probe, where

29


30


Fig. 2.11 Simulated data for Experiment 1 are shown as solid lines; the empirical data are shown as dashed lines for comparison. either the stimulus is drawn from a set of color patches, with digit names as responses, or the response is drawn from a set of color names with digits as the stimuli. Depending on which of these two primes one chose, the probe in such a prime–probe pair would display either a stimulus repetition or a response repetition, but not both. This is precisely how we designed the prime–probe pairs for this experiment.

2.6.1 Methods 2.6.1.1 Time line The temporal relationships within and between trials were the same as they were in Experiment 1. 2.6.1.2 Stimuli and responses The stimuli were presented on a CRT screen, and consisted of either four rectangular color patches (red, blue, green, and yellow) or four digits (2, 4, 6, 8). The responses were verbal and consisted of either four color names (‘red’, ‘blue’, ‘green’, and ‘yellow’) or four digit names (‘two’, ‘four’, ‘six’, and ‘eight’). For Type 1 tasks, we used either color patch stimuli and mapped them onto digit name responses, or digit stimuli and mapped them onto color name responses. For Type 2 tasks, we used the same sets of stimuli and responses but paired them differently: color patch stimuli were mapped onto color name responses—congruently and incongruently, and digit stimuli were mapped onto digit name responses—congruently or incongruently. 2.6.1.2.1 Type 1 → Type 2, and Type 2 → Type 1, prime–probe pairs Consider 1rst the case in which the prime-probe pairs consisted of Type1–Type 2 tasks, respectively. In our illustrations (Fig. 2.12) we show the stimuli and responses for the probe as color patches and color names respectively; of course, as was true of experiment 1, in this experiment there was another set of probe trials for which the stimuli and responses consisted of digits and digit names.



Fig. 2.12 Illustrative stimuli and responses for different prime–probe pairs in Experiment 2 in which the primes are Type 1 and the probes are Type 2 tasks. For this illustration the prime stimuli are color patches. There is another set of prime–probe pairs (not shown) for which the prime stimuli are digits. With those primes, the S–R pairing on probe trials remains the same as what is shown except that what are here identi1ed as response set switches (see text) become stimulus switches and vice versa. The particular colors and digits are for illustrative purposes only. Horizontal striations indicate the color blue, diagonal striations the color red. Response set switches are indicated by a dotted line, stimulus set switches are indicated by a solid line. Whether the probe is a stimulus or response, repetition or non-repetition is indicated on the right. Stimulus repetitions with Type 2 color probes, whether congruent or incongruent, were produced by Type 1 primes for which the stimuli were color patches and the responses digit names. Similarly, response repetitions with Type 2 color probes were produced by Type 1 primes for which the responses were color names and the stimuli were digits. When the order of task types in the prime–probe pairs was reversed, and consisted of Type 2– Type 1 tasks, respectively (see Fig. 2.13), the procedure for obtaining stimulus and response repetitions was identical to what we just saw. For example, if the Type 1 probe stimulus was a color patch and the response a digit name, a stimulus repetition was produced by a Type 2 color prime, with either congruent or incongruent mapping. Similarly, a response repetition was produced by a Type 2 digit prime, with either congruent or incongruent mapping.

2.6.1.3 Conditions, blocks, and procedures Thirty-two subjects participated in the experiment; for half the subjects the prime stimulus was color, for the other half it was digits. These two groups were further subdivided into four groups of

31


32


Fig. 2.13 Illustrative stimuli and responses for different prime–probe pairs in Experiment 2 in which the primes are Type 2 and the probes are Type 1 tasks. For this illustration the prime stimuli are color patches. There is another set of prime–probe pairs for which the prime stimuli are digits. With those primes, the S–R pairing on probe trials remains the same as what is shown except that what are here identi1ed as response set switches become stimulus switches (see text) and vice versa. The particular colors and digits are for illustrative purposes only. Horizontal striations indicate the color blue, diagonal striations the color red. Response set switches are indicated by a dotted line, stimulus set switches are indicated by a solid line. Whether the probe is a stimulus or response, repetition or non-repetition is indicated on the right.

four subjects each. Each of these subgroups was identi1ed by the mapping instructions that it received for the Type 2 task: one subgroup was given the congruent mapping, and each of the other three subgroups was given a different incongruent mapping. Each of the four subjects in these subgroups received a different mapping for the Type 1 task. Each subject was run on six experimental blocks of 32 trials each at one RSI, followed by six more experimental blocks at the other RSI. This order was balanced. Half the subjects in each subgroup of four started with a Type 1 prime, the other half started with a Type 2 prime. Prime stimulus and S–R mapping were between subject variables. At the start of each series of trials each subject was run on one practice block. The prime–probe transition frequencies were balanced within 64 prime–probe pairs presented in two sub-blocks of 32 pairs.

2.6.1.4 Subjects Thirty-two University of Michigan students volunteered for the experiment and were paid for their participation. They were all right-handed, native English speakers with self-reported normal hearing and vision. Their color vision tests were normal.



Fig. 2.14 Results of Experiment 2. The left panel shows the data when the prime–probe pair consisted of Type 1–Type 2 tasks, respectively; the right panel shows the results when the prime–probe pairs consisted of Type 2–Type 1 tasks. ‘Congruent’ and ‘incongruent’ refer to the S–R mapping for the Type 2 tasks, irrespective of order. The dotted lines are the data for the stimulus repetitions and non-repetitions, when the response set was switched; the solid lines are the data for the response repetitions and non-repetitions, when the stimulus set was switched.

2.6.2 Type 1 → Type 2 2.6.2.1 Results First we present the results for the Type 2 probes preceded by Type 1 primes averaged over the digit and color patch stimuli (see Fig. 2.14). 1. As expected, there is a highly signi1cant effect of mapping: the RT for congruent mapping is over 365 ms faster than for incongruent mapping [F(1, 30) = 44.31, p < 0.0001]. 2. The interaction between mapping and the size of the repetition effect is also signi1cant [F(1, 30) = 5.48, p < 0.0260]: when the S–R mapping is congruent, the repetition effect is 25 ms. [F(1, 23) = 9.06, p < 0.0197]; when it is incongruent it is 96 ms [F(1, 23) = 31.01, p < 0.0001], a fourfold increase. 3. Note that when we speak of repetition effects in this experiment we are speaking of repetitions of either the stimulus or the response, with corresponding shifts in response and stimulus sets respectively (see Fig. 2.14). That is, when a stimulus repetition or non-repetition occurs (the dotted lines in Fig. 2.14), the prime stimulus and the probe stimulus are both drawn from the same stimulus set (they are either both color patches or both digits). In contrast, the prime response and the probe response are each drawn from different response sets (digit names for the one and color names for the other, or vice versa). This means that the subject must shift from one response set to another (i.e. digit names to color names or vice versa). The symmetric situation holds for response repetitions and non-repetitions (the solid lines in Fig. 2.14). Here, the responses on the prime and probe are

33


34


both drawn from the same response set (they are either both color names or both digit names) and it is the prime and probe stimuli that are each drawn from a different stimulus set (color patch for one and digit for the other, or vice versa). This means that subjects must shift from one stimulus set to another (digits to colors, or vice versa). These shifts appear to have exacted a cost. Shifting from one response set to another (digit names to color names, or vice versa) takes about 27 ms longer than shifting from one stimulus set to another (digits to color patches or vice versa) [F(1, 30) = 6.76, p < 0.0143]. These set shifting costs (25 ms for congruent mapping ([F(1, 7) = 10.15, p < 0.0154], and 29 ms for incongruent mapping [F(1, 7) = 4.32, p < 0.0489]) are additive with the effects of repetition and mapping [set-shift × congruence : F(1, 30) = 0.03, p < 0.8643].

2.6.2.2 Discussion Our results, in contrast to earlier reports, and using a different experimental paradigm, show that stimulus and response repetition effects are both fairly large, and roughly equal in size. The locus of the repetition effect thus appears to be equally apportioned between stimulus and response processes. These 1ndings also discon1rm a prediction recently made by Hommel (1998; Hommel et al. in press). In a recent paper in which he extends the notion of feature integration (Treisman 1988) to include action features to construct, what he calls ‘event 1les’, Hommel (1998; Hommel et al. in press) makes speci1c predictions about the relative costs and bene1ts of certain kinds of repetitions. In particular, according to Hommel’s view, if one takes as a baseline the total non-repetition condition, when neither the stimulus nor the response are repeated, then the RT for the total repetition condition, when both the stimulus and the response are repeated, should show a distinct bene1t. The partial repetition condition, on the other hand, when either the stimulus or the response, but not both, are repeated, would show no bene1t, at best, and possibly a cost. Our results, in which we obtain clear bene1ts from stimulus repetitions and response repetitions, each in the absence of the other, are clearly inconsistent with these predictions. The results of the interaction between mapping and repetition are qualitatively similar to the results obtained in Experiment 1. That is, the probe which was a Type 2 task, unsurprisingly, behaved like a Type 2 task: there was a very large mapping effect (365 ms), and the repetition effect was much smaller for the congruent (25 ms) than for the incongruent (96 ms) mapping conditions. There are, of course, differences between the two experiments as well. First, note that the overall RT in this experiment is marginally longer than in Experiment 1 [F(1, 40) = 3.38, p < 0.0735]. (Even though this difference appears to interact with mapping, this between-subjects difference is not statistically signi1cant [F(1, 40) = 0.16, p < 0.6899].) Recall that in the present experiment the Type 2 probes were preceded by Type 1 primes. In the previous experiment these Type 2 probes were preceded by primes that were Type 2 as well and also used the same stimulus and response sets. In this experiment the RT for congruent mapping was 79 ms slower than in the different carrier condition of Experiment 1, and the overall RT for incongruent mapping was 149 ms. slower than in Experiment 1. We suggest that these differences are attributable to task switching. Let us anticipate the results of the second half of this experiment in which the primes were congruent/incongruent Type 2 tasks, and the probes were Type 1. There we 1nd a similar effect, with the primes and probes reversed: the cost of switching from a congruent prime to a neutral probe is much less than the cost of switching from an incongruent prime to the same neutral probe. So, whether one switches to or from a trial with congruent mapping, RT appears to be faster than when one is switching to or from a trial with incongruent mapping.



Fig. 2.15 Simulation of the results of Experiment 2 with Type 2 probes. Panel A shows the simulated results when the effects of stimulus and response set switching is not taken into account. Panel B shows the simulated results when a constant is added for set switching: 60 ms and 80 ms have been added for the stimulus and response set switches, respectively, for the congruent conditions; 165 and 200 ms have been added for the stimulus and response set switches, respectively for the incongruent conditions. Panel C shows the empirical results on the same scale (these are the same data as are shown on Fig. 2.14). The dotted lines are the data for the stimulus repetitions and non-repetitions, when the response set was switched; the solid lines are the data for the response repetitions and non-repetitions, when the stimulus set was switched.

Another result that we believe is due to switching is the 1nding that switching from one response set to another generated a longer RT than switching from one stimulus set to another. One 1nal observation worth noting is that Rogers and Monsell (1995) reported that in their study the response repetition effect vanishes following a task switch. We, on the other hand, obtained a robust response repetition effect in Type 2 probe trials following a Type 1 prime. As we indicated in the introduction to these experiments we have not tried to make our model account for either this or any other effects of switching. We would, of course, have liked to use the same parameter values for the simulation of these results as we used to 1t the data of the previous experiment. However, because of the effects of task and set switching, and their interactions, we were unable to do that. Nevertheless, the new simulations capture the repetition effects quite nicely as is evident by comparing the slopes of the empirical data in Panel C (Fig. 2.15) with the slopes of the simulated data in Panel A (Fig. 2.15). If we now treat the vertical displacements as due to task and set switching, and add these as arbitrary constants (Panel B, Fig. 2.15), which we confess is far from theoretically satisfying, then the overall 1t is quite good.

2.6.3 Type 2 → Type 1 2.6.3.1 Results Next we turn to the results of the Type 1 probes preceded by Type 2 primes (see Fig. 2.14, right panel). 1. One of the most striking aspects of these data is the large and reliable effect that mapping of the preceding prime had on these neutral probes. This is a result that we have already alluded to: when

35


36


the mapping for the prime was congruent, the RT for the probe was 92 ms faster than when the mapping was incongruent [F(1, 30) = 4.19, p < 0.0494]. 2. Also, the time to switch between response sets was 100 ms longer than the time to switch between stimulus sets [F(1, 30) = 40.20, p < 0.0001]. This, of course, was much larger than the set switching effect that we had seen with Type 2 probes, and was observed with incongruent primes only [F(1, 23) = 43.18, p < 0.0001]; when the prime was congruent, set switching failed to have a signi1cant effect [F(1, 23) = 0.01, p < 0.9228]. 3. Finally we note that in Experiment 1, when Type 1 probes are preceded by the same type primes, so that there was no task switching, the overall RT is much faster than in this experiment— as expected. Now we come to the repetition effects. When the prime was congruent, the repetition effect was 54 ms [F(1, 7) = 43.6, p < 0.0003]; when the prime was incongruent it was 31 ms [F(1, 23) = 10.31, p < 0.0039]. The difference between these two repetition effects is not signi1cant [Rep × Cong interaction: F(1, 30) = 1. 72, p < 0.1999]. Thus, even though the congruence and incongruence of the Type 2 primes seem to have in2uenced the overall RT of the Type 1 probe, they did not have a differential in2uence on the repetition effect. Because these effects all appear to be due to switching of one kind or another we shall have nothing further to say about them from the point of view of the DO model, and we leave them with the reader to ponder as empirical results that pose theoretical puzzles.

2.7 Irrelevant stimuli and sequential effects In this next section we shall look at the results of some experiments in which the prime and probe trials have irrelevant stimuli that either do or do not overlap with some other aspect of the task. These are either neutral Type 1 tasks in which there is no DO, Type 3, or Simon tasks, in which the irrelevant stimulus dimension overlaps with the response, or Type 4, Stroop-like tasks, in which the irrelevant stimulus dimension overlaps with the relevant stimulus dimension. All these experiments used the same procedures; we will, therefore, describe them just once at the start. As was true of all the experiments up to now, the experimental unit consisted of trial pairs: a prime and a probe trial.

2.7.1 General procedures 2.7.1.1 Stimuli and responses In all cases the relevant stimuli were the letters B, J, Q, Z. The responses consisted of joystick movements up, down, left or right. The S–R mapping was arbitrary. The irrelevant stimuli were presented as 2ankers to the left and right of the relevant letters, and differed depending on the task type. There were four possible irrelevant stimuli for each task type, which generated 16 different stimuli of each type: For the Type 1 tasks, the irrelevant stimuli were diacritical marks and a plus sign (#, %, &, +); for the Type 3 tasks, the irrelevant stimuli were up, down, left, and right arrows; and for the Type 4 tasks, the irrelevant stimuli were the letters B, J, Q, Z. 2.7.1.2 Experimental factors and design Given the task type (Type 1, 3, or 4) for the prime and probe, the factors of interest were:



1. the consistency of the prime and probe; 2. the repetition/non-repetition of this consistency state; 3. the repetition/non-repetition of the relevant and/or irrelevant stimulus; 4. RSI. Because in two-choice tasks some of these factors are confounded with each other as well as with negative priming, we used four-choice tasks and constructed a transition matrix (see Fig. 2.16) in which the 1rst three factors were explicitly represented. This matrix, which revealed a surprising number of constraints, makes the confounding that necessarily occurs in two-choice tasks very clear. For example, whether the relevant stimulus in a two-choice task repeats or not, the repetition/non-repetition of the irrelevant stimulus is confounded with the repetition/non-repetition of consistency and negative priming preconditions. This matrix was the starting point for the design of all our experiments.

2.8 Experiment 3 In this 1rst experiment we were interested in examining the question of whether the consistency or inconsistency effects of probe trials was affected by the consistency or inconsistency of primes. According to the DO model there is no reason why such contingencies should occur. However, such effects have been reported in the literature for Type 3 tasks, so we wanted to verify these reports before proceeding (e.g. Mordkoff 1998).

2.8.1 Design We used Type 3 and Type 4 tasks, presented in different experimental blocks. Each block also contained a Type 1 task. The prime in a Type 3 block was, therefore, S–R consistent, inconsistent, or neutral. The prime in a Type 4 block was S–S consistent, inconsistent, or neutral. The probe, similarly, was either consistent, inconsistent, or neutral. Each block, therefore, contained nine different primeto-probe transitions, whether it was a Type 3 or a Type 4 block (see Fig. 2.17). From the master transition matrix (see Fig. 2.16) it was also evident that in order for these nine conditions to be comparable and not be confounded with other variables, neither the relevant nor the irrelevant stimuli of the prime could be repeated in the probe. Each block included four randomized instances of each of the nine prime-to-probe transitions; there were two blocks per task type and RSI (700 and 1500 ms). Twelve subjects participated in the experiment. Four different mappings were used for each task type, and mapping was as between subjects variable.

2.8.2 Results We start with the results of the Type 3 task (see Fig. 2.18). When both the prime and the probe are Type 3 tasks, there is a highly signi1cant S–R consistency effect of 80 ms that is totally immune to differences in the consistency of the prime [consistent prime, 87 ms; inconsistent prime, 73 ms; the prime × probe interaction is not signi1cant: F(1, 8) = 0.25, p < 0.6275]. However, when the prime is neutral, the consistency effect of the probe jumps to 140 ms. This 75% increase is achieved by having both a faster RT for consistent probes, and a slower RT for inconsistent probes. When the probe is

37


38


Fig. 2.16 Generic prime-to-probe transition matrix for four-choice tasks with irrelevant stimuli. The capital letters A, B, C, and D designate either relevant stimuli or responses, depending on the task being represented; for Types 1 and 4, they represent relevant stimuli, for Type 3 they represent responses. This generated sixteen large square areas representing all the transitions between the four capital letters (relevant stimuli for Types 1 and 4, or responses for Type 3). The lower-case letters a, b, c, d designate irrelevant stimuli that overlap either with the relevant stimuli (Type 4) or with the responses (Type 3). The lower-case letters w, x, y, z designate irrelevant stimuli that have no DO with any aspect of the task (Type 1). Each of these large square areas is thus subdivided into four quadrants that represent the two-by-two combination of overlapping (DO) and non-overlapping (N) prime–probe pairs: DO→DO; DO→N; N→N; and N→DO. Inside these four quadrants are sixteen individual cells identi1ed by the letters c and i, as well as by dashes; they have the following meaning: ‘c’ stands for a consistent trial, ‘i’ for an inconsistent trial, and ‘-’ for a neutral trial. These letters and dashes appear in pairs, where the 1rst position in the pair denotes the nature of the prime



neutral, the RT falls between the RTs for consistent and inconsistent probes and is completely unaffected by the prime [F(1, 8) = 0.07, p < 0.8008]. We turn next to the results of the Type 4 task (see Fig. 2.18). When both the prime and the probe are Type 4 tasks, there is a highly reliable S–S consistency effect of 62 ms [F(1, 8) = 6.70, p < 0.0322] which is not signi1cantly altered by neutral primes (59 ms) [F(1, 8) = 0.44, p < 0.5260]. (There also appears to be a 31 ms interaction between prime and probe consistency which, however, is not signi1cant [F(1,8) = 0.82, p < 0.3907] and appears to be due entirely to the effects of the prime on inconsistent probes: the RTs for consistent probes all fall within a range of 11 ms for the various primes that they are paired with. The RTs for inconsistent probes span a range of 47 ms.) To summarize. First, just as there are clear differences in performance between Type 3 and Type 4 tasks in terms of overall mean RTs, we again see differences in performance between these task types when merely considering the sequence of consistent inconsistent trials. Second, the results for Type 3 tasks are clear and systematic: the consistency or inconsistency of the prime has no effect whatsoever on the size of the consistency effect of the probe. However, whether or not the prime has DO has an enormous in2uence on the size of the S–R consistency effect: a neutral prime almost doubles the size of that effect. The results with the Type 3 task are inconsistent with Mordkoff’s (1998) earlier reports. However, as we have indicated, these reports, which are based on two-choice data, may have included confoundings between the repetition effects of relevant and irrelevant stimuli with other factors in the experiment. The results with Type 4 tasks are not as clean and, obviously, need further work.

2.9 Experiment 4 In this next experiment we examine the basic repetition effects of relevant and irrelevant stimuli in four-choice tasks of Types 1, 3, and 4.

trial and the second position the nature of the probe. Thus, for example, a ‘cc’ in a cell identi1es this cell as the transition between a consistent prime and a consistent probe; a ‘ci’ cell would be the transition between a consistent prime and an inconsistent probe; ‘i-’ would be the transition between an inconsistent prime and a neutral probe, etc. We now come to the repetition/non-repetition properties of the prime–probe pairs in this matrix. The only repetitions of relevant stimuli or response occur in the four large square areas on the main diagonal. The remaining twelve large, offdiagonal, square areas represent non-repetitions of relevant stimuli or responses. The cells on the main diagonal of these sixteen large square areas all represent repetitions of the irrelevant stimuli; the off-diagonal cells are all non-repetitions of the irrelevant stimuli. This matrix makes it relatively easy to identify some transitions with special properties that may be interesting. For example, consider the large A × A square area. The 1rst column represents transitions in which the irrelevant stimulus on the probe trial is the same as either the relevant stimulus, or the response on the prime. The 1rst row represents transitions in which the irrelevant stimulus on the prime becomes the relevant stimulus, or the response on the probe. Each of the sixteen large square areas has one row and one column with these same properties. Other interesting transitions may be those in which the relevant and irrelevant stimuli on the prime are switched on the probe; by de1nition, of course, these can only occur in the large, off-diagonal squares.

39


40


Fig. 2.17 Basic design for Experiment 3 showing the nine different prime-to-probe transitions in a block. Task types were blocked so that some experimental blocks had task Types 3 and 1, and other blocks had task Types 4 and 1.

Fig. 2.18 Results of Experiment 3 for Type 3 and Type 4 blocks; each of these blocks included Type 1 neutral trials. On the abscissa are the three values of the probe: consistent, inconsistent and neutral. The parameters for the data lines are the nature of the prime: circles ( ) indicate consistent primes; squares ( ) indicate inconsistent primes; triangles ( ) indicate neutral primes.

2.9.1 Methods and procedures The relevant stimuli were the four letters and the responses were the up, down, left, and right movements of a joystick. The irrelevant stimuli differed depending on the type of task: for the Type 1 task,



they were diacritical marks, for the Type 2 task they were directional arrows, and for the Type 4 task they were letters (see the general description of the stimuli and responses at the beginning of this section). The Type 1 task was run on one group of 12 subjects, the Type 3 and 4 tasks were run on another group of 12 subjects in a balanced order. For each task type there were four different mappings, each assigned to a different group of subjects. We used a simple 2 × 2 design: the repetitions and non-repetitions of the relevant stimuli were crossed with the repetitions and non-repetitions of the irrelevant stimuli. In order to obtain this factorial combination in the Types 3 and 4 tasks, both the prime and the probe trials in each pair had to be inconsistent; in the Type 1 task, of course, this issue was moot (see Fig. 2.16). There were two experimental blocks for each task type and RSI value (700 ms and 1500 ms). Each block contained eight randomized presentations of the four repetition/non-repetitions primeto-probe transitions for a total of 32 pairs.

2.9.2 Results The results of the Type 1 task are illustrated in Fig. 2.19. There was a highly signi1cant repetition effect of 90 ms [F(1, 8) = 38.26, p < 0.0003] for the relevant stimulus which, of course, includes repetition of the response. This is almost indistinguishable from the repetition effect that we observed with Type 1 tasks in Experiment 1 (100 ms), where we used very different stimuli and responses, and where the overall RT was also more than 120 ms longer than in this experiment. The repetition of the irrelevant stimulus had no signi1cant effect [F(1, 8) = 0.01, p < 0.9393], and there was no signi1cant interaction [F(1, 8) = 0.01, p < 0.9393]. The repetition of irrelevant stimuli in Type 1 neutral tasks, therefore, has no effect on performance. In the Type 3 task repeating the relevant stimulus had a signi1cant 116 ms effect [F(1, 8) = 69.34, p < 0.0001], whereas the repetition of the irrelevant stimulus had no signi1cant effect [F(1, 8) = 0.09, p < 0.7753], and there was no interaction. (Even though the repetition effect for the relevant stimulus is 22 ms greater when the irrelevant stimulus does not repeat than when it repeats, this interaction is not signi1cant [F(1, 8) = 0.89, p < 0.3740].) Thus, as was true for the Type 1, neutral, task, repeating or not repeating the irrelevant stimulus has no effect on performance in Type 3 tasks. In the Type 4 task, repeating the relevant stimulus had a signi1cant effect of 110 ms [F(1, 8) = 95.87, p < 0.0001]. And, unlike the results obtained with the Types 1 and 3 tasks, there is a 26 ms repetition effect for the irrelevant stimulus [F(1, 8) = 17.47, p < 0.0031]. There was no signi1cant interaction between the repetitions of the relevant and irrelevant stimulus.

2.9.3 Discussion These results are inconsistent with Hommel’s event 1le view (Hommel 1998; Hommel et al. in press) in at least three different ways: 1rst, according to that view, if the relevant stimulus repeats then repeating the irrelevant stimulus should show a bene1t compared to the non-repetition of that irrelevant stimulus. We fail to con1rm this in both Type 1 and Type 3 tasks. Second, the event 1le view predicts an interaction between the repetition of relevant and irrelevant stimuli. The results of our Type 4 task fail to con1rm this. Third, according to our reading, Hommel’s event 1le position would make identical predictions for Type 1, Type 3, and Type 4 tasks. Our results show that performance on these three tasks is quite different and appears to be based on the patterns of dimensional overlap.

41


42


Fig. 2.19 Results of Experiment 4 for task Types 1, 3, and 4. On the abscissa is indicated whether the relevant stimulus repeats or not. The parameter for the data line is whether the irrelevant stimulus repeats or not: the circle indicates a repetition, the square a non-repetition.

These results are consistent with one of the DO model’s principal assertions, namely: whether and how irrelevant stimuli affect performance, depends on what they overlap with. In this case we see that the repetition of irrelevant stimuli in Type 3 and Type 4 tasks clearly affects performance in very different ways—ways that, as we will now show, the model is able to account for.

2.9.3.1 The Information Reduction Hypothesis for irrelevant stimuli Let us 1rst consider the 1nding that the overall RT for Type 1 is faster than for Types 3 and 4. Recall that in this experiment, in order to avoid the confounding between factors, the Type 3 and Type 4 trials were all inconsistent; the Type 1 trials, in contrast, were all neutral . The RTs for S–R (Type 3) and S–S (Type 4) inconsistent trials are known to be slower than for neutral trials. The DO model’s account for these consistency effects was summarized in Sections 2.2.2.1 and 2.2.2.2 at the beginning of this chapter. Now consider the effects of repetitions for the relevant and irrelevant stimuli. Recall that the basic way in which the model accounts for the effects of relevant stimulus, and response repetitions is by lowering the threshold in the stimulus and response units when stimuli or responses are repeated (see Fig. 2.8). That is, less information is required to reach threshold after a repetition than after a non-repetition. We suggest that the Information Reduction Hypothesis is equally applicable to the effects of irrelevant stimulus repetitions: whatever the process that distinguishes between relevant and irrelevant stimuli (see Fig. 2.4), that process requires less information when irrelevant stimuli are repeated than when they are not repeated. This proposition is easily implemented in the model: if less information is required, it is reasonable to assume that less time would be required to process that reduced amount of information. Following a repetition, therefore, we shorten the time parameter (τ) (see Kornblum et al. 1999) which, in the model, determines how long it takes to distinguish between relevant and irrelevant inputs (see Fig. 2.4). The effect of this time reduction on the irrelevant stimulus activation curve is to reduce the



Fig. 2.20 Illustration of the Information Reduction Hypothesis for irrelevant stimuli. According to the hypothesis, when an irrelevant stimulus is repeated less information is needed to distinguish between the inputs of the relevant and the irrelevant stimuli. This translates into less time (τ) being required to make that distinction, which means that the input for the irrelevant stimulus (see also Fig. 2.4) will start decreasing sooner after a repetition than after a non-repetition. This means that the peak of the irrelevant stimulus activation curve will be shallower (a) and occur earlier (b) for repetitions (right) than for non-repetitions (left).

Fig. 2.21 Illustration of how the magnitude of the repetition effect for irrelevant stimuli depends on what the irrelevant stimuli overlap with, and whether the bulk of the irrelevant stimulus activation curve is in the stimulus identi1cation or the response selection stage. As illustrated, the bulk of the irrelevant stimulus activation curve is in the stimulus identi1cation stage. Note that when the irrelevant stimulus repeats, that curve is shallower and peaks earlier than when it does not repeat, which reiterates what was shown on Fig. 2.20. The shaded and unshaded portions of the irrelevant activation curves show the amount of irrelevant activation in the stimulus and response stages, respectively. The fact that the difference between the two shaded portions of the curves (in the stimulus stage) is greater than between the two unshaded portions of the curves (in the response stage) generates the greater repetition effect of irrelevant stimuli for Types 4 (with S–S overlap) than for Types 3 (with S–R overlap), according to the Information Reduction Hypothesis.

43


44


Fig. 2.22 Simulation of the results for Experiment 4. See Fig. 2.19 for the empirical results.

level to which the irrelevant stimulus activation curve rises following a repetition, and to move its peak earlier in time (see Fig. 2.20). Because the overall irrelevant stimulus activation is now less for repetitions (see the bottom right panels in Fig. 2.20) than for non-repetitions (see the bottom left panel in Fig. 2.20), the in2uence of the irrelevant stimuli on performance will necessarily be less for repetitions than for non-repetitions. However, the magnitude of this effect depends on whether the irrelevant stimulus activation curve affects stimulus processing, as in the Type 4 tasks, or response processing, as in the Type 3 tasks. For example, suppose that most of the irrelevant stimulus activation curve lies in the stimulus identi1cation stage (see shaded areas of the curve in Fig. 2.21). This would produce a relatively large irrelevant stimulus repetition effect (e.g. Type 4) because the difference between the shaded areas for repetitions and non-repetitions is large. In the meantime, the amount of activation in the response stage (the unshaded areas of the curve in Fig. 2.21) is very small whether it is repetition or a non-repetition. As a result, there will be a very small, and perhaps undetectable, effect of irrelevant stimulus repetition in the Type 3 condition. Figure 2.22 shows the actual simulation of the data, illustrating numerically how these principles generate the reaction time for Experiment 4. The correspondence with the empirical data (Fig. 2.19) is quite good.

2.10 Summary and conclusions We began this chapter by outlining the representational and functional principles of the DO model, spelled out how these principles generated a taxonomy of tasks, selected four tasks from this taxonomy, and showed how, based on these principles, the structure of these tasks could be represented by a common processing architecture, and performance with them accounted for by the model. One of the effects of S–R compatibility that we had not considered in our model up to this point, however, was its interaction with the repetition effect (Bertelson 1963). These effects are ubiquitous and have a pervasive in2uence on RT. If the DO model is to be considered as having contributed to our understanding of performance in S–R compatibility tasks, then we needed to 1nd out whether, and, if so, how the model handled sequential effects. If it had turned out that these effects were beyond the model’s ability to deal with,



then, as Luce pointed out, the model would have been incomplete (at best) and probably wrong to boot (see Luce 1986). We reported the results of four experiments. In the 1rst two experiments we examined the sequential effects of relevant dimensions, congruent and incongruent mapping, and the repetition of physically identical, as well as conceptually similar, but physically different, stimuli in task Types 1 and 2. In the third experiment we looked at sequential effects of consistency (consistent, inconsistent, and neutral) in task Types 1, 3, and 4. In the fourth experiment we examined the sequential effects of relevant and irrelevant stimuli in task Types 1, 3, and 4. In the 1rst experiment we found a large repetition effect that interacted with congruent/incongruent mapping as well as with the repetition/non-repetition of conceptually similar stimuli. That is, the overall RT was longer and the repetition effect larger for incongruent than for congruent mapping. In addition, the mapping effect for repetitions was larger with conceptually similar (different carrier) than with physically identical (same carrier) stimuli In the second experiment, which was aimed at identifying the locus of the repetition effect, we again found an interaction between the repetition effect and congruent/incongruent mapping. This interaction was present whether the stimulus or the response was repeated, each in the absence of the other, which placed the locus of the repetition effect in both the stimulus and the response processing modules. This 1nding contrasts with earlier reports (e.g. Bertelson 1965; Pashler and Bayliss 1991) that attributed the bulk of the repetition effect to the repetition of the response. These results were accounted for by the DO model’s newly formulated Information Reduction Hypothesis, which states: information requirements on repeated trials are less than on non-repeated trials. According to this hypothesis, when a relevant stimulus or a response is repeated, the stimulus or response threshold on the repeated trial drops so that both the information and the time required to reach this lower threshold are reduced—hence the repetition effect. The results of Experiment 3 showed the expected differences in the effects of irrelevant stimuli for task Types 1, 3, and 4, none for Type 1, and robust consistency effects for Types 3 and 4. However, there was no signi1cant sequential effect of consistency for either Type 3 or Type 4 prime–probe pairs. The only sequential effect of consistency was the 1nding that the size of the S–R consistency effect in the Type 3 tasks was greater when the prime was neutral than when it was another Type 3 task. In Experiment 4, we obtained signi1cant repetition effects of the relevant stimulus, and response, in task Types 1, 3, and 4. Repetition of the irrelevant stimulus produced no signi1cant effects for task Types 1 and 3; however, that effect was signi1cant for task Type 4. These results were also accounted for by the Information Reduction Hypothesis. According to the hypothesis, when an irrelevant stimulus is repeated the information, and hence the time, required to distinguish between the relevant and irrelevant stimuli are both reduced. This was implemented in the model by reducing the value of the parameter (τ) following a repetition. Because a shorter value of (τ) causes the irrelevant stimulus input to start falling sooner than it otherwise would, the resulting irrelevant stimulus activation curve, following a repetition, has a shallower peak and is also shifted earlier in time so that a proportionately greater portion of the curve coincides in time with the stimulus module. The net result is for the repetition of an irrelevant stimulus that overlaps with the relevant stimulus (Type 4) to have a greater effect than the repetition of an irrelevant stimulus that overlaps with the response Type 3). Thus, the underlying reasoning for the repetition effects of relevant stimuli, irrelevant stimuli, and responses is the same: repetition leads to reduced information requirements which, in turn, leads to

45


46


faster processing. Depending on whether the relevant or irrelevant stimuli have DO, the repetition effect is accounted for by modifying one or the other of two parameters in the DO model, contingent on the occurrence of a repetition, thus leaving the basic mechanisms of the model intact.

Acknowledgements We are grateful for support from the Air Force Of1ce of Scienti1c Research Grant F496020-94-10020 and from The Horace H. Rackham School of Graduate Studies at the University of Michigan. We thank Anthony Whipple for technical support and discussions, and Greta Williams for assistance in carrying out these studies.

Notes 1. It is interesting to note that theories that deal with fundamental (i.e. irreducible) concepts (e.g. gravity) express the lawful relationships between the entities identi1ed (and de1ned) in the representational part of the theory. Such theories have no processing component because, in principle, these relationships are irreducible. Ecological theories, and so-called dynamic theories in psychology, often take this approach—prematurely and erroneously, in our opinion. Boyle’s law illustrates this point well. When it was 1rst formulated it expressed the systematic relationship between the pressure, volume, and temperature of an enclosed gas and was thought to be fundamental. It was not until Dalton’s atomic theory that a mechanism was discovered that could give rise to this relationship. This mechanism became the functional part of Boyle’s model. 2. We have included the Type 8, or Stroop, task in this table because of the broad interest that people have in it and also to show how, in accordance with DO principles, it could be parsed into separate components. In the rest of the article, however, we shall have nothing further to say about this task. 3. This time plays an important role later on in this paper in enabling the model to account for the sequential effects of irrelevant stimuli. 4. Thorough reviews of this literature exist that interested readers may wish to consult (Audley 1973; Kirby 1980; Kornblum 1973; Luce 1986). 5. Within the framework of the DO model there is no way to literally implement the version of the Residual Activation Hypothesis in which a process is bypassed without doing violence to the model itself and radically altering its structure. However, the duration of any process in the DO model could, in principle, be made arbitrarily small.

References Allport, A., Styles, E.A., and Hsieh, S. (1994). Shifting intentional set: Exploring the dynamic control of tasks. In C. Umiltà and M. Moscovitch (Eds.), Attention and performance XV, pp. 421–452. Cambridge, MA: MIT Press. Audley, R.J. (1973). Some observations on theories of choice reaction time: Tutorial review. In S. Kornblum (Ed.), Attention and performance IV, pp. 509–546. New York: Academic Press. Barber, P., and O’Leary, M. (1997). The relevance of salience: Towards an activational account of irrelevant stimulus–response compatibility effects. In B. Hommel and W. Prinz (Eds.), Theoretical Issues in Stimulus– Response Compatibility, pp. 135–172. Amsterdam: North Holland, Elsevier. Bertelson, P. (1961). Sequential redundancy and speed in a serial two-choice responding task. Quarterly Journal of Experimental Psychology, 12, 90–102.



Bertelson, P. (1963). S–R relationships and reaction times to new versus repeated signals in a serial task. Journal of Experimental Psychology, 65, 478–484. Bertelson, P. (1965). Serial choice reaction time as a function of response versus signal-and-response repetition. Nature, 206, 217–218. Bertelson, P. and Renkin, E. (1966). Reaction times to new vs. repeated signals in a serial task as a function of response–signal time interval. Acta Psychologica, 25, 132–136. Bertelson, P. and Tysseyre, F. (1966). Choice reaction time as a function of stimulus vs. response relative frequency of occurrence. Nature, 212, 1069–1070. Falmagne, J.C. (1965). Stochastic models for choice-reaction time with application to experimental results. Journal of Mathematical Psychology, 2, 11–127. Falmagne, J.C. and Theios, J. (1969). On attention and memory in reaction time experiments. In W.G. Köster (Ed.), Attention and performance II. A special issue of Acta Psychologica, 30, 316–323. Falmagne, J.C., Cohen, S.P., and Dwivedi, A. (1975). Two-choice reactions as an ordered memory scanning process. In P.M.A. Rabbitt and S. Dornic (Eds.), Attention and performance V, pp. 296–344. New York: Academic Press. Fitts, P.M. and Seeger, C.M. (1953). S–R compatibility: Spatial characteristics of stimulus and response codes. Journal of Experimental Psychology, 46, 199–210. Hommel, B. (1998). Event 1les: Evidence for automatic integration of stimulus–response episodes. Visual Cognition, 5, 183–216. Hommel, B., Müsseler, J., Ascherslebem, G., and Prinz, W. (in press). The theory of event coding (TEC): a framework for perception and action planning. Behavioral and Brain Sciences. Hyman, R. (1953). Stimulus information as a determinant of reaction time. Journal of Experimental Psychology, 45, 188–196. Keele, S.W. (1967). Compatibility and time sharing in serial reaction time. Journal of Experimental Psychology, 75, 529–539. Kirby, N. (1980). Sequential effects in choice reaction time. In A. Welford (Ed.), Reaction times, pp. 129–172. London: Academic Press. Kornblum, S. (1969). Sequential determinants of information processing in serial and discrete choice reaction time. Psychological Review, 76, 113–131. Kornblum, S. (1973). Sequential effects in choice reaction time: A tutorial review. In S. Kornblum (Ed.), Attention and performance IV, pp. 259–288. New York: Academic Press. Kornblum, S. (1992). Dimensional overlap and dimensional relevance in stimulus–response and stimulus–stimulus compatibility. In G.E. Stelmach and J. Requin (Eds.), Tutorials in motor behavior, Vol. 2, pp. 743–777. Amsterdam: Elsevier. Kornblum, S. (1994). The way irrelevant dimensions are processed depends on what they overlap with: The case of Stroop- and Simon-like stimuli. Psychological Research/Psychologische Forschung, 56, 130–135. Kornblum, S. and Lee, J.W. (1995). Stimulus–response compatibility with relevant and irrelevant stimulus dimensions that do and do not overlap with the response. Journal of Experimental Psychology: Human Perception and Performance, 21, 855–875. Kornblum, S., Hasbroucq, T., and Osman, A. (1990). Dimensional overlap: Cognitive basis for stimulus– response compatibility—A Model and taxonomy. Psychological Review, 97, 253–270. Kornblum, S., Stevens, G.T., Whipple, A., and Requin, J. (1999). The effects of irrelevant stimuli: 1. The time course of stimulus–stimulus and stimulus–response consistency effects with Stroop-like stimuli and Simonlike tasks, and their factorial combinations. Journal of Experimental Psychology: Human Perception and Performance, 25, 688–714. Leuthold, H. and Sommer, W. (1993). Stimulus presentation rate dissociates sequential effects in event-related potentials and reaction times. Psychophysiology, 30, 510–517. Luce, R.D. (1986). Response times. New York: Oxford University Press. McLeod, C.M. (1991). Half a century of research on the Stroop effect: An integrative review. Psychological Bulletin, 109, 163–203. Marcel, T. and Forrin, B. (1974). Naming latency and the repetition of stimulus categories. Journal of Experimental Psychology, 103, 450–460. Monsell, S. and Driver, J. (2000). Control of cognitive processes. In S. Monsell and J. Driver (Eds.) Attention and performance XVIII. Cambridge, MA: MIT Press. Mordkoff, J.T. (1999). The gating of irrelevant information in selective-attention tasks. Abstracts of the Psychonomic Society, 3, 21.

47


48


Pashler, H. and Bayliss, G. (1991). Procedural learning: 2. Intertrial repetition effects in speeded-choice tasks. Journal of Experimental Psychology: Learning, Memory and Cognition, 17, 33–48. Rabbitt, P.M.A. (1968). Repetition effects and signal classi1cation strategies in serial choice–response tasks. Quarterly Journal of Experimental Psychology, 20, 232–240. Remington, R.J. (1969). Analysis of sequential effects in choice reaction times. Journal of Experimental Psychology, 82, 250–257. Remington, R.J. (1971). Analysis of sequential effects for a four-choice reaction time experiment. Journal of Psychology, 77, 17–27. Rogers, R.D. and Monsell, S. (1995). Costs of a predictable switch between simple cognitive tasks. Journal of Experimental Psychology: General, 124, 207–231. Schvaneveldt, R.W., and Chase, W.S. (1969). Sequential effects in choice reaction time. Journal of Experimental Psychology, 80, 1–8. Shaffer, L.H. (1965). Choice reaction with variable S–R mapping. Journal of Experimental Psychology, 70, 284–288. Shiu, L.-P. and Kornblum, S. (1996). Negative priming and stimulus–response compatibility. Psychonomic Bulletin and Review, 3, 510–514. Simon, J.R. (1990). The effects of an irrelevant directional cue on human information processing. In R.W Proctor and T.G. Reeve (Eds.), Stimulus–response compatibility: An integrated perspective, pp. 31–86. Amsterdam: North-Holland. Smith, M.C. (1968). Repetition effect and short-term memory. Journal of Experimental Psychology, 77, 435–439. Soetens, E. (1998). Localizing sequential effects in serial choice reaction time with the information reduction procedure. Journal of Experimental Psychology: Human Perception and Performance, 24, 547–568. Soetens, E., Boer, L.C., and Hueting, J.E. (1985). Expectancy or automatic facilitation? Separating sequential effects in two-choice reaction time. Journal of Experimental Psychology: Human Perception and Performance, 11, 598–616. Squires, K.C., Wickens, C., Squires, N.K., and Donchin, E. (1976). The effect of stimulus sequence on the waveform of the cortical event-related potential. Science, 193, 1142–1146. Stevens, G.T. (2000). The locus of Eriksen, Simon and Stroop Effects: New data and a comparison of models. Ph.D. Dissertation, University of Michigan, Ann Arbor. Stevens, G.T. and Kornblum, S. (2000). Goals and dimensional overlap: The effects of irrelevant response dimensions. Poster presented at the XIXth International Symposium on Attention and Performance, Kloster Irsee, Germany, July 16–22, 2000. Stevens, G.T. and Kornblum, S. (2001). The locus of consistency effects in Eriksen and Stroop-like tasks. Manuscript in preparation. Stroop, J.R. (1935). Studies of interference in serial verbal reactions. Journal of Experimental Psychology, 18, 643–661. Treisman, A. (1988). Features and objects: The fourteenth Bartlett Memorial Lecture. Quarterly Journal of Experimental Psychology, 47A, 201–237. Vervaeck, K.R. and Boer, L.C. (1980). Sequential effects in two-choice reaction time: Subjective expectancy and automatic after-effect at short response–stimulus intervals. Acta Psychologica, 44, 175–190. Williams, J. (1966). Sequential effects in disjunctive RT: Implications for decision models. Journal of Experimental Psychology, 71, 665–672. Zhang, H. and Kornblum, S. (1998). The effects of stimulus–response mapping and irrelevant stimulus– response and stimulus–stimulus overlap in four-choice Stroop tasks with single carrier stimuli. Journal of Experimental Psychology: Human Perception and Performance, 24, 3–19.



Appendix A Experiment 1 Same carrier

Congruent Neutral Incongruent

Different carrier

Rep

Non-Rep

Rep

Non-Rep

0.0 2.1 0.0

0.7 1.9 3.7

0.7 0.7 1.4

1.4 3.2 4.9

Experiment 2 Type 2 probe

Type 1 probe

Rep

Non-Rep

Rep

Non-Rep

1.0

4.2

2.1

Cong

Stim. Rep/Non-Rep

1.0

Resp. Rep/Non-Rep

0.0

1.0

0.0

3.1

Incong

Stim. Rep/Non-Rep Resp. Rep/Non-Rep

0.4 3.2

1.1 2.5

3.5 2.5

1.7 2.5

Experiment 3 Prime

Probe Type 3 Consist

Consistent Neutral Inconsistent

Neut

4.2 2.1 2.1

2.1 2.1 1.0

Type 4 Inconsist

Consist

0.0 0.0 0.0

4.2 0.0 3.1

Neut 1.0 2.1 0.0

Inconsist 2.1 3.1 4.2

Experiment 4 Irrel S

Rel S Type 1

Rep Non-Rep

Type 3

Type 4

Rep

Non-Rep

Rep

Non-Rep

Rep

Non-Rep

0.0 1.6

3.1 3.1

0.5 1.0

2.6 1.6

1.0 0.5

3.1 1.6

Error rates at the 700 ms RSI for the four experiments reported.

49


50


Appendix B Experiment 1 Type 2 → Type 2; Congruent RSI Prime

Probe

700

1500

3000

Same carrier

red → ‘red’ blue → ‘blue’

red → ‘red’ red → ‘red’

463 (27) 484 (28)

460 (24) 488 (30)

473 (17) 490 (25)

Rep Non-Rep

Different carrier

RED → ‘red’ BLUE → ‘blue’


489 (29) 490 (27)

494 (31) 495 (24)

494 (19) 492 (25)

Rep Non-Rep

Type 2→Type 2; Incongruent RSI Prime

Probe

700

1500

3000

Same carrier

red → ‘green’ blue → ‘yellow’ green → ‘blue’ blue → ‘yellow’

red → ‘green’ red → ‘green’ red → ‘green’ yellow → ‘red’

618 (42) 853 (125) 890 (145) 871 (81)

629 (86) 849 (157) 905 (199) 863 (168)

730 (68) 872 (112) 912 (123) 895 (122)

Rep Non-Rep NR (S→R) NR (R→S)

Different carrier

RED → ‘green’ BLUE → ‘yellow’ GREEN → ‘blue’ BLUE → ‘yellow’

red → ‘green’ red→‘green’ red → ‘green’ yellow → ‘red’

753 (45) 885 (98) 933 (154) 915 (122)

738 (61) 842 (157) 935 (236) 915 (193)

798 (85) 875 (120) 901 (112) 908 (122)

Rep Non-Rep NR (S→R) NR (R→S)

Type 1→Type 1 RSI Prime

Probe

700

1500

3000

Same carrier

red → ‘two’ blue → ‘four’

red → ‘two’ red → ‘two’

582 (37) 682 (38)

593 (54) 695 (62)

623 (50) 691 (43)

Rep Non-Rep

Different carrier

RED → ‘two’ BLUE → ‘four’


635 (62) 690 (44)

650 (76) 711 (65)

677 (52) 700 (42)

Rep Non-Rep

Mean RTs (and standard deviations) for Experiment 1 at three values of RSI. Because all possible prime–probe pairs were randomized and equiprobable, and the interval between prime–probe pairs was three seconds, RTs for RSI values of three seconds were obtained by considering the probes of regular prime–probe pairs as primes, and the primes of the next regular pair as probes of new pairs. Even though the RTs for the 700 ms RSI are discussed in detail in the text, we have included them in this table for ease of comparison. The stimuli and responses for the primes and probes are prototypical, generic descriptions. Thus, for example, the probe stimuli in this table are all shown as color



patches; however, we know (see text) that the probe stimuli were either color patches or color words. This generic description is intended to encompass both cases, and the data shown are averaged over both cases. Note also that we distinguish between three cases of non-repetitions. The 1rst is the case of pure, or total, non-repetitions in which no aspect of the prime is repeated in the probe. When we speak of ‘non-repetitions’ in the text, these are the trials to which we refer. The second is the case in which the label of, or the congruent responses to, the stimulus on the prime becomes the response on the probe (S → R). We view these as negative priming (NP) trials (see Shiu and Kornblum 1996). At RSI of 700 ms, the RT for this case is signi1cantly longer than for the total non-repetitions [F(1, 5) = 10.85, p < 0.0216], which con1rms Shiu and Kornblum’s (1996) earlier 1ndings. The third is the case in which the response on the prime trial becomes the label of the probe stimulus (R → S). Even though the RT for this case is also longer than for the total non-repetitions (but shorter than on NP trials) this difference is not statistically signi1cant [F(1, 5) = 2.12, p < 0.2051]. The statistical results of RSI for this experiment are summarized below:

Type 1 tasks

• Main effect: the longer the RSI, the slower the RT [F(2, 6) = 7.61, p < 0.0226)]; • Interesting to note that as RSI increases the repetition effect decreases. This effect is not statistically signi1cant. Nevertheless, the fact that this appears to be due principally to the fact that as RSI increases the RT for repetitions increases more than for non-repetitions is consistent with the Information Reduction Hypothesis: as RSI increases the threshold goes back to normal thus reducing the advantage of the reps.

Type 2 tasks

• RSI has no signi1cant main effect with Type 2 tasks, either congruent or incongruent; • RSI has an 8 ms interaction with carrier for congruent mapping, which is signi1cant [F(2, 10) = 8.01, p < 0.0084].

Appendix C Experiment 2 Type 1 → Type 2; Congruent Prime

Probe

RSI 700

1500

3000


2 → ‘two’ 4 → four’

533 (53) 565 (58)

555 (85) 566 (64)

547 (51) 562 (53)

Resp Rep Resp Non-Rep


red → ‘red’ blue → ‘blue’

565 (71) 583 (71)

568 (80) 580 (73)

550 (55) 564 (70)

Stim Rep Stim Non-Rep

51


52


Appendix C Continued Type 1→Type 2; Incongruent Prime

Probe

RSI 700

1500

3000

red → ‘two’ red → ‘two’ red → ‘two’

4 → ‘two’ 6 → ‘eight’ 2 → ‘six’

861 (132) 962 (173) 920 (169)

902 (162) 954 (178) 912 (182)

949 (190) 974 (187) 986 (209)

Resp Rep Resp Non-Rep Non-Rep (R→S)

red → ‘two’ red → ‘two’ red → ‘two’

red → ‘green’ blue → ‘yellow’ yellow → ‘red’

895 (148) 986 (186) 964 (183)

916 (179) 988 (212) 980 (216)

968 (194) 990 (205) 1049 (224)

Stim Rep Stim Non-Rep Non-Rep (S→R)

Type 2→Type 1; Congruent Prime

Probe

RSI 700

1500

3000

red → ‘red’ red →‘red’

2 →‘red’ 4 →‘blue’

793 (110) 860 (114)

835 (156) 907 (187)

830 (132) 862 (150)

Resp Rep Resp Non-Rep


red →‘two’ blue →‘four’

808 (126) 847 (96)

823 (104) 862 (133)

853 (168) 864 (149)

Stim Rep Stim Non-Rep

Type 2→Type 1; Incongruent Prime

Probe

RSI 700

1500

3000

red → ‘green’ red → ‘green’ red → ‘green’’

8 → ‘green’ 4 → ‘blue’ 2 → ‘red’

844(112) 894 (121) 872 (119)

845 (115) 882 (111) 894 (139)

892 (110) 897 (110) 900 (127)

Resp Rep Resp Non-Rep Non-Rep (S→R)

red → ‘green’ red → ‘green’ red → ‘green’

red →‘two’ blue → ‘four’’ green → ‘six’

963 (128) 974 (126) 931 (133)

971 (153) 966 (120) 951 (163)

1039 (144) 1036 (149) 1018 (147)

Stim Rep Stim Non-Rep Non-Rep (R→S)

Mean RTs (and standard deviations) for Experiment 2 at three values of RSI. As was true of Appendix B, the stimuli and responses for the primes and probes are prototypical cases, or generic descriptions. Thus, for example, the prime stimuli in this table are all shown as red color patches; however, as is made clear in the text, not only were there four different color patches, but the stimuli on prime trials could also be digits. The generic descriptions in this table are, therefore, intended to encompass both cases, and the data shown are averaged over both cases. Again, similarly to what we did in Appendix B, when the mapping is incongruent we distinguish between three different cases of non-repetitions: total non-repetitions; non-repetitions in which the label of, or the congruent response to, the prime stimulus becomes the response on the probe; and non-repetitions in which the response on the prime becomes the label on the probe. In contrast to the



results of Experiment 1, none of the differences between the total non-repetitions and the other nonrepetition case within a particular incongruent condition are signi1cant. The statistical results of RSI for this experiment are summarized below:

• Main effect: The longer the RSI the slower the RT [F(2, 62) = 6.24, p < 0.0034]. Type 1 → Type 2

• The increase of RT with RSI was greater for repetitions than for non-repetitions: this interaction is signi1cant for stimulus rep/non-rep [F(2, 46) = 4.24, p < 0.0204], and response rep/non-rep [F(2, 46) = 6.8, p < 0.0026].

Type 2 → Type 1

• RSI interacted with response repetition whether the mapping of the prime was congruent [F(2, 14) = 3.95, p < 0.0436], or incongruent [F(2, 46) = 4.37, p < 0.0183].

Appendix D Experiment 3 RSI 700

1500

Con.

Incon.

Neut.

Con.

Incon.

Neut.

Type 3 ( +1)

Con. Incon. Neut.

572 (80) 586 (75) 557 (71)

659 (63) 659 (81) 697 (108)

616 (86) 614 (63) 614 (68)

606 (118) 627 (105) 637 (103)

681 (105) 687 (107) 710 (145)

634 (98) 709 (134) 633 (90)

Type 4 ( +1)

Con. Incon. Neut.

594 (77) 609 (52) 598 (92)

640 (78) 687 (128) 657 (91)

605 (77) 619 (115) 589 (78)

627 (96) 627 (87) 627 (99)

682 (101) 695 (115) 710 (120)

634 (99) 655 (112) 604 (100)

Mean RT (and standard deviations) for Experiment 3 at the two values of RSI that were used. RSI did not have any statistically signi1cant effects in this experiment, although the same trend is observable here as in the other experiments: the longer the RSI, the slower the RT. Experiment 4 RSI 700

1500

Rel. Rep

Rel. Non-Rep

Rel. Rep

Rel. Non-Rep

Type 1 nn

Irr. Rep Irr. Non-Rep

470 (43) 469 (30)

559 (77) 561 (45)

506 (60) 497 (63)

585 (60) 593 (65)

Type 3 ii


498 (75) 491 (58)

606 (105) 616 (114)

520 (94) 521 (82)

619 (108) 612 (121)

53


54


Experiment (Continued) RSI 700

Type 4 ii


1500

Rel. Rep

Rel. Non-Rep

Rel. Rep

Rel. Non-Rep

484 (63) 516 (78)

599 (78) 620 (71)

538 (67) 543 (73)

616 (90) 631 (88)

Mean RT (and standard deviations) for Experiment 4 at the two values of RSI that were used. The row labels are for the irrelevant (Irr.) transitions; the column labels are for the relevant (Rel.) transitions. Included in the Task type identi1cation for the rows are reminders of consistency status of the prime–probe pairs. Thus, nn indicates that both the prime and probe were neutral, ii indicates that both the prime and the probe were inconsistent. The statistical results of RSI for this experiment are easily summarized: the longer the RSI the slower the RT. This is signi1cant for Type 1 [F(1, 8) = 6.49, p < 00343], Type 3, and Type 4 [F(1, 8) = 10.05, p < 0.0132]. There are no signi1cant interactions. However, In Type 3 the effect of relevant stimulus repetitions appears to decrease as RSI increases, and in Type 4 there appears to be that same trend except that now it is for the irrelevant stimulus whose effect of repetition decreases as RSI increases.


I Space perception and spatially oriented action




3 Perception and action: what, how, when, and why Introduction to Section I Glyn W. Humphreys

3.1 Perception and action can dissociate Over the past decade, a good deal of evidence has accumulated that perception and action can dissociate (see Milner and Goodale 1995, for one summary). Dissociations can occur even when perception and action are tested using the same object, and even when input from the same modality is used (e.g. vision). This leads to a counter-intuitive conclusion: that the information, and the underlying brain systems, may differ when we perceive and recognize a cup (on the one hand) and when we make a grasp action to pick it up (on the other). The papers reported in this section of the book are concerned with the relations between perception and action, and they provide state-of-the-art summaries of work on this important topic. The papers detail not only how perception and action can dissociate but also how they interact. This is clearly important if we are to understand how coherent behaviour emerges, and the papers presented here provide interesting suggestions as to how forms of integration can take place. In the chapters reported by Rossetti and Pisella, as well as Bridgeman, the evidence on dissociations between perception and action, even with single objects, is discussed. As with all of the chapters in this section, the use of visual information for perception and action is emphasized. One source of evidence here comes from studies of perceptual illusions. For example, perceptual judgements to a part of a display can be strongly in2uenced by surrounding parts (e.g. as in the Titchener–Ebbinghaus size illusion). In contrast, actions to pick up a local stimulus can be much less affected by the surrounding context (e.g. so that grasp apertures do not show the size illusion; see Aglioti, DeSouza, and Goodale 1995; Haffenden and Goodale 1998). The paper by Bridgeman here illustrates this. Bridgeman reports work on an adapted form of the Roelofs effect (Roelofs 1935), in which perceptual judgements about the location of a target are affected by its position relative to a surrounding frame, instead of being based on the target’s absolute position with respect to the viewer. Bridgeman shows that this Roelofs effect disappears if observers make a rapid jab to the target’s location. The effect of the rectangle context on action is less than on perception. This occurs even when observers are prevented from 1xating on the target, suggesting that actions are being directed by a relatively rich sensorimotor representation rather than simple information about where the eyes are currently looking (see Bridgeman, this volume, Chapter 5). Rossetti and Pisella also review neuropsychological evidence for the perception–action dissociation. This includes optic ataxic patients with parietal damage who show intact perception but impaired


58


action to objects (e.g. misreaching under visual guidance), and agnosic patients (often with occipitotemporal damage) who have impaired perception along with intact action (e.g. orienting the hand appropriately when reaching even when unable to make simple orientation discriminations in perceptual judgement tasks). The distinction between these two classes of patient supports the separation between an occipito-parietal (dorsal) route in vision which supports visually-guided action, and an occipito-temporal (ventral) route supporting perceptual judgements and object recognition. The dorsal and ventral visual pathways in the cortex were originally linked to the processes of computing ‘where’ (spatial location) and ‘what’ objects were (Ungerleider and Mishkin 1982). However, the evidence on intact action in agnosia has been used to motivate the further argument that the dorsal route is involved not just in computing where an object is but also ‘how’ an action should be made (e.g. how an action should be shaped to pick up an object; see Milner and Goodale 1995). In these cases patients appear to have intact access to forms of representation supporting one but not the other task. To this can be added other evidence. For example, work on patients with visual neglect after parietal damage has indicated that more neglect may be expressed when patients have to point to the centre of a rod than when they have to pick it up (Robertson, Nico, and Hood 1995). Bridgeman (this volume, Chapter 5) suggests that, even though pointing is a motor action, it often plays a role in communication and so may be controlled through a recognition rather than an action pathway. If so, then the point–grasp dissociation in neglect would again 1t with the idea of one pathway being disturbed (the perceptual recognition route) and the other intact (the action route). I return to discuss an alternative view of this below.

3.2 When perception influences action Although there is considerable evidence that ‘pulls apart’ perception and action, other work highlights occasions on which the processes involved in object recognition in2uence action. Several findings are discussed by Rossetti and Pisella (this volume, Chapter 4), and I mention only two that pick up on the dissociation between optic ataxia and neglect noted above. One is that, in studies of agnosia, patients can be impaired when required to use a representation of the relative locations of stimuli to direct action (e.g. placing the 1ngers in the holes of a ten-pin bowling ball! See Dijkerman, Milner, and Carey 1998). In this case, the perceptual impairment in the patient seems to carry-over into their actions. A second is that the poor reaching behaviour in optic ataxic patients can be improved when they grasp a known relative to an unknown object (Jeannerod, Decety, and Michel 1994). Here intact perceptual processes in the patient seem to improve action. Other examples come from the chapters by Rieser and Pick, and by Graziano and Botvinick (this volume, Chapters 8 and 6, respectively). Rieser and Pick discuss effects of perceptual representations on the reconstruction of action, when participants have to negotiate large environments. In a series of ingenious experiments, they demonstrate that vision helps recalibrate locomotive activity. They propose that ‘when walking with vision or without it, people tend to perceive their locomotion relative to the surrounding environment as a frame of reference, and their perception serves to update their representation of their spatial orientation. The resulting representation of spatial orientation is unitary—it re2ects the perception of locomotion, it serves as the basis for control of continuing locomotion, and it serves as the basis for the control of all other environmentally-directed actions.’ Graziano and Botvinick report on physiological studies in monkeys that demonstrate interactions between visual and proprioceptive input—as when cells have visual receptive 1elds that are tuned to the position of a hand in space (and not on the retina). They argue that ‘the body schema is used to


Perception and action: what, how, when, and why

cross-reference between different senses, as a basis for spatial cognition and for movement planning’. Here they also maintain a common representation underlying visual perception and motor actions.

3.3 When action affects visual perception As well as visual perception and object recognition sometimes in2uencing action, effects can go in the reverse direction. The chapters by Graziano and Botvinick and by Jordan et al. (this volume, Chapter 7) illustrate this. Jordan et al. discuss experiments showing that perceived displacements in the location of a visual target are biased in the direction of a planned action. This effect is not due to the action being made per se, and it extends in time even beyond the duration required to perform the action. They suggest that ‘action plans contribute to, and shape, perceptual space’. Graziano and Botvinick report cells that only respond to the visual presence of stimuli if they are close to a hand being used for action. Analogies can also be found in experimental studies with humans. Tipper and colleagues (Tipper, Lortie, and Baylis 1992), for instance, have provided evidence that visual attention can be locked to the position of the hand with respect to a target for action. If you reach with your hand directly out to an object then irrelevant stimuli lying in front of the object may be suppressed; however if you move your hand back to the object then irrelevant stimuli before the target are not suppressed but those behind are. In these instances, a planned action in2uences perception. There is also neuropsychological evidence along the same lines. For example, Berti and Frassinetti (2000) report a patient who showed neglect of ‘near’ but not ‘far’ space (e.g. poor bisection of a line close to the body but not when it was shown far away, even when matched for visual angle). Interestingly, neglect could be induced even for distant stimuli if the patient responded using a long pointer. Berti and Frassinetti argue that the neglected representation of ‘near space’ was reconstituted by using a pointer for action. The opposite pattern has been found by Ackroyd and colleagues (Ackroyd, Riddoch, Humphreys, and Townsend, in press). Here the patient had neglect of far and left space in visual detection tasks. However, when given a pointer to hold, the neglected areas decreased. Importantly, this patient demonstrated aspects of visual neglect, rather than his neglect being only ‘motoric’ in nature (e.g. when required to use mirrored feedback he responded to stimuli seen on his right, though this involved moving towards his impaired, left side). The extension of the patient’s body space, by using a pointer, ameliorated the degree of visual neglect. The physiological basis of such effects may be the bimodal cells discussed by Graziano and Botvinick (this volume, Chapter 6), who also report work showing that the receptive 1elds of such cells extend to an implement held in the hand used for an action.

3.4 When do perception and action interact? The chapters by Rieser and Pick, Graziano and Botvinick, and Jordan et al. here all provide examples of interactions between perception and action, in one direction or the other. Rossetti and Pisella, in their very thorough overview chapter, consider the implications. One proposal they consider is that perception–action interactions are contingent on the time parameters governing behaviour. We can understand perception–action relations by understanding when behaviour is constrained by each factor. They suggest that there is a fast action route (through dorsal cortex) and a slow action route (through ventral cortex). The dorsal route may be used for actions that operate on-line and perhaps

59


60


also in an open-loop fashion, using visual feedback. The ventral route may be used for action, but only when actions are delayed or not contingent on on-line feedback. They present evidence from their laboratory on this. For example, normal participants show rapid use of perturbations in the locations of targets when guiding their on-line reaching, but fail to use changes in target colour in the same way. Location information may be computed via the dorsal visual stream, but colour only through the ventral stream. The evidence on the contrast and between pointing and grasping in neglect may be understood in the same way. Thus Martin Edwards and I (Edwards and Humphreys 1999) found that the improved behaviour when a neglect patient grasped rods was due to corrections taking place at the end of the reach trajectory, due to on-line feedback. It is possible to argue that, in such cases, neglect is due to an impaired representation used in perceptual judgements. This can be overcome by on-line feedback via the action route. Pointing may be worse than grasping either because pointing is always dependent on the perceptual-recognition system (e.g. being used in communicative acts) or because pointing is less dependent on feedback in any case. The chapter by Rieser and Pick takes a different view, suggesting instead that there is always an integrated representation of perception and action. They present evidence for such integrated representations. However, it is not clear whether this evidence emerges because of the time parameters in the studies. These time parameters are typically long—as participants carry out tasks walking around their environment, and this may encourage involvement of the slow ventral route (Rossetti and Pisella, this volume, Chapter 4). Also action recall rather than on-line control is sometimes measured, and this may be crucial. A further possibility is that there may be effects of spatial distance and/or scale. The studies of Rieser and Pick typically take place over large-scale areas, and actions can be dictated by stimuli presented far from the body—unlike the hand actions made close to the body used in many studies demonstrating dissociations between perception and action (e.g. Bridgeman, this volume, Chapter 5). Such dissociations may occur only under conditions in which immediate actions are made in which a participant directly interacts with their environment. Other ways to consider when perception and action interact are proposed by Bridgeman and Jordan et al. (both this volume, Chapters 5 and 7). Bridgeman suggests that differences may emerge due to the kinds of spatial representation being computed. He suggests that the dorsal route operates in absolute, egocentric coordinates, whilst the ventral route codes relative location information between stimuli (an allocentric coding scheme). Some of the work on impaired action in agnosia is consistent with this (Dijkerman et al. 1998). On this view, perception and action dissociate when responses are based on an egocentric reference frame but they interact when actions use an allocentric frame. We may think of Rieser and Pick’s studies in this light too, since their work on locomotion typically examines the use of allocentric coding of the environment. Jordan et al. (this volume, Chapter 7) consider a third alternative, which is that what counts is whether behaviour is controlled by temporally proximal or temporally distal events. Perception and action can dissociate when actions are made to temporally proximal events, but they are integrated when actions are to be made to temporally distal events—and they remain integrated even when, later in time, actions are made to the stimulus then present. On this view, perception and action are brought together in anticipatory planning.

3.5 How do perception and action interact? As well as considering the issue of when perception and action interact, we may also consider how such an interaction takes place. Graziano and Botvinick (this volume, Chapter 6) suggest that such


Perception and action: what, how, when, and why

interactions may be mediated through the body schema, which is constantly re-calibrated by on-line perceptual information. Jordan et al. (this volume, Chapter 7) propose the ‘common coding’ scheme suggested by Prinz (1997). They argue that the planning of actions to distal events recruits processes also involved in the perception of the consequences of the action. It may be that common coding is effected through the kind of body scheme representation outlined by Graziano and Botvinick. Though how this form of representation, thought to underlie perception-action interactions, links to the idea of allocentric coding for such interactions (Bridgeman, this volume, Chapter 5; Rieser and Pick, this volume, Chapter 8), is somewhat unclear and remains for future work to specify. Whatever account is 1nally formulated, the evidence reviewed in this part of the book indicates that there are both dissociations and interactions between perception and action. Rossetti and Pisella’s overview chapter provides a clear account of many of the relevant 1ndings along with a well-articulated view of how perception and action can be interrelated. The chapter by Bridgeman emphasizes the dissociation between perception and action, whilst the chapters by Rieser and Pick, Graziano and Botvinick, and Jordan et al. emphasize perception–action interactions. The work reported here stimulates questions about the best way to conceptualize both the dissociations and interactions, as well as highlighting the need for us to develop more detailed processing models of perceptuo-motor integration.

Acknowledgement This work was supported by grants from the Medical Research Council and the Wellcome Trust.

References Ackroyd, K., Riddoch, M.J., Humphreys, G.W., and Townsend, S. (in press). When near becomes far and left becomes right: Using a tool to extend extrapersonal visual space in a patient with severe neglect. Neuropsychologia. Aglioti, S., DeSouza, J.F., and Goodale, M.A. (1995). Size-contrast illusions deceive the eye but not the hand. Current Biology, 5, 679–685. Berti, A. and Frassinetti, F. (2000). When far becomes near: Remapping of space by tool use. Journal of Cognitive Neuroscience, 12, 415–420. Dijkerman, H.C., Milner, A.D., and Carey, D.P. (1998). Grasping spatial relationships: Failure to demonstrate allocentric visual coding in a patient with visual form agnosia. Consciousness and Cognition, 7, 424–437. Edwards, M.G. and Humphreys, G.W. (1999). Pointing and grasping in unilateral visual neglect: Effect of on-line visual feedback in grasping. Neuropsychologia, 37, 959–973. Haffenden, A. and Goodale, M.A. (1998). The effect of pictorial illusion on prehension and perception. Journal of Cognitive Neuroscience, 10, 122–136. Milner, A.D. and Goodale, M.A. (1995). The visual brain in action. London: Academic Press. Prinz, W. (1997). Perception and action planning. European Journal of Cognitive Psychology, 9, 129–154. Robertson, I., Nico, D., and Hood, B.M. (1995). The intention to act improves unilateral neglect: two demonstrations. NeuroReport, 7, 246–248. Roelofs, C. (1935). Optische Localisation. Archiv für Augenheilkunde, 109, 395–415. Tipper, S.P., Lortie, C., and Baylis, G.C. (1992). Selective reaching: Evidence for action-centred attention. Journal of Experimental Psychology: Human Perception and Performance, 18, 891–905. Ungerleider, L.G. and Mishkin, M. (1982). Two cortical visual systems. In D. Ingle, M.A. Goodale, and R.J.W. Mans1eld (Eds.), Analysis of visual behavior, pp. 49–86. Cambridge, MA: MIT Press.

61


4 Several ‘vision for action’ systems: a guide to dissociating and integrating dorsal and ventral functions (Tutorial) Yves Rossetti and Laure Pisella

Abstract. There is a well-established argument for a double-dissociation between vision for action and vision for conscious identi1cation. The distinction between these two visual systems applies to both the attributes being processed and the outputs of the processing. However, numerous direct and indirect interconnections and convergence have been described between cortical and subcortical visual pathways as well as between the dorsal and the ventral streams. This chapter presents an attempt to solve this apparent contradiction between neuroanatomy and behaviour, by organizing our knowledge about several aspects of how vision can be involved in the various aspects of action. First, several cognitive → motor interactions can be observed, which suggest that action can rely on cognitive representations. Reciprocally, examples of sensorimotor → cognitive interactions have been provided by the reorganization of cognitive representation of space induced by sensorimotor plasticity. Second, it is shown that introducing a memory delay between the target presentation and the action emulates a cognitive representation of the target that is used for action. Conversely, adding a speed constraint to a simple pointing task seems to allow a speci1c activation of the sensorimotor system: for fast movements no in2uence of cognitive representations or of intention is observed. Neuropsychological data suggest that the most typical function of the dorsal stream is the on-line control of an ongoing goal-directed action. It is concluded that, depending on the time-scale considered, no interaction or two-way interactions between the dissociated vision for action and vision for identi1cation can be observed. These functional data are fully compatible with the temporal constraints of the complex anatomical network involved in the processing of visual information, in relation to fast (magnocellular) and slow (parvocellular) streams. Recipes are proposed to isolate or integrate a sensorimotor or a cognitive sensory system, according to the type of stimulus, the type of response, and the temporal link between the stimulus and the response.

4.1 Perception and action: a perfect couple? The question of the relationship between action and perception is central to many areas of philosophy, psychology, and neuroscience. The perception–action couple has been said by ecological psychologists to be inalterably inseparable, whereas speci1c experimental conditions and neurological patients have suggested that a divorce can be achieved. One of the two partners, Action, is the more forthright and does not trouble itself with non-concrete aspects of life. Therefore everybody agrees about what it is up to. But so many people have 2irted with Perception that everyone has their own view of it. Some think that it is only able to deal with proper mental objects which can be manipulated by the mind, while others believe it is a simple-minded character who is under the direct in2uence of the senses. Still others have rather argued that it is a thoughtful creature that is very aware of objects or events, and tends to interpret them in the most appropriate fashion. It has


Several ‘vision for action’ systems: A guide to dissociating and integrating dorsal and ventral functions

also been claimed that perception has an impressionable character, being easily in2uenced by its environment. With such a many-faceted personality, it is no wonder that Perception and Action have a rather chaotic relationship. In order to go beyond these vaudevillesque considerations and try to consider their relationship in a less con2ictual way, let us specify which view of perception will be adopted in the present chapter. Following the usage of Milner and Goodale (1995), we will consider perception as an integrated process that gives rise to identi1cation. The usual way to investigate the content of perception implies that this content is mediated by conscious awareness before an output (e.g. a verbalization) can be produced. In more general terms, this way of dealing with sensory inputs has been termed ‘cognitive’ processing by Bridgeman (1991) and Paillard (1987, 1991), as opposed to ‘sensorimotor’ processing. Most of the current debates on sensorimotor versus cognitive processing refer respectively to dorsal and ventral streams of the visual brain. As will be detailed in a further section (4.3.2.), the dorsal stream is de1ned here as the projections from the primary visual area to the posterior parietal cortex, in particular to the superior parietal lobule. The ventral stream can be de1ned as the projections from the occipital visual areas to the inferior temporal cortex. Based on neuropsychological observations reviewed below, these two streams have sometimes been associated with implicit (non-conscious) versus explicit (conscious) processing. This issue will be discussed at the end of the present chapter (Section 4.8.4.) Following this introductory section (4.1), we will 1rst summarize the evidence for the dissociability between perception and action in both normals and brain-damaged patients (4.2). Then we will demonstrate that the complex neuroanatomical networks involved in vision, perception, and action do not show a strict segregation between two cortical visual pathways (4.3). Our main aim will be to review the different ways by which this apparent gap between anatomy and behaviour can be 1lled (4.4). On the one hand, several lines of evidence for double interactions between the two visual systems will be presented. It will turn out that both visual systems can contribute to action, suggesting that neuroanatomical data are right (4.5). On the other hand, we review several aspects of the effect of time variables on space processing by the visual system, which suggest that pure anatomical data are not suf1cient to account for behavioural observations. We will argue that dissociations between cognitive and sensorimotor processing can result from temporal limits of visual processing (4.6). Then some simple and more complex experimental recipes will be proposed for either isolating or integrating dorsal and ventral types of function (4.7). Among others, the method of choice for isolating the sensorimotor mode of vision in normals appears to be to apply time constraints to the task. The purest expression of the dorsal processes may consist of an ‘automatic pilot’ able to online drive the hand to a selected visual target, irrespective of the subject’s own intention. To end with, a few concluding remarks will be made on neglected aspects of visuomotor processing (4.8). Instead of seeing dichotomies between sensorimotor and cognitive, dorsal and ventral, implicit and explicit processes, we propose that transitions between these aspects can be viewed as continuous gradients.

4.2 Dissociations The discovery of re2ex reactions at the end of the nineteenth century has given rise to an enormous amount of experimental and theoretical work. The discovery of unconscious nervous processes, as already postulated by von Helmholtz in the case of visual perception, opened new areas of investigation of the mind (e.g. psychoanalysis) and of perception and behaviour. It is interesting to

63


64


note that the pioneering work of Helmoltz or Freud not only emphasized the distinction between the conscious and the unconscious but also clearly addressed the issue of the interaction between these two aspects of mental life. Unfortunately, for about one century there has been more and more attraction towards the power of unconscious processes as opposed to the conscious mental life, and the report of dissociation between the conscious and the unconscious has become more fashionable than it really deserves to be. This bias has applied to the study of sensory and motor processes separately, and especially to the distinction between implicit processing for action and explicit processing for perception (review: Milner and Goodale 1995; Place 2000; Rossetti and Revonsuo 2000b). As a consequence, more is known about the dissociation than about the interaction between sensorimotor and cognitive processes.

4.2.1 The double-step paradigm The double-step paradigm refers to experimental conditions where a visual target is 1rst presented to the subject (step one: between 1xation point and the target), and then displaced during the action (step two: between the initial target position and the secondary target position). Psychophysical studies have revealed that human subjects are unaware of displacements occurring in the visual world if these displacements are synchronized with the saccade (see e.g. Bridgeman, Hendry, and Stark 1975). Several experiments have explored the consequence of this saccadic suppression phenomenon, which refers to the apparent loss of perception occurring during saccades (Campbell and Wurtz 1978), on arm movement production. In one early experiment, subjects were asked to point at a target that had been displaced during the saccade (by a stroboscopic induced motion) and then extinguished (Bridgeman et al. 1979). These authors observed that the saccadic suppression effect was not followed by related visuomotor errors. Moreover, it was found that a pointing movement following a target jump remained accurate, irrespective of whether this displacement could be verbally reported or not. These experiments therefore suggested that two psychophysically separable visual systems can be distinguished—one system for a ‘cognitive’ response, and a second one for sensorimotor behaviour. This distinction has been more recently referred to as ‘hand sight’ (Rossetti, Pisella, and Pélisson 2000). Following this work, a long series of experiments was initiated by Prablanc and colleagues to explore on-line arm movement control. In a 1rst experiment, they required normal subjects to orient their gaze and point to visual targets presented in full darkness (at 1xation point offset). These targets could be unexpectedly displaced forward or backward during the saccade, so that a shorter or a longer hand movement had to be performed for the 1nger to land on the target. Since eye movements are usually initiated before arm movements, these target jumps occurred well before the hand had reached the target. The use of virtual images of the targets (seen in a mirror) allowed a continuous presentation of the target, without the reaching hand hiding it to the eyes (and thus this differed from the Bridgeman et al. 1979 study). The interesting question raised here was whether the motor system would be able to update the hand movement in conditions where the target jump had been performed unbeknown to the subject (Fig. 4.1). Their results were straightforward: (1) as in the Bridgeman et al. study (1979), subjects altered the amplitude of their movements such as to compensate for most of the target displacement (‘hand-sight’); (2) this hand path correction did not imply a signi1cant increase in movement time; (3) not only did subjects not detect the target jump, but they also remained unable to detect their own movement corrections; (4) forced-choice guesses about the direction of the jump could not discriminate between forward and backward target perturb-



ations (Goodale, Pélisson, and Prablanc 1986; Pélisson Prablanc, Goodale, and Jeannerod 1986). This seminal work has been followed by many studies on motor control versus conscious perception. First, Prablanc and Martin (1992) replicated the same experiment for perturbations in direction and performed a detailed kinematic analysis of perturbed as well as unperturbed reaches. Using an analysis of the direction of the tangential velocity vector, they showed that the two types of trials could be discriminated as early as about 110 ms following movement onset. Knowing that this duration includes both sensory processing time and motor pattern activation time, this result suggested that the speci1c access of the motor system to visual information was operating at an extremely fast rate. In addition, they observed that the transition from the unperturbed pattern of trajectory to the updated one was produced very smoothly, suggesting that movement correction was integrated in the initial motor programme. Recent work using transcranial magnetic stimulation applied to the posterior parietal cortex has shown that inhibition of this structure was disrupting the on-line correction system (Desmurget et al. 1999). A further experiment triggered the target jump at different times with respect to the saccade peak velocity (Komilis, Pélisson, and Prablanc 1993). It revealed that identical corrections were observed in conditions where the subjects could or could not detect the target jump, provided it was applied early enough in the movement (i.e. no later than the hand peak velocity). These 1ndings were then extended to more complex actions. For example, smooth corrections were also observed for grasping movements perturbed either in target orientation (Desmurget et al. 1995; Desmurget and Prablanc 1997) or in target location (Gréa, Desmurget, and Prablanc 2000) at the onset of the movement. We shall mention other experiments performed with simulated perturbations of target objects and on the respective timing of the processes involved in following sections (4.6.2 and 4.7).

4.2.2 Illusions When a large structured background is displaced during visual 1xation of a small target, the latter appears to move in the opposite direction. This phenomenon can be observed for both smooth (induced motion) and step (induced displacement) background shifts. Bridgeman, Kirch, and Sperling (1981) extended a 1nding made on eye movements (Wong and Mack 1981) and compared the amount of the perceptual illusory effect with the pointing response to the extinguished target. They showed that the motor system was much less affected by the apparent motion than the cognitive system. It was concluded that apparent target displacement affected only perception whereas real target displacement affected only motor behaviour, which provides a case for a double dissociation between cognitive and motor function (see Bridgeman 2000; this volume, Chapter 5). In about the last 1ve years, a substantial number of experiments have been performed to explore the effect of visual illusion on a grasping action. We shall only review a few of them here and come back in a further section (4.7.6) to this issue, which is becoming controversial. Aglioti, DeSouza, and Goodale (1995) made use of size-contrast illusions (or Titchener’s circle illusion). In this illusion, two circles in the centre of two circular arrays, composed of circles of either smaller or larger size, appear to be different in size even though they are physically identical. The circle surrounded by larger circles appears smaller than the one surrounded by smaller circles. Using this principle, one can build con1gurations with central circles of physically different sizes that will appear perceptually equivalent in size. Using this version of the illusion adapted in pseudo-3-D, Aglioti et al. required subjects to grasp the central circle between thumb and index 1nger, and measured their

65



Single step

(a)

Eye and target position

Target

Eye

Target

Eye

Double step

Eye velocity

Hand and target position Hand velocity Vision of the hand

On

On

Off

Off

d (b)

d

Frequency

15 10

5 0 25

(c)

Duration (ms)

66

30

35

40

45

500

400 0

30

32

40

44

50

Amplitude (cm)

Fig. 4.1 Saccadic suppression and hand pointing performance. (a) Experimental procedure: Schematic representation of single- and double-step trials randomly presented during an experimental session. In all trials, the target was displaced from a central position to a randomly selected position in the right hemi1eld, and vision of the hand was turned off at the onset of the hand response. In double-step trials, the peripheral target jumped again to the right at the time the saccadic eye response reached its peak velocity, i.e. nearly at hand movement onset. The second target step represented 10% of the 1rst step amplitude and was not detected consciously by subjects. (b) Spatial distribution of hand pointings. Distributions of the endpoints of hand pointing responses, pooled over 4 subjects, towards single-step targets at 30and 40cm and towards double-step targets (30–32 and 40–44 cm). Note that pointings to double-step targets undershoot the 1nal target location (a characteristic of hand movements performed without visual feedback) to the same extent as pointings to single-step targets, demonstrating the existence of corrective processes compensating for the target position perturbation. (From Pélisson et al. 1986.)



maximal grip aperture during the reaching phase of the movement. Strikingly, they observed that grip size was largely determined by the true size of the circle to be grasped and not by its illusory size. In a later study, Haffenden and Goodale (1998) compared the scaling of the grasp to a matching condition, in which subjects had to indicate the central circle size with thumb and index 1nger without reaching it. The effect of the illusion on this ‘matching’ task was very similar to the mean difference in actual size required to produce perceptually identical circles, whereas it was signi1cantly smaller in the grasp condition. This result suggests that matching object size with the 1ngers relies on an object representation similar to the perceptual representation. In contrast, the motor representation for grasp remained much less affected by the illusion. Another such experiment was performed by Gentilucci et al. (1996) to explore the effect of the Müller–Lyer illusion on pointing behaviour (see Fig. 4.21). The Müller–Lyer illusion induces the perception of longer or shorter length of a line ended by arrows and has been widely used by psychologists and philosophers to argue about the cognitive penetrability of visual perception (see Rossetti 1999). When the two arrows are directed to the centre of the line, it appears shorter. When they are oriented away from the line, it appears longer. Gentilucci et al. (1996) compared pointing responses made from one to the other end of lines linked to the two types of arrows used in the Müller–Lyer illusion, the subject having to look at the 1gure for two seconds prior to initiating the movement. Mean endpoints were signi1cantly, though slightly, in2uenced by the visual illusion, so that movement distance was increased or shortened by a few millimeters, according to the type of illusion produced. As in the Haffenden and Goodale (1998) study, the in2uence of the illusion on the goal-directed action was much less than on perception, because the perceptual effect usually covered about 20% of the physical line length used by Gentilucci et al. (Rossetti, unpublished). Interestingly, early movement kinematics were also altered, which suggests that the illusion affected the programming of the movement, and not only its 1nal execution. We shall come back later to précis the effects of illusion on action and examine possible points of controversies between authors. For the moment let us consider that visual illusions seem to affect the perceptual system in a more pronounced way than the action system, which may further support the idea of a dissociation between perception and action.

4.2.3 Masking Visual masking has been used extensively as a probe to study conscious experience and cognition (reviews in: Price 2001; Bar 2000), and may explain some of the effects observed during saccadic suppression (Matin, Clymer, and Matin 1972). We will consider here some speci1c implications of masking for action control. Taylor and McCloskey (1990) investigated the triggering of preprogrammed motor responses to masked stimuli. Three stimuli were tested: one small central LED with a 5ms pulse, a large stimulus composed of the central LED plus four surrounding LEDs, and a sequential stimulus, where the central LED was lit 50 ms prior to the onset of the surrounding LEDs. (c) Duration of hand pointings. Relationship between hand pointing duration (mean and standard deviation) and target step amplitude for three single-step targets (30, 40, and 50 cm) and for two double-step targets (30–32 and 40–44 cm). Same responses as in (b) It can be seen that the same relationship accounts for both types of trial, indicating that motor correction in response to a target perturbation (see panel (b)) is not related to an increased pointing duration. (Modi1ed from Goodale et al. 1986, and Pélisson et al. 1986.)

67


68


This last stimulus could evoke both metacontrast (masking by a surrounding shape) and backward masking (masking with a subsequent light of greater intensity than the small test light). Three motor responses of various complexities (from a single muscle group contraction to a predetermined movement sequence) were used. Reaction times (RT), as measured by EMG, were not affected by the masking of the small stimulus in the sequential condition. Comparison of RTs obtained for the large and for the sequential stimulus showed that motor response registered in the sequential condition was triggered by the short, small stimulus preceding the masking surrounding. Although the simple response evoked a shorter RT, a similar effect of the masked stimulus was observed for the three types of movements tested. This experiment thus con1rmed that motor reaction to a visual stimulus can be dissociated from the verbal report about detection of this stimulus (see also Fehrer and Biederman 1962). As stated by Taylor and McCloskey (1990, p. 445), ‘the ability to react to such stimulus with a voluntary movement implies that sensory processing during reaction time does not have to be completed before motor prcessing can commence’. Indeed, motor RTs are usually shorter than the 500 ms delay that may be required before a conscious sensation can be elicited. Although these results con1rmed that unconscious operations proceed faster than conscious ones, they cannot tell whether conscious perception and motor reaction are processed along parallel pathways with different thresholds, or whether these two responses can be elicited at different stages of serial sensory processing. It appears that masking and metacontrast affect conscious perception of the stimulus although the ability to trigger a motor response remains largely intact. Neumann and Klotz (1994) have speci1cally explored several aspects of this phenomenon. They showed that similar effects could be observed on RT (measured by keypressing) even in a two-choice situation that required integrating form information with position information. In addition, this priming effect in2uenced the error rate as well as speed of the motor response, and could appear despite of the use of variable stimulus–response couplings, showing that it is not restricted to preprogrammed responses. Taylor and McCloskey (1996) also replicated this 1nding in their experimental design. Interestingly it has been shown that the brain-activation pattern triggered by a masked stimulus is very similar to that triggered by unmasked ones. In a very elegant experiment, Dehaene et al. (1998) have shown that a masked stimulus used in a semantic priming task could activate up to the primary motor area (see also Eimer and Schlaghecken 1998). Similarly, recordings of the Lateralised Readiness Potential (LRP) in motor areas provide a physiological basis for the Simon effect (review in Hommel 2000). The double-step paradigm, applied to both reportable and non-reportable target perturbations, as well as experiments exploring the effect of visual illusions or masking on action, suggest that the neural pathways leading to visual awareness are distinct from those involved in visuomotor processing. The implicit processing of sensory information during action may affect the release of a preprogrammed motor output as well as motor planning or on-line control of the execution. The experimental study of neurological cases allows researchers to speculate on the possible anatomical substrate for this dissociation.

4.2.4 Optic ataxia Descriptions of the effects of lesions of a restricted area of the posterior parietal lobe were reported in groups of patients by Jeannerod (1986) and Perenin and Vighetto (1988). These patients had dif1culties in directing actions to objects presented in their peripheral visual 1eld although they were not impaired in the recognition of these objects, a neurological de1cit that was termed ‘optic



ataxia’. Visually directed reaching movements made by these patients are inaccurate, often systematically in one direction (usually to the side of the lesion). In addition, these movements are kinematically altered: their duration is increased, their peak velocity is lower, and their deceleration phase is longer. This alteration of movement kinematics becomes particularly apparent when vision of the hand prior to and during the movement is prevented. Restoration of visual feedback reduces the reaching errors, but the movements remain slower than normal (Jeannerod 1986). Object grasping and manipulation are also altered by posterior parietal lesions. Patients misplace their 1ngers when they have to visually guide their hand to a slit (Perenin and Vighetto 1988). During prehension of objects, they open their 1nger grip too wide with no or poor preshaping, and they close their 1nger grip when they are in contact with the object (Jakobson, Archibald, Carey, and Goodale 1991; Jeannerod 1986). They exhibit de1cits not only in their ability to reach to the object, but also in adjusting the hand orienting and shaping during reaching. In contrast, they seem to remain able to indicate the orientation of a stimulus by a wrist movement that is not aimed to the stimulus (matching task: see Jeannerod, Decety, and Michel 1994). These results strongly suggest that the posterior parietal cortex plays a crucial role in the organization of object-oriented actions, whether the visual processig required for a given action is concerned with spatial vision (location) or with object vision (size or shape) (see Jeannerod 1988; Jeannerod and Rossetti 1993; Milner and Goodale 1995; Rossetti 1998, 2000). One interpretation of optic ataxia is that patients present a de1cit in programming hand movements (Jakobson et al. 1991). Recent evidence rather suggests that de1cits result primarily from a disruption of on-line motor control (Gréa et al. 2002; Pisella et al. 2000).

4.2.5 Visual agnosia Pathological conditions may also result in disconnecting parietal mechanisms for processing object attributes from those for programming the hand con1guration. Jeeves and Silver (1988) reported the case of a patient with callosal agenesis who was unable to grasp objects correctly if they were brie2y presented within either half of his visual 1eld. The hands remained wide open throughout the movement and did not adapt to the object size. Jeeves and Silver speculated that, due to the absence of callosal control, the crossed corticospinal pathway (normally responsible for the control of 1nger movements) could not be activated by visuomotor mechanisms. Instead, the patient had to use the ipsilateral motor pathway, which was inappropriate for carrying the correct commands. These results have prompted a reappraisal of the respective functions of the two cortical pathways. The posterior parietal cortex exerts a role in organising object-oriented action, whether movements are executed by the proximal or the distal channel. This role must be dissociated from the role of other cortical structures specialised for object identi1cation and recognition. An observation by Goodale et al. (1991) provides another piece of evidence for this dissociation between perception and action, showing a reciprocal pattern to optic ataxia. These authors reported the case of a patient who developed a profound visual-form agnosia following a bilateral lesion of the occipito-temporal cortex. DF was unable to recognize object size, shape, and orientation (Fig. 4.2). This patient was also unable to purposively size her 1ngers according to the size of visually inspected target objects based on a representation of these objects (matching tasks). In contrast, when instructed to pick up objects by performing prehension movements, the patient was quite accurate and her maximum grip size correlated normally with object size. This observation suggests that, during action, DF could still process visual information about the object properties she could not perceive. If these results are compared with those following posterior parietal

69


70


Fig. 4.2 Action and object processing in visual agnosia and blindsight. Polar plots illustrating the orientation of a hand-held card in two tasks of orientation discrimination, from an agnosic patient (DF), a blindsight patient (JCG), and an age-matched control subject. On the perceptual matching task, subjects were required to match the orientation of the card with that of a slot placed in different orientations. On the ‘posting’ task, they were required to reach out and insert the card into the slot. The correct orientation has been normalized to the vertical. (Adapted from Goodale et al. 1991, and Perenin and Rossetti 1996.) lesions, impairments in perceptual recognition of objects and in object-oriented action appear to be clearly dissociated. Optic ataxia and visual agnosia patients would support the case for a double dissociation between perceptual recognition of objects and object-oriented action (see Milner and Goodale 1995) (this conclusion will, however, be questioned in a further section). It may be emphasized here that DF had her primary visual area spared. As a consequence, processing of visual information may have been disrupted only in the ventral pathway and spared in the dorsal pathway, which would explain why she could perform visually directed movements. The question therefore arises whether blindsight patients, with V1 lesions, would also exhibit a similar dissociation between perception and action.

4.2.6 Action-blindsight In addition to optic ataxia and visual agnosia, mentioned above, blindsight is another neurological de1cit that is interesting to consider in the framework of the dissociation between implicit and explicit sensorimotor processing. Early studies on patients with lesion of the primary visual area (V1), considered o be amputed from the half of their visual 1eld, showed that they remained able to orient eyes and/or the hand to visual stimuli brie2y presented within their blind 1eld (see Weiskrantz 1986). It has been recently shown that some patients could orient their hand and size their 1nger grip appropriately when reaching out to unseen visual objects (Fig. 4.2) (Jackson 2000; Perenin and



Rossetti 1996). The neuroanatomical substrate proposed to explain this action-blindsight (Rossetti et al. 2001) was the projection from the superior colliculus to the posterior parietal cortex via the pulvinar (Bullier et al. 1996; Perenin and Rossetti 1996). Therefore this fascinating non-conscious vision, emerging during a goal-directed action, is considered to provide one more instance of dissociation between the dorsal (parietal) and the ventral (temporal) streams of the visual system (e.g. Milner 1998; Milner and Goodale 1995; Rossetti 1998; Rossetti et al. 2000).

4.2.7 Action-numbsense A patient with a left parietal thalamo-subcortical lesion was studied for signs of residual processing of somesthetic modalities. The patient was unaware of any tactile stimuli applied to his arm skin and

Fig. 4.3 Numbsense: direct pointing versus pointing on a drawing. A set of 8 stimulus locations was used in this experiment: the posterior ungual phalanx of the 1ve 1ngers + the palm centre + the wrist + the middle forearm of a patient exhibiting action-numbsense following a central lesion of somatosensory afference. Pointing with the left index 1nger was made toward the locus stimulated (1) directly on the right forearm, (2) on an arm drawing of the right forearm (scale 1). Patient JA was blindfolded when answering directly on the arm. When pointing on the drawing, JA could see the A4 sheet with the arm drawing placed next to his unseen target arm. An opaque curtain was used to prevent him from seeing his target right arm. In addition, the whole arm and face of the investigator delivering the stimuli remained out of sight throughout the experiment, so that no indice (e.g. gaze direction) was available to the patient. (From Rossetti et al. 2001.)

71


72


failed to demonstrate any signi1cant performance in a verbal forced-choice paradigm. However, he generated above-chance levels of performance when pointing at the stimulus location on the numb arm (Rossetti, Rode, and Boisson 1995, 2001). This observation is similar to that of Paillard, Michel, and Stelmach (1983), who presented a tactile equivalent of blindsight. The question under investigation was whether the residual ability of the patient was linked to the mode of response (motor vs. verbal) or to the representation subservient to these responses (motor vs. symbolic). Interestingly, when the patient had to point to stimulus locations on a drawing of an arm, no signi1cant performance was observed (chance level Fig. 4.3). This dissociation indicates that only a representation of the stimulus linked to the body scheme was preserved, whereas more elaborate representations of the stimulus had vanished. In addition, the patient was unable to localize verbally his right index 1nger when it was passively positioned in a horizontal plane, but demonstrated signi1cant performance when pointing to this 1nger with the left hand. Therefore numbsense can apply to proprioception as well. These results reinforce the interpretation proposed above for actionblindsight: there seems to be a sensory system speci1c for action.

4.2.8 Conclusion These observations suggest the existence of a speci1c representation for those (extrinsic as well as intrinsic) object attributes which are used for controlling movement. In the action of grasping an object, the role of the sensorimotor representation will be to transform the visual qualities of the object into corresponding action-speci1c motor patterns for the hand to achieve the proper action. This mode of representation thus relates to the object as a goal for an action. The object attributes are represented therein as affordances, that is to the extent that they afford speci1c motor patterns (see Riddoch et al. 2001). This pragmatic, or sensorimotor representation seems to specify the metric properties of the action goal in a veridical way, because the hand has to interact with real objects rather than with distorted representations (it would therefore match the theoretical properties of an ‘immaculate perception’,1 Rossetti 1999). It is different to the mode used during the process of overt recognition, and by which an object can be named, categorized, and memorized. This process implies a representation of the semantic type, where the object appears as an identi1able entity and remains invariant across different vantage points. Its elementary attributes (size, orientation, colour, texture, etc.) are bound together to form a speci1c unit. At variance with this cognitive representation, the pragmatic representation implies no binding of attributes into a single percept (see Revonsuo and Rossetti 2000). Instead, each attribute of the graspable object is represented in itself and contributes to the motor con1guration of the arm and hand. The above hypothesis implies that the cortical mechanisms for object recognition or for objectoriented action are selectively activated by the task in which the subject is involved. If the task involves recognizing, memorizing, or forming a visual image of an object, only the ventral visual pathway should be activated. If, on the other hand, the task involves 1nger movements for grasping or manipulating an object, the dorsal pathway should be activated. Taken altogether, the 1ndings made on the ability of neurological patients as well as normals to process sensory information speci1cally for action purposes suggest that vision (or somaesthesia) for action and vision for perception can be dissociated. The dissociations found in optic ataxia, visual agnosia, and blindsight further suggest that the neurological substrates for these two functions could be located selectively in the dorsal and in the ventral streams of visual processing.



4.3 Neuroanatomy of visual-to-motor connections 4.3.1 Cortical versus subcortical vision About 100 years ago, anatomical studies 1rst suggested the existence of three visual pathways by which the retina was connected to the cortex. Apart from the main route through the lateral geniculate body, von Monakow identi1ed one pathway through the pulvinar and another through the superior colliculus. These pathways terminated in cortical areas outside the striate area (see Polyak 1957). Cajal (1909) described a ‘descending’ or ‘motor’ pathway arising from the fourth layer of the superior colliculus and terminating in the ocular motor nuclei and the adjacent reticular formation. This pathway was thought to carry orienting as well as pupillary re2exes. Subcortical vision was thus considered by Cajal to be pure motor vision. Accordingly, extensive lesions of this structure were shown to produce severe impairment in eye movements and visuomotor behaviour (Sprague and Meikle 1965). The distribution of retinofugal 1bres between the retinogeniculate and the retinotectal pathways was interpreted within the framework of a dichotomy between two visual systems endowed with complementary functions. Schneider (1969) proposed that the geniculostriate pathway is a system essential in the learning of pattern discrimination, and that the retinotectal pathway is a system for mediating spatial orientation. Using hamsters, he dissociated ‘cortical blindness’ from ‘tectal blindness’. Following ablation of visual areas 17 and 18, animals became unable to learn simple pattern discrimination (e.g. vertical vs. horizontal stripes), although they remained able to orient toward stimuli (e.g. sun2ower seeds) presented across their visual 1eld. By contrast, following large undercuttings through the midbrain tectum, spatial orientation ability was lost, whereas pattern discrimination was still possible. This anatomical and functional duality became known as the now classical oppoition between a system specialized for answering the question ‘What is it?’ and another one specialized for answering the question ‘Where is it?’ (Schneider 1969). A model of visuomotor coordination built on the notion of two visual channels for movement control was also presented by Trevarthen (1968). This author studied visuomotor behaviour in split-brain monkeys and concluded that the subcortical visual system subserved ‘ambient’ vision, while the cortical system subserved ‘focal’ vision. Pathological destruction of the visual cortex in humans was classically thought to produce total blindness, except for pupillary responses to light and very crude visual perception limited to sudden changes in illumination. This opinion, however, was called into question on the basis of experimental 1ndings in monkeys. Although destriated monkeys also appeared to be profoundly impaired in their ordinary visual behaviour, they were still able to avoid obstacles and to generate motor responses for reaching objects appearing in, or moving across, their visual 1eld (Humphreys and Weiskrantz 1967). These anatomical 1ndings represented a strong argument for the role of subcortical structures in mediating residual visual function in destriated monkeys. Mohler and Wurtz (1977) showed that partially destriated monkeys, which were able to orient visually toward stimuli presented within their scotoma, lost this ability after subsequent destruction of the retinotopically corresponding zones of the superior colliculi. Thus in monkey, the superior colliculi, and possibly other brainstem areas receiving input from the retina, may play a critical role either in mediating pure ‘subcortical vision’ or in relaying visual input to other structures onto which they project, including extrastriate cortex. In humans, clinical observations suggestive of incomplete or ‘relative’ blindness within scotoma of cortical origin had been mentioned previously by several authors (see Weiskrantz 1986, for

73


74


review). Systematic experimental evidence of residual visual abilities following lesions of the striate cortex was 1rst reported by Pöppel, Held, and Frost (1973). This experiment used a new methodological approach derived from the monkey studies and based on forced-choice responses. Cortically lesioned subjects were requested not to try to see stimuli that were presented within their scotoma, but rather to turn their eyes or point their hand each time a stimulus was presented (see also Weiskrantz, Warrington, Sanders, and Marshall 1974). The amplitude and direction of the responses de1nitely correlated with target positions. Similar results were obtained by Perenin and Jeannerod (1978) and Ptito, Lepore, Ptito, and Lassonde (1991) in hemidecorticated subjects. In this situation the complete loss of cortex on one side stressed the role of subcortical vision. The fact that subjects tested for ‘blindsight’ remain unaware of the stimuli, and usually experience ‘guessing’ rather than ‘seeing’, would be in accordance with the classical idea that subcortical vision is ‘unconscious’. As can be seen in Fig. 4.5, the subcortical network also projects to cortical visual systems (see Bullier, Schall, and Morel 1994; Girard 1995).

4.3.2 Two cortical visual systems In rodents, lesions of the striate cortex appeared to affect orientating behaviour toward targets located within the rostral visual 1eld, whereas this ability was spared after collicular lesions. By contrast, the superior colliculus was necessary for orienting toward targets placed in the far peripheral visual 1eld (see Goodale 1983). Thus, orientating seems a more complex function than suggested by Schneider’s results, and cannot be completely dissociated from pattern discrimination, especially in the most central parts of the visual 1eld. Later experiments performed on monkeys suggested that both modes of vision were mediated by two diverging corticocortical pathways for processing ‘what’ versus ‘where’. One pathway was the ventral occipitotemporal route, linking striate cortex to prestriate areas and from there reaching inferotemporal cortex on both sides via callosal connections. Interruption of this pathway abolished object discrimination without affecting perception of spatial relations between objects. The other, dorsal, pathway diverged from the ventral one by linking the prestriate areas to the posterior part of the parietal lobe. Interruption of this pathway produced visual spatial disorientation characterized not only by misperception of the relative positions of spatial landmarks (Ungerleider and Mishkin 1982), but also by localization de1cits during object-oriented action (Ungerleider 1995) (see Fig. 4.4). As mentioned above, cases of optic ataxia and visual agnosia have raised the possibility that the anatomical dorsal–ventral division may instead relate to a distinction between the processing of ‘what’ vs. of ‘how’ to direct an action (Goodale and Milner 1992; Jeannerod and Rossetti 1993; Milner and Goodale 1995). This renewed conception of parallel visual systems was no longer based on the modalities of visual coding (what vs. where), but rather on the modes of representation of the target object, that is directly linked to the type of response produced by the subject (what vs. how) (Jeannerod 1994). In monkeys, posterior parietal lesions produce a reaching de1cit, characterized by the fact that the animals usually misreach with the arm contralateral to the lesion in either part of the visual 1eld (e.g. Faugier-Grimaud et al. 1978, 1985; Hartje and Ettlinger 1973). In addition, as discovered by Faugier-Grimaud et al. (1978), after lesions limited to the inferior parietal lobule (monkey area 7), the contralesional 1nger movements are impaired during grasping. These 1ndings are consistent with the properties of neuronal populations recorded in this cortical region; neurones coding for the direction of reaching arm movements were described in this area by Hyvärinen and



Fig. 4.4 Several sketches of the visual system. Several representative conceptions of the main neural pathways in the visual system. A–C: From the main visual input to area 17, two segregated streams of processing have been described projecting respectively to the posterior parietal cortex (dorsal pathway) and to the inferotemporal cortex (ventral pathway). While the ventral pathway is specialised in processing colour and form and is assumed to play a key role in object identi1cation (‘what’), the dorsal pathway is known to be primarily involved in the computation of places and movement (‘where’) (Morel and Bullier 1990; Ungerleider and Desimone 1986; Ungerleider and Mishkin 1982) and in the sensorimotor processing of the object metrics (‘how’) (Schwartz 1994). D: Both pathways project onto frontal structures involved in action. A. Ungerleider and Desimone (1986) B. from Morel and Bullier (1990) C. from Schwartz (1994) D. Ungerleider (1995) Poranen (1974) and by Mountcastle et al. (1975). More recently, another population of cells, selectively activated during manipulation by the animal of objects of given con1gurations, were described by Taira et al. (1990). The production of typical visuomotor de1cits by lesions and the recording of typical sensorimotor activities in the posterior parietal cortex strengthened the conception of

75


76


a dorsal visual system specialized for action (review in Sakata and Taira 1994; Jeannerod, Arbib, Rizzolatti, and Sakata 1995; Milner and Dijkerman 2001; Milner and Goodale 1995; Pisella and Rossetti 2000; Rossetti 1998; Sakata et al. 1997).

4.3.3 An occipito-frontal visuomotor network The study of the dorsal–ventral dissociation in the motor context led researchers to distinguish speci1c motor abilities, which are dissociable from conscious experience and preserved by patients with lesions of the ventral stream, from other types of motor responses. The dorsal–ventral distinction 1nally evolved toward the conception of two parallel visual streams in the occipito-frontal network for visual-to-behavioural motor responses (Rushworth et al. 1997; Schwartz 1994; Ungerleider 1995; see Fig. 4.4). Occipito-parieto-frontal networks have been precisely identi1ed for reach and grasp movements (Jeannerod et al. 1995; Rossetti et al. 2000; Sakata et al. 1997; Tanné et al. 1995). However, the visual processing of both the dorsal and the ventral streams has to join the motor structures in order to allow the subject to produce behaviour adapted to his or her environment. Indirect projections of the ventral stream toward the motor regions exist: the temporal area TE can connect the primary motor area after a relay in the prefrontal and then in the premotor regions (Tanné et al. 1995; see Fig. 4.5). The temporal lobe can also be implicated in action via its connections to the basal ganglia. Two types of behavioural arguments support this idea of a ‘dual route’ to visuomotor action (Milner and Dijkerman 2001). The 1rst line of studies distinguished between ‘sensorimotor’ and ‘cognitive’ representations underlying actions, either in normal subjects or in neurological patients. The involvement of the dorsal or ventral stream in action was based on the modes of representation of the goal of the movement: egocentric versus allocentric coding, goal-directed action versus matching, implicit versus explicit processing, grasping of meaningless shape versus meaningful objects. We shall describe this distinction in more detail in the following sections. The second line of studies concerned conditional motor tasks. Not all sensorimotor transformations consist of goal-directed actions and involve the computation of sensorimotor coordinates and the shaping of the hand with respect to object properties. Other aspects of motor behaviour depend on object identity. The functional role of a stream connecting areas involved in object perception and recognition with the motor structures accounts for the usual associations between a speci1c stimulus and a motor behaviour (like braking associated with red lights). Rushworth, Nixon, and Passingham (1997) conclude from lesion studies in monkeys that neither part of the parietal lobe may play a major role in the selection of movements made to arbitrarily and conditionally associated visual stimuli. Relatedly, a patient with bilateral posterior parietal lesion exhibited no dif1culty performing instructed motor responses (stop or redirect ongoing action) to visual stimuli, but lost the automatic visuomotor guidance of action (Pisella et al. 2000; see Fig. 4.14). In search of the detailed neuroanatomical basis for the ventral and dorsal systems, Fig. 4.5 presents an attempt to synthesize the cortical neuronal networks described in monkeys between V1 and M1 allowing visual inputs to be transformed into motor output. Although the dorsal and the ventral streams can be individuated from this network, this illustration displays a possible substrate for common participation of these two systems in action. Strikingly, M1 receives only mixed projections and no pure projections, either from the dorsal system or from the ventral one. Although distinctions can be



Fig. 4.5 (See also the color plate of this figure.) Overview of the visual-to-motor network. Cortical neuronal networks allowing visual inputs to be transformed into motor output. This illustration displays the possible substrates for dissociation and interactions between ventral and dorsal pathways driving information from V1 to M1. The dorsal and the ventral streams are depicted in green and red, respectively, as well as their efferences. Blue arrows arise from areas receiving convergent dorsal and ventral inputs, either directly or indirectly. Further projections from areas receiving these mixed convergent inputs have also been represented in blue. Even though the posterior parietal cortex and the inferior temporal cortex receive a single direct projection from each other, they were not considered as mixed recipient areas. By contrast, areas in the frontal lobe receive parallel dorsal, ventral, and mixed projections. Interestingly, at the motor end of this network there is no pure projection from either the dorsal or the ventral stream of visual processing. Abbreviations: AIP: anterior intraparietal area; BS: brainstem; Cing.: Cingulate motor areas; d: dorsal; FEF: frontal eye 1eld; FST: 2oor of the superior temporal sulcus; Hipp.: Hippocampus; LIP: lateral intraparietal area; M1: primary motor cortex; MIP: mesial intraparietal area; MST: medial superior temporal area; MT: medio-temporal area; PF: prefrontal cortex; PM: premotor cortex; SC: superior colliculus; SEF: supplementary eye 1eld; SMA: supplementary motor area; STS: superior temporal sulcus; STP: superior temporal polysensory area; TE: temporal area; TEO: temporo-occipital area; v: ventral; V1: primary visual cortex; VIP: ventral intraparietal area. (updated from Rossetti, Pisella, and Pélisson 2000—derived from Colby et al. 1988; Morel and Bullier 1990; Schall et al. 1995; Schwartz 1994; Tanné et al. 1995; Van Hoesen 1982.) described between cortical and subcortical vision, between dorsal and ventral stream, and between two occipito-frontal routes, the important point raised here is that all subsystems considered in these distinctions are interconnected.

77


78


4.4 The gap between anatomy and behaviour The above two lines of evidence make a strong case for the dissociability perception and action, on the one hand, and for the interconnection of visual-to-motor networks, on the other. The examples of dissociation presented here suggest that there must be two independent visual systems which can give rise either to action or to conscious perception. However, taken as a whole, the anatomical data do not support as clear a segregation between two major pathways as they are often considered to establish. Instead, Fig. 4.5 suggests that interaction between the two streams can take place at many levels before visual information reaches the motor output, and that a pure input from the dorsal stream onto the motor areas cannot be isolated within the visuomotor network. A crucial point to note here is that most of the evidence used to support a dissociation between two visual subsystems comes from animal lesion or neuropsychological patients, whereas most of the arguments for an interaction between these two systems comes from experimental work performed in normal subjects. Is it possible to bring together these two lines of evidence? There must be ways to go beyond the surface of this lack of a direct anatomical correlate of a behavioural dissociation. If there is no doubt, as argued above, that perception and action can be dissociated (at least in some circumstances) then the anatomical data presented in Fig. 4.5 should be revised or improved. In particular, additional features of this anatomical network have to be identi1ed, which would explain this behavioural dissociation. Conversely, if, as argued above, there are numerous interconnections between the dorsal and the ventral anatomical pathways, then interactions should be observed between the two behavioural responses.

4.5 Anatomy is right Numerous examples of interaction between action and perception systems will be listed below, suggesting that the anatomical evidence for an interconnection between the two systems is followed by functional correlates. Altogether four types of cognitive→sensorimotor interaction can be observed, whereas one type of reciprocal sensorimotor→cognitive interaction has been described (see Fig. 4.6).

4.5.1 Perception can trigger action Since they don’t perceive objects, blindsight patients would never initiate spontaneous actions toward these objects. Their motor ability in pointing and orienting objects has always been observed in forced-choice experiments, where the action was initiated upon a go-signal (see Rosetti 1998; Weiskrantz et al. 1974, 1989). The same observation has been made for numbsense patients (Paillard et al. 1983; Rossetti et al. 1995, 2001). This experimental detail has a strong theoretical impact. It means that in order to allow the sensorimotor system to release an action, the cognitive system has to provide instructions to initiate this action.

4.5.2 Perception can inhibit action We have seen that perception and sensorimotor processing can be dissociated in action-blindsight and numbsense patients. It is interesting to note, however, that the preserved motor abilities of these patients were disrupted when the elaboration of a cognitive representation of the action goal was activated during the action (see Rossetti 1998). When asked to produce a verbal response simultaneously



Cognitive representation

Triggers

Configures

Structures

Inhibits

Sensori-motor transformation

Stimulus

Response

Fig. 4.6 Summary of the various types of interactions observed between the sensorimotor and the cognitive representations of a stimulus leading to behavioural responses.

to the action, the motor performances of blindsight patients dropped to chance level (Fig. 4.7). And the same observation has been made with a numbsense patient, for both tactile and proprioceptive targets (Fig. 4.7): the simultaneous motor + verbal condition was responsible for random responses (see Rossetti et al. 1995). Experiments in normals have also shown that the co-activation of a verbal representation during a motor response changes the con1guration of the movement endpoint errors. Immediate pointing toward proprioceptive targets was tested with blindfolded subjects by Rossetti and Régnier (1995). For each target, constant and variable errors were computed. Variable errors were assessed by a con1dence ellipse of the endpoint distribution (Fig. 4.8). On each trial a target was presented on one out of six possible locations lying on a circle centred in the starting point. Because subjects were trained to point to these positions in a preliminary session and to associate a number (from 1 to 6) to each target, they could mentally extract the pattern of the target array and use it to encode the target location in an allocentric frame. In this case (as for delayed action, see Fig. 4.10), the distribution of endpoints would tend to align with this target array, that is, perpendicular to movement direction. If they encoded the target position in an egocentric reference frame, then their pointing distribution should remain unaffected by the target array and should be elongated in the movement direction (as in Vindras et al. 1998). In a ‘motor’ condition, subjects simply had to point toward the proprioceptive target after the encoding 1nger had just been removed from it. In a ‘motor + verbal’ condition, subjects similarly pointed toward the same proprioceptive targets but were instructed to give simultaneously a forced-choice verbal response of the target number. In the ‘motor’ condition, as in the condition of arbitrary ‘number’ verbalization, the orientations of the ellipse main axes were randomly distributed (Fig. 4.9(a)). This lack of in2uence from the context of the target array was interpreted as the pure

79


80


Fig. 4.7 Effect of verbalization on the reach performance of a patient with a lesion in the primary visual area and a patient with a lesion of the primary somaesthesic afferents (tested for touch and proprioception). The vision modality was assessed by testing the reach performance of the blindsight patient PJG toward visual targets presented in his blind visual 1eld. The reaching performance of the numbsense patient JA was evaluated toward tactile and proprioceptive stimuli on the affected side. In these three modalities, the correct reaching responses decreased when a forcedchoice verbalization of the target was produced simultaneously to the immediate motor response. (Adapted from Rossetti 1998.)



Fig. 4.8 Constant and variable pointing errors. Endpoint recording was made of each individual movement, and each target was used to compute constant and variable pointing errors. Constant errors were measured in direction (angle) and amplitude relative to the ideal reach that would hit the target. Variable errors were assessed by con1dence ellipses (95%) of the scatter of 1nger end positions. The ellipse surface provided an estimate of the global pointing variability. The orientation of the ellipse major axis was computed relative to mean movement direction (angle beta). This most pertinent parameter revealed the sensorimotor or cognitive representation underlying the pointing movement. This parameter was shown to be affected by the various conditions of target coding (review: Pisella et al. 1996; Rossetti 1998, see Fig. 4.9) and by the delay of response (Rossetti and Régnier 1995, see Fig. 4.10.)

activation of a sensorimotor representation in this condition (Rossetti and Régnier 1995). In the ‘motor + verbal’ condition (with a speci1c spatial verbalization of the ‘target number’), the orientation of the ellipse, perpendicular to movement direction (see Fig. 4.9(a)), was interpreted as the result of the in2uence of an allocentric representation of the target position. Therefore the cognitive integration of the whole target pattern played a role in the immediate action only when a spatial verbal representation was activated (Rossetti 1998). In order to demonstrate that this result could not simply be attributed to an attentional bottleneck or dual task effect, several control experiments were performed (Fig. 4.9(b), and see Fig. 4.20 in the recipes provided in section 4.7). This type of cognitive →sensorimotor interaction effect was later con1rmed by the application of this motor–verbal paradigm to the Roelofs effect (Bridgeman 1997, 2000, this volume, Chapter 5).

81


82


Fig. 4.9 Effect of verbalization on the orientation of con1dence ellipses. (a) Histogram of the beta distribution for immediate (delay = 0s) and delayed pointing movements (delay = 8s) when subjects verbally report a number during their pointing responses. This number can result from downward counting (‘number’ condition) or from a guess about the target location (‘target number’ condition). With the ‘target number’ verbalization speci1c to the spatial location of the target, ellipses orientation tended to be more aligned with the arc array (beta reaches 90 deg). In contrast, with the arbitrary ‘number’ verbalization, the in2uence of the context of target presentation appears only for delayed pointing movements. Immediate movements are coded in an egocentric reference frame, independent of the target array as in the condition without simultaneous verbalization [schematized in (b)]. (b) Schematization of the in2uence of various verbalizations on the ellipse orientation of immediate pointings. Without verbalization, ellipses are randomly oriented. Ellipses tended to align with the arc array (beta reaches 90 deg in mean) only for the speci1c verbalization of the target spatial location (‘target code’) but not for the two arbitrary verbalizations: arbitrary number (‘downward counting’) or learned ‘texture code’ of the target. (Derived from Pisella et al. 1996; Rossetti 1998 and Rossetti and Régnier 1995.)



Fig. 4.10 Effect of delay on the orientation of con1dence ellipses of pointing errors. (a) Histogram of the beta distribution for immediate (delay = 0s) and delayed pointing movements (delay = 8s) when targets were presented on an arc array or a line array. For immediate pointing, the con1dence ellipses are in2uenced by the direction of the pointing movement (as shown in Fig. 4.9). After a delay, ellipse orientation tends to align with the target array, revealing an allocentric coding of the target location. In the case of the arc array, beta tends to reach 90deg for delayed pointing movements. On the contrary, in the case of a line array aligned with the direction of the movement, ellipses orientation doesn’t change between immediate and delayed pointing movements (remains about 180deg). These results for 8s delay are schematized in (b). (b) Schematization of the in2uence of the target array on the ellipse orientation of delayed pointings. The result shown in (a) was replicated with various target arrays. (Derived from Pisella et al. 1996; Rossetti 1998 and Rossetti and Régnier 1995.)

83


84


As in the case of blindsight and numbsense, the observations made in normals con1rm that when a motor and a cognitive representation of a target are simultaneously elaborated, the cognitive representation seems to impose, at any level before the output, its spatial parameters on the final output.

4.5.3 Recognition of the objects to grasp con1gures the motor program The preserved motor performances in blindsight patients are classicaly observed after a training session performed in the intact visual 1eld (e.g. Rossetti 1998). This suggests that a con1guration of the motor act by the ventral stream is a preliminary step necessary for blind sensorimotor processes. Non-motor parameters have been shown to in2uence action in normal subjects after learned associations between objects’ features and motor parameters. Clever experimentation demonstrated that after such semantic associative learning, the processing of the colour of an object (Haffenden and Goodale 2000b), or of words written on an object, can interfere with the sensorimotor processing of the physical properties pertinent for grasp, and consequently affect the grip size. The size–weight illusion provides another such example (Flanagan and Beltzner 2000). In the same vein, Jeannerod et al. (1994) reported that a patient with a bilateral lesion of the dorsal stream (AT) exhibited normal actions toward familiar objects, whereas her actions toward similar unfamiliar 3D forms were severely impaired. These results suggest that goal-directed actions could be con1gurated by the visual recognition pathway. Even for normal subjects, the motor parameters of everyday actions made toward familiar objects can therefore be programmed on the basis of the knowledge about the usual physical characteristics of these objects.

4.5.4 Sensorimotor representation structures spatial cognition Both visuomotor adaptation in neglect patients and the natural ageing process have been shown to affect the elaboration of explicit cognitive representations of spatial information (Pisella and Rossetti 2000; Bhalla and Prof1tt 2000). The therapeutic improvement of neglect symptoms mediated by a pointing procedure of adaptation to a prismatic deviation (Rossetti et al. 1998b, 1999a) has demonstrated that the spatial cognition of brain-damaged patients can be restructured by adapting sensorimotor transformations. This profound action was shown to affect sensorimotor coordination and classical neuropsychological testing through visuomotor tasks (e.g. cancellation task, copying task, bisection task), as well as pure cognitive representation such as mental imagery (Rode, Rossetti, Li, and Boisson 1999; Rode et al. 2001). In addition, we have shown recently that this type of structuring effect of visuomotor adaptation can also alter spatial cognition (midline judgements) in normal individuals (Colent et al. 2000). Bhalla and Prof1tt (2000) too have shown that evaluation of the slope of a hill was in2uenced by subjects’ age; that is, by a subject’s adaptation to his/her reduced sensorimotor and power abilities. This effect is not observed over a shorter time range such as after jogging. This is coherent with Piagetian theory about child development, that is, the brain is structured by sensorimotor associations and learning (‘sensorimotor scheme’), which are progressively transferred to cognition (considered as covert action schemes). As shown by these examples of intermediate or long-term interactions between action and cognition, a much larger time-scale seems to be necessary to observe an in2uence from the sensorimotor level to the cognitive level. It is likely that the cerebellum is involved in this type of profound revision of these several levels of space representation, because this brain structure is the best candidate for the neurological substrate of adaptation (review: Jeannerod and Rossetti 1993). But other structures may be involved as well, as



suggested by the lack of contextual in2uence on delayed pointing performed by blindfolded subjects in the experimental situation depicted in Fig. 4.10 (Rossetti et al. 1996). Based on this short synthesis of the empirical data available on the interplay between cognition and action systems, it appears that explicit and implicit processing of space involved in action cannot be considered as fully dissociated from cognition and that there are multiple possible interferences between them. These interactions are not symmetrical. Cognitive→sensorimotor interactions are observed at the level of the duration of a single slow action, whereas sensorimotor→cognitive interactions can be observed over a much longer time-scale (as a result of many successive actions).

4.6 Anatomy is not sufficient Whereas our knowledge of brain anatomy is mainly based on monkey data and does still need development and re1nement, human behaviour can be observed directly. If there is a mismatch between neuroanatomy and behaviour, then we have to look at anatomy in a more functional way. Let us now consider the neuroanatomical networks connecting sensors and effectors in this light. The dissociation between a dorsal and a ventral stream of visual processing has been initially based on the connectivity of visual areas. More recently, the functional properties of the cells participating in each of these streams have been described. Recent single-unit studies in the monkey also provide evidence for a temporal dissociation between two visual pathways. Comparison of visual response latencies at various locations in the monkey visual system have led Nowak and Bullier (1997) to distinguish two groups of visual areas. Parietal areas from the dorsal stream, projecting onto the premotor cortex, exhibit particularly short visual latencies (about 40–80ms) as compared with other pre-striate areas. In their extensive review of the literature, the dorsal pathway projections towards frontal areas is therefore referred to as the ‘Fast Brain’, whereas temporal areas are described as the ‘Slow Brain’ (about 100–150ms). As stressed by Nowak and Bullier (1997), the visual latencies do not match a hierarchical model of purely anatomical organization, but rather follow the distribution of magnocellular and parvocellular inputs. The speed of the occipito-parietal processing seems to be explained by the faster conductivity of the magnocellular channel almost exclusively activating this dorsal stream and by the numerous bypass connections existing in it, whereas the ventral stream seems to be connected in a more serial fashion (Nowak and Bullier 1997; Schall, Morel, King, and Bullier 1995). In addition, neuroanatomical tracing has shown that parietal areas of the dorsal stream project directly to the dorsal premotor cortex, whereas the ventral stream projects only indirectly to ventral premotor cortex via the ventral prefrontal cortex (Schmolesky et al. 1998; Schwartz 1994; Tanné et al. 1995; see Fig. 4.5). Can we now identify some behavioural correlates of the functional properties of neuroanatomical projections?

4.6.1 Immediate versus delayed actions Differences between immediate and delayed actions have been reported in normal and braindamaged subjects. With respect to the temporal issues raised here, it is interesting to note that visuomotor performance (how) in brain-damage patients and healthy subjects can depend upon the delay and the speed of the motor response.

85


86


The effect of a delay can be easily tested in normals with various simple tasks. In general, there is a global decrease in performance when the delay introduced between the stimulus presentation and the response is increased. This effect is mainly observable in terms of an increase in response variability. Interestingly, it has also been observed that the effect of delay duration is not linear (see Fig. 4.11(a)). Using a simple experimental design, Rossetti et al. (1994) had subjects point with various delays to visual targets 2ashed on a monitor. Nine target locations were used, and organized along an arc centred on the starting position (see Rossetti 1998). Several accuracy parameters were investigated (Fig. 4.8). First, the global variability, as assessed by the surface of the con1dence ellipse 1tting the movement endpoints, continuously increased with the delay. Second, the evolution of the orientation of the main axis of the con1dence ellipses 1tted for each target followed instead a two slope function: it tended to be aligned with movement direction in the absence of a delay and then rapidly increased for the 500ms delay (see Fig. 4.11(b)). Between the 500ms and the 8s delay, a nearly horizontal plateau was reached, with ellipse orientation tending to be aligned with the target array, that is orthogonal to movement direction (see Rossetti et al. 2000: Fig. 4.10). Third, the orientation of the constant error vector in space also followed a similar two-slope trend. As shown in Fig. 4.11(a), it is rather striking that experiments investigating the effect of a delay on eye movement accuracy in the monkey made similar observations (Krappmann 1998; White, Sparks, and Stanford 1994). Although the parameters used in the monkey saccade experiment were not identical to the one used in the human pointing experiment, it is interesting to observe that a similar time course could be observed in both studies. These results indicate that a different type of sensorimotor process is at work in the immediate and in the delayed condition. A short-lived egocentric representation of the target location seems to be used to guide immediate actions. However, an allocentric coding of the visual target seems to participate in the delayed action, which is affected by the global spatial context of the experiment that has been extracted by a trial-to-trial integration over time. In addition, similar results have been observed for delayed pointing to proprioceptively de1ned targets (Fig. 4.10) (Pisella et al. 1996; Rossetti and Procyk 1997; Rossetti and Régnier 1995; Rossetti et al. 1996). Neuropsychological data have also reported the effect of delay on the motor performances of patients with a lesion of the dorsal or ventral stream. The agnosic patient DF can correctly reach and grasp objects that she cannot describe, but loses this preserved motor ability when her action is delayed by only 2s (Goodale et al. 1994). Goodale et al. reported that many kinematic landmarks of the grasping movement were affected by a 2-s delay introduced between stimulus viewing and movement onset. In particular, the opening and closure of the 1nger grip was altered and maximal grip size was reduced as compared with normal movements. Strikingly, movements delayed by 30s and pantomimed movements performed beside the object were similar to those observed after 2s. Conversely, an ataxic patient (AT) described by Milner et al. (1999a) performed imprecise reach and grasp movements when instructed to act immediately to objects, but she was (paradoxically) improved when a delay was introduced between the stimulus presentation and the pointing response (see also Milner and Dijkerman 2001). Action-blindsight and numbsense have also been shown to be disrupted when a delay is introduced between the stimulus and the response (see Rossetti 1998, Fig. 4.12). These results converge towards the idea that when action is delayed and the object has disappeared, the parameters of object position and characteristics that are used by the action system can only be accessed from a sustained cognitive representation. This type of representation obviously relies on different reference frames with respect to the immediate action system. Furthermore, the neuropsychological data suggest that the dorsal stream is able to build a short-lived sensorimotor representation of the target that is only available for immediate actions.



(a)

Monkey 333 Monkey 340 Mean variable error (deg)

Mean constant error (deg)

6

4

2

0 4 2 Delay (sec)

6

90

60

30

0

0

1

2

3 4 5 Delay (sec)

6

2

0 4 2 Delay (sec)

0

Mean variable error orientation (deg)

(b)

Mean constant error orientation (deg)

0

4

7

8

6

90

60

30

0

0

1

2

3 4 5 Delay (sec)

6

7

8

Fig. 4.11 Time course of error distributions. Effect of increasing delay intervals (between the target extinction and the go-signal) on the precision of two types of motor responses. Constant errors (left) and variable errors (right) de1ned in Fig. 4.8 were plotted as a function of the delays. Plotted points are the errors averaged across all target locations. Similar tendency was observed in two studies on (a) ocular and (b) manual motor errors. (a): Constant saccadic errors showed a drastic increase between 0 and 1s and stabilized for longer delays in the two monkeys (White et al. 1994). Variable errors follow the same evolution with delays at least for one monkey. (b): Both constant and variable pointing errors toward visual targets increased sharply in humans between 0 and 1s and then reached a plateau. (Drawn from Rossetti et al. 1994.)

4.6.2 Fast versus slow actions Following the initial 1nding that movements could be updated on-line unbeknown to the subject (Pélisson et al. 1986), related studies stressed the high speed of motor correction and investigated the delay of subjective awareness of the perturbations (Castiello and Jeannerod 1991; Castiello, Paulignan, and Jeannerod 1991). In these experiments, a simple vocal utterance (‘Tah!’) was used

87


88


Fig. 4.12 Effect of delayed tasks on action-numbsense. Action-numbsense was tested in JA for tactile stimuli delivered to the hand. Four different sessions tested the effect of 4 delays introduced between the stimulus delivery and the go-signal: 0, 1, 2, and 4 s. In each session 48 stimuli were randomly delivered to 6 possible locations (stars). The number of correct responses (black dots) decreased abruptly from an above chance (immediate and 1s) to a close-to-chance level (2 and 4s). This result suggests that the residual somatosensory information used to direct JA’s response was available only for a very short period (about 1s). (From Rossetti 1998.) by the subject to signal his or her awareness of the object perturbation. Comparison of the hand motor reaction time and the vocal reaction time showed that the vocal response consistently occurred after the motor corrections. As in preliminary experiments (Paulignan et al. 1991), the onset of motor correction was about 110 ms after the object displacement, and about 280 ms after the change in object size. However, the vocal responses occurred in both cases about 420 ms after the object’s perturbation. It was concluded that conscious awareness of the object perturbation lagged behind the motor reaction to this perturbation. These results also stressed the important role played by time factors with respect to the action–perception debate. The spontaneous variation of response speed in a patient with action-blindsight allowed us to note that faster sessions gave rise to a more signi1cant performance than the slower sessions (see Rossetti 1998). A similar effect was described in an experiment investigating the effect of movement time on the type of action control (Pisella et al. 2000: Exp. 1). In a ‘location–stop’ pointing experiment, one green target was initially presented and subjects were requested to point at it at instructed rates. This target remained stationary on 80% of the trials or could unexpectedly jump to the right or to the left at the time of movement onset. Subjects were instructed to point at the target, but to systematically interrupt their ongoing movement when the target jumped. The direction of the target



Fig. 4.13 An automatic pilot for the hand. (a) Schematization of the ‘location–stop’ protocol, predictions, and results. Left column: Protocol. During a pointing task with a 300-ms movement duration constraint, the target could be displaced unexpectedly to the right or to the left at movement onset (in 20% of the trials). The instruction was to stop the ongoing movement whenever the target jumped. Central column: Predictions. Slow movements should allow subjects to stop their movement and not to touch the screen, whereas too-fast movements should not be affected by the target jump and thus should touch the programmed target location. Right column: Results. Three types of motor responses were successively observed when movement time increased. As expected, subjects touched the 1rst target location for the fastest movements, whereas

89


90


jump was thus irrelevant for this task. A strict compliance with the ‘stop’ instruction would imply that subjects would either succeed in stopping their movement or fail to interrupt their action and therefore reach the primary position of the target (Fig. 4.13(a)). In striking contrast to this prediction, a signi1cant percentage of corrective movements were performed in the direction of the target jump in spite of the ‘stop’ instruction (Figs 4.13(a), 4.16). After touching the displaced target, subjects were fully aware of their mistakes and spontaneously expressed a strong frustration. We explored whether the ongoing hand movement was corrected or interrupted with respect to movement times. Sampled movement times ranged from about 100 to 450ms with a Gaussian distribution; they corresponded to movement speeds because the distance between targets and starting point was constant in the experiment. Fig. 4.13(b) shows the number of corrected movements with respect to movement durations. Since they occurred in a restricted temporal window, escaping the slower processes of voluntary interruption, the involuntary corrections resulted from a failure to inhibit an automatic process of on-line visuomotor guidance. This ‘automatic pilot’ (see also Place 2000), systematically activated during movement execution, led subjects to produce disallowed corrective movements over a narrow range of movement times between about 150 to 300 ms. Over a given temporal window (about 200–240 ms), the same rate of correction was found in this location–stop condition and in a control ‘location–go’ condition where subjects had to correct the action in response to the target jump. Only movements slower than 300ms could be fully controlled by voluntary processes. In contrast to this normal pattern, a patient with a bilateral lesion of the posterior parietal cortex (Fig. 4.14) showed a lack of on-line automatic corrective processes, whereas intentional motor processes were preserved (Pisella et al. 2000: Exp. 3). This allows us to conclude that fast movements are controlled by a posterior parietal ‘automatic pilot’ (PAP) located in the dorsal stream. By contrast, slow movements are controlled by intentional motor processes that remain largely independent of the posterior parietal cortex. Accordingly, frontal patients tested on the same tasks exhibited a complete loss of intentional inhibition of their automatic corrections (see Pisella et al. 2000). Thus the notion of automatic pilot extended that of ‘hand-sight’ in the sense that it refers not only

they had enough time to intentionally stop their movement during slow trials. However, an intermediate class was also observed, in which subjects performed a signi1cant number of unwilled corrections. (b) Distribution of the unwilled automatic corrections performed in response to unexpected target jumps. Comparison with a control ‘location–go’ condition in which another group of subjects faced the same stimuli set but were instructed to perform corrections in response to the target jumps. The percentage of corrected pointing responses was calculated with respect to the total number of perturbed and unperturbed trials performed by 6 subjects. Corrected movements in response to target jumps appeared for the movement duration of 150ms and became signi1cant with respect to the motor variability observed in fast, unperturbed trials (speed–accuracy trade-off law) for the movement duration of 200 cms in both location-stop and location–go conditions. Automatic corrections were produced by the location–stop group until movement times of about 300ms, which allow voluntary control to fully prevail over automatic visual guidance. A total of 9% of all the perturbed trials were redirected toward the second location of the pointing target in this stop condition. For movement durations between 200 and 240ms, the correction responses were produced at the same rate by the location–go group (in accordance with instruction) and by the location–stop group (irrepressible reaction despite the instructed stop response). This indicates that these fast motor corrections result from the same automatic pilot in both groups of subjects. (Adapted from Pisella et al. 2000.)



Fig. 4.14 Speci1c disruption of automatic corrections following a bilateral parietal lesion. This set of 1gures illustrates the pointing performance of three control subjects compared with a patient, IG with bilateral optic ataxia, in the ‘location–stop’ and in a ‘location–go’ condition where correcting movement toward the second target location was instructed. For each condition, the horizontal bars indicate the 95% con1dence intervals of movement time computed for all stationary targets. In the lower part of the 1gure, the vertical dotted line indicates the upper edge of the 95% con1dence interval of movement time computed for all non-interrupted perturbed trials (displayed as a dotted horizontal bar). When correction was instructed (upper panel), control subjects mainly produced corrections without increasing their movement time with respect to unperturbed trials (horizontal bar), whereas most of the corrections produced by the patient IG caused a large increase of movement duration. When the stop response was instructed (lower panel), the patient produced no corrective responses, whereas about 10% of the perturbed trials elicited disallowed corrective responses in the controls. For the interruption response, the patient exhibited a performance similar to normal subjects (vertical dotted lines). (Adapted from Pisella et al. 2000.)

91


92


Fig. 4.15 Interaction between reaction time and movement on-line control. Mean latencies of the responses given to target jumps with respect to three classes of movement times during a pointing task. Motor correction was instructed in response to target perturbation in location. A post-hoc classification of corrected versus un corrected movements showed that they corresponded to significantly different reaction time values. to unconscious visual processing by the action system, but also to an autonomous use of visual information which bypasses and even counteracts intention.

4.6.3 Interaction between movement preparation and execution In the ‘location–go’ condition described above, subjects faced the same unpredictable target jumps occurring at movement onset but were instructed to redirect their ongoing movement toward the second location of the target. Figure. 4.15 shows that whether perturbed trials were corrected following the target jump depended on their movement times and their reaction times. Three motor phases were identi1ed with respect to movement times. Movements faster than 150ms were ballistic and always reached the initial location of the target. No reaction was observed in response to the target jump. For durations between 150 to 300 ms, both corrected and uncorrected movements were observed. On-line reaction to the perturbation became possible but errors were still observed. All movements lasting more than 300 ms correctly responded to the target jump. Figure 4.15 shows that for the same movement time interval, corrected pointings tended to exhibit shorter latencies, that is, shorter durations available for movement programming. These shortlatency movements may have been less precisely or less rigidly programmed and they were consequently more sensitive to the on-line visuomotor guidance. In contrast, movements that bene1ted



from a longer and 1ner programming phase were less reoriented on-line by the automatic pilot in response to the target jump. Long-latency movements appeared therefore to be more rigidly executed as they were better programmed. Accordingly, they showed no 2exibility to respond to the target perturbation and reached the initial target location.

4.6.4 Conclusion As a conclusion, in addition to the anatomical connectivity between areas, we have to take the timing of the activation of cortical networks into account. As the action system seems to bene1t from overall faster processing than the identi1cation system, dissociation between action and cognition can result from time factors. Timing and speed-constraints seem to be the keys for a ventral/dorsal dissociation in the cerebral network. The participation of a given area in an anatomical network does not imply that it is absolutely necessary to the task being considered. Temporal constraints may be a reason for a connected area not to participate in a process (but other reasons can be evoked, such as a particular con1guration of the network, as shown at the beginning of Section 4.6).

4.7 Recipes The current chapter is based on a controversy between behavioural evidence and neuroanatomical data and we have proposed that temporal variables can provide a way to reconcile these two lines of evidence. However, numerous other parameters have been evoked throughout the previous sections. We found it useful to take an inventory of these parameters and attempt to classify them in order to compile a user’s manual for researchers interested in the relationship between sensorimotor and cognitive processes. For example, these directions for use should be particularly helpful in the 1eld of visual illusions, where a controversy is growing between the arguments for dissociation versus integration of the two visual systems. To enable valid comparisons between experiments, attention should be paid to the use of comparable conditions among the many axes that will be listed below. These recipes could also prove bene1cial in a rehabilitation context. In patients with one of the two systems damaged, they can be used to set optimal conditions allowing them to achieve better performance by recruiting residual visual-to-motor networks. Activating these residual functions could provide a means to activate the organization of compensatory mechanisms, and thus become a 1rst step of re-adaptation processes.

4.7.1 Brain lesion One obvious way to speci1cally isolate the dorsal stream is to study an individual with a lesioned ventral stream. Reciprocally, one obvious way to speci1cally isolate the ventral stream is to study an individual with a lesioned dorsal stream. As shown above, the neuropsychological double-dissociation observed between optic ataxia and visual agnosia has provided a key argument for building up the notion of a perception/action dichotomy on the top of the ventral/dorsal segregation. In the absence of one of the two visual systems, the processing performed by the other one can be expressed in the most pure fashion. In the same way, primary sensory lesions, such as encountered in action-blindsight and action-numbsense, seem speci1cally to restrict visual processing to the action system. It would be tempting to propose that it should be easier to study the properties of the dorsal stream in patients with blindsight rather than in patients with a bilateral ventral stream lesion. Indeed, neurologists

93


94


much more frequently encounter hemianopsia, which is a prerequisite to blindsight, than the kind of ventral stream lesion exhibited by DF (see Milner and Goodale 1995). However, the dissociation observed in this patient is more pronounced than that observed in action-blindsight patients (see Rossetti 1998). Although both types of patient fail to produce signi1cant verbal guesses about the object property being tested, Fig. 4.2 illustrates the greater variability observed in the motor task for a blindsight patient. This difference can be attributed to the different lesion sites. A V1 lesion is responsible for a full loss of cortical visual inputs, whereas a ventral lesion should leave the dorsal stream intact. In the case of blindsight, the only input to the dorsal stream appears to arise from subcortico-cortical projections (see Fig. 4.5), which obviously do not provide a normal input. Conversely, a difference may be noted between the two types of patient for the matching task, the performance also being poorer in action-blindsight than in visual agnosia (see Fig. 4.2).

4.7.2 Speci1c inputs As speci1c dorsal versus ventral brain lesions can result in a pure activation of one of the two visual systems, the functional properties of the anatomical network suggest that some visual features can selectively activate these systems. For example, it is known that most of the neuronal activity related to colour processing is found in the ventral stream, whereas the processing of location would be more speci1c to the dorsal stream (Heywood, Shields, and Cowey 1988; Ungerleider and Mishkin 1982). In a pointing experiment, Pisella et al. (2000: Exp. 2) tested whether the parietal automatic visuomotor guidance would extend to other visual attributes than target jump. In particular, would it also be observed when the change in target location is encoded through a chromatic perturbation? To test this, a green and a red target were presented simultaneously in the two positions used for the location–stop task. The subjects were instructed to point at the green one, and the colour of the two targets could be unexpectedly interchanged at movement onset. As the ‘location–stop’ group, subjects in the ‘colour-stop’ group were instructed to interrupt their ongoing movement in response to the perturbation. In contrast with the ‘location-stop’ group, no automatic corrective movements were observed in the ‘colour–stop’ group (Fig. 4.16). It was concluded that only intentional corrections can be produced on the basis of a colour cue and that the visuomotor transformations of the hand’s ‘automatic pilot’ may be speci1c to location processing and spatial vision. This speci1city for ‘dorsal attributes’ can be related to the partition of magnocellular and parvocellular inputs between the dorsal and the ventral stream. Another related explanation is that this speci1city is due to the processing time required for ‘ventral attributes’, which is not compatible with the expression of automatic processes. Indeed, irrepressible visuomotor corrections result not only from their automaticity (the processes of visuomotor guidance are inherent to movement execution) but also from their high speed relative to the slow process of voluntary control. It may be hypothesized that normal subjects produce unintentional corrections because the slow intentional inhibition process from the frontal lobe leaves enough time for the fast automatic corrective processes to alter the motor output. It is interesting to consider the strong contrast between the absence of a colour effect on the PAP (Pisella et al. 2000) and the signi1cant colour guesses performed by blindsight patients (Stoerig and Cowey 1992). The main difference between these two experimental situations seems to lie with the type of response investigated. In the case of the Pisella et al. study, movements are all programmed and initiated before the relevant event (target change in colour or location) appears. Therefore the target perturbation is only relevant to movement guidance and it is found that colour processing does not affect on-line motor control. In the case of colour-blindsight, simple responses have to be



simply initiated (and there is no proper execution control in simple key pressing). This discrepancy suggests that different sensory coding can be involved in different motor responses, and speci1cally in action preparation versus execution. In the same way, the use of simulated object jumps in conditions where a coloured light indicated which object to grasp from an array of several objects (e.g. Paulignan et al. 1991) should be reinterpreted in the light of the Pisella et al. (2000) results. The abrupt changes observed in the trajectories obtained by Paulignan et al. indicate that the ongoing movement was interrupted and replaced by a secondary movements directed towards the new object to grasp. Indeed, conditions where a real object displacement (and not a colour code) was triggered at movement onset gave rise to much smoother trajectories (Gréa et al. 2000), which are compatible with the activation of the automatic pilot. This is con1rmed by the observation that these corrections are completely disrupted following a posterior parietal lesion (Gréa et al. 2002). To summarize, using colour stimuli is more likely to activate the cognitive than the sensorimotor system. The on-line control of action has speci1c access to metric object properties, but remains unaffected by categorical properties such as colour. The issue of depth apprehension will be addressed in a further section (context processing).

4.7.3 Speci1c outputs The study of the dorsal–ventral dissociation in the motor context enables us to distiguish speci1c motor abilities, which are preserved by visual agnosia and blindsight patients, from other types of motor responses. These patients, with impairment of the ventral stream, remain able to direct an action and even to adapt the hand to unidenti1ed or even unseen objects. The preserved, immediate goal-directed action thus seems to be implemented in the dorsal stream. In contrast, matching, delayed, slow, and indirect motor responses are performed very poorly by these patients. Several types of indirect (in space) action have been tested. In action-numbsense, the patient was unable to indicate where he was stimulated on a drawing, whereas he pointed at an above chance level to these unperceived stimuli in a direct pointing task (Rossetti et al. 1995, 2001; Fig. 4.3). A related type of response is derived from the classical antisaccade paradigm. When subjects are requested to make a reverse movement with respect to what would be a direct action, they usually perform with a reduced accuracy. Interestingly, the pattern of errors observed for visually guided antisaccades is similar to that of the memory guided antisaccades (Krappmann 1998). This result is again coherent with the observation that movements shifted in space share their representation with movements shifted in time with respect to direct action. This idea was further con1rmed by experiments investigating antisaccades in a visual agnosic patient (DF). Dijkerman et al. (1997) observed that DF was performing accurate saccades in natural conditions but was strongly impaired for delayed and antisaccades (Milner, Dijkerman, and Carey 1999b). More recently the same idea was applied to hand pointing. As in the Pisella et al. study (2000), it was shown that fast movements are under the control of an automatic process. Interestingly this automatic guidance could participate in the direct pointing performance, but counteracted the anti-pointing task derived from the antisaccade paradigm (Day and Lyon 2000). Schneider and Deubel (this volume, Chapter 30) have also shown that the automatic orientation of attention that is observed around the location of the target of a saccade being prepared, is not observed in the case of antisaccades. Other experiments have also shown that anti-pointing performance in response to masked stimuli would be poorer than direct pointing (Price et al. 2001).

95


96


Fig. 4.16 Speci1c inputs. (a) In response to target jumps, unwilled correction movements occurred even when countermanded (location–stop condition). Corrections were performed to a signi1cant extent as compared to unperturbed trials despite the opposite instruction. (b) A colour switch between two targets (‘colour-cued’ target jump) was not able to elicit automatic corrections. Although the change in target location was physically the same as in the ‘location– stop’ condition, no signi1cant corrections toward the new green target were observed compared with unperturbed trials. In this condition, responses to the perturbation always complied with the stop instruction. (Adapted from Pisella et al. 2000.) Another type of action to consider is pantomime. In the Goodale et al. (1994) study of delayed actions in DF and normals, it was observed that the kinematic structure of pantomimed movements was similar to that of movements delayed by 2 or 30s. This study further supported the view that brain mechanisms underlying perceptual representations are quite independent of those activated during action, and stressed the necessity for motor representations to have an on-line access to target parameters. The effect of subtle changes in the type of motor output requested from subjects can also strongly affect the type of processing involved. A rather clear example has been provided by Bridgeman (this volume, Chapter 5) when he compared communicative pointing to instrumental pointing. Subjects



could be asked either to point at an object or to really act on it to push it down. The latter condition provided a stronger difference from the perceptual tasks used in the same experiments, suggesting that the former type of response could result from a less pure activation of the sensorimotor system. One should note that the issue of the reference frames used by the two systems (an egocentric one for the action system and an allocentric one for the perceptual system) has been more or less implicitly addressed in several sections of the present review, and is strongly dependent on the type of task performed (for recent reviews, see Desmurget et al. 1998; Goodale and Haffenden 1998). From the strong effect of the response on the type of process involved in an action, it seems easier to activate the cognitive mode rather than purely the sensorimotor mode of action control. Any experimental conditions departing from the natural, direct goal-directed action appear to less selectively activate both sensorimotor and cognitive systems in various proportions. As will be addressed below, this is true not only for spatial but also for temporal factors. One interesting parameter to consider when different tasks are compared is the respective amount of programming versus on-line control that is required by the task. Clearly reaching tasks, directly aimed at a real object (Day and Lyon 2000; Goodale et al. 1986; Pisella et al. 2000; Prablanc and Martin 1992), would generate the maximal activation of on-line control processes.

4.7.4 Delaying the action ‘Delaying an action can transform a theory.’ Agreeing with this quote from Goodale et al. (1992), the review presented in the previous sections demonstrates that there are numerous examples of a dramatic effect of delay on sensorimotor processes. Probably time appears to represent the most crucial dimension in the control of sensorimotor–cognitive interactions. Two conclusions can be reached on this issue: (1) sensorimotor representations are short-lived; (2) cognitive representations take over when a delay is introduced. In terms of recipes, one should note that the titration of the effect of delay on action seems to depend on the sensory modality. As shown in Fig. 4.12, the crucial delay to switch from a dominant sensorimotor activation to a dominant cognitive control would lie between 1 and 2 s in the tactile modality. Other experiments with the same action-numbsense patient showed that this delay was of about 4-s for the proprioceptive modality (Rossetti 1998; Rossetti et al. 2001). In the visual modality, most experiments used delays of several seconds (e.g. Bridgeman, this volume, Chapter 5; Goodale et al. 1994; Milner et al. 1999a). A 2-s delay appears to be long enough to strongly affect the motor output. Fewer experiments tested delays below this value, but it seems that a delay value of about 500ms would be suf1cient to generate sensible changes in the type of motor output (see Fig. 4.11; see Krappmann 1998; Rossetti 1998; White et al. 1994). There is a large amount of converging evidence, arising from three sensory modalities, that the sensorimotor representation can only be expressed within a short delay following stimulus presentation, providing a very simple recipe to study sensorimotor versus cognitive control of action. Another aspect worth mentioning here is that introducing a delay between the stimulus and the response should rapidly disrupt the ability to process on-line control of the action.

4.7.5 Speed constraints In the same way as normal actions can be delayed, a speed constraint can be added on any given task in order to explore short time-range phenomena. For example, the Pisella et al. (2000) and the Day and Lyon (2000) experiments showed that only very fast movements could be considered as

97


98


fully driven by an automatic pilot. Different values have been reported in the literature about the minimum movement duration compatible with on-line corrections (see Day and Lyon 2000; Pisella et al. 2000; Rossetti 1998). This value depends on the type of task, on movement amplitude, and on the experimental set-up. For example the value of about 200ms reported by Pisella et al., obtained for vertical pointing against gravity (and thus with higher torque values) on a computer screen, is higher than the usual value obtained in the horizontal plane, closer to 100ms (e.g. Desmurget et al. 1996; Gréa et al. 2000; Prablanc and Martin 1992). In terms of recipe, values of movement time between 200 and 250ms in the vertical plane (Pisella et al. 2000) and between 125 and 160ms in the horizontal plane (Day and Lyon 2000) seem to correspond to movements dominated by the parietal automatic pilot for the hand.

4.7.6 Visual illusions The two experiments on the effect of static visual illusions on action presented earlier in this chapter (Aglioti et al. 1995; Gentilucci et al. 1996) have given rise to a flurry of research on this topic. Other related experiments, using illusion-like phenomena linked to motion perception, are summarized by Bridgeman (2000). Until now we have referred to only a few experiments on visual illusion and action, but we have counted at least 12 articles published on this topic between 1999 and mid 2000. Most of these studies replicated the initial 1ndings made by the 1rst two papers, namely that the effect of visual illusions appears to be stronger on perceptual reports than on action performance. Nevertheless, a number of interesting features emerge from this hot topic, which may help the reader to prepare his own illusion recipe. First, a variety of visual illusions have been investigated, for which there is a relatively weak effect of the illusion on action. In addition to the Titchener’s circle (or Ebbinghaus size-contrast illusion) effect on grasping (Aglioti et al. 1995; Haffenden and Goodale 1998, 2000a) and pointing (Fisher, unpublished), the Müller–Lyer illusion has been studied during both pointing (Gentilucci et al. 1996) and grasping tasks (Otto-de Haart, Carey, and Milne 1999; Westwood, Chapman, and Roy 2000). Grip aperture during prehension was also shown to remain unaffected by the Ponzo illusion (Jackson and Shaw 2000) or less affected than perception by the horizontal–vertical illusion (Vishton et al. 1999: Exp. 1, but see below). Other illusions have been used to show that the positioning of the grasp remained less sensitive than perceptual tasks (Ellis, Flanagan, and Lederman 1999). Also, the developmental aspect of these effects was addressed by investigating the sensitivity of children to illusion (Kovacs 2000). It was found that children were less sensitive to the perceptual illusion than adults. The authors suggested that this may indicate a slower maturation of the ventral stream with respect to the dorsal stream of visual processing, which would be another way of supporting the dissociation between the two visual systems. Second, several limits of the relative insensitivity of the action system to visual illusion should be considered. In order for a visual illusion to produce no effects on action, one has to speculate that the neurological substrate that is responsible for generating the illusion lies somewhere around the ventral visual system. However, some illusions have been described as affecting very early visual processes, even at the retinal level (see Gentilucci et al. 1996). If an illusion does affect the visual processes taking place between the retina and the primary visual cortex, then the effect of this illusion (or at least of some of its components) would feed both the dorsal and the ventral stream, and one would expect action to be affected by this illusion. In the same vein, one has to address the



issue of the visual subsystem that is involved in the action. For example, it has been shown that the Ponzo illusion would affect grip size but not the force programmed to lift a target object (Jackson and Shaw 2000; see also Brenner and Smeets 1996). It is interesting to note that grip size is one of the metric properties of space that has been proposed to be processed within the dorsal stream (see Milner and Goodale 1995; Rossetti 1998), whereas the object weight has to be inferred from previous learning of the relationship between objects’ appearance and their weight. This association has to involve other structures than just the dorsal stream, for example the ventral stream and its connections to memory systems (see Jeannerod et al. 1994). Therefore it involves more off-line processing (prior to the action) than on-line motor guidance. Third, the issue of relative versus absolute coding of object metrics has been raised by Pavani et al. (1999), Vishton et al. (1999), and Franz et al. (2000). They observed that previous studies with the size-contrast illusion (the Titchener circles) presented subjects with two stimuli to compare in the perceptual task, whereas only one object was used in the pointing task (Aglioti et al. 1995). They performed experiments in which only one illusory object was presented at a time. Pavani et al. (1999) and Franz et al. (2000) used the size-contrast illusion and found that both perceptual and motor responses were affected by the illusion. Vishton et al. (1999), using the horizontal–vertical illusion, extended the initial observation made on the size-contrast illusion when two stimuli were presented at once, but showed that the perceptual effect could be suppressed when only one element of the display was presented. They also observed that grip scaling was strongly in2uenced by the illusion when subjects had to scale their movements to both vertical and horizontal dimensions of the triangular 1gure. The more complex visuomotor processing required by this latter task suggests that it may be assimilated to matching tasks in which the hand has to reproduce an object size between two 1ngers. In contrast to natural grasping action, these tasks have been proposed to involve ventral stream processing (Haffenden and Goodale 1998; Milner and Goodale 1995). Other experiments have shown that such tasks are much more in2uenced by illusory context than natural direct actions (Westwood et al. 2000). However, it remains that a relative judgement made between two stimuli seems to be more sensitive to illusion than the absolute estimate required by a simple action to one stimulus. Fourth, another possible confound has been raised to account for the discrepancy observed between several studies. Haffenden and Goodale (2000a) have explored the effect of the targetsurrounding elements gap in the Titchener illusion. They 1rst showed that a smaller target-2anker gap produced an apparent effect on grip scaling (see also Pavani et al. 1999). Then they investigated the effect of two neutral rectangular 2ankers that could be presented along the horizontal or the sagittal axis of the object. While a perceptual effect was observed in the two conditions, a motor effect appeared only when the 2anker lay on the sagittal direction, that is when it could interfere with the two grasping 1ngers. The effect of these 2ankers on action also varied with the target2anker distance, suggesting that elements from the 2-D stimulus could be considered as potential obstacles by the action system. Alternatively the difference between the effects of a visual context aligned with the depth or the frontal axis presented on figure 4.17 may explain such a result. These findings are likely to explain the variety of the effects reported in the literature. Altogether there seems to exist a reproducible effect of visual illusion on perception while the action system is less sensitive to illusory context. The variety of results found in the literature suggests that careful attention should be paid to the design of the stimuli, however, since the perceptual and the action systems may be sensitive to different stimulus properties. For example, the action system appears to be sensitive to motion-induced illusions (Smeets and Brenner 1995) and to depth cues (see below).

99


100


4.7.7 Context processing A tentative explanation for the effects of illusions on perception and action is that only the former system would be in2uenced by visual context. Arguments for this interpretation come from experiments performed with illusory set-ups. For example, the Roelofs effect was used by Bridgeman to compare perceptual and motor effects (Bridgeman, this volume, Chapter 5). As for illusions, a pronounced effect was found for perceptual estimates, whereas no signi1cant in2uence was observed on pointing responses. It has therefore been argued that the sensorimotor system would not be in2uenced by the visual context, as this seems to be the case for immediate (vs. delayed) pointing (e.g. Bridgeman 1997, 2000, this volume, Chapter 5; Rossetti 1998). However, several examples can be found of an in2uence of visual context on action. Several levels of context complexity may be considered here: intrinsic, extrinsic, and inferred contexts. First, the minimum level of visual context that can be tested during a simple action seems to be the size of a target. When subjects have to point to targets of different sizes, they exhibit a spontaneous modulation of their movement time as a function of both target size and distance. The expression of this relationship has been provided by Fitts (1954). If one accepts that the size of a target is the simplest context in the task of pointing to the centre of the target, then the effect of target size on pointing may be considered as an effect of intrinsic context. Not only normals but numerous types of brain damaged patients do comply with Fitts’s speed–accuracy trade-off law. This is also true for schizophrenic patients, who are known to be impaired for context processing (Saoud et al. 2000). Second, the effect of target size applies to sequential movements. The 1rst stroke of a two-stroke sequence is altered even if the size of only the distal target of a two-stroke movement is manipulated (e.g. Marteniuk et al. 1987). This robust effect even applies to schizophrenic patients (Saoud et al. 2000). Third, another example is derived from the experiments designed by Bridgeman about the Roelofs effect, where an illusory displacement of a target, caused by a change in position of a surrounding rectangular frame, affected phenomenological experience of target position, but not pointing responses (Bridgeman 1991; review in: Bridgeman 2000, this volume, Chapter 5). One of the conclusions from this work was that the sensorimotor system would be insensitive to the visual context. However, this interpretation stands in contrast with the reported 1nding that visual context in2uences the perception of the egocentric distance of a visual stimulus (e.g. Coello and Magne 2000). In order to unravel this issue, a recent study addressed the question of whether the dissociation between verbal identi1cation and reaching holds when the illusory target’s displacement is radial instead of horizontal, as was the case in the original report (Bridgeman 1991). The task for the participants was to verbally determine whether the second target was presented in the same location as the 1rst one, or to make a non-visually controlled pointing movement towards the second target. The results showed that a similar illusion of target displacement was obtained with verbal reports when the illusion stemmed from a horizontal or radial rectangle displacement (thus con1rming Bridgeman’s 1ndings). However, the striking outcome was that no effect of the rectangle position was observed in motor performance, but a very signi1cant effect was obtained with the displacement made along the radial axis (Fig. 4.17). This latter effect con1rmed the involvement of visual context in distance but not direction perception (Coello and Magne 2000). In addition, the amount of the illusion-induced effect was identical in the perceptual and the motor tasks. This 1nding has an important implication for theories of visual perception, in the sense that it shows that the distinction between visual perception for identi1cation and visual perception for action is task-and-stimulus



Fig. 4.17 Context processing in the depth dimension. Participants were shown for 0.4s a visual target (diameter 8 mm) presented at 23cm and centred according to a radial or horizontal rectangle. Black and white symbols depict the results obtained for the two test-targets. Then both the target and the rectangle vanished for a period of 0.5ms, before reappearing for 0.4s at a similar or different location. Furthermore, the target was presented alone, or inside an uncentred rectangle (96×38mm), displaced either on one side or the other with respect to the target. The task was to verbally determine whether the 2-D target was in the same location as the 1rst, or to make a nonvisually controlled pointing movement towards it. Perceptual and motor responses exhibited the same bias only when the rectangular frame was presented in the radial orientation: the pointing distance was overestimated when the perceptual matching of the target was overestimated, and vice versa. (From Coello et al. 2001.) dependent. It should be emphasized though, that the absence of the visual target at the time of movement completion may also have reduced the in2uence of on-line motor control on movement guidance. Other experiments have shown that the depth dimension has a speci1c status with respect to the relations between perception and action. Dissociations can be observed between cognitive and sensorimotor responses to depth stimuli (see Dijkerman et al. 1998). However, interesting comparisons can be made concerning the ability of DF (the case with bilateral visual agnosia) to perform orientation judgements. She seems to perform better on perceptual judgements of orientation in depth under binocular viewing conditions than she would have when judging the orientation of an object (or slot) in the picture plane (Dijkerman et al. 1996; Goodale et al. 1991). On the basis of DF’s performance it seems that there is less difference between perception and action with regard to the orientation of objects in the depth plane than in the frontal plane (see also Carey, Dijkerman, and Milner 1998). Altogether, the effect of context on action seems to strongly depend on the experimental procedure. As shown by simple or sequential pointing, intrinsic properties of the goal of an action do affect motor parameters. More distal contexts, such as those responsible for the generation of static illusions or illusory movement, appear to be less effective on action, unless the depth dimension is considered. As mentioned in the visual illusion recipe section, the most relevant issue here appears to be the

101


102


level of processing concerned in each type of context. Intrinsic context such as target size or position in depth refers to a primary metric property of objects, whereas more complex contexts, such as a target array or an illusory set-up, imply integration mechanisms over time or space. In the latter case of the complex integration of a spatio-temporal context (cf. Fig. 4.17), only delayed actions are strongly affected. One should pay attention to these parameters when building an experiment to investigate the role of context on action.

4.7.8 Actual versus represented stimulus A remaining question about the difference between immediate and delayed tasks such as those presented in Fig. 4.10 is whether the observed effects can be attributed to the duration of the target presentation or to the delay from target onset to the response. In other words, does the availability of the target just prior to the initiation of the action in the immediate condition account for the difference between the two conditions? This question was explored in a simple experiment where subjects had their target 1nger held in place for a given delay prior to the go-signal versus shortly positioned on the target location, followed by a similar delay. A 2-s memory delay was suf1cient for the cognitive representation to affect the pointing distribution (as for the 5-s delay shown in Fig. 4.11). In contrast, no such effect was observed for the 2-s duration of the target presentation; that is, when no memorization was required (see Fig. 4.18). In this case, both types of visual representation are available: (1) the cognitive one, because the 2-s delay enabled the subject to encode the target location with respect to the target array, and (2) the on-line sensorimotor one, because the object has just disappeared upon the movement onset. The result (Fig. 4.18) shows that only the sensorimotor one seems to contribute to the motor output. Therefore priority seems to be given to the sensorimotor mode of processing when it is available (as shown by the Pisella et al. 2000, results described earlier). This result indicates that the effect of delay cannot be solely attributed to the slowness of the cognitive system, but is also due to the absence of a real target object that has to be represented because sensorimotor processes are short-lived (see Rossetti 1998). A similar logic can be used to explore the nature of the representation involved in action in neurological patients. In order to test for the ability of optic ataxia patients to process visuomotor information on-line, we performed another experiment with our patient with a bilateral posterior parietal lesion (IG). When IG was asked to delay her pointing or grasping action, she improved her poor ability to perform accurate actions (review in: Milner and Dijkerman 2001). Knowing that the effect of a memory delay is different from that of a long presentation of a stimulus in normals (see Fig. 4.18), we investigated this possibility with IG. She exhibited a better performance in both tasks, suggesting that both the long presentation and the delay enabled her to form a representation of the object that could be used by the action system. Given her lesion, this representation was postulated to be formed via the ventral stream (see also Milner et al. 1999a). An interesting question that leads from this is whether it is possible to generate a con2ict between this sustained cognitive representation and short-lived motor representations. Together with Prof. David Milner and Dr Chris Dijkerman, an experiment was designed in which an object was presented for 2s, then hidden behind a panel for 8s, then shown again. This procedure improved her grip scaling compared with an immediate action condition. Then a special session was performed in which, in some trials, the small object could be unexpectedly replaced by a large one, or vice versa. The speci1c question asked here was whether the grip formed by IG would follow the size of the present object or that of



Fig. 4.18 Long presentation versus memory delay. Mean of ellipse orientations (beta in deg, see Fig. 4.8) as a function of various delays between target onset and go-signal, with the target remaining present during the delay (delay = ‘target duration’) or not (‘memory delay’). When the target remained present, for immediate pointings (control), 500ms or 2s delays, the orientation of the ellipses remained approximately around beta = 135deg, i.e. as for immediate pointing movements. In the absence of the target as soon as there was 2 s of memory delay the beta tended to displace toward the orientation of 90deg (illustrated by the interrupted line). This graph shows that the ellipse orientation 1tting the pointing distribution is contingent upon the presence or absence of the target during the delay. the internal representation formed after the presentation of the initial object in the same trial. The results clearly showed that her grip was initially scaled to her internal representation rather than to the actual object she was aiming at (see Fig. 4.19). Control subjects tested in the same conditions exhibited an on-line grip formation that was adapted to the object present in front of them at the time of the grasp (see also Milner and Dijkerman 2001). In addition to the effect found on grip size, the time to maximal grip aperture was reached earlier in the large→small condition than in the large condition for each of the six control subjects, whereas IG exhibited a similar value in the two conditions. On the particular trials where she ‘correctly’ opened her hand widely on ‘small→large’ catch trials, the wide grip aperture actually occurred abnormally late in the movement. This strongly suggests

103


104


Fig. 4.19 Represented versus actual object. This 1gure presents the four steps constituting one noncongruent trial of the delayed grasping task, producing a kind of object-size perturbation. In these non-congruent trials, occurring only in 20% of the trials, the object of size A presented before a 8-s delay is replaced by an object of size B for the initiation of the grasping movement. The 1gure illustrates the results obtained for controls and for patient IG, who has bilateral optic ataxia. IG produced a movement with a grip size based on the 1rst object visualized. She executed her movement quite rigidly as it was programmed during the memory delay, before 1nally adapting her grasp to the second object. Control subjects were less in2uenced in their grip size by the 1rst object presented, they rapidly adapted their 1nger aperture to object B and seemed to behave as in the case of immediate movements toward object B. Control subjects can use the on-line information to perform their movement, whereas the patient with lesion of the dorsal stream seemed to use slower and more rigid sensorimotor processes. that she was unable to process the change in size that had occurred during the delay fast enough to be able to update her ongoing movement. These results clearly con1rm that sensorimotor and cognitive representation have different time constraints: while sensorimotor representations can be elaborated in real time and are very short-lived, the cognitive representation needs more time to emerge and can be evoked off-line in the organization of action.

4.7.9 Motor and perceptual co-activation We have referred earlier to experiments where there appears to be a simultaneous activation of the sensorimotor and the cognitive systems in normal individuals and patients (Bridgeman 1997;



Fig. 4.20 The speci1c effect of object feature verbalization. As in Fig. 4.9, subjects were requested to produce verbal estimates during a simple pointing to a proprioceptive target. A change in the distribution of the con1dence ellipse main axes was only observed when the verbalization concerned the target location (relevant to the action) and not when it concerned the target texture. Rossetti and Régnier 1995; reviews in Bridgeman 2000; Rossetti 1998). In a further experiment derived from the Rossetti and Régnier study, subjects were trained to identify two target features: location and texture (Pisella et al. 1996). During this training, a texture could be presented on any target location, without any systematic association between location and texture. Then on two different days, experimental sessions where performed in which subjects were requested, on each trial, to provide a verbal guess about the current target being presented. The number of verbal errors made during the pointings was similar in the two conditions, suggesting that the two tasks were equally dif1cult. When the guess applied to target texture, no speci1c effect was observed on ellipse orientation (see Fig. 4.20). By contrast, a signi1cant modulation of the endpoint spatial distribution was observed when the guesses applied to the target location. This result suggests that for an interaction to be observed between the cognitive and the motor system, experimental conditions have to activate verbal representation of particular features that are relevant to the action being performed. Only in such conditions does the representation elaborated in the cognitive system seem to overwrite the sensorimotor one. Another observation worth taking into consideration before trying this recipe was made on our action-blindsight (Rossetti 1998) and numbsense (Rossetti 2001) observations. We have seen earlier that the simultaneous motor–verbal task disrupted action performance. We have also had several opportunities to note that, once the subject was asked to provide a verbal guess at the end of a trial, then the following series of trial would show a disruption of performance as well. This is as if the only fact that the patient knew he might have to describe the stimulus activated cognitive processes which interfered with the residual sensorimotor ability.

105


106


4.7.10 Complex recipes Several of the ingredients described in the previous recipes can be combined into more complex recipes. For example, the effect of illusion and that of delays can be coupled. A detailed subject-bysubject analysis of the experiment testing the effect of the Roelofs effect on action showed that only half of the subjects exhibited a motor effect of the visual illusion (Bridgeman 1991). This observation became all the more interesting when it was observed that interposing an 8-s delay before the response forced all of the subjects to use spatial information that is biased by the perceptual illusion (Bridgeman 2000), replicating an earlier 1nding made on eye movements (Wong and Mack 1981). This result suggested that subjects might switch from motor to cognitive modes of sensory processing at differing delays after stimulus offset. In an elegant experiment, Gentilucci et al. (1996) also showed that introducing delays between line observation and onset of movement proportionally increased the effect of the Müller–Lyer illusion on the pointing response (see Fig. 4.21). It is also very interesting to notice that the in2uence of the illusion becomes particularly noticeable as the delay between stimulus presentation and movement onset increases. Another combination of ingredients from the previous recipes was made between a speci1c input and a speci1c output, namely colour and movement interruption. To co-activate the direct sensorimotor representation of the target responsible for automatic on-line motor control and the higher level cognitive representation of the same stimulus, we designed an experimental device producing double perturbations of a visual target at movement onset. These double perturbations involved a change in location, mainly processed in the dorsal stream and known to trigger a fast updating of the ongoing action, and a simultaneous change in colour, known to be mainly processed in the ventral stream and implying a categorization process. Unperturbed trials (80%), simple perturbations in either location or colour, and double perturbations were intermixed in each session (Pisella et al. 1998b; Pisella et al. 1999). Subjects had to point to the green targets, and to redirect their movement if the green target had jumped. The change in colour was associated with an instruction to immediately stop the ongoing movement. The double perturbation contrasts a combination of a dorsal visual input (location) and a dorsal motor response (automatic visuomotor guidance of a goal-directed movement) with a combination of a ventral visual input (colour) and a ‘ventral’ motor response (stop is a conditional motor response to colour change). On the one hand, a faster processing of the location attribute as with compared with the colour attribute is expected (Pisella et al. 1998; Rossetti et al. 1997). On the other hand, the ‘dorsal’ visuomotor guidance inherent to goal-directed execution is swift and automatic, contrary to the arbitrary association of a stop response with the red colour (Pisella et al. 2000). The double-perturbation condition therefore brings together the association of speci1c inputs and speci1c responses that offers the greatest temporal difference, allowing us to dissociate the systematic corrections driven by the dorsal ‘automatic pilot’ from the intentional motor control of the movement execution. A long temporal window of automatic behaviour should therefore be observed. The results of six subjects con1rmed these predictions. For pointing movements with durations shorter than about 200 ms, no effect of the perturbation was observed. For movement times ranging from 200 to about 280 ms, subjects behaved fully automatically in response to double-perturbed trials: they systematically redirected their movement toward the forbidden displaced target and touched it as frequently as when correction was actually instructed (in response to the simple location perturbation). The instructed response to movement interruption appeared progressively only for slower movements (Fig. 4.22). But 1nally, in this condition, subjects produced automatic disallowed corrections for movement times ranging from 200 to 450ms (a total of 15% of all trials) and expressed a strong frustration linked to the sensation of being unable to control their action intentionally.



Fig. 4.21 Visual illusions and delayed action. Pointing biases induced by the two con1gurations of the Müller–Lyer illusion. Movement amplitude tended to increase in the open con1guration and to decrease in the closed con1guration, i.e. in the same direction as the perceptual illusion. Values plotted on this 1gure were normalized by subtracting the value obtained for the control con1guration. The effect of the illusion on pointing was very weak in the full-vision condition. It is noticeable that the effect of the illusion on movement amplitude increased when less information was available to the subject and when a delay was introduced between the stimulus presentation and the response. (From Rossetti 1998 based on Gentilucci et al. 1996.) These two latter cases of sensorimotor and cognitive coactivations took place at different timescales. In most of the data reviewed here, it has been shown that interactions between sensorimotor and cognitive processing of spatial information can occur only from the cognitive to the sensorimotor system (Bridgeman 1999; Rossetti 1998). In motor + verbal tasks tested with normals and patients, for example, the response production was affected in an ‘all or none’ pattern, the semantic representation of the target overwhelming the sensorimotor representation. More recent experiments have shown that the reverse pattern of interaction can be observed on a longer time-scale, through

107


108


Fig. 4.22 Double perturbation experiment. (a) Histograms of the percentage of corrected movements for each type of trials. Corrections were produced as instructed in response to the simple target jump (location perturbation) but also, despite the associated stop instruction, in response to the double perturbation. (b) The automatic corrections escaping subjects’ intentional motor control in response to the double perturbation were produced for movement times between 200 and 450ms. Surprisingly, in a temporal window including movement durations from 200 to 280ms, subjects showed the same rate of correction in accordance (in the simple location perturbation) and in contradiction with instruction (in the double perturbation). This result indicated that corrections from this restricted temporal window surely result from the same automatic processes of on-line visuomotor guidance.

adaptative changes (Bhalla and Prof1tt 2000; Rossetti et al. 1998b). This suggests that several time ranges have to be considered in the discussion of interactions between sensorimotor and cognitive processing of spatial information. As reviewed above, the effect of time can be fully explored by comparing immediate and delayed responses. However, some portion of reaction times inevitably pertains to the production of motor responses, whether produced immediately or delayed. Although



reaction times may be short (about 250ms in normals), one cannot exclude the possibility that this period of time is likely to favour an early effect of delay and/or of early semantic representations on sensorimotor processes. The experiments with speci1c target perturbations during ongoing aiming actions allowed us to explore possible interactions within a shorter time range following the stimulus onset. These series of experiments led us to distinguish programming and control of sensorimotor processes with respect to the direction of interactions between explicit and implicit processes. Programmed sensorimotor processes are more likely to be overwhelmed by cognitive processes. Automatic processes of on-line motor control, however, seem to escape to the in2uence of cogitive processes during a given temporal window following the target perturbation. In short, in order to obtain more involvement of the cognitive system in a motor output one may combine several of the ingredients listed above. Such complex recipes typically should associate a time constraint (delayed or speeded action) with any one of the spatial dimension parameters (illusion, context, speci1c stimuli, or ouputs).

4.8 Concluding remarks We have examined the link between behavioural evidence for a dissociation of the visual brain in two sensorimotor and cognitive subsystems and the neuroanatomical evidence for a complex and interconnected visual-to-motor network. We have found not only that neuropsychological evidence favours two separate visual systems, but also that it indicates that action, under speci1c circumstances, can be organized from one or the other of these two systems, con1rming that there is ‘a dual route to visuomotor action’ (Milner and Dijkerman 2001). The numerous results exposed here about the in2uence of delay or verbalization of speci1c inputs or outputs, in normals and patients, strongly support the existence of two distinct ways of encoding spatial information for action. Then we have provided recipes to control for the respective contribution of the two visual-to-motor systems to a given motor output. As a conclusion, it appears useful to consider a few neglected aspects of this complex organization of action toward visual goals.

4.8.1 A gradual time effect Our review of the literature demonstrates that an effect of time can be observed over several time ranges. Figure 4.23 attempts to describe these time ranges. Let us 1rst consider the level of a single action. For the shorter reaction time, and for tasks performed without delay, we have seen that implicit processing provides the only representation that can affect the motor outcome. This exclusive in2uence of short-lived sensorimotor representations may be carried out by the dorsal stream of visual processing. Then the voluntary control, which is activated more slowly, can in2uence the motor response, and gradually supplant the pure sensorimotor system. This in2uence may be supported by the ventral stream, probably through indirect projections made to the motor cortex via the prefrontal and premotor cortex. Different results can be reached if we consider a broader time-scale, which allows for slow adaptative processes to take place through sensorimotor plasticity. Both visuomotor adaptation in neglect patients (Rossetti et al. 1998b, 1999b) and ageing (see Bhalla and Prof1tt 2000) have been shown to affect the elaboration of explicit cognitive representation of spatial information.

109


110


Stimulus

Single action

Plasticity

Online only ... Voluntary control ... Adaptation

Implicit only Dorsal

Explicit Ventral?

Input Implicit

Time

Explicit

Cerebellum?

Fig. 4.23 Time constraints and implicit–explicit processing. Tentative synthesis of the temporal constraints on the relationship between sensorimotor and cognitive processing. As reviewed in the present chapter, sensorimotor processing can be performed faster than cognitive processing, which is particularly useful for on-line motor control. The plausible neural substrate for this particular speed seems to be the dorsal pathway. Slower action can be supervised by voluntary control, which implies a transfer from the explicit to the implicit system. For example, the verbalization of the movement goal location during the action induces a loss of the implicit residual abilities observed in blindsight and numbsense patients and a change in the reference frame used to control the action. Interestingly, these effects are similar to the effect of a delay between the stimulus delivery and the action. These interactions might involve the posterior or the anterior portion of the ventral pathway. To evoke a transfer from the implicit to the explicit mode of sensory processing, one has to study a more extended time-scale. This time-scale allows for plasticity to gradually develop and thus implies that several movements are involved. Prism adaptation provides an example of such transfer from an implicit sensorimotor level to an explicit space representation. (From Pisella and Rossetti 2000.)

4.8.2 An action–perception gradient Although the dissociation between perception and action found in neurological patients is often used to argue for a clear segregation between two visual systems, it should be noticed that the performance exhibited by these patients cannot be taken as identical to normal performance. The motor production obtained in visual agnosia, and especially in blindsight and numbsense patients, is much more variable than that of normals. In our view, this suggests that both visual systems have to cooperate in order to produce normally accurate behaviour. On the one hand, neuropsychological investigations have shown that matching responses (such as indicating an object size or orientation with the hand) provide similar results to purely perceptual tasks (e.g. Goodale and Milner 1992; Goodale et al. 1991; Jeannerod and Rossetti 1993; Milner and Goodale 1995; Milner et al. 1998; Perenin and Rossetti 1996). On the other hand, an elegant series of experiments investigating slope perception in normals (see Bhalla and Prof1tt 2000) have shown a dissociation between ann (action-like) matching response and a perceptual–verbal report. The distinction between instrumental and communicative pointing described by Bridgeman (this volume, Chapter 5) also emphasizes such a progression from pure sensorimotor responses to verbal reports, via intermediate processes such as pointing to designate. Taken as a whole these results therefore



suggest that motor matching responses may lie somewhere in between the action and perception sides of vision, thus offering a third term to the dichotomy that is usually defended (Rossetti, Pisella, and Pélisson 2000). If one accepts the idea of a continuum between a pure sensorimotor system and a pure perceptual system, then the results summarized in this review suggest that the main parameter of this gradient would be the amount of on-line processing participating in a given task. Thus the sensorimotor end of this continuum would be pure on-line motor control. The only way to observe pure sensorimotor activation appears to be to favour the hand automatic pilot. A pure automatic pilot produces a greater output variability than a natural behaviour, because an unconstrained movement is usually slow enough to also allow some participation of the cognitive system in the action. In addition, this automatic pilot is not capable of planning action. By contrast, a pure ‘perceptual control of action’ (Milner and Dijkerman 2001) is only seen for very slow actions or in patients with damage to the dorsal system. This indicates that both routes to action have to cooperate under natural circumstances, even if each of them in turn can dominate a given aspect of behaviour. For example, a movement can be initiated toward a target that has been selected by the cognitive system and then be corrected on-line by a sensorimotor system such as the parietal automatic pilot. The example of the Ponzo illusion shows that the force applied to lift an object at the end of the movement is not controlled by the same mechanism as the grip aperture (Brenner and Smeets 1996). Nevertheless, both systems participate in the action, grip force probably simply being speci1ed before the action is initiated while grip aperture is controlled on-line. As a whole, performing an action cannot be considered as only a sensorimotor process and, once again, the two ends of the perception–action gradient do participate in daily behaviour.

4.8.3 A dorsal–ventral gradient? Within the purest ‘vision for action’ system there seem to be an anatomical gradient between the area responsible for space-based action and those responsible for object-based action. Electrophysiological data in monkeys (Sakata and Taira 1994) and lesion studies in human subjects (Binkofski et al. 1998) seem to agree that, within the intraparietal cortex, the more rostral part is primarily involved in simple actions such as pointing, whereas the more ventral part is involved in grasping. This anatomical organization seems to constitute the 1rst step of a more global gradient between two extremes whose prototypes would be visual pointing (simple space-based action) and object recognition (object-based cognitive processing) (see Pisella and Rossetti 2000; Rossetti et al. 2000). As argued above, the action end of this continuum would be the parietal automatic pilot (Pisella et al. 2000). At the neuropsychological level, the dichotomy of the visual system is best argued from the double dissociation between optic ataxia and visual agnosia (see Goodale and Milner 1992; Jeannerod and Rossetti 1993; Milner and Goodale 1995; Rossetti 1998). However, the very notion of this ‘double-dissociation’ may be questioned. Unlike visual agnosia, it should be kept in mind that optic ataxia does not impair patients in everyday life. In order to observe a motor de1cit one has to require the patients to perform goal-directed actions in peripheral vision, that is, in non-natural conditions. This condition is well-known to increase the need for on-line control, because it allows only for a crude target position encoding prior to movement onset. The main problem encountered by these patients in central vision may be the automatic sensorimotor control performed by the PAP.

111


112


In this case, one has to consider that the double-dissociation lies between cognitive identi1cation and automatic motor control rather than between cognitive processing and action in general. Along this line, it is interesting to consider the effect of lesions lying in the brain area just between the focus of lesions producing optic ataxia and that of lesions producing visual agnosia, that is the temporo-parietal junction. The best-documented de1cit following such a lesion in a human is hemispatial neglect. On the one hand, neglect has been considered to be dissociated from optic ataxia, in that visuomotor de1cits reported for the latter are not observed for the former (Milner 1997). On the other hand, visual neglect has been shown to involve some de1cit on the action side (e.g. Mattingley et al. 1998), which implies that neglect can also be distinguished from visual agnosia. To conclude, we propose that the action-perception gradient can be matched onto a dorsal–ventral gradient. The end of this latter gradient would be represented by the PAP.

4.8.4 Implications for visual awareness One of the key distinctions described between sensorimotor and cognitive processing is the poor level of visual awareness associated with the sensorimotor processing. Whereas movement guidance can be based on a partial analysis of the target attributes, identi1cation and consciousness imply that a binding of all object properties is achieved (see Revonsuo and Rossetti 2000). However, it would be oversimplistic to identify action and implicit processing on one side and identi1cation and consciousness on the other. Not all implicit processing takes place in the posterior parietal cortex, and not all actions rely only on this structure. Unconscious control of more complex sequential actions may involve other structures, such as the basal ganglia (e.g. Rolls 1999). It has also been known for a long time that unconscious processes, either at a low level (Helmholtz) or at a high semantic level (Freud), can affect not only actions but many aspects of cognition as well, and that semantic processes are not necessarily conscious. Still, particular aspects of sensory processing for the purpose of action, and especially for movement guidance, remain fully independent from conscious perception. Therefore action can be directed to unconsciously perceived stimuli, even though conscious awareness allows for a perceptual control of action. Speci1cally, consious visual perception and intention select the goal for the action, but the realization of the action may escape from their control. In the same way as unconscious processes may participate in goal speci1cation, as is clearly shown by the Simon effect (see Hommel 2000), such processes play an even stronger role in the action control. Although dissociation has proved to be a useful tool to understand the basic functions of the brain, which have been atomised by most scienti1c approaches, the understanding of complex functions requires a more global account of how our mind works—an account that implies a synthetic rather than an analytic approach (see Rossetti and Revonsuo 2000b). We propose that the temporal dimension is one of the keys to understanding complex interconnected networks such as the visual brain. Because of the links between the perception–action debate and issues of conscious awareness and intentionality, we suggest here that the strength of temporal factors may be relevant to these issues as well. As proposed by Milner and Dijkerman (2001), it may be that the role of consciousness is primarily to delay action in order to gain behavioural ef1ciency. Further, if an animal becomes able to slow down, delay, or inhibit immediate actions, it may also become able to reach for a better (hidden or internal) goal. Restricting the use of automatic processes such as the PAP to the regulation of action can improve the execution of intentional actions, without interfering with decision processes.



Acknowledgements The authors wish to thank Yann Coello, Chris Dijkerman, Glyn Humphreys, Robert MacIntosh, David Milner, Denis Pélisson, Gilles Rode, Caroline Tilikete, Alain Vighetto, and an anonymous referee for comments and stimulating discussions on the issues presented here. This work was supported by INSERM, CNRS, Programme Cognitique (COG118), the Center for Consciousness Research (University of Arizona), and by the McDonnell Pew Foundation.

Note 1. . . . Contrasting with naïve conceptions of perception as a pure bottom-up process, the idea has been proposed by von Helmholtz that perception results from unconscious inductive inferences. Although physiological studies of the visual system have long been focused on how visual images are constructed through hierarchically organized stages of processing, the same idea of a dialogue between bottom-up and top-down processes is now being applied to the understanding of vision. This two-way description of vision and perception in general is also widely acknowledged by psychologists and philosophers, so much so that the idea that ‘there is no such thing as immaculate perception’ has been defended (Kosslyn and Sussman, 1995). The most cited experimental evidence for the implication of descending in2uences on perception is the case of ambiguous 1gures, for which perception can alternate between two possible interpretations of the visual input, even through the memorized image can be subjected to other interpretation. Visual illusions are also often considered as a clear example of the interpretation (and contamination) of retinal information involved in perception. (Rossetti 1999, p. 141.)

References Aglioti, S., DeSouza, J.F.X., and Goodale, M.A. (1995). Size-contrast illusions deceive the eye but not the hand. Current Biology, 5(6), 679–685. Bar, M. (2000). Conscious and nonconscious processing of visual identity. In Y. Rossetti and A. Revonsuo (Eds.), Beyond dissociation: Interaction between dissociated implicit and explicit processing, pp. 153–174. Amsterdam: Benjamins. Bhalla, M. and Prof1tt, D.R. (2000). Geographical slant perception: Dissociation and coordination between explicit awareness and visually guided actions. In Y. Rossetti and A. Revonsuo (Eds.), Beyond dissociation: Interaction between dissociated implicit and explicit processing, pp. 99–128. Amsterdam: Benjamins. Binkofski, F., Dohle, C., Posse, S., Stephan, K.M., Hefter, H., Seitz, R.J., and Freund, H.-J. (1998). Human anterior intraparietal area subserves prehension: A combined lesion and functional MRI activation study. Neurology, 50, 1253–1259. Brenner, E. and Smeets, J.B.J. (1996). Size illusion in2uences how we lift but not how we grasp an object. Experimental Brain Research, 111, 473–476. Bridgeman, B. (1991). Complementary cognitive and visuomotor image processing. In G. Obrecht and L.W. Stark (Eds.), Presbyopia Research: From molecular biology to visual adaptation, pp. 189–198. New York: Plenum Press. Bridgeman, B. (1992). Conscious vs unconscious processes: The case of vision. Theory Psychology, 2(1), 73–88. Bridgeman, B. (2000). Interactions between vision for perception and vision for behavior. In Y. Rossetti and A. Revonsuo (Eds.), Beyond dissociation: Interaction between dissociated implicit and explicit processing, pp. 17–39. Amsterdam: Benjamins.

113


114


Bridgeman, B., Hendry, D., and Stark, L. (1975). Failure to detect displacement of the visual world during saccadic eye movements. Vision Research, 15, 719–722. Bridgeman, B., Lewis, S., Heit, F., and Nagle, M. (1979). Relation between cognitive and motor-oriented systems of visual perception. Journal of Experimental Psychology: Human Attention and Performance 5, 692–700. Bridgeman, B., Kirch, M., and Sperling, A. (1981). Segregation of cognitive and motor aspects of visual function using induced motion. Perception and Psychophysics, 29, 336–342. Bridgeman, B., Peery, S., and Anand, S. (1997). Interaction of cognitive and sensorimotor maps of visual space. Perception and Psychophysics, 59(3), 456–469. Bullier, J., Girard, P., and Salin, P.-A. (1994). The role of area 17 in the transfer of information to extrastriate visual cortex. In A. Peters and K.S. Rockland S. (Eds.), Cerebral cortex 10, pp. 301–330. New York: Plenum Press. Bullier, J., Schall, J.D., and Morel, A. (1996). Functional streams in occipito-frontal connections in the monkey. Behavioural Brain Research, 76, 89–97. Cajal, S.R. (1909). Histologie du système nerveux de l’homme et des vertébrés. Paris: Maloine. Campbell, F.G. and Wurtz, R.H. (1978). Saccadic ommission: Why we do not see a grey-out during a saccadic eye-movement. Vision Research, 18, 1297–1303. Carey, D.P., Dijkerman, H.C., and Milner, A.D. (1998). Perception and action in depth. Consciousness and Cognition, 7 (3), 438–453. Castiello, U. and Jeannerod, M. (1991). Measuring time to awareness. Neuroreport, 2, 797–800. Castiello, U., Paulignan, Y., and Jeannerod, M. (1991). Temporal dissociation of motor responses and subjective awareness. A study in normal subjects. Brain, 114, 2639–2655. Coello, Y. and Magne, P. (2000). Determination of target position in a structured environment: Selection of information for action. Eur. J. Cog. Psychol., 12, 489–519. Coello, Y., Magne, P., and Plenacoste, P. (2000). The contribution of retinal signal to the speci1cation of target distance in a visuo-manual task. Current Psychology Letters, 3, 75–89. Coello, Y., Richaud, S., Magne, P., and Rossetti, Y. (2001) Vision for location discrimination and vision for action: Anisotropy in the induced Roelofs effect. Unpublished manuscript. Colby, C.L., Gattas, R., Olson, C.R., and Gross, C.G. (1988). Topographic organisation of cortical afferents to extrastriate visual area PO in the macaaque: A dual tracer study. Journal of Comparative Neurology, 269, 392–413. Colent, C., Pisella, L., Bernieri, C., Rode, G., and Rossetti, Y. (2000). Cognitive bias induced by visuomotor adaptation to prisms: A simulation of unilateral neglect in normals? Neuroreport, 11, 9, 26, 1899–1902. Day, B.L. and Lyon I.N. (2000).Voluntary modi1cation of automatic arm movements evoked by motion of a visual target. Experimental Brain Research, 130(2), 159–168. Dehaene, S., Naccache, L., Le Clec, H. G., Koechlin, E., Mueller, M., Dehaene-Lambertz, G., van de Moortele, P.-F., and Le Bihan, D. (1998). Imaging unconscious priming. Nature, 395, 597–600. Desmurget, M. and Prablanc, C. (1997). Postural control of three-dimensional prehension movements. Journal of Neurophysiology, 77, 452–464. Desmurget, M., Prablanc, C., Rossetti, Y., Arzi, M., Paulignan, Y., Urquizar, C., and Mignot, J.C. (1995). Postural and synergic control for three-dimensional movements of grasping. Journal of Neurophysiology, 74, 905–910. Desmurget, M., Prablanc, C., Arzi, M., Rossetti, Y., Paulignan, Y., and Urquizar, C. (1996). Integrated control of hand transport and orientation during prehension movements. Experimental Brain Research, 110, 265–278. Desmurget, M., Epstein, C.M., Turner, R.S., Prablanc, C., Alexander, G.E., and Grafton, S.T. (1999). Role of the posterior parietal cortex in updating reaching movements to a visual target. Nature Neuroscience, 2(6), 563–567. Dijkerman, H.C. and Milner, A.D. (1998). The perception and prehension of objects oriented in the depth plane. II. Dissociated orientation functions in normal subjects. Experimental Brain Research, 118, 408–414. Dijkerman, H.C., Milner, A.D., and Carey, D.P. (1996). The perception and prehension of objects oriented in the depth plane. I. Effect of visual form agnosia. Experimental Brain Research, 112, 442–451. Dijkerman, H.C., Milner, A.D., and Carey, D.P. (1997). Impaired delayed and anti-saccades in a visual form agnosic. Experimental Brain Research, Sup. 117, 566 (abstract). Driver, J. and Mattingley, J.B. (1998). Parietal neglect and visual awareness. Nature Neuroscience 1(1), 17–22. Eimer, M. and Schlaghecken, F. (1998). Effects of masked stimuli on motor activation: Behavioral and electrophysiological evidence. Journal of Experimental Psychology: Human Perception and Performance, 24(6), 1737–1747. Ellis, R.R., Flanagan, J.R., and Lederman, S.J. (1999). The in2uence of visual illusions on grasp position. Experimental Brain Research, 125(2), 109–114.



Faugier-Grimaud, S., Frenois, C., and Stein, D.G. (1978). Effects of posterior parietal lesions on visually guided movements in monkeys. Neuropsychologia, 16, 151–168. Faugier-Grimaud, S., Frenois, C., and Peronnet, F. (1985). Effects of posterior parietal lesions on visually guided movements in monkeys. Experimental Brain Research, 59, 125–128. Fitts, P.M. (1954). The information capacity of the human motor system in controlling the amplitude of movement. Journal of Experimental Psychology, 47, 381–391. Flanagan, J.R. and Beltzner, M.A. (2000). Independence of perceptual and sensorimotor predictions in the size– weight illusion. Nature Neuroscience, 3(7), 737–741. Franz, V.H., Gegenfurtner, K.R., Bülthoff, H.H., and Fahle, M. (2000). Grasping visual illusions: No evidence for a dissociation between perception and action. Psychological Science, 11, 20–25. Gentilucci, M., Chief1, S., Daprati, E., Saetti, M.C., and Toni, I. (1996). Visual illusion and action. Neuropsychologia, 34(6), 369–376. Girard, P. (1995). Anatomic and physiologic basis of residual vision after damage to the primary visual area. Rev. Neurol., (Paris), 151:8–9, 457–465. Goodale, M.A. (1983). Neural mechanisms of visual orientation in rodents: Target vs places. In A. Hein and M. Jeannerod (Eds.), Spatially oriented behavior, pp. 35–62. New York: Springer-Verlag. Goodale, M.A. and Haffenden, A. (1998). Frames of reference for perception and action in the human visual system. Neuroscience and Biobehavioral Reviews, 22, 161–172. Goodale, M.A. and Milner, A.D. (1992). Separate visual pathways for perception and action. Trends in Neurosciences, 15(1), 20–25. Goodale, M.A. and Servos, P. (1992). Now you see it, now you don’t: How delaying an action can transform a theory. Behavioral and Brain Sciences, 15(2), 335–336. Goodale, M.A., Pélisson, D., and Prablanc, C. (1986). Large adjustments in visually guided reaching do not depend on vision of the hand or perception of target displacement. Nature, 320(6064), 748–750. Goodale, M.A., Milner, A.D., Jakobson, L.S., and Carey, D.P. (1991). A neurological dissociation between perceiving objects and grasping them. Nature, 349, 154–156. Goodale, M.A., Jakobson, L.S., and Keillor, J.M. (1994). Differences in the visual control of pantomimed and natural grasping movements. Neuropsychologia, 32, 1159–1178. Gréa, H., Desmurget, M., and Prablanc, M. (2000). Postural invariance in three-dimensional reaching and grasping movements. Experimental Brain Research, 134, 2, 156–162. Gréa, H., Pisella, L., Rossetti, Y., Prablanc, C., Desmurget, M., Tilikete, C., Grafton, S., Vighetto, A. (2002). A lesion of the posterior parietal cortex disrupts on-line adjustments during aiming movements. Neuropsychologia, in press. Haffenden, A. and Goodale, M.A. (1998). The effect of pictorial illusion on prehension and perception. Journal of Cognitive Neurosciences, 10, 122–136. Haffenden, A. and Goodale, M.A. (2000a). Independent effects of pictorial displays on perception and action. Vision Research, 40(10–12), 1597–1607. Haffenden, A. and Goodale, M.A. (2000b). Perceptual associations and visuomotor programming. Journal of Cognitive Neuroscience, 12, 6, 950–964. Hartje, W. and Ettlinger, G. (1973). Reaching in light and dark after unilateral posterior parietal ablations in the monkey. Cortex, 9, 346–354. Heywood, C.A., Shields, C., and Cowey, A. (1988). The involvement of the temporal lobes in colour discrimination. Experimental Brain Research, 71(2), 437–441. Hommel, B. (2000). Intentional control of automatic stimulus-response translation. In Y. Rossetti and A. Revonsuo (Eds.), Beyond dissociation: Interaction between dissociated implicit and explicit processing, pp. 221–242. Amsterdam: Benjamins. Humphreys, N.K. and Weiskrantz, L. (1967). Vision in monkey after removal of striate cortex. Nature, 215, 595–597. Humphreys et al. (2000). A P18, in press. Hyvärinen, J. and Poranen, A. (1974). Function of the parietal associative area 7 as revealed from cellular discharges in the alert monkey. Brain, 97, 673–692. Jackson, S.R. (2000). Perception, awareness and action: Insights from blindsight. In Y. Rossetti and A. Revonsuo (Eds.), Beyond dissociation: Interaction between dissociated implicit and explicit processing, pp. 73–98. Amsterdam: Benjamins. Jackson, S.R. and Shaw, A. (2000). The Ponzo illusion affects grip-force but not grip-aperture scaling during prehension movements. Journal of Experimental Psychology: Human Perception and Performance, 26(1), 418–423.

115


116


Jakobson, L.S., Archibald, Y.M., Carey, D.P., and Goodale, M.A. (1991). A kinematic analysis of reaching and grasping movements in a patient recovering from optic ataxia. Neuropsychologia, 29, 803–809. Jeannerod, M. (1986). The formation of 1nger grip during prehension. A cortically mediated visuomotor pattern. Behavioural Brain Research, 19, 99–116. Jeannerod, M. (1988). The neural and behavioural organization of goal-directed movements. Oxford: Oxford University Press. Jeannerod, M. (1994). The representing brain: Neural correlates of motor intention and imagery. Behavioral and Brain Sciences, 17(2), 187–245. Jeannerod, M. and Rossetti, Y. (1993). Visuomotor coordination as a dissociable function: Experimental and clinical evidence. In C. Kennard (Ed.), Visual perceptual defects, Baillière’s clinical neurology, international practise and research, pp. 439–460. Baillière Tindall/Saunders. Jeannerod, M., Decety, J., and Michel, F. (1994). Impairment of grasping movements following a bilateral posterior parietal lesion. Neuropsychologia, 32(4), 369–380. Jeannerod, M., Arbib, M.A., Rizzolatti, G., and Sakata, H. (1995). Grasping objects: The cortical mechanisms of visuomotor transformation. Trends in Neurosciences, 18(7), 314–320. Jeeves, M.A. and Silver, P.H. (1988). The formation of 1nger grip during prehension in an acallosal patient. Neuropsychologia, 26, 153–159. Komilis, E., Pélisson, D., and Prablanc, D. (1993). Error processing in pointing at randomly feedback-induced double-step stimuli. Journal of Motor Behavior, 25(4), 299–308. Kosslyn, S.M. and Sussman, A.L. (1995). Roles of imagery in perception: Or, there is no such thing as immaculate perception? In M.S. Gazzaniga (Ed.), The cognitive neurosciences, pp. 1035–1042. Cambridge, MA: MIT Press. Kovacs, I. (2000). Human development of perceptual organization. Vision Research, 40(10–12), 1301–1310. Krappmann, P. (1998). Accuracy of visually directed and memory-guided antisaccades in man. Vision Research, 38, 2979–2985. Marteniuk, R.G., MacKenzie, C.L., Jeannerod, M., Athenes, S., and Dugas, C. (1987). Constraints on human arm movement trajectories. Canadian Journal of Psychology, 41(3), 365–378. Matin, E., Clymer, A.B., and Matin, L. (1972). Metacontrast and saccadic suppression. Science, 178, 179–182. Mattingley, J.B., Husain, M., Rorden, C., Kennard, C., and Driver, J. (1998). Motor role of human inferior parietal lobe revealed in unilateral neglect patients. Nature, 392, 179–182. Milner, A.D. (1998). Streams and consciousness: Visual awareness and the brain. Trends in Cognitive Sciences, 2(1), 25–30. Milner, A.D. and Dijkerman, C. (2001). Direct and indirect visual routes to action. In B. de Gelder, E.H.F. de Haan, and C.A. Heywood (Eds.), Varieties of unconscious processing: new 1ndings and new comparisons. Oxford University Press, in press. Milner, A.D. and Goodale, M.A. (1995). The visual brain in action (Oxford Psychology Series 27). Oxford: Oxford University Press, 248 pp. Milner, A.D., Harvey, M., and Pritchard, C.L. (1998). Visual size processing in spatial neglect. Experimental Brain Research, 123, 192–200. Milner, A.D., Paulignan, Y., Dijkerman, H.C., Michel, F., and Jeannerod, M. (1999a). A paradoxical improvement of misreaching in optic ataxia: New evidence for two separate neural systems for visual localization. Proceedings of the Royal Society of London B, 266, 2225–2229. Milner, A.D., Dijkerman, H.C., and Carey, D.P. (1999b). Visuospatial processing in a pure case of visual-form agnosia. In N. Burgess, K. Jeffery, and J. O’Keefe (Eds.), The hippocampal and parietal foundations of spatial cognition, pp. 443–466. Oxford: Oxford University Press. Mohler, C.W. and Wurtz, R.H. (1977). Role of striate cortex and superior colliculus in visual guidance of saccadic eye movements in monkeys. Journal of Neurophysiology, 40, 74–94. Mon-Williams, M. and Bull, R. (2000). The Judd illusion: Evidence for two visual streams or two experimental conditions? Experimental Brain Research, 130(2), 273–276. Morel, A. and Bullier, J. (1990). Anatomical segregation of two cortical visual pathways in the macaque monkey. Visual Neuroscience, 4, 555–578. Mountcastle, V.B., Lynch, J.C., Georgopoulos, A., Sakata, H., and Acuna, C. (1975). Posterior parietal association cortex of the monkey: Command functions for operations within the extrapersonal space. Journal of Neurophysiology, 38, 871–908. Neumann, O. and Klotz, W. (1994). Motor responses to nonreportable, masked stimuli: Where is the limit of direct parameter speci1cation? In C. Umiltà and M. Moscovitch (Eds.), Attention and performance XV: Conscious and nonconscious information processing, pp. 123–150. Cambridge, MA: MIT Press.



Nowak, L. and Bullier, J. (1997). The timing of information transfer in the visual system. In J. Kaas, K. Rochland, and A. Peters (Eds.), Extrastriate cortex in primates, pp. 205–241. New York: Plenum Press. Otto-de Haart, E.G., Carey, D.P., and Milne, A.B. (1999). More thoughts on perceiving and grasping the Müller–Lyer illusion. Neuropsychologia, 37(13), 1437–1444. Paillard, J. (1987). Cognitive versus sensorimotor encoding of spatial information. In P. Ellen and Thinus-Blanc (Eds.), Cognitive processes and spatial orientation in animal and man, pp. 43–77. Dordrecht: Nijhoff. Paillard, J. (1991). Motor and representational framing of space. In J. Paillard (Ed.), Brain and space, pp. 163–181. Oxford: Oxford University Press. Paillard, J., Michel, F., and Stelmach, G. (1983). Localization without content. A tactile analogue of ‘Blindsight’. Archives of Neurology, 40, 548–551. Paulignan, Y., MacKenzie, C.L., Marteniuk, R.G., and Jeannerod, M. (1991). Selective perturbation of visual input during prehension movements. 1. The effect of changing object position. Experimental Brain Research, 83, 502–512. Pavani, F., Boscagli, I., Benvenuti, F., Rabuffetti, M., and Farne, A. (1999). Are perception and action affected differently by the Titchener circles illusion? Experimental Brain Research, 127(1), 95–101. Pélisson, D., Prablanc, C., Goodale, M.A., and Jeannerod, M. (1986). Visual control of reaching movements without vision of the limb. II. Evidence of fast unconscious processes correcting the trajectory of the hand to the 1nal position of a double-step stimulus. Experimental Brain Research, 62, 303–311. Perenin, M.-T. (1997). Optic ataxia and unilateral neglect: Clinical evidence for dissociable spatial functions in posterior parietal cortex. In P. Thier and H.O. Karnath (Eds.), Parietal lobe contribution to orientation in 3D space, pp. 289–308. Berlin, Heidelberg, New York: Springer Verlag. Perenin, M.-T. and Jeannerod, M. (1978). Visual function within the hemianopic 1eld following early cerebral hemidecortication in man. I. Spatial localisation. Neuropsychologia, 16, 1–13. Perenin, M.-T. and Rossetti, Y. (1996). Grasping in an hemianopic 1eld. Another instance of dissociation between perception and action. Neuroreport, 7(3), 793–797. Perenin, M.-T. and Vighetto, A. (1988). Optic ataxia:6ain, 111(3), 643–674. Pisella L. and Rossetti Y. (2000). Interaction between conscious identi1cation and non-conscious sensorimotor processing: temporal constraints. In Y. Rossetti and A. Revonsuo (Eds.), Beyond dissociation: Interaction between dissociated implicit and explicit processing, pp. 129–151. Amsterdam: Benjamins. Pisella, L., Attali, M., Frange, H., Régnier, C., Gaunet, F., and Rossetti, Y. (1996). Représentations en action: une même représentation spatiale pour la mémorisation et la verbalisation? In Actes du 6e colloque de l’Association pour la Recherche Cognitive, pp. 37–43. Villeneuve d’Ascq. Pisella, L., Arzi, M. and Rossetti, Y. (1998a). The timing of color and location processing in the motor context. Experimental Brain Research, 121, 270–276. Pisella, L., Rossetti, Y., and Arzi, M. (1998b). Dorsal vs. ventral parameters of fast pointing: effects of stimulus attribute and of response type. European Journal of Neuroscience, 10(sup.10), 192 (Abstract). Pisella, L., Tilikete, C., Rode, G., Boisson, D., Vighetto, A., and Rossetti, Y. (1999). Automatic corrections prevail in spite of an instructed stopping response. In M. Grealy and J.A. Thomson (Eds.), Studies in perception and action, pp. 275–279. Mahwah, NJ: Erlbaum. Pisella, L., Gréa, H., Tilikete, C., Vighetto, A., Desmurget, M., Rode, G., Boisson, D., and Rossetti, Y. (2000). An automatic pilot for the hand in the human posterior parietal cortext: Toward a reinterpretation of optic ataxia. Nature Neuroscience, 3(7), 729–736. Place, U.T. (2000). Consciousness and the zombie within: A functional analysis of the blindsight evidence. In Y. Rossetti and A. Revonsuo (Eds.), Beyond dissociation: Interaction between dissociated implicit and explicit processing, pp. 295–330. Amsterdam: Benjamins. Polyak, S. (1957). The vertebrate visual system. Chicago: University of Chicago Press. Pöppel, E., Held, R., and Frost, D. (1973). Residual visual function after brain wounds involving the central visual pathways in man. Nature, 243, 295–296. Prablanc, C. and Martin, O. (1992). Automatic control during hand reaching at undetected two-dimensional target displacements. Journal of Neurophysiology, 67, 455–469. Price, M.C. (2001). Now you see it, now you don’t. Preventing consciousness with visual masking. In P.G. Grossenbacher (Ed.), Finding consciousness in the brain: A neurocognitive approach (Advances in Consciousness Research, Vol. 8), pp. 25–60 . Amsterdam: John Benjamins, pp. 25–60. Price, M.C. et al. (2001). Manuscript submitted for publication. Ptito, A., Lepore, F., Ptito, M., and Lassonde, M. (1991). Target detection and movement discrimination in the blind 1eld of hemispherectomized patients. Brain, 114, 497–512.

117


118


Revonsuo, A. and Rossetti, Y. (2000). Dissociation and interaction: Windows to the hidden mechanisms of consciousness. In Y. Rossetti and A. Revonsuo (Eds.), Beyond dissociation: Interaction between dissociated implicit and explicit processing, pp. 351–366. Amsterdam: Benjamins. Riddoch, M.J., Humphreys, G.W., Edwards, M.G. (2000). Visual affordance and object selection. In: J. Driver and S. Monsell (Eds.), Control of cognitive processes (Attention and Performance XVIII). Rode, G., Rossetti, Y., Li, L., and Boisson, D. (1999). The effect of prism adaptation on neglect for visual imagery. Behavioural Neurology, 11, 251–258. Rolls, E.T. (1999). A theory of consciousness and its application to understanding emotion and pleasure. In: The brain and emotion, pp. 244–265. Oxford: Oxford University Press. Rossetti, Y. (1998). Implicit short-lived motor representation of space in brain-damaged and healthy subjects. Consciousness and Cognition, 7, 520–558. Rossetti, Y. (1999). In search of immaculate perception: Evidence from motor perception of space. In S. Hameroff, A. Kaszniak, and D. Chalmers (Eds.), Towards a science of consciousness, pp. 141–148. Cambridge, MA: MIT Press. Rossetti, Y. (2000). Implicit perception in action: Short-lived motor representations of space. In P.G. Grossenbacher (Ed.), Finding consciousness in the brain: A neurocognitive approach, pp.131–179. Amsterdam: Benjamins. Rossetti, Y. and Procyk, E. (1997). What memory is for action: The gap between percepts and concepts. Behavioral and Brain Sciences, 20(1), 34–36. Rossetti, Y. and Régnier, C. (1995). Representations in action: Pointing to a target with various representations. In B.G. Bardy, R.J. Bootsma, and Y. Guiard (Eds.), Studies in perception and action III, pp. 233–236. Mahwah, NJ: Erlbaum. Rossetti, Y. and Revonsuo, A. (Eds).(2000a). Beyond dissociation: Interaction between dissociated implicit and explicit processing, 366 pp. Amsterdam: Benjamins. Rossetti, Y. and Revonsuo, A. (2000b). Beyond dissociations: Recomposing the mind-brain after all? In Y. Rossetti and A. Revonsuo (Eds.), Beyond dissociation: Interaction between dissociated implicit and explicit processing, pp. 1–16. Amsterdam: Benjamins. Rossetti, Y., Lacquaniti, F., Carrozzo, M., and Borghese, A. (1994). Errors of pointing toward memorized visual targets indicate a change in reference frame with memory delay. Unpublished manuscript. Rossetti, Y., Rode, G., and Boisson, D. (1995). Implicit processing of somaesthetic information: A dissociation between where and how? Neuroreport, 6, 506–510. Rossetti, Y., Gaunet, F., and Thinus-Blanc, C. (1996). Early visual experience affects memorization and spatial representation of proprioceptive targets. Neuroreport, 7(6), 1219–1223. Rossetti, Y., Pisella, L., and Arzi, M. (1997). Stimulus location is processed faster than stimulus colour. Perception, 26(suppl.), 110 (abstract). Rossetti, Y., Pisella, L., Rode, G., Perenin, M.-T., Régnier, C., Arzi, M., and Boisson, D. (1998a). The fast-brain in action versus the slow-brain in identi1cation. European Brain and Behaviour Society Meeting, Cardiff, UK. Rossetti, Y., Rode, G., Pisella, L., Farné, A., Li, L., Boisson, D., and Perenin, M.-T. (1998b). Prism adaptation to a rightward optical deviation rehabilitates left hemispatial neglect. Nature, 395, 166–169. Rossetti, Y., Rode, G., Pisella, L., and Boisson, D. (1999a). Plasticité sensori-motrice et récupération fonctionnelle: Les effets thérapeutiques de l’adaptation prismatique sur la négligence spatiale unilatérale. Médecine/ Sciences, 15, 239–245. Rossetti, Y., Rode, G., Pisella, L., Farne, A., Ling, L., and Boisson, D. (1999b). Sensorimotor plasticity and cognition: Prism adaptation can affect various levels of space representation. In M. Grealy and J.A. Thomson (Eds.), Studies in perception and action, pp. 265–269. Mahwah, NJ: Erlbaum. Rossetti, Y., Pisella, L., and Pélisson, D. (2000a). Eye blindness and hand sight: Temporal aspects of visuomotor processing. Visual Cognition, 7, 785–808. Rossetti, Y., Rode, G., and Boisson, D. (2000b). Numbsense and blindsight. In B. de Gelder, E. de Haan, and C. Heywood (Eds.), Varieties of unconscious processing. Oxford: Oxford University Press, in press. Rossetti, Y., Rode, G., and Boisson, D. (2001). Numbsense: A case study and implications. In B. De Gelder, E. De Haan, and C. Heywood (Eds.), Varieties of unconscious processing. Oxford: Oxford University Press, in press. Rushworth, M.F.S., Nixon, P.D., and Passingham, R.E. (1997). Parietal cortex and movement, I. Movement selection and reaching. Experimental Brain Research, 117, 292–310. Sakata, H. and Taira, M. (1994). Parietal control of hand action. Current Opinion in Neurobiology, 4, 847–856. Sakata, H., Taira, M., Kusunoki, M., Murata, A., and Tanaka, Y. (1997). The parietal association cortex in depth perception and visual control of hand action. Trends in Neurosciences, 20(8), 350–356.



Saoud, M., Coello, Y., Dumas, P., Franck, N., d’Amato, T., Dalery, J., and Rossetti, Y. (2000). Visual pointing and speed/accuracy trade-off in schizophrenia. Cognitive Neuropsychiatry, 5(2), 123–134. Schall, J.D., Morel, A., King, D.J., and Bullier, J. (1995). Topography of visual cortex connections with frontal eye 1eld in macaque—Convergence and segregation of processing streams. Journal of Neuroscience, 15, 4464–4487. Schmolesky, M.T., Wang, Y., Hanes, D.P., Thompson, K.G., Leutgeb, S., Schall, J.D., and Leventhal, A.G. (1998). Signal timing across the macaque visual system. Journal of Neurophysiology, 79, 3272–3278. Schneider, G.E. (1969). Two visual systems. Science, 163, 895–902. Schwartz, A.B. (1994). Distributed motor processing in cerebral cortex. Current Opinion in Neurobiology, 4, 840–846. Smeets, J.B.J. and Brenner, E. (1995). Perception and action are based on the same visual information. Journal of Experimental Psychology: Human Perception and Performance, 21, 19–31. Sprague, J.M. and Meikle, T.H. (1965). The role of the superior colliculus in visually guided behavior. Experimental Neurology, 11, 115–146. Stoerig, P. and Cowey, A. (1992). Wavelength discrimination in blindsight. Brain, 115, 425–444. Taira, M., Mine, S., Georgopoulos, A.P., Murata, A., and Sakata, H. (1990). Parietal cortex neurons of the monkey related to the visual guidance of hand movements. Experimental Brain Research, 83, 29–36. Tanné, J., Boussaoud, D., Boyer-Zeller, N., and Rouiller, E. (1995). Direct visual pathways for reaching movements in the macaque monkey. Neuroreport, 7, 267–272. Taylor, T.L. and McCloskey, D. (1990). Triggering of preprogrammed movements as reactions to masked stimuli. Journal of Neurophysiology, 63, 439–446. Taylor, J.L. and McCloskey, D.I. (1996). Selection of motor responses on the basis of unperceived stimuli. Experimental Brain Research, 110, 1, 62–66. Trevarthen, C.B. (1968). Two mechanisms of vision in primates. Psychologische Forschung, 31, 299–337. Ungerleider, L.G. (1995). Functional brain imaging studies of cortical mechanisms for memory. Science, 270 (5237), 769–775. Ungerleider, L. and Desimone, R. (1986). Cortical projections of visual area MT in the macaque. Journal of Comparative Neurology, 248, 190–222. Ungerleider, L. and Mishkin, M. (1982). Two cortical visual systems. In D.J. Ingle, M.A. Goodale, and R.J.W. Mans1eld (Eds.), Analysis of motor behavior, pp. 549–586. Cambridge, MA: MIT Press. Van Hoesen, G.W. (1982). The parahippocampal gyrus: New observations regarding its cortical connections in the monkey. Trends in Neurosciences, 345–350. Vindras, P., Desmurget, M., Prablanc, C., and Viviani, P. (1998). Pointing error re2ect biases in the perception of the initial hand position. Journal of Neurophysiology, 79(6), 3290–3294. Vishton, P.M., Rea, J.G., Cutting, J.E., and Nuñez, L.N. (1999). Comparing effects of the horizontal–vertical illusion on grip scaling and judgment: Relative versus absolute, not perception versus action. Journal of Experimental Psychology: Human Perception and Performance, 25(6), 1659–1672. Weiskrantz, L. (1986). Blindsight. A case study and implications. Oxford: Oxford University Press. Weiskrantz, L., Warrington, E.K., Sanders, M.D., and Marshall, J. (1974). Visual capacity in hemianopic 1eld following a restricted occipital ablation. Brain, 97, 709–728. Westwood, D.A., Chapman, C.D., and Roy, E.A. (2000). Pantomimed actions may be controlled by the ventral visual stream. Experimental Brain Research, 130(4), 545–548. White, J.M., Sparks, D.L., and Stanford, T.R. (1994). Saccades to remembered target locations: An analysis of systematic and variable errors. Vision Research, 34(1), 79–92. Wong, E. and Mack, A. (1981). Saccadic programming and perceived location. Acta Psychologica, 48, 123–131.

119


5 Attention and visually guided behavior in distinct systems Bruce Bridgeman

Abstract. Recent research from several laboratories has revealed two distinct representations of visual space: a cognitive or ‘what’ system specialized for perception, and a sensorimotor or ‘how’ system specialized for visually guided behavior. To know how these two pathways normally operate and cooperate, they must be studied in normal humans. This has become possible with the development of psychophysical methods that isolate the two pathways and measure spatial information separately in each. The pathways are distinguished by the response measure, a symbolic response probing the cognitive system and an isomorphic motor reaction probing the sensorimotor system. The two systems represent visual space in different ways, the cognitive system relying on context even when that strategy leads to errors of localization, while the sensorimotor system possesses a quantitative calibration of visual space that is insensitive to context. Only the contents of the cognitive system are accessible to awareness, operationally de1ned as the ability to describe visual information. When con2icts arise between cognitive and sensorimotor information, it is the cognitive information that is available for making judgments and decisions. In this context, only the cognitive system can direct attention to particular objects or regions in the visual 1eld, and only that system can initiate behaviors based on current goals. The sensorimotor system has calibrated egocentrically based spatial information, but cannot initiate actions in the service of behavioral goals. Attention serves as a pathway for the cognitive system to motivate actions, which are then carried out under the guidance of sensorimotor information.

5.1 Introduction—cognitive and sensorimotor visual systems It seems obvious that to interact effectively with an object, we must perceive its location and properties accurately. We have the impression that vision is a uni1ed sense, with all of its richness and variety tied to a single, coherent whole. Perceived positions of objects and surfaces, color, motion, and control of action are smoothly integrated. This intuition is deceptive, however: several lines of evidence now have demonstrated that humans can achieve accurate motor behavior despite experiencing inadequate or erroneous perceptual information from the same environment at the same time. Under some conditions, perception is not required to visually guide an action. The following reviews the accumulation of several decades of empirical work to test this idea. The review is designed to complement the review by Rossetti and Pisella (this volume, Chapter 4), and therefore some important lines of research are given less attention here. The earliest experiments on separation of cognitive and sensorimotor systems were done in animals and in human neurological patients. Hamsters with a lesioned superior colliculus could perform a simple pattern recognition task, but could not run a maze. Other hamsters, with visual cortex lesions, could run the maze but could not do the pattern recognition task (Schneider 1967).


Attention and visually guided behavior in distinct systems

Monkeys with lesions of the striate visual cortex could not perform pattern recognitions in their scotomic 1elds, but could perform many visually guided behaviors (Trevarthen 1968). The result implied that in primates, as in hamsters, pattern recognition and visually guided behavior could be affected separately by selective lesions. Subsequent work extended some of these observations to humans. Although human patients with scotomata from damage to the visual cortex fail to report the presence of objects in their ‘blind’ areas, they are able to point at or direct their eyes to these objects with little error (Pöppel, Held, and Frost 1973; Sanders, Warrington, Marshall, and Weiskrantz 1974; Weiskrantz 1996). Weiskrantz has termed the phenomenon ‘blindsight’ because the patients were perceptually blind in the affected 1eld but retained some ability to guide actions by sight, and even to pick up some other kinds of limited visual information without awareness. Ungerleider and Mishkin (1982) modi1ed Schneider’s dichotomy in the context of monkey neurophysiology into cortical what versus where systems, assigning the what to the inferotemporal cortex and the where to the posterior parietal cortex. Bridgeman (1991) revised the dichotomy again, noting that meaningful what questions can be asked of both pathways—one merely receives different answers from them under some circumstances. Paillard (1987) described a similar distinction, introducing the terms cognitive and sensorimotor, which are used here and in Rossetti and Pisella (this volume, Chapter 4). According to these views, visually guided behavior such as grasping or reaching is handled by a sensorimotor pathway that takes information from early vision and processes it in a pathway separate from the one that underlies the rich spatial sense of perception. Perception is de1ned here as sensory information that is actually experienced or, more operationally, information that can be described and remembered by the observer. According to this de1nition, if a visual stimulus is masked in such a way that an observer denies seeing it, the stimulus is not considered to be perceived, even if it can affect later perceptual judgments or actions. Broader de1nitions, asserting that any information input to an organism that can affect behavior represents perception, lead to the conclusion that perception also occurs in insects, one-celled protozoa, and even thermostats. At this point the de1nition would become so broad as to add nothing to the discussion of human neurological organization. Further, the experiments described below show that under some conditions normal humans can simultaneously hold two contradictory spatial values for the same stimulus, one perceived and the other not perceived, without becoming aware of the con2ict and without resolving it. According to Milner and Goodale (1995), a ventral channel mediates perception (what), while a dorsal channel subserves visually guided behavior (how). This dual arrangement allows spatially directed behavior to be rapid and ef1cient, implemented by a dedicated processor operating solely on the here-and-now goal of action. The cognitive pathway, in contrast, specializes in recognizing and remembering the identities of objects and patterns and their spatial interrelationships, based on comparisons with prior knowledge. The dorsal/ventral or parietal/temporal summary of the neuroanatomy of the two pathways is oversimpli1ed, however, since some cortical regions that should be identi1ed with the ‘ventral’ system are anatomically dorsal to the striate cortex, though the pathways remain anatomically distinct. The language is still useful to discuss the two systems, with the provisos that the anatomy is more complex than the terms imply and that there are information links between the systems at several levels (Rossetti and Pisella, this volume, Chapter 4). Since the sensorimotor and cognitive pathways normally lead to motor actions and perceptual experiences that are consistent with one another, tests of dissociability require conditions that disturb

121


122


this congruence as a result of either experimental intervention in normal subjects or certain types of brain injury in neurological patients. For example, Goodale, Milner, Jakobson, and Carey (1991) described a patient who was unable to identify the orientation of a slot perceptually, but could correctly place objects in it. Milner and Goodale (1995) review many other instances of behavior in the absence of perception, as well as perception in the absence of a behavioral capability (i.e. double dissociation). For example, some patients show visual apraxia, an inability to reach for and grasp objects appropriately despite being able to identify them. This de1cit is not the result of general motor damage, since grasping that is not guided by vision remains normal. In our interpretation, information in the cognitive pathway is unavailable to control accurate grasping. Conversely, Rossetti and Pisella (this volume) report patients who can grasp objects appropriately but are unable to describe them. However, neither the possession of two anatomically disparate visual streams nor evidence of perception–action dissociation in brain-damaged patients guarantees that such a dissociation applies to the intact brain, for a system that is uni1ed in normals might become fragmented after brain damage. For example, the injured brain might erect a ‘1rewall’ to preserve at least a portion of its usual function. Thus, rigorous evidence for perception–action dissociation in normal humans can be obtained only by studying normal subjects.

5.1.1 Cognitive and sensorimotor visual systems Several methods have been used to tease apart cognitive and sensorimotor systems in normal humans. Early experiments on the separation of systems showed that normal subjects were unable to perceive jumps of targets that take place during saccadic eye movements (a cognitive-pathway function). But the subjects could still point accurately to the new locations of the same targets (a sensorimotor-pathway function), even if their pointing movements were controlled open-loop (Bridgeman, Lewis, Heit, and Nagle 1979; Goodale, Pélisson, and Prablanc 1986). In these conditions accurate information about the new location of the target was entering the nervous system, but was not available to perception. Since each pathway could be probed without affecting the representation in the other, one can conclude that the two pathways must be storing spatial information independently. Bridgeman and Stark (1979) refuted the possibility that this result was due to differing response criteria by showing that the dissociation between perception and action occurred even with a criterion-free forced-choice perceptual measure. A more rigorous way to separate cognitive and sensorimotor systems is by double dissociation, introducing a signal only into the sensorimotor pathway in one condition and only into the cognitive pathway in another (Bridgeman, Kirch, and Sperling 1981). A 1xed target was projected in front of a subject, with a frame surrounding it. When the frame was displaced left or right, subjects saw an illusion of stroboscopic induced motion—the target appeared to jump in the opposite direction. After target and frame were extinguished, the subjects pointed to the last target position. They always pointed to the same location, regardless of the direction of the stroboscopic induced motion. The illusion did not affect pointing, showing that the illusory displacement signal was present only in the cognitive system. Another condition of the same experiment inserted displacement information selectively into the sensorimotor system by nulling the cognitive signal. Each subject adjusted the real target jumps until the target appeared stationary, with a real displacement in phase with the background jump equaling the induced displacement out of phase with the background. Thus, the cognitive pathway



speci1ed a stable target. Nevertheless, subjects pointed in different directions when the target disappeared in the left position than when it disappeared in the right position, showing that the difference in real target positions was still represented in the sensorimotor pathway. This is a double dissociation because the apparent target displacement in the 1rst condition affected only the cognitive measure, while the real displacement in the second condition affected only the sensorimotor measure.

5.1.2 Experiments and ambiguities Apparent dissociations might appear if a moving stimulus is sampled at different times for different functions, even though a uni1ed visual representation underlies each function. A target evaluated at a longer latency, for example, will be sampled when it has moved further along its path. Recently, methods have been developed, using static illusions, that can test dissociations of cognitive and sensorimotor function without the possible confounding effects of motion. One such method is based on the Ebbinghaus illusion, also called the Titchener circles illusion. A circle appears to be larger if it is surrounded by smaller circles than if it is surrounded by larger circles. The Ebbinghaus illusion has been applied to cognitive/sensorimotor dissociation by making the center circle into a threedimensional poker-chip-like object and asking subjects either to judge the size of the circle or to grasp it (Aglioti, DeSouza, and Goodale 1995). The grasp was adjusted closer to the real size of the circle than to its illusory size. Subjects were able to see their hands in this experiment, however, so it is possible that subjects adjusted their grasp not to the non-illusory true size of the circle, but to the visible error between the grasp and the edge of the circle. The adjustments did not occur until just before the movement was completed, nearly 2 s after it started. In a subsequent experiment that avoids the feedback confound, Haffenden and Goodale (1998) measured the illusion either by asking subjects to indicate the apparent size of a circle or to pick it up, in both cases without vision of hand or target. Both measures used distance between thumb and fore1nger as the dependent variable, so that output mode was controlled, and only the source of the information varied. The illusion appeared for both estimations but was much smaller for the grasp, indicating that the sensorimotor system was relatively insensitive to the illusion. The interpretation of these results has been called into question by Franz, Gegenfurtner, Bülthoff, and Fahle (2000), who failed to replicate the smaller grasp-adjustment effects in the motor condition. Goodale has responded with new experiments showing that it is primarily the physical distance between the test circle and the inducing circles that affects grasp aperture, while perceived size of the test circle is affected primarily by the size contrast between the test and inducing circles. In previous experiments, inducing circle size and distance from the test circle had been confounded. A different method for contrasting grasp and perception, using the Müller–Lyer illusion, showed that while the illusion is signi1cantly smaller when measured with grasp than with perception, there is some illusion under both conditions (Daprati and Gentilucci 1997). Again, relatively slow grasp movements may be responsible, and vision of both hand and stimulus was allowed. The difference dissipated when the observers were forced to delay their behavior, indicating a short-lived sensorimotor representation consistent with other results described below. In summary, there is behavioral evidence in normal subjects for a distinction between processing in two visual pathways, but we still know very little about processing in the sensorimotor pathway. In addition, there is a contrast in the parameters examined, some methods addressing the properties of objects and others their locations. But the saccadic suppression and induced motion methods are

123


124


vulnerable to the ambiguities of sampling a moving target. If information used for perception is sampled from a uni1ed visual representation at a different time than information used for action, one could explain some differences between perceptual and motor measures without resorting to a two-visual-systems hypothesis. The illusion methods use static stimuli, but show a quantitative rather than a qualitative distinction between cognitive and sensorimotor processing, and thus are vulnerable to scaling and distortion artifacts. A new method overcomes these limitations, producing large and consistent contrasts between cognitive and sensorimotor systems, differentiated by response measure. The dissociation is based on another perceptual illusion, the Roelofs effect: if a rectangular frame is presented off-center, so that one of its edges is directly in front of the subject, that edge will appear to be offset in the direction opposite the rest of the frame. A rectangle presented on the left side of the visual 1eld, for example, with its right edge in the center, will appear less eccentric than it is, and the right edge will appear somewhat to the right of the subject’s center (Roelofs 1935). We have extended and generalized the Roelofs effect to apply it to the study of the two-visualsystems theory. First, the frame need not have one edge centered in front of the subject; illusions occur whenever the frame is presented asymmetrically in the visual 1eld. Second, if a target is presented within the offset rectangle, its location tends to be misperceived in the direction opposite the offset of the frame. Misperception of frame position induces illusions of target position; this is an induced Roelofs effect, but will be called simply a Roelofs effect here. In our experiments, the motor task is isomorphic with stimulus position. This means that there is a continuous, 1:1 relationship between target position and hand position when the subject touches the target. If the target deviates 5 deg. to the right, the hand does also, and no remapping or symbolic operation intervenes between stimulus and response. Roelofs effects can be observed reliably if subjects describe the target’s position verbally, a task that addresses the cognitive system. A jab at the target, however, made just after it disappears from view, is not affected by the frame’s position. This task addresses the sensorimotor system. Motor behavior for many subjects remains accurate despite the perceptual mislocalization (Bridgeman 1992; Bridgeman, Peery, and Anand 1997). Here a question arose because the Roelofs result showed a dissociation for only about half the subjects, while the earlier studies based on saccadic suppression and on induced motion showed dissociations for all subjects. The earlier work did not require a direct position judgment of the perceptual system, however, but only an indication of whether a target had moved or not. The perceptual task was a simple detection, not a position discrimination, and the relatively undemanding nature of the task may have enabled subjects to stay in a more direct motor mode for the motor task. Indeed, in the Roelofs experiments the subjects felt that the motor trials were less dif1cult than the cognitive trials, because no decision had to be made. Pointing was perceived to be easier, possibly because a representation inaccessible to consciousness was doing the work. Since the experiments described below follow up on earlier studies (Bridgeman et al. 1997), we were able to take advantage of the results of those studies to improve our experimental design. In the earlier studies we presented targets in 1ve different positions. With both cognitive and sensorimotor measures, though, the responses to the 1ve positions fell close to a straight line; nearly all of the variance in responses as a function of target position was accounted for by a linear regression. Thus the positions were redundant, and in the current experiments we did not need to present 1ve target positions: two target positions would give us the same information, and allow us to increase the number of trials per condition.



Though the motor task is formally isomorphic, it can also be interpreted as a communicatory act. In effect the observer might be informing the experimenter by pointing where he thinks the target is located, so that the task might be closely linked to cognitive representations. An alternative is to require an instrumental act, in which a subject must do something to the world rather than simply indicate a position to another person. Behavior with a purely instrumental goal might be different from behavior with a communicatory goal, even if both the stimuli and the motor movements themselves are identical. Thus in our next experiment subjects jabbed a three-dimensional target object, a long vertical bar, pushing it backward and making a clicking noise. Their intention was not to communicate anything, but only to hit the bar. With this improvement in our technique we achieved a cleaner separation of cognitive and motor systems. For a quick jab at a three-dimensional target, rather than a pointing motion, almost all subjects showed independence from Roelofs effects in immediate action, along with the robust Roelofs effects that we have observed previously in verbal estimation of position.

5.2 Preliminary experiment: dissociating cognitive and sensorimotor pathways Using these improved techniques, we began the job of characterizing the psychophysics of the sensorimotor system. A preliminary experiment (Bridgeman, Gemmer, Forsman, and Huemer 2000) is necessary to interpret the results of the main experiment. Because many of the methods and procedures are common to the two experiments, they will be described in detail.

5.2.1 Method Observers sat with heads stabilized before a white hemicylindrical screen that provided a homogeneous visual 1eld 180° wide × 50° high. A lever box located in front of the screen presented 5 vertical white levers. The center lever, marked with a black stripe, functioned as the target. Each lever was hinged at its base and spring-loaded. A long baf2e hid the microswitch assembly without revealing the position of the lever array. In the motor condition, the task was to jab the black target rapidly with the right fore1nger. The remaining levers served to record the locations of inaccurate responses. A jab between the locations of two levers would trip both of them, as the distance between the edges of the levers was about 7 mm, less than the width of a 1nger. A rectangular frame 38° wide was projected via a galvanic mirror under computer control, either centered on the subject’s midline, 6° left, or 6° right of center. Inside the frame, the lever array occupied one of two positions, 3.5° left of center or 3.5° right of center. On each trial the frame and target were positioned in darkness during the intertrial interval. Then a computer-controlled shutter opened for 1 s. Re2ected light from the projected frame made the screen and the levers visible as well. As soon as the shutter closed, the observer could jab the target or verbally indicate its position in complete darkness. Responses were recorded by the computer on an absolute scale (lever 1, 2, 3, 4, or 5).

5.2.1.1 Cognitive measure For the cognitive system the observer verbally estimated the position of the target spot on the center lever. The choices were ‘far left’, ‘left’, ‘center’, ‘right’, or ‘far right’, so that the response was a

125


126


5-alternative forced choice. In the present series of experiments the cognitive measure serves as a control to assure that a cognitive illusion is present, differentiating the cognitive and sensorimotor systems. Instructions in the verbal condition emphasized egocentric calibration. Quoting from the instructions that were read to each observer, ‘In this condition you will be telling the experimenter where you think the target is in relation to straight ahead.’ Further, ‘If the target looks like it’s directly in front of you, you will indicate this by saying “center”.’ Thus center was de1ned in terms of the subject’s body rather than the apparatus or the frame. Each subject received at least 20 trials of practice with no frame present, so that only egocentric information could be used in the judgment.

5.2.1.2 Sensorimotor measure The observer rested the right fore1nger on a foam pad mounted on the centerline of the apparatus just in front of the chin rest, then jabbed the target with the fore1nger as soon as the target disappeared. Thus both cognitive and sensorimotor measures were open-loop, without error feedback. Before the experimental trials began, observers practiced jabbing the target—some were reluctant to respond vigorously at 1rst for fear of damaging the apparatus. Subjects then received at least 10 practice trials in the jab condition. 5.2.1.3 Trial execution A computer program randomly selected target and frame positions, with the exception that an identical set of positions could not occur on two successive trials. In each trial one of the two target positions and one of the three frame positions was presented, exposed for 1 s, and extinguished. Since the projected frame provided all of the illumination, target and frame exposure were simultaneous. A computer-generated tone told the subject to respond. For no-delay trials the tone sounded as the shutter extinguished the frame, while on other trials the tone began after a delay. The delay trials, while intermixed with no-delay trials, were aimed at a different experimental question, and will not be considered further here. Two target positions × three frame positions × two response modes × three delays resulted in 36 trial types. Each trial type was repeated 10 times for each subject, resulting in a data base of 360 trials/subject. There was a brief rest and a chance to light adapt after each block of 60 trials. Data were collated on-line and analyzed statistically off-line. Two-way ANOVAs were run for each subject and each response mode. Factors were frame position and target position. Summary statistics were analyzed between subjects. Nine University of California undergraduates participated in the experiment, all right-handed with normal or corrected-to-normal visual acuity. Four were male and 1ve female.

5.2.2 Results 5.2.2.1 Cognitive measure The induced Roelofs effect, measured as a main effect of frame position, was signi1cant under all conditions. Observers tended to judge the target to be further to the left than its actual position when the frame was on the right, and vice versa. Six of seven individual subjects showed a signi1cant Roelofs effect, and the magnitude of the Roelofs effect averaged across subjects was 2.23° (S.E. 52 min arc).



Fig. 5.1 Jabbing at a target under three stimulus conditions. Position of the background frame does not affect behavior. Drawn from data of Bridgeman et al. (2000).

5.2.2.2 Sensorimotor measure The results can best be summarized with the generalization that subjects hardly ever missed the target, regardless of target position or frame position (Fig. 5.1). Seven of eight subjects showed no signi1cant Roelofs effect. Averaged across subjects, the magnitude of the Roelofs effect was 20 min arc (S.E. 22 min arc). 5.2.2.3 Comparing the two measures Overall, there was a signi1cant difference between cognitive and motor measures, as expected from the robustness of Roelofs effects with the cognitive measure and the absence of Roelofs effects with the motor measure. The sizes of the Roelofs effects under various conditions can be compared by measuring the difference between average response with the target on the right and with the target on the left in Fig. 5.1. The cognitive measure shows a large and consistent deviation, replicating Bridgeman et al. (1997), while the sensorimotor measure (illustrated) shows no deviation. 5.2.3 Discussion This experiment showed that the sensorimotor pathway can code veridical information about target position, unaffected by visual context, even when perception shows an illusion of position. The rules are different for the two systems. Cognition is conscious and must use context, even when that leads to errors of localization. The sensorimotor system, in contrast, is insensitive to context, and its spatial values are held unconsciously. Con2icting spatial values can exist in the two systems simultaneously. These results contrast with results obtained previously with the Roelofs method (Bridgeman et al. 1997). In the earlier experiments, using a projected target and frame, only half of the subjects pointed

127


128


to the targets in the motor measure without the in2uence of a Roelofs effect, while in the present experiments almost all of them did. The difference between the current study and the earlier one is that, with the addition of the mechanical target, subjects are more prone to execute an instrumental rather than a communicative action. Dissociation occurred here in the frontal plane, where grasp directions differed only on the x-axis. Rossetti and Pisella (this volume, Chapter 4) have found a critical limitation of this phenomenon, for grasp in the z-axis (changing distance from the observer) is affected by context. Depth cues are very different from the more retinally based x-axis information, and Rossetti and Pisella’s result may be due to a context effect in the convergence movements that help to localize targets in depth. Recordings of eye movements during a depth grasping experiment would be necessary to test this hypothesis. This experiment by itself does not prove that the normal brain possesses a true sensorimotor map of visual space, though. A possible mechanism of the sensorimotor store is that subjects might perform the motor action by 1xating the target visually when it is visible, then pointing where they are looking when the target is extinguished. This would mean a zero-dimensional storage of information of spatial information limited to the location of a single point, held in gaze position rather than in an internal neurological register. Further, since oculomotor 1xation is a good measure of spatially selective attention, 1xating the target also facilitates attention to it. If this interpretation is correct, subjects should be unable to perform the motor task if they are prevented from ever 1xating the target. In the next experiment, extending the Roelofs effect paradigm, we seek to control for possible attention and 1xation effects by preventing observers from 1xating the target. This is the motivation of the main experiment, testing whether it is necessary for spatial values in the sensorimotor system to be held in an internal neurological store.

5.3 Main experiment: gaze position and the motor pathway In a condition where subjects are not allowed to 1xate the target in a Roelofs-effect design, one can form two hypotheses. If the sensorimotor pathway normally stores target position only as a gaze angle, then it cannot use spatial information from gaze position and will be forced to call upon the cognitive pathway for spatial location information. If the pathway includes a true map of visual space, however, context-free spatial information would be available even from a target that has never been 1xated. We monitor eye movements to be sure that subjects never 1xate the target. Further, we prevent covert orienting to the target by requiring subjects to perform a continuous oculomotor task throughout the exposure period. In this way we break the normally tight relationship between 1xation and spatial allocation of attention.

5.3.1 Method For this experiment we need 1xation points that de1ne eye movements, but give the subject no information about target or frame positions. A pair of 1xation points is added to the display, in positions statistically uncorrelated with target or frame positions, to elicit horizontal saccades.

5.3.1.1 Apparatus In order to present the target, frame, and 1xation points simultaneously, and also to improve the accuracy or our motor recordings, we moved to an electronic apparatus with all stimuli displayed on



Fig. 5.2 Aiming at a target in the electronic apparatus. The display screen appears to the observer to be located at the plane of the touch pad. Contact with the pad offers 800-pixel resolution, compared to the 1ve pixels of the preliminary experiment.

a CRT screen. The screen is mounted at the top of the apparatus, with its face down, and is viewed through a front-surface mirror mounted at a 45° angle in front of the eyes, so that the display appears to be in the frontal plane directly in front of the subject. A touch pad mounted vertically in the apparent plane of the display records jab responses made with a stylus, at an 800-pixel horizontal resolution (Fig. 5.2). The frame’s width is 24° and its height is 12°. The saccade targets are 2° diameter circles, 23° apart, displayed 2.5° above the frame. Because of the smaller available stimulus aperture, frame positions are at 4° left of center, center, and 4° right of center. Target positions, are 2° left and 2° right of center. Two target positions × 3 frame positions × 3 1xation point positions yielded 18 trial types. Gaze position was monitored continuously by a Bouis photoelectric infrared eye tracker aligned to the left eye. The head was stabilized with a bite bar attached to the frame of the tracker. With this apparatus, eye position can be measured in two dimensions at a 400 Hz sampling rate in complete darkness.

5.3.1.2 Procedure Except for the change in apparatus, the procedure and design are similar to those in Experiment 1, with the following additions. Before each trial block, the eye tracker was calibrated by having subjects 1xate each corner of the frame in its centered position. At the start of each trial, subjects were instructed to look up, above the frame’s edge. When the eye monitor indicated upward gaze, the experimenter triggered the computer to display the trial, while the subject alternately 1xated the two 1xation points, alternating as quickly as possible as long as the points were visible (Fig. 5.3). Continuous saccades were required to prevent surreptitious attention shifts to the target position. Target, frame, and 1xation points appeared and disappeared simultaneously. For analysis, the mean response averaged across subjects for each target, frame, and 1xation point position was entered into a factorial ANOVA. This format trades some power for the ability to compare cognitive and sensorimotor data directly, with equal power in each measure, despite the different number of observers in each condition. Cognitive and sensorimotor conditions were analyzed separately, and then combined into a single analysis. Two target positions × three frame positions × three 1xation point positions resulted in 18 trial types. Each trial type was repeated 10 times for each subject, so that each cell in the analysis is based on 10 observations/subject × the number of subjects in the respective condition.

129


130


Fig. 5.3 Stimuli and saccadic eye movement scanpath in the gaze experiment. Eye movements alternate as many times as possible between the left and the right 1xation points during the 1-s exposure period. In this example, the saccade 1xation points are biased to the right. In other trials, they were centered or biased to the left.

Seven University of California undergraduates participated in the cognitive condition, and six in the motor condition, all right-handed with normal or corrected-to-normal visual acuity. Each subject was run in only one condition, cognitive or motor.

5.3.2 Results In general, signi1cance levels were lower in this experiment than in the preliminary experiment because of greater variability, though mean effect sizes were comparable. Preventing direct visual 1xation reduced the quality of the spatial information available.

5.3.2.1 Cognitive measure For this analysis the three 1xation point positions were considered as repetitions of each target/ frame condition. In an ANOVA with target and frame as the factors, the cognitive observers showed a signi1cant effect of target position, F(1, 12) = 30.88, p = 0.0001, and a marginally signi1cant effect of frame, F(2, 12) = 3.74, p = 0.0547. A Fisher’s PLSD test for the frame at a signi1cance level of 5% showed that the difference between position estimates at frame positions of 4° left and 4° right of center was signi1cant at p = 0.018, mean difference = 1.40°, critical difference = 1.12°. The interaction was not signi1cant at p < 0.05 (Fig. 5.4). 5.3.2.2 Sensorimotor measure The motor observers, in contrast, showed no Roelofs effect, frame F(2, 12) < 1, p = 0.96, but had a statistically signi1cant target effect, F(1, 12) = 404.78, p < 0.0001. There was no signi1cant interaction. Thus the motor behavior remained independent of frame position (Fig. 5.5). 5.3.2.3 Comparing the measures When cognitive and sensorimotor data were combined in a single ANOVA with target, frame, and measure as the factors, the measure factor was signi1cant, F(1, 24) = 5.46, p = 0.028. The only signi1cant



Fig. 5.4 Verbal estimates of target position without direct 1xation on the target. Perceived position is biased by the position of the background frame, even though the instructions did not mention the frame. Error bars +/− 1 S.E., where not shown, were smaller than the size of the corresponding symbol. interaction at p < 0.05 was frame × measure, F(2, 24) = 3.55, p = 0.045, showing that the frame effect was larger in the cognitive condition than in the sensorimotor condition.

5.3.3 Discussion The single most important 1nding is that preventing direct 1xation on the target does not cause a Roelofs effect in motor activity. Since the observers in the motor condition showed no Roelofs effect, while those in the cognitive condition did, we can conclude that the sensorimotor representation was controlling the jab for the motor observers. The result indicates that the sensorimotor representation is at least two-dimensional, a true map rather than a simple matching of gaze and jab positions. This experiment shows that oculomotor 1xation and spatially selective attention are not responsible for accurate pointing behavior in an illusory visual context.

5.4 Decision and the motor pathway At this point we are certain that the sensorimotor pathway can represent spatial information without the disturbing in2uence of a biased visual context that invariably affects perception. Further, the information used for motor control is held in a true neural map of visual space. What, then, is the function of the cognitive system? Surely it has functions beyond representing a more error-prone version of the sensorimotor map. One unique function of the cognitive system is its ability to make decisions, to determine which of several alternative behaviors to execute. This is a nonlinearity in vision, because a decision once made is an all-or-none affair, no longer subject to the subtleties of coding and perception that may have led to the decision. Since the nonlinearity of a decision seems to be a hallmark of the cognitive system, forcing observers to make decisions about which of several possible targets to hit might

131


132


Fig. 5.5 Motor responses to targets without direct 1xation on the target. Perceived position is not biased by the position of the background frame. Error bars +/− 1 S.E. bring control by the cognitive system into a sensorimotor task, and bring the Roelofs effect along with it. An experiment by Bridgeman and Huemer (1998), using the same apparatus as used in the preliminary experiment but putting targets on the second and fourth bars rather than only the third bar, tested this hypothesis. Subjects were cued at the end of the stimulus display period in each trial whether they would jab the left or the right target bar. The results showed that observers could jab either the left or the right target bar in accordance with a post-stimulus cue, without in2uence of the Roelofs effect. The hypothesis was not con1rmed—no Roelofs effect was found (Fig. 5.6). Observers were able to make a cognitive decision about which target to attend to, and then to use spatial information in the context-free sensorimotor pathway to guide their actions. The jab was not as veridical as that of the result in Fig. 5.1, and the subjects sometimes hesitated before responding, but there was no statistically signi1cant effect of the surrounding frame. The combination of decision and jab was apparently more dif1cult than a simple jab, resulting in larger errors.

5.5 Conclusions The results of these experiments can be interpreted in terms of two visual pathways. One pathway is based on egocentric coordinates to govern motor behavior, while another uses information from visual context to represent spatial relationships in perception. These experiments also lend support to the claim that the price in performance the cognitive pathway must pay in order to take advantage of visual context information is a susceptibility to illusions of spatial context. While we have shown that direct 1xation driven by attentional selection is not the mechanism responsible for accurate visually guided pointing in a context that creates illusory perceptions in the cognitive system, this result shows only that 1xation is not responsible. Other aspects of attention may be responsible for the continued accuracy of motor behavior in these experiments.



Fig. 5.6 Aiming at a target under three stimulus conditions. Motor responses to two targets are adjusted to be superimposed in the graph. There is no signi1cant Roelofs effect. Redrawn from data of Bridgeman and Huemer (1998). The visual mechanism by which motor behavior is governed has been shown to be extremely robust, both by these and previous studies. Indeed, the reappearance of a Roelofs effect for motor responses after a delay (Bridgeman et al. 1997) shows that the cognitive system can provide information to the motor system when necessary, and this so far appears to be the only form of realtime communication between the two systems. Rossetti and Pisella (this volume; Rossetti and Régnier 1995) review other extensive evidence for the ephemerality of the motor representation under delay. To date there is no evidence that the cognitive pathway has access to spatial location information in the motor pathway, except for longer-term adaptation effects. This observation supports the inference that spatial information can 2ow in only one direction, from cognitive to motor, for immediate control of behavior. In normal visual conditions (motor actions directed at still-visible targets), spatial information remains segregated in the two pathways. However, sensory in2uences that operate at the stage of early vision, before the two pathways divide, will of course, affect both pathways. It is clear that vision begins from a single anatomical base, and ends by affecting either perception or motor behavior. Many of the controversies in this 1eld can be interpreted as arguments about how late the split of these functions occurs. Some prefer to assume a uni1ed system until just before the output stages, while the evidence reviewed here shows separate and distinct representations of visual space for cognitive and motor systems. The cognitive system must take cognizance of spatial context, even when that strategy leads to errors in localization. Extending this principle to the temporal domain, Rossetti and Pisella (this volume) have shown that jabbing a lighted target is also affected by context if the jab is delayed, forcing the observer to rely on the cognitive pathway to control the jab. An array of target lights was arranged in a horizontal arc in front of the observer: errors for delayed jabbing at a given target tended to spread along the arc, in the direction of the other alternatives jabbed in other trials. In another condition, the same observers jabbed targets aligned in a row extending outward from the observer. Now the errors

133


134


were spread in depth, again in the direction of the rest of the array but orthogonal to the error distribution in the horizontal-arc condition. This difference in error distributions was found even for the point where the two rows intersected, so that the observer was jabbing the same point under each condition. Thus information about context taken from other trials in2uenced aiming at the target point, leading to systematic errors in jabbing. If the jab was immediate, however, observers could use their direct sensorimotor pathway to hit the target with less error, and independent of the context established in other trials. Spatial and temporal context thus follow similar rules, affecting only the spatial coding in the cognitive pathway. This experiment is also signi1cant from a methodological standpoint because it relies on error analysis, rather than spatial illusions, to differentiate cognitive and sensorimotor control. Both these experiments and the Roelofs effect experiments reviewed above require a subject to act on only one target in an otherwise clear or nearly clear 1eld. The real world, however, is always 1lled with a myriad of possible objects to grasp, swat, push, or prod, and the sensorimotor pathway lacks the motivation or plan to decide which of the objects to engage. In hindsight, a sensorimotor system that could not receive speci1c instructions on which of many alternatives is relevant would be of little value. Any other result would have spelled real trouble for the two-visual-systems hypothesis. To return to the questions in the introduction, these results show that attention to a target, in the sense of verbalization of its location, does not always enhance performance. In fact, such performance is subject to illusions that lead to mistakes in localization, while an unconscious pathway can control motor action without suffering from the illusions. Accurate motor localization can occur despite simultaneous mislocalization represented in attended, and remembered, pathways. Vision is not all of a piece, and the location of what we perceive is not necessarily the location of action. The decision to engage the world must come from cognitive sources, but the calibration of the engagement itself can come from separate, unperceived pathways. What seemed uni1ed is segregated in the brain into separate processing streams that follow different rules.

References Aglioti S., DeSouza J.F., and Goodale M.A. (1995). Size-contrast illusions deceive the eye but not the hand. Current Biology, 5, 679–685. Bridgeman, B. (1991). Complementary cognitive and motor image processing. In G. Obrecht and L.W. Stark (Eds.), Presbyopia research: From molecular biology to visual adaptation. New York: Plenum Press. Bridgeman, B. (1992). Conscious vs. unconscious processes: The case of vision. Theory and Psychology, 2, 73–88. Bridgeman, B. and Huemer, V. (1998). A spatially oriented decision does not induce consciousness in a motor task. Consciousness and Cognition, 7, 454–464. Bridgeman, B. and Stark, L. (1979). Omnidirectional increase in threshold for image shifts during saccadic eye movements. Perception and Psychophysics, 25, 241–243. Bridgeman, B., Lewis, S., Heit, G., and Nagle, M. (1979). Relation between cognitive and motor-oriented systems of visual position perception. Journal of Experimental Psychology: Human Perception and Performance, 5, 692–700. Bridgeman, B., Kirch, M., and Sperling, A. (1981). Segregation of cognitive and motor aspects of visual function using induced motion. Perception and Psychophysics, 29, 336–342. Bridgeman, B., Peery, S., and Anand, S., (1997). Interaction of cognitive and sensorimotor maps of visual space. Perception and Psychophysics, 59, 456–469. Bridgeman, B., Gemmer, A., Forsman, T., and Huemer, V. (2000). Processing spatial information in the sensorimotor branch of the visual system. Vision Research, 40, 3539–3552.



Daprati, E. and Gentilucci, M. (1997). Grasping an illusion. Neuropsychologia, 35, 1577–1582. Franz, V., Gegenfurtner, K., Bülthoff, H.H., and Fahle, M. (2000). Grasping visual illusions: No evidence for a dissociation between perception and action. Psychological Science, 11, 20–25. Goodale, M.A., Pélisson, D., and Prablanc, C. (1986). Large adjustments in visually guided reaching do not depend on vision of the hand or perception of target displacement. Nature, 320, 748–750. Goodale, M.A., Milner, A.D., Jakobson, L.S., and Carey, D.P. (1991). A neurological dissociation between perceiving objects and grasping them. Nature, 349, 154–156. Haffenden, A. and Goodale, M. (1998). The effect of pictorial illusion on prehension and perception. Journal of Cognitive Neuroscience, 10, 122–136. Milner, D. and Goodale, M. (1995). The visual brain in action. Oxford: Oxford University Press. Paillard, J. (1987). Cognitive versus sensorimotor encoding of spatial information. In P. Ellen and C. ThinusBlanc (Eds.), Cognitive processes and spatial orientation in animal and man. Dordrecht, The Netherlands: Martinus Nijhoff. Pöppel, E., Held, R., and Frost, D. (1973). Residual visual function after brain wounds involving the central visual pathways in man. Nature, 243, 295–296. Roelofs, C. (1935). Optische Localisation. Archiv für Augenheilkunde, 109, 395–415. Rossetti, Y. and Régnier, C. (1995). Representations in action: Pointing to a target with various representations. In B.G. Bardy, R.J. Bootsma, and Y. Guiard (Eds.), Studies in perception and action III, pp. 233–236. Mahwah, NJ: Lawrence Erlbaum. Rossetti, Y., Rode, G., and Boisson, D. (1995). Implicit processing of somaesthetic information: A dissociation between where and how. Neuroreport, 6, 506–510. Sanders, M.D., Warrington, E.K., Marshall, J., and Weiskrantz, L. (1974). ‘Blindsight’: Vision in a 1eld defect. Lancet, 20, 707–708. Schneider, G.E. (1967). Contrasting visuomotor functions of tectum and cortex in the golden hamster. Psychologische Forschung, 31, 52–62. Trevarthen, C. (1968). Two mechanisms of vision in primates. Psychologische Forschung, 31, 299–348. Ungerleider, L. and Mishkin, M. (1982). Two cortical visual systems. In D. Ingle, M. Goodale and R.J.W. Mans1eld (Eds.), Analysis of visual behavior. Cambridge, MA: MIT Press. Weiskrantz, L. (1996). Blindsight revisited. Current Opinion in Neurobiology, 6, 215–220.

135


6 How the brain represents the body: insights from neurophysiology and psychology Michael S.A. Graziano and Matthew M. Botvinick

Abstract. To reach for the computer mouse, to sit upright in a chair or hold a journal in order to read it, indeed, to do most of the actions that we commonly perform, we rely on a representation of the spatial con1guration of the body. How and where in the brain is the body represented and what are the psychological properties of this body schema? In this article we review 1rst the neurophysiology and then the psychology of the body representation. One 1nding that has emerged from both approaches is that the body representation is not merely a registration of proprioceptive inputs about joint angle. Instead, the brain contains a sophisticated model of the body that is continually updated on the basis of multimodal input including vision, somesthesis, and motor feedback. Neurophysiological studies in the monkey brain show that parietal area 5 is a critical node for processing the body’s con1guration. Neurons in area 5 combine signals from different modalities in order to represent limb position and movement. Psychological studies show that the body schema is used to cross-reference between different senses, as a basis for spatial cognition and for movement planning.

I thrust my arms wildly above and around me in all directions. I felt nothing . . . I cautiously moved forward, with my arms extended, and my eyes straining from their sockets . . . Poe, The Pit and the Pendulum Sometimes, too, just as Eve was created from a rib of Adam, so a woman would come into existence while I was sleeping, conceived from some strain in the position of my limbs. Proust, Swann’s Way ‘Easy!’ I said. ‘Be calm! Take it easy! I wouldn’t punch that leg like that.’ ‘And why not!’ he asked, irritably, belligerently. ‘Because it’s your leg,’ I answered. ‘Don’t you know your own leg?’ Sacks, The Man who Mistook his Wife for a Hat

6.1 Introduction Without an internal representation of the body, a mental model of the relative positions of the head and limbs, we would be unable to perform the most vital or trivial actions; unable to move toward and around the objects that surround the body; unable to process the locations of those objects in relation to the body; disoriented and without any sense of physical self. In the 1rst quote above, the protagonist of the story tries to understand the layout of his environment by touch and by use of his body representation. A touch on his hand will do him no good unless he knows the position of his outstretched arm. In the second quote, the position of the limbs implies the presence and shape of a


How the brain represents the body: insights from neurophysiology and psychology

nearby object. The third quote is about a brain-damaged patient whose internal body representation no longer includes his own leg. The internal representation of the body has been studied on two different disciplinary fronts: a neurophysiological one and a psychological one. Until recently, these two approaches were surprisingly isolated from each other. The psychological approach emphasized the multisensory nature of the body representation. This work demonstrated that vision, touch, and proprioception are combined and cross-referenced in a sophisticated mental schema of the body. Neurophysiology, in contrast, emphasized proprioception, one component of body representation, and focused especially on the use of proprioception in the control of movement. Only recently have these two 1elds become more integrated and begun to converge on similar themes. The psychological studies have turned more toward exploring the spatial coordinate systems that organize the representation of the body and the control of movement. In neurophysiology, recent experiments have focused on how vision, proprioception, and touch are integrated by single neurons in the parietal lobe and premotor cortex. The purpose of the present article is to review both areas of research side by side, providing an overview of each and describing some of the relations between the two. The 1rst half of the article reviews neurophysiological studies on body representation, mainly in the monkey brain. These experiments examine a set of interconnected somatosensory and motor brain areas, emphasizing area 5 in the superior parietal lobe. The second half of the article reviews the psychological data, emphasizing how the body representation coordinates information within and across perceptual modalities. Both parts of the review share certain underlying themes: the representation of the body is multimodal, and it has a close relationship to the representation of space around the body and the control of movement through that space.

6.2 The neuronal basis of the body representation 6.2.1 Proprioceptive pathways from the periphery to area 5 Proprioception begins in receptors primarily in the joints and muscles (for review, see Burgess, Wei, Clark, and Simon 1982; Iggo and Andres 1982). Information about muscle stretch and joint angle is then transmitted through the dorsal column nuclei of the medulla and the ventrobasal complex of the thalamus, to two principal regions of the cerebral cortex, area SI on the postcentral gyrus and area SII in the lateral sulcus (for a review of these pathways, see Mountcastle 1984). Of these two cortical areas, SI is far better studied and understood, perhaps partly because it is on the top of the brain and easier to reach with a recording electrode. In area SI information from deep, proprioceptive receptors arrives mainly in the subregions termed 3a and 2, while information from the cutaneous receptors arrives mainly in subregions 3b and 1 (Kaas, Nelson, Sur, Lin, and Merzenich 1979; Mountcastle 1984). These subdivisions of SI contain even smaller partitions, cortical columns a few millimeters wide, that receive information from only one receptor type. Information-processing columns, now known to exist all over the cortex, were 1rst discovered in SI (Kaas et al. 1979; Mountcastle 1997). SI projects to a wide range of cortical sites including area 5 in the superior parietal lobe. All four subregions of SI project to area 5, but the strongest projection is from subregion 2 (Pandya and Kuypers 1969; Pearson and Powell 1985; Vogt and Pandya 1978). That is, area 5 receives input mainly from the deep, proprioceptive receptors. Most of the work on body representation in the

137


138


Fig. 6.1 Side view of the macaque monkey brain showing some of the cortical areas involved in representing the physical con1guration of the body. The intraparietal sulcus is shown opened up to expose the buried cortex (stippled area). MIP = medial intraparietal area, AIP = anterior intraparietal area, 5A = anterior area 5, M1 = primary motor cortex, SI = primary somatosensory cortex, SII = second somatosensory cortex. monkey brain has concentrated on area 5. This cortical area is not uniform. As shown in Fig. 6.1, it has several subdivisions, including area MIP on the medial bank of the intraparietal sulcus, V6A on the anterior bank of the parieto-occipital sulcus, a newly suggested area PRR that may overlap V6A and MIP, and a region on the gyral surface termed here Anterior 5 or 5A. Until recently, however, most of the work on area 5 did not distinguish between these different regions. The following sections review the effects of lesions to area 5, its physiology, and its possible role in body representation and the control of movement.

6.2.2 Lesions to area 5 In 1884, Ferrier and Yeo reported the effects of parietal lesions in monkeys (see Gross 1998 for a review of the history). They argued that visual cortex must be located in the parietal lobe and not the occipital lobe, because monkeys with parietal lesions were unable to reach accurately for pieces of food. Some time after, Balint (1909) observed similar behavior in humans with damage to parietal cortex. Balint and others (e.g. Holmes 1918) realized that the de1cit was probably one of sensory-motor integration, spatial attention, or body representation, not vision. Since then, the parietal lobe syndrome in all its spatial, motor, and attentional manifestations has been studied extensively in humans and monkeys (Andersen 1987; Critchley 1953; De Renzi 1982; Holmes 1918; Kolb and Whishaw 1990; Newcombe and Ratcliff 1989). One of the many de1cits often seen in human patients is a disturbance of body representation. For example, some patients will neglect one side of the body, failing to shave or dress on that side; other patients will notice the limbs contralateral to the damaged parietal lobe readily enough, but will mistakenly think that the



limbs are not attached to them and belong to someone else. Whether these de1cits are associated with one or another subregion of the parietal lobe is dif1cult to tell in humans, because of the size and uncertain borders of the lesions. Some of these de1cits in body representation are discussed further in the second half of this chapter. Lesions to the superior parietal lobe in monkeys cause de1cits in almost all aspects of somesthesis, not just body representation but touch as well (Ettlinger and Wegener 1958; Moffet and Ettlinger 1970; Ridley and Ettlinger 1975; Ruch, Fulton, and German 1938). Murray and Mishkin (1984) argued that many of these de1cits were the result of accidental damage to SI itself, not to area 5. Their results indicated that lesions carefully restricted to area 5 had minimal effects on texture, roughness, and shape discrimination. In contrast, lesions to areas SI and SII had a devastating impact on all of these behaviors. They proposed that area 5 processes the spatial component of somesthesis, such as the position of the arm, while area SII processes the perceptual, object-recognition, and memory component of somesthesis that comes with feeling an object with the hand. This dichotomy of the somatosensory system was proposed as a parallel to the visual system. In 1982, Ungerleider and Mishkin proposed that the cortical visual system was divided into two components of which the ‘dorsal stream’ subserves spatial vision and the ‘ventral stream’ subserves object recognition and memory. In the view of Murray and Mishkin (1984), area 5 is the dorsal stream, or spatial module of the somatosensory system. Several recent studies by Passingham and colleagues (Rushworth, Nixon, and Passingham 1997a,b) con1rmed that area 5 is necessary for the accurate spatial guidance of the arm, especially in the dark when only somatosensory cues are available. In the next several sections, we describe the properties of single neurons in area 5 and discuss how they might encode the spatial con1guration of the body and help to control movement.

6.2.3 Early single neuron studies of area 5 In 1971, Duffy and Burch1el studied the activity of neurons in area 5 of awake monkeys. They found that most neurons responded to proprioceptive signals—to joint angle and muscle stretch. These cells had highly complex properties. Some combined a tactile receptive 1eld on the skin with a response to joint rotation. Some responded to rotation of more than one joint, and many had bilateral responses. This convergence of different joints and different somatosensory submodalities onto individual neurons was never seen in SI. Typically, neurons in SI have small receptive 1elds on the contralateral side of the body, and respond to stimulation of one class of peripheral somatosensory receptor only. The differences between area 5 and SI led Duffy and Burch1el to suggest that area 5 represents a higher stage in somatosensory processing, and especially in the processing of body representation. Sakata et al. (1973) con1rmed and extended these 1ndings in area 5. These authors made two important original observations, both of which have been largely neglected. First, they found a subset of area 5 neurons that responded to a touch on the hand, but only if the joints of the arm were placed in certain positions. They argued that such neurons would be able to encode the spatial location of a felt object. Second, they found neurons ‘which responded to certain visual as well as to somesthetic stimuli.’ This visual input has never been systematically studied, although it was noted by other investigators (Colby and Duhamel 1991; MacKay and Crammond 1987; Mountcastle et al. 1975). Mountcastle and colleagues (1975) provided the 1rst coherent view of the functions of area 5 and its role in behavior. This landmark paper described the properties of neurons in both area 5, in the

139


140


superior parietal lobe, and the adjacent area 7 in the inferior parietal lobe. In addition to con1rming the 1ndings of previous studies, this study made the novel discovery of motor functions in the parietal lobe. In area 5, some of the neurons that responded during passive movement of the arm showed greater response when the monkey moved its arm on its own volition. Some neurons showed no somatosensory activity at all, responding only during the monkey’s goal-directed reaches. Another class of neurons responded when the monkey grasped or manipulated objects with its 1ngers. In area 7, neurons responded in association with eye movement. Some responded during active 1xation, others during saccadic or smooth pursuit eye movements. In the words of the authors, ‘These regions receive afferent signals descriptive of the position and movement of the body in space, and contain a command apparatus for operation of the limbs, hands and eyes within immediate extrapersonal space.’ The sensory–motor command hypothesis of Mountcastle et al. was controversial at 1rst, especially as applied to area 7 (e.g. Robinson, Goldberg, and Stanton 1978), but in the past 20 years has gradually gained a quali1ed acceptance. Areas 7 and 5 have now been parceled into more than 10 functionally different areas (for review, see Andersen, Snyder, Bradley and Xing 1997; Colby and Duhamel 1991). Some of these areas are involved in eye movement and 1xation, such as LIP and V6A (Andersen, Bracewell, Barash, Gnadt, and Fogassi 1990; Galletti, Battaglini, and Fattori 1995; Nakamura, Chung, Graziano, and Gross 1999). Some parietal areas are more involved in arm movement, such as areas MIP, PRR, and 7 m (Colby and Duhamel 1991; Ferraina et al. 1997; Snyder, Batista, and Andersen 1997). Parietal area AIP may be involved in grasping objects with the hand (Sakata and Taira, 1994). Even in the human literature, the parietal lobe has come to be viewed as a sensory–motor structure rather than as a purely visual, proprioceptive, or spatial structure (Goodale et al. 1994; Rossetti and Pisella, this volume, Chapter 4; Gallese et al., this volume, Chapter 17). As described in the next section, the single neuron experiments in area 5 that followed Mountcastle focused almost exclusively on the effort to distinguish sensory from motor; perception of limb position from the command to move.

6.2.4 Body representation and movement control in area 5 Area 5 projects to and receives projections from primary motor cortex, premotor cortex, and supplementary motor cortex, among other areas (Johnson et al. 1996; Jones, Coulter, and Hendry 1978; Jones and Powell 1970; Strick and Kim 1978); that is, it is closely connected to the motor system. To what extent is it a sensory structure or a motor structure? As described above, Mountcastle, Lynch, Georgopoulos, Sakata, and Acuna (1975) found neurons that responded best, sometimes only, during active rather than passive movements of the arm. But do these responses represent motor commands, as Mountcastle et al. proposed, or are they somatosensory signals, perhaps enhanced when the monkey is paying attention to its arm? Seal, Gross, and Bioulac (1982) examined this issue in monkeys that were trained to 2ex or extend the elbow joint. These experimenters cut the sensory nerves from the arm, and found that about 38% of the neurons in area 5 still responded just before and during arm movements. These neurons therefore responded independently of any somatosensory stimulation; their activity was internally generated. More recent studies have con1rmed that neurons in area 5, both on the surface and in the intraparietal sulcus, are active during reaching movements (Batista, Buneo, Snyder, and Andersen 1999; Lacquaniti et al. 1995; Snyder et al. 1997). Neurons in a proposed region of the intraparietal sulcus, area PRR, may be especially active in association with reaching (Snyder et al. 1997). These neurons



respond in anticipation of the arm movement. In a delayed reaching task, the neurons respond during the delay period after the monkey is instructed where to reach but before the ‘go’ signal. One speculation is that the activity of these neurons represents a motor plan. However, these experiments do not distinguish between activity that ultimately causes a movement and activity that represents an arm position predicted on the basis of motor feedback. Kalaska, Caminiti, and Georgopoulos (1983) found that when monkeys are planning to make an arm movement, area 5 neurons begin to respond on average 60 ms after the neurons in primary motor cortex. Thus at least some of the motor-related activity in area 5 could be the result of efference copy. Though motor in origin, these signals could serve a sensory function, helping to encode body posture. Lesions to motor cortex do not abolish the motor-related activity in area 5 (Bioulac, Burbaud, and Varoqueaux 1995); but it is dif1cult to rule out the possibility that another motor area is sending this efference-copy signal. The critical experiment to determine whether the activity in area 5 encodes body representation or controls movement has not yet been found, and may never be. Such a distinction between sensory and motor now appears to be too simple. Area 5 may contribute to both roles. A more meaningful question might be: How far along the sensory–motor transformation does area 5 lie? For example, how do the response properties in area 5 compare to those in primary motor cortex? Several groups have studied exactly this question. Georgopoulos and Massey (1985) found that the neuronal selectivity for the direction of hand movement was greater in primary motor cortex than in area 5, while selectivity to the static position of the hand in space was greater in area 5 than in primary motor cortex. Kalaska and colleagues (Kalaska and Hyde 1985; Kalaska, Cohen, Prud’homme, and Hyde 1990) trained monkeys to move a handle along speci1c trajectories while external force loads were applied to the handle. In this way, the location and direction of hand movement were dissociated from the muscular forces that the monkey used. The results showed that selectivity to the position and trajectory of the hand through space was greater in area 5 than in primary motor cortex, while selectivity to the muscular forces applied by the arm was greater in primary motor cortex than in area 5. In summary, the differences between area 5 and primary motor cortex are relative, not absolute. Area 5 neurons may play relatively more of a representational role, keeping track of the positions and movements of limbs, while primary motor cortex may play relatively more of a dynamic role, initiating and guiding the movements. However, these two functions overlap extensively. Not only do some area 5 neurons have motor properties, but most primary motor neurons have sensory properties, responding to tactile stimuli and joint rotation (Gentilucci et al. 1988). Indeed, primary motor cortex receives direct projections from almost every stage of the somatosensory system, including SI and even the somatosensory thalamus (for review, see Mountcastle 1984). The somatosensory–motor system, therefore, is organized as a set of highly interconnected nodes that collectively participate in the sensory guidance of movement. The evidence so far suggests that among these many nodes, area 5 is relatively more specialized for encoding the spatial con1guration of the body.

6.2.5 Visual representation of arm position in area 5 The studies reviewed so far focused on the role of proprioception and motor control in body representation. However, other sources of information are also important in body representation. According to psychophysical studies discussed in the second half of this article, vision is sometimes

141


142


the dominant sense of arm position. Does area 5 use visual input to help encode the position of the arm? Graziano, Cooke, and Taylor (2000) examined the visual representation of arm position in monkey area 5 by manipulating two variables: (1) the position of the monkey’s arm while it was out of view, under an opaque plate; (2) the position of a visible false arm, placed on top of the plate (see Fig. 6.2). The false arm was a monkey arm prepared by a taxidermist and arranged in a realistic fashion, positioned to appear as if it were extending from the shoulder of the experimental monkey. The monkey 1xated on a central spot during these tests. About 25% of the neurons tested in area 5 were signi1cantly affected by the visual position of the false arm. The proportion was signi1cantly higher

Fig. 6.2 Diagram of apparatus for testing whether neurons are sensitive to the felt or seen position of the arm. The monkey’s real arm was held in an adjustable arm holder covered from view while a realistic fake arm was in view. The real arm and the visible fake arm were placed on the left or right resulting in four experimental conditions. The monkey was trained to 1xate on a central lightemitting diode.



Fig. 6.3 Convergence of visual and somatosensory signals about arm position on an area 5 neuron. The neuron 1red at a higher tonic rate when the monkey felt its arm to be on the left. It also 1red at a higher tonic rate when the monkey saw the fake arm to be on the left. Each point is an average of 10 trials. Error bars are standard error. See Fig. 6.2 for methods. in MIP (35%) than in 5A (18%), suggesting that there might be a hierarchy of areas in which the visual sense of arm position is more fully developed in MIP (see Fig. 6.1 for location of MIP and 5A). Data from a typical example neuron is shown in Fig. 6.3. The tonic 1ring rate of the neuron was signi1cantly higher when the real arm was on the left. The 1ring rate was also signi1cantly higher when the fake arm was on the left. That is, this neuron integrated the felt position of the real arm and the seen position of the false arm. This result suggests that area 5 neurons encode the position of the arm in a supramodal fashion, using both somesthesis and vision. Similar tests using objects other than a fake arm, such as a white rectangle the same approximate size as the arm, or a piece of fruit to which the monkey appeared to attend, did not affect the activity of the neurons in the same fashion. In the same study, Graziano et al. (2000) found that neurons in SI were not sensitive to the seen position of the false arm. That is, in the ascending somatosensory pathway from the periphery to SI to area 5 and beyond, the 1rst stage at which visual information about arm position is integrated with somatosensory information appears to be in area 5. This 1nding is consistent with the view that area 5 is a central node in representing the con1guration of the body. It receives all necessary signals, including proprioception, motor feedback, and vision, and combines these signals to encode the relative positions of body parts. Area 5 projects to many cortical regions, including premotor and motor cortex, where information about body con1guration would be useful for planning movements. In the following section we describe the properties of neurons in premotor cortex that integrate the body representation with the representation of space surrounding the body. These neurons encode the locations of objects in space relative to the body, perhaps for the purpose of guiding movements.

143


144


6.2.6 Premotor cortex: a convergence of body representation, visual space, and movement control Neurons in the caudal premotor cortex of monkeys, just posterior to the bend in the arcuate sulcus, process and encode the locations of visual, tactile, auditory, and remembered stimuli and may help to guide movements of the head and arms (Gentilucci et al. 1988; Graziano, Hu, and Gross 1997a,b; Graziano, Reiss, and Gross 1999). About 40% of the neurons are bimodal, responding both to tactile and to visual stimuli (Gentilucci et al. 1988; Graziano et al. 1997a; Rizzolatti et al. 1981). The tactile receptive 1elds are arranged to form a somatotopic map. The visual receptive 1elds are usually adjacent to the tactile ones and extend outward from the skin about 20 cm (see Fig. 6.4). The area therefore contains a somatotopically organized map of the visual space that immediately surrounds the body. For most bimodal cells, the visual receptive 1eld is anchored to the site of the tactile receptive 1eld on the body. When the monkey’s eyes move, the visual response may change magnitude, but the location of the visual receptive 1eld does not change (Fogassi et al. 1992, 1996; Gentilucci et al. 1983; Graziano, Yap, and Gross 1994; Graziano et al. 1997a). If the tactile receptive 1eld is on the head, then rotating the head will cause the visual receptive 1eld to move in tandem with the head (Graziano et al. 1997a,b). If the tactile receptive 1eld is on the arm, moving the arm to different positions will cause the visual receptive 1eld to move in the same direction as the arm (Graziano et al. 1994, 1997a). The arm-related neurons are in2uenced by the sight of a fake arm as well as by the felt position of the real arm (Graziano 1999). In a recent experiment mapping the precentral gyrus, these multimodal neurons were found in a relatively restricted zone in the caudal part of premotor cortex (Graziano and Gandhi 2000). Other studies on more rostral and ventral regions in premotor cortex have found neuronal properties that may be somewhat different; but because of differences in experimental technique, the studies are dif1cult to compare (Mushiake, Tanatsugu, and Tanji 1997). The bimodal neurons in caudal premotor cortex bind together body representation with the visual space surrounding the body and the tactile space on the body. They encode the locations of objects in body-part centered coordinates. One possibility is that these neurons form a mechanism for guiding movements of the limbs and head away from nearby objects, for 2inching and avoiding. That is,

Fig. 6.4 Receptive 1elds of two bimodal, visual-tactile neurons in the polysensory zone in premotor cortex. (A). The tactile receptive 1eld (shaded) is on the snout, mostly contralateral to the recording electrode (indicated by the arrowhead) but extending partially onto the ipsilateral side of the face. The visual receptive 1eld (boxed) is contralateral and con1ned to a region of space within about 10 cm of the tactile receptive 1eld. (B). The tactile receptive 1eld for this neuron is on the hand and forearm contralateral to the recording electrode (indicated by the black dot) and the visual receptive 1eld (outlined) surrounds the tactile receptive 1eld. (Adapted from Graziano and Gross 1998.)



these multimodal receptive 1elds may form a type of protective shell around the body, alerting the brain of any potentially noxious object impinging on near space. As described in the following sections, similar interactions among vision, touch, body representation, and the control of movement can be seen in behavioral studies on humans.

6.3 The psychology of the body representation 6.3.1 The body schema In this section, we turn from neurophysiological to psychological work on body representation. One of the most important general 1ndings of psychological experiments is that the body representation involves more than the mere registration of peripheral inputs. Rather, it involves the interpretation of these inputs in the context of a rich internal model of the body’s structure. In what follows, we refer to this internal model as the ‘body schema’. While this term has been used in many different ways in past work (for discussion, see Gallagher 1986), we use the term broadly to mean an implicit knowledge structure that encodes the body’s form, the constraints on how the body’s parts can be con1gured, and the consequences of this con1guration on touch, vision, and movement. The body schema plays a central role in interrelating concurrent perceptual inputs, allowing for the reconstruction of missing information, enabling the detection and resolution of con2icts, and ensuring an integrated, globally consistent, multimodal representation of the body’s con1guration. The body schema may even be used to interpret the seen body con1guration of others (Shiffrar and Pinto, this volume, Chapter 19). In the following sections we discuss different types and combinations of information that are coordinated through the body schema.

6.3.2 Converting proprioceptive inputs into a representation of body position The body schema is not simply a representation of joint angles, but a complex integration of vision, proprioception, touch, and motor feedback. The relative weights applied to these various sources of information probably depend on the quality of information from each source (Stark and Bridgeman 1983). In this section we discuss one source of information about body con1guration; proprioception. In the subsequent sections, we will discuss the interactions between proprioception, touch, vision, and motor feedback. Proprioception derives originally from the local forces acting on muscle spindles, joint receptors, and tendon receptors. However, behavioral data, in line with everyday experience, suggests that this raw sensory information is ultimately combined with knowledge of the body’s segmental structure in order to produce a representation of the body’s current spatial con1guration. One indication of this transformation from simple joint information to a more complex body representation is that human subjects are more accurate at judging the spatial orientation of limb segments than they are at estimating the angles of individual joints (Soechting 1982). Another indication is the phantom limb phenomenon. Here, an amputated arm or leg continues to be experienced as present, and occupying its former location in space. This phenomenon is thought to involve continued input to areas of cortex formerly responsible for representing the position of the missing limb (for a review, see Ramachandran and Hirstein 1998). The fact that this input is translated into a detailed limb representation indicates that peripheral information is interpreted with reference to a centrally maintained model of the body’s form.

145


146


In order to transform peripheral signals into a representation of posture, the brain must integrate information from different, potentially distant, parts of the body. For example, the judgement of forearm orientation requires information concerning shoulder and elbow positions, at the least. This integration is re2ected in the behavior of neurons in cortical area 5, which as noted above combine information about multiple joints. The same type of integration can be seen in human behavioral studies, such as those performed by Lackner (Lackner 1988; Lackner and Taublieb 1983). Lackner took advantage of an illusion produced by vibration. If the experimenter applies vibration at about 100 Hz to a muscle or its tendon, and prevents movement resulting from the associated re2ex contraction, the subject experiences the illusion of movement around the joints crossed by that muscle. For example, vibration of the biceps produces an illusion of elbow extension, and vibration of the triceps one of elbow 2exion. Lackner (1988) found that under appropriate circumstances vibration of a single muscle group could produce the perception of rather global shifts in posture. For example, for subjects seated on the 2oor with the right hand under the right buttock, biceps vibration produced not only the illusion of arm extension, but also of a tilting of the body toward the left, as would occur if the arm really did extend and push against the 2oor (see also Roll, Roll, and Velay 1991). Similarly, while biceps vibration in the standing position ordinarily produces an illusion of movement only in the vibrated arm, if the subject grasps the left wrist with the right hand, vibration of the right biceps produces the illusion of movement of both arms (Craske, Kenny, and Keith 1984). These effects suggest that proprioceptive information deriving from multiple joints, tendons, and muscles is integrated into an internally consistent model of body position that takes into account the constraints imposed by the body’s structure.

6.3.3 Representing the size and shape of body parts In order to convert joint and muscle information into a representation of body position, the brain’s model of the body must include information about the size and shape of the body’s parts. The brain must also be able to update this model as body shape changes over the course of development. An experiment by Craske, Kenny, and Keith (1984) demonstrated how the perception of the body’s dimensions can be recalibrated to maintain consistency with other sources of information. In this study, subjects seated in the dark extended both arms and used their right index 1nger to touch a position on their left arm. A mechanical device was used to produce a mismatch between the position of the right hand and the location stimulated on the left arm, such that the touch actually occurred 12.5 cm closer to the shoulder. After a period of exposure to this mismatch, subjects reported feeling that their left arm was longer than their right. That is, the representation of arm length had been updated to resolve a con2ict between proprioception and touch. Another demonstration of the same phenomenon was provided by Lackner (1988). In this experiment, each subject was asked to grasp his or her own nose, with eyes closed. Vibration was then applied to the biceps of the grasping arm. The resulting illusion of arm extension was accompanied by a feeling that the nose had become elongated. ‘Oh my gosh!,’ one subject exclaimed, ‘My nose is a foot long! I feel like Pinocchio.’ Lackner’s illusion, like the other phenomena we have described above, highlights two important aspects of the body schema. First, the body schema contains geometrical information about the hinged and segmented structure of the body, such as that extension of the arm means an increase in the distance of the hand from the face, and that the tip of the nose is connected with the rest of the face. Second, the brain will update and even distort this model of the body in order to resolve con2icts of information.



6.3.4 Body position and the interpretation of touch One important role for body representation is to support the perception of objects in the environment surrounding the body. For example, in order to perceive the shape, location, and orientation of an object being felt with the hand, it is necessary to have an accurate representation of the hand’s posture and location. This ability to integrate the sense of touch with the body representation might be related to the neurons in area 5 that have combined tactile and proprioceptive receptive 1elds, as discussed above. Psychological experiments in humans have also explored the relationship between touch and body representation. For example, Driver and Grossenbacher (1996) had subjects perform a tactile discrimination task with one hand while attempting to ignore concurrent stimulation to the other hand. They found that subjects performed better when the two hands were held farther apart, indicating that attentional mechanisms were working within a spatial representation of touch that incorporated hand position. Closely related 1ndings come from work on patients with hemispatial neglect after parietal lobe injury. Driver, Mattingley, Rorden, and Davis (1997) studied tactile extinction, the failure to detect the more contralesional of two simultaneous touches on the body. They found that if both hands were touched, the patients showed greater tactile extinction when the two hands were held close together than when they were held far apart. Aglioti, Smania, and Peru (1999) found that for some neglect patients, if the two hands were crossed, such that the left hand was to the right of the trunk and the right hand to the left of the trunk, the tactile neglect switched hands, to remain on the contralesional side of the trunk. In a related study, Moscovitch and Behrmann (1994) asked patients to hold out one hand with the palm up. When both sides of the wrist were touched, the patients neglected the touch on the contralesional side. The patients were then asked to turn the hand so that the palm faced down. Under this condition, the patients neglected a touch on the opposite side of the wrist, that is, still toward the contralateral side of the body. The integration of touch with body position information is especially important for stereognosis, the use of touch to judge the size and shape of objects. Illusions in stereognosis can occur when body position is misperceived. One such illusion, described originally more than two thousand years ago by Aristotle (described in Benedetti 1985), is caused by holding a small ball between the crossed third and fourth 1ngers. This situation produces the perception that the ball has doubled, and that two separate objects are contacting the 1ngertips. Benedetti (1985, 1988) in somewhat more recent studies of this phenomenon showed that the doubling occurs because the tactile input is interpreted as if the 1ngers were uncrossed. This ‘tactile diplopia’ that occurs when objects are explored with the hand in a highly unfamiliar posture disappears if the subject is given extended experience with that hand position (Benedetti 1991). This result may be related to the reorganization in somatosensory cortex that occurs after practice with a particular task (Jenkins et al. 1990) and after changes in hand structure, for example after surgically induced syndactyly (Allard, Clark, Jenkins, and Merzenich 1991).

6.3.5 Coordinating seen and felt body position Just as the body schema links touch and proprioceptive information, it also links proprioception with vision and oculomotor function. One demonstration of this connection is that subjects can 1xate the position of their 1ngertip in the dark, and moreover can track its motion with smooth

147


148


pursuit eye movements (Jordan 1970). Furthermore, illusions of arm movement can produce illusions of visual motion; if a diode is af1xed to the 1ngertip, and an illusion of arm 2exion is induced by muscle vibration, the light appears to move in the direction of the perceived arm movement (DiZio, Lathan, and Lackner 1993). Conversely, visual inputs can in2uence proprioception, as demonstrated by the phenomenon of visual capture. Here, viewing one’s hand through a prism results in a distortion of proprioceptive perception such that the hand is felt to lie in the location where it is seen (Hay, Pick, and Ikeda 1965; Welch 1986). Another version of visual capture occurs in patients with phantom limbs. If these patients view the intact arm in a mirror, such that its re2ected image appears in the location formerly occupied by the missing limb, then movement of the intact arm can induce the perception of identical movements in the phantom (Ramachandran and Hirstein 1998). After prolonged exposure to a visual-proprioceptive mismatch, the mechanisms serving to coordinate the two modalities can themselves be altered. This adaptation is shown by many experiments with prism-induced visual displacements (Redding and Wallace 1997; Welch 1986). When pointing to targets viewed through a laterally displacing prism, subjects initially misreach in the direction of the visual displacement. However, after continued exposure to the prism, reaching becomes more accurate. After such adaptation, if the prism is removed and the subject is asked to reach for targets viewed normally, misreaching tends to occur in the direction opposite to the previous visual displacement. If a prism-adapted subject is asked to close his or her eyes and position the adapted hand so that it feels straight ahead of the nose, the subject will misplace the hand off the body midline, suggesting that adaptation to the initial visual–proprioceptive mismatch has led to a recalibration of the felt position of the arm (Harris 1965).

6.3.6 Interrelating multiple perceptual modalities A number of studies indicate that the body schema can coordinate even rather complex relationships among sensory modalities, including three-way interactions among touch, vision, and proprioception. For example, directing the eyes toward a particular part of the body, even in the dark, enhances the tactile sensitivity of that part (Tipper et al. 1998). Another demonstration of such three-way coordination was provided by Driver and Spence (1998). They found that a brief touch on the hand enhanced the subsequent processing of visual stimuli near the hand. The touch seemed to draw visual attention to the region of space near the hand. This enhanced visual region was anchored to the hand and moved to different locations in space when the hand was moved, even when the hands were crossed. Likewise, a 2ash of light presented near the hand enhanced the subsequent processing of tactile stimuli applied to the hand. In both versions of the experiment, crossmodal attention between vision and touch was operating on representations that had already taken body con1guration into account. Crossmodal attention has also been studied in patients with brain injury. Di Pellegrino, Ladavas, and Farne (1997) reported on a patient with right fronto-temporal damage and symptoms of tactile extinction. This subject was asked to detect a tactile stimulus applied to the contralesional hand. When a visual stimulus was simultaneously presented near the ipsilesional hand, the subject no longer reported the tactile stimulus. That is, the tactile stimulus had been extinguished by the competing visual stimulus. The critical region of visual space, in which the competing stimulus was most effective, surrounded the ipsilesional hand and moved if the hand was moved. That is,



touch, vision, and proprioception were bound together in a framework provided by the body schema. A similar close association between vision, touch, and proprioception occurs in the bimodal visuotactile neurons in premotor cortex, discussed above, an area that may have been damaged in the patient considered in the di Pellegrino study. An experiment by Botvinick and Cohen (1998) showed that perceptual information can sometimes be distorted in order to maintain consistency in the three-way relationship among vision, touch, and proprioception. This study reported a novel illusion, produced by introducing a spatial mismatch between seen and felt touch. The effect was elicited in the following manner (see Fig. 6.5): a rubber replica of a human hand was placed on a table in front of the subject. The subject’s own hand was positioned out of view, behind a screen. As the subject watched the rubber hand, the experimenter stroked it with a brush and, at the same time, brushed the subject’s own hand in an identical manner. Subjects reported a spatial fusion between the seen and felt touch, as if they were feeling the touch of the brush in the location where they saw the rubber hand touched. They often described this illusion by saying that it felt as if the rubber hand had become ‘their’ hand (see Fig. 6.6). The rubber hand illusion provides another example of the body schema mediating in the resolution of a con2ict; the perception of felt touch was brought into spatial alignment with the visually perceived touch, much as a ventriloquist’s voice is aligned with his dummy’s moving mouth. Botvinick and Cohen (1998) reasoned that this realignment of visual and tactile representations should involve a distortion of proprioceptive information, causing the subject’s arm to be represented in a position that would place his hand in the position of the rubber hand. They predicted that if proprioceptive information was indeed being distorted in this way, then prolonged exposure to the illusion should give rise to effects on reaching similar to those observed in prism adaptation experiments. Indeed, the subjects did show a reaching bias consistent with a recalibration of proprioception. Furthermore, the magnitude of the reaching bias correlated positively with the reported duration of the illusion.

Fig. 6.5 Arrangement used in eliciting the rubber hand illusion. The subject’s hand, out of view, was stroked with a paint brush while a rubber hand, in view, was synchronously stroked.

149


150


Fig. 6.6 Questionnaire results from Botvinick and Cohen (1998). Subjects were asked to rate the accuracy of each statement on a seven-point scale ranging from ‘disagree strongly’ (---), through ‘neither agree nor disagree’ (0), to ‘agree strongly’ (+ + +). Each point is the mean rating of 10 subjects. Error bars show response range.

6.3.7 The body in action: representing the relation of target and effector As reviewed above, neurophysiological 1ndings in the monkey indicate that the brain does not draw a sharp boundary between its representation of the body and its representation of movement. Behavioral studies in humans point to the same conclusion. Movements appear to be planned in spatial coordinate frames that are referenced to the different parts of the body. In particular, the act of reaching toward a target is closely related to the sense of arm position. When we reach toward a target, we normally have proprioceptive feedback from the arm, a continuous view of both the hand and the target. A number of experiments have investigated the effect of removing one or another of these sources of information. For example, vision of the hand throughout the reach improves accuracy (Desmurget et al. 1995; Prablanc et al. 1979). The importance of the view of the hand is especially clear in patients who have lost proprioceptive sense in their arms due to nerve degeneration. These patients have no other sense of arm position than vision. If these patients reach toward a target without the sight of their hands, they make large errors in both direction and extent (Ghez et al. 1995). Vision of the hand can in2uence not only the accuracy of pointing, but also the path used to reach the target. Wolpert, Ghahramani, and Jordan (1995) showed that if the hand trajectory as seen by the subject is distorted by increasing its curvature, subjects adapt by reaching along paths curved in the opposite direction, apparently seeking to produce reaches that follow a straight line in visual space. Vision of the hand can affect reaching performance even if the hand is viewed only prior to reach initiation. A glimpse of the hand in its resting position prior to movement has been shown in a number of studies to improve reach accuracy (Desmurget



et al. 1997; Prablanc et al. 1979). Furthermore, if the subject wears a displacing prism and is given a brief initial view of the hand, the subsequent reach is misdirected, indicating again that the position of the hand, in this case mislocalized by the subject, is incorporated into the motor program (Rossetti, Desmurget, and Prablanc 1995). Collectively, these studies show that the location of the hand is continuously monitored and used during reaching. One hypothesis is that the brain computes the current location of the target relative to the hand, and then uses this hand-centered spatial information to guide the movement of the hand (Graziano and Gross 1998). Several studies have found that the errors in reaching to a visual or remembered target tend to be along the line between the starting position of the hand and the target (Chief1 et al. 1999; Gordon et al. 1994; Soechting and Flanders 1989). This result suggests that the visuomotor system does indeed compute the distance and direction of the target from the hand, with greater error in the computation of distance. McIntyre et al. (1998) found that the pattern of errors during reaching supported both an eye-centered and a hand-centered reference frame. They point out that their 1ndings are consistent with a 1nal transformation into a hand-centered frame. Tipper, Lortie, and Baylis (1992) found evidence that attention to visual stimuli during a reaching task may be linked to the position of the hand. In their study, subjects reached for a target while avoiding a distracting stimulus. The reaction times were longer when the distracter lay roughly between the hand and the target. The critical region of visual space, in which the distracter had maximum effect, was anchored to the hand and moved if the hand was placed in different starting locations. In several conceptually related experiments (Anzola, Bertoloni, Buchtel, and Rizzolatti 1977; Wallace 1971), subjects pressed a button in response to a 2ashed light. If the 2ash of light was in the space near the hand, the subjects responded more quickly. For example, when subjects were asked to respond to a light on the right side of visual space by pressing a button with the left hand, and to a light on the left side using the right hand, they were faster when the hands were crossed than when they were uncrossed. In summary, there is accumulating behavioral evidence that during movement of the arm and hand, stimuli are encoded in a spatial reference frame that is anchored to the hand, at least at some point during the movement planning. This result matches the 1ndings in monkey premotor cortex. As described above, some of the multimodal neurons in premotor cortex that are related to arm movement also have visual receptive 1elds that are anchored to the arm and hand, apparently encoding the locations of nearby objects in arm- and hand-centered coordinates. The behavioral results suggest that humans may also have hand-centered visual receptive 1elds. This type of spatial coding, in body-part centered coordinates, would bind together the representation of the body and of the visual space around the body with the control of movement.

6.3.8 Development of the body schema: the roles of nature and nurture As the body changes size and shape over the life-span, the internal model of the body must change accordingly. A wealth of experimental and clinical data, some of which we have already reviewed, shows just how plastic the body schema can be, even in adulthood. A classic example of this adaptation is the phenomenon of prism adaptation, discussed above. The body schema can also be modi1ed by injuries that deform the limbs. In cases where the deformed limb is amputated, the patient often reports experiencing a phantom limb with the same deformity (Ramachandran and Hirstein 1998). While experience seems certain to play an important role in the construction and continual modi1cation of the body schema, at least some elements of the body schema may not depend on

151


152


experience, or at least may be present very early in life (Bahrick and Watson 1985). Rochat and colleagues (Morgan and Rochat 1997; Rochat 1998) investigated the development of the body schema in infancy, and found that infants as young as three months were able to match proprioceptive events to visual ones. Other researchers found evidence of visual–proprioceptive matching even shortly after birth (Meltzoff and Moore 1983). One of the most compelling arguments for an innate body schema is the phenomenon of the aplastic phantom. Here, a person congenitally lacking a limb experiences a full-2edged phantom in place of the missing limb (Saadah and Melzack 1994). While 1ndings such as these are intriguing, the relative roles of nature and nurture in establishing and calibrating the body schema remain poorly understood, and present an important area for future research. In investigating how the body schema emerges during infancy, some developmental psychologists have also asked the question of how the body comes to be distinguished from other objects as belonging to the self. Rochat (1998) suggested that the detection of correlations among visual, proprioceptive, tactile, and motor signals provides the basis for the identi1cation of the body as self. It is interesting that focal brain lesions, especially to the right parietal lobe, can cause the denial of ownership of intact body parts (Kolb and Whishaw 1990); perhaps the loss of the sense of self in these patients is due to a disruption of Rochat’s crossmodal mappings.

6.4 Conclusions We have described neurophysiological work in monkeys and psychological work in normal and brain-damaged humans on the internal representation of the body. Between these different approaches, an enormous amount is now known about the body representation. The critical brain areas have been identi1ed; they include area 5 in the superior parietal lobe, and possibly other areas such as premotor and motor cortex. A main 1nding of both the neurophysiological and the psychological approach is that the body representation is not merely a simple code for joint angles. Instead, proprioceptive information is combined with visual, tactile, and motor-feedback signals. All of this information is interpreted in the context of an internal model of the geometry of the body. The body schema appears to be a device for cross-referencing between sensory modalities, and for guiding movement of the limbs through space. Just as the body schema itself lies at the crossroads of multiple sensory modalities and in the communication among multiple cortical regions, its study traverses multiple disciplines. Progress in understanding the body schema will come from the continued, rich interrelations between psychophysics, neuropsychology, and neuroscience.

Acknowledgement We thank J. Cohen for his help.

References Aglioti, S., Smania, N., and Peru, A. (1999). Frames of reference for mapping tactile stimuli in brain-damaged patients. Journal of Cognitive Neuroscience, 11, 67–79.



Allard, T., Clark, S.A., Jenkins, W.M., and Merzenich, M.M. (1991). Reorganization of somatosensory area 3b representations in adult owl monkeys after digital syndactyly. Journal of Neurophysiology, 66, 1048–1058. Andersen, R.A. (1987). Inferior parietal lobule function in spatial perception and visuomotor integration. In F. Plum and V.B. Mountcastle (Eds.), Handbook of physiology, Vol. 5, pp. 483–518. Bethesda, MD: American Physiological Society. Andersen, R.A., Bracewell, R.M., Barash, S., Gnadt, J.W., and Fogassi, L. (1990). Eye-position effects on visual, memory, and saccade-related activity in areas LIP and 7a of macaque. Journal of Neuroscience, 10, 1176–1196. Andersen, R.A., Snyder, L.H., Bradley, D.C., and Xing, J. (1997). Multimodal representation of space in the posterior parietal cortex and its use in planning movements. Annual Review of Neuroscience, 20, 303–330. Anzola, G.P., Bertoloni, G., Buchtel, H.A., and Rizzolatti, G. (1977). Spatial compatibility and anatomical factors in simple and choice-reaction time. Neuropsychologia, 15, 295–301. Bahrick, L.E. and Watson, J.S. (1985). Detection of intermodal proprioceptive–visual contingency as a potential basis of self-perception in infancy. Developmental Psychology, 21, 963–973. Balint, R. (1909). Seelenlähmung des ‘Schauens’, optische Ataxie und räumliche Störung der Aufmerksamkeit. Monatsschrift für Psychiatrische Neurologie, 25, 51–81. Batista, A.P., Buneo, C.A., Snyder, L.H., and Andersen, R.A. (1999). Reach plans in eye-centered coordinates. Science, 285, 257–260. Benedetti, F. (1985). Processing of tactile spatial information with crossed 1ngers. Journal of Experimental Psychology: Human Perception and Performance, 11, 517–525. Benedetti, F. (1988). Exploration of a rod with crossed 1ngers. Perception and Psychophysics, 44, 281–284. Benedetti, F. (1991). Reorganization of tactile perception following the simulated amputation of one 1nger. Perception, 20, 687–692. Bioulac, B., Burbaud, P., and Varoqueaux, D. (1995). Activity of area 5 neurons in monkeys during arm movements: Effects of dentate nucleus lesion and motor cortex ablation. Neuroscience Letters, 192, 189–192. Botvinick, M. and Cohen, J.D. (1998). Rubber hand ‘feels’ what eye sees. Nature, 391, 756. Burgess, P.R., Wei, J.Y., Clark, F.J., and Simon, J. (1982). Signalling of kinesthetic information by peripheral sensory receptors. Annual Review of Neuroscience, 5, 171–187. Chief1, S., Allport, D.A., and Woodin, M. (1999). Hand-centred coding of target location in visuo-spatial working memory. Neuropsychologia, 37, 495–502. Colby, C.L. and Duhamel, J.R. (1991). Heterogeneity of extrastriate visual areas and multiple parietal areas in the macaque monkey. Neuropsychologia, 29, 517–538. Craske, B., Kenny, F.T., and Keith, D. (1984). Modifying an underlying component of perceived arm length: Adaptation of tactile location induced by spatial discordance. Journal of Experimental Psychology: Human Perception and Performance, 10, 307–317. Critchley, M. (1953). The parietal lobes. New York: Hafner. De Renzi, E. (1982). Disorders of space exploration and cognition. New York: Wiley. Desmurget, M., Rossetti, Y., Jordan, M., Meckler, C., and Prablanc, C. (1997). Viewing hand prior to movement improves accuracy of pointing performed toward unseen contralateral hand. Experimental Brain Research, 115, 180–186. Desmurget, M., Rossetti, Y., Prablanc, C., Stelmach, G.E., and Jeannerod, M. (1995). Representation of hand position prior to movement and motor variability. Canadian Journal of Physiological Pharmacology, 73, 262–272. di Pellegrino, G., Ladavas, E., and Farne, A. (1997). Seeing where your hands are. Nature, 388, 730. DiZio, P., Lathan, C.E., and Lackner, J.R. (1993). The role of brachial muscle spindle signals in assignment of visual direction. Journal of Neurophysiology, 70, 1578–1584. Driver, J. and Grossenbacher, P.G. (1996). Multimodal spatial constraints on tactile selective attention. In T. Inui and J.L. McClelland (Eds.), Attention and performance, XVI: Information integration in perception and communication, pp. 209–235. Cambridge, MA: MIT Press. Driver, J. and Spence, C. (1998). Crossmodal attention. Current Opinion in Neurobiology, 8, 245–253. Driver, J., Mattingley, J.B., Rorden, C., and Davis, G. (1997). Extinction as a paradigm measure of attentional bias and restricted capacity following brain injury. In P. Thier and H.-O. Karnath (Eds.), Parietal lobe contributions to orientation in 3D space, pp. 401–429. Berlin: Springer-Verlag.

153


154


Duffy, F.H. and Burch1el, J.L. (1971). Somatosensory system: Organizational hierarchy from single units in monkey area 5. Science, 172, 273–275. Ettlinger, G. and Wegener, J. (1958). Somaesthetic alternation, discrimination and orientation after frontal and parietal lesions in monkeys. Quarterly Journal of Experimental Psychology, 10, 177–186. Ferraina, S., Johnson, P.B., Garasto, M.R., Battaglia-Mayer, A., Ercolani L., Bianchi L., Ferraresi, P., Lacquaniti, F., and Caminiti, R. (1997). Combination of hand and gaze signals during reaching: Activity in parietal area 7m of the monkey. Journal of Neurophysiology, 77, 1034–1038. Ferrier, D. and Yeo, G.F. (1884). A record of the experiments on the effects of lesions of different regions of the cerebral hemispheres. Philosophical Transactions of The Royal Society, London, 175, 479–564. Fogassi, L., Gallese, V., di Pellegrino, G., Fadiga, L., Gentilucci, M., Luppino, M., Pedotti, A., and Rizzolatti, G. (1992). Space coding by premotor cortex. Experimental Brain Research, 89, 686–690. Fogassi, L., Gallese, V., Fadiga, L., Luppino, G., Matelli, M., and Rizzolatti, G. (1996). Coding of peripersonal space in inferior premotor cortex (area F4). Journal of Neurophysiology, 76, 141–157. Gallagher, S. (1986). Body image and body schema: A conceptual clari1cation. Journal of Mind and Behavior, 7, 541–554. Gallese et al., this volume, Chapter 17. Galletti, C., Battaglini, P.P., and Fattori, P. (1995). Eye position in2uence on the parieto-occipital area PO (V6) of the macaque monkey. European Journal of Neuroscience, 7, 2486–2501. Gentilucci, M., Scandolara, C., Pigarev, I.N., and Rizzolatti, G. (1983). Visual responses in the postarcuate cortex (area 6) of the monkey that are independent of eye position. Experimental Brain Research, 50, 464–468. Gentilucci, M., Fogassi, L., Luppino, G., Matelli, M., Camarda, R., and Rizzolatti, G. (1988). Functional organization of inferior area 6 in the macaque monkey. I. Somatotopy and the control of proximal movements. Experimental Brain Research, 71, 475–490. Georgopoulos, A.P. and Massey, J.T. (1985). Static versus dynamic effects in motor cortex and area 5: Comparison during movement time. Behavioral Brain Research, 18, 159–166. Ghez, C., Gordon, J., Ghilardi, M.F., and Sainburg, R. (1995). Contributions of vision and proprioception to accuracy in limb movements. In M.S. Gazzaniga (Ed.), The cognitive neurosciences, pp. 549–564. Cambridge, MA: MIT Press. Goodale, M.A., Meenan, J.P., Bülthoff, H., Nicolle, D.A., Murphy, K.J., and Racicot, C.I. (1994). Separate neural pathways for the visual analysis of object shape in perception and prehension. Current Biology, 4, 604–610. Gordon, J., Ghilardi, M.F., and Ghez, C. (1994). Accuracy of planar reaching movements. I. Independence of direction and extent variability. Experimental Brain Research, 99, 97–111. Graziano, M.S.A. (1999). Where is my arm? The relative role of vision and proprioception in the neuronal representation of limb position. Proceedings of the National Academy of Science USA, 96, 10418–10421. Graziano, M.S.A., Cooke, D.F., and Taylor, C.S.R. (2000). Coding the location of the arm by sight. Science, 290, 1782–1786. Graziano, M.S.A. and Gandhi, S. (2000). Location of the polysensory zone in the precentral gyrus of monkeys. Experimental Brain Research, 135, 259–266. Graziano, M.S.A. and Gross, C.G. (1998). Spatial maps for the control of movement. Current Opinion in Neurobiology, 8, 195–201. Graziano, M.S.A., Yap, G.S., and Gross, C.G. (1994). Coding of visual space by premotor neurons. Science, 266, 1054–1057. Graziano, M.S.A., Hu, X.T., and Gross, C.G. (1997a). Visuo-spatial properties of ventral premotor cortex. Journal of Neurophysiology, 77, 2268–2292. Graziano M.S.A., Hu, X.T., and Gross, C.G. (1997b). Coding the locations of objects in the dark. Science, 277, 239–241. Graziano, M.S.A., Reiss, L.A.J., and Gross, C.G. (1999). A neuronal representation of the location of nearby sounds. Nature, 397, 428–430. Gross, C.G. (1998). Brain, vision, memory: Tales in the history of Neuroscience. Cambridge, MA: MIT Press. Harris, C.S. (1965). Perceptual adaptation to inverted, reversed, and displaced vision. Psychological Review, 72, 419–444. Hay, J.C., Pick, H.L., and Ikeda, K. (1965). Visual capture produced by prism spectacles. Psychonomic Science, 2, 215–216.



Holmes, G. (1918). Disturbances of visual orientation. British Journal of Ophthalmology, 2, 449–516. Iggo, A. and Andres, K.H. (1982). Morphology of cutaneous receptors. Annual Review of Neuroscience, 5, 1–31. Jenkins, W.M., Merzenich, M.M., Ochs, M.T., Allard, T., and Guic-Robles, E. (1990). Functional reorganization of primary somatosensory cortex in adult owl monkeys after behaviorally controlled tactile stimulation. Journal of Neurophysiology, 63, 82–104. Johnson, P.B., Ferraina, S., Bianchi, L., and Caminiti, R. (1996). Cortical networks for visual reaching: Physiological and anatomical organization of the frontal and parietal lobe arm regions. Cerebral Cortex, 6, 102–119. Jones, E.G. and Powell, T.P.S. (1970). An anatomical study of converging sensory pathways within the cerebral cortex of the monkey. Brain, 93, 793–820. Jones, E.G., Coulter, J.D., and Hendry, S.H.C. (1978). Intracortical connectivity of architectonic 1elds in the somatic sensory, motor, and parietal cortex of monkeys. Journal of Comparative Neurology, 181, 291–348. Jordan, S. (1970). Ocular pursuit movement as a function of visual and proprioceptive stimulation. Vision Research, 10, 775–780. Kaas, J.H., Nelson, R., Sur, M., Lin, C.-S., and Merzenich, M.M. (1979). Multiple representations of the body within the primary somatosensory cortex of primates. Science, 204, 521–523. Kalaska, J.F. and Hyde, M.L. (1985). Area 4 and 5: Differences between the load-dependent discharge variability of cells during active postural 1xation. Experimental Brain Research, 59, 197–202. Kalaska, J.F., Caminiti, R., and Georgopoulos, A.P. (1983). Cortical mechanisms related to the direction of twodimensional arm movements: Relations in parietal area 5 and comparison with motor cortex. Experimental Brain Research, 51, 247–260. Kalaska, J.F., Cohen, D.A.D., Prud’homme, M., and Hyde, M.L. (1990). Parietal area 5 neuronal activity encodes movement kinematics, not movement dynamics. Experimental Brain Research, 80, 351–364. Kolb, B. and Whishaw, I.Q. (1990). Fundamentals of human neuropsychology (3rd edn). Freeman: New York. Lackner, J.R. (1988). Some proprioceptive in2uences on the perceptual representation of body shape and orientation. Brain, 111, 281–297. Lackner, J.R. and Taublieb, A.B. (1983). Reciprocal interactions between the position sense representations of the two forearms. Journal of Neuroscience, 3, 2280–2285. Lacquaniti, F., Guigon, E., Bianchi, L., Ferraina, S., and Caminiti, R. (1995). Representing spatial information for limb movement: Role of area 5 in the monkey. Cerebral Cortex, 5, 391–409. MacKay, W.A. and Crammond, D.J. (1987). Neuronal correlates in posterior parietal lobe of the expectation of events. Behavioral Brain Research, 24, 167–179. McIntyre, J., Stratta, F., and Lacquaniti, F. (1998). Short-term memory for reaching to visual targets: Psychophysical evidence for body-centered reference frames. Journal of Neuroscience, 18, 8423–8435. Meltzoff, A.N. and Moore, M.K. (1983). Newborn infants imitate adult facial gestures. Child Development, 54, 702–709. Moffet, A.M. and Ettlinger, G. (1970). Tactile discrimination performance in monkey: The effect of unilateral posterior parietal ablations. Cortex, 6, 47–67. Morgan, R. and Rochat, P. (1997). Intermodal calibration of the body in early infancy. Ecological Psychology, 9, 1–23. Moscovitch, M. and Behrmann, M. (1994). Coding of spatial information in the somatosensory system: Evidence from patients with neglect following parietal lobe damage. Journal of Cognitive Neuroscience, 6, 151–155. Mountcastle, V.B. (1984). Central nervous mechanisms in mechanoreceptive sensibility. In I. Darian-Smith (Ed.), Handbook of physiology, I/Vol. 3: Sensory processes, pp. 789–878. Bethesda, MA: American Physiological Society. Mountcastle, V.B. (1997). The columnar organization of the neocortex. Brain, 120, 701–722. Mountcastle, V.B., Lynch, J.C., Georgopoulos, A., Sakata, H., and Acuna, C. (1975). Posterior parietal association cortex of the monkey: command functions for operations within extrapersonal space. Journal of Neurophysiology, 38, 871–908. Murray, E.A. and Mishkin, M. (1984). Relative contributions of SII and area 5 to tactile discrimination in monkeys. Behavioral Brain Research, 11, 67–83. Mushiake, H., Tanatsugu, Y., and Tanji, J. (1997). Neuronal activity in the ventral part of premotor cortex during target-reach movement is modulated by direction of gaze. Journal of Neurophysiology, 78, 567–571.

155


156


Nakamura, K., Chung, H.H., Graziano, M.S.A., and Gross, C.G. (1999). A dynamic representation of eye position in the parieto-occipital sulcus. Journal of Neurophysiology, 81, 2374–2385. Newcombe, F. and Ratcliff, G. (1989). Disorders of visuo-spatial analysis. In F. Boller and J. Grafman (Eds.), Handbook of neuropsychology, pp. 333–356. New York: Elsevier. Pandya, D.N. and Kuypers, H.G.J.M. (1969). Cortico-cortical connections in the rhesus monkey. Brain Research, 13, 13–36. Pearson, R.C.A. and Powell, T.P.S. (1985). The projections of the primary somatosensory cortex upon area 5 in the monkey. Brain Research Reviews, 9, 89–107. Prablanc, C., Echallier, J.E., Jeannerod, M., and Komolis, E. (1979). Optimal pointing response of eye and hand motor systems in pointing to a visual target: II. Static and dynamic visual cues in the control of hand movement. Biological Cybernetics, 35, 183–187. Prablanc C., Pélisson, D., and Goodale M.A. (1986). Visual control of reaching movements without vision of the limb. I. Role of retinal feedback of target position in guiding the hand. Experimental Brain Research, 62, 293–302. Ramachandran, V.S. and Hirstein, W. (1998). The perception of phantom limbs. Brain, 121, 1603–1630. Redding, G.M. and Wallace, B. (1997). Adaptive spatial alignment. Mahwah, NJ: Erlbaum. Ridley, R.M. and Ettlinger, G. (1975). Tactile and visuo-spatial discrimination performance in the monkey: The effects of total and partial posterior parietal removals. Neuropsychologia, 13, 191–206. Rizzolatti, G., Scandolara, C., Matelli, M., and Gentilucci, M. (1981). Afferent properties of periarcuate neurons in macaque monkeys. II. Visual responses. Behavioral Brain Research, 2, 147–163. Robinson, D.L., Goldberg, M.E., and Stanton, G.B. (1978). Parietal association cortex in the primate: Sensory mechanisms and behavioral modulations. Journal of Neurophysiology, 41, 910–932. Rochat, P. (1998). Self-perception and action in infancy. Experimental Brain Research 123, 102–109. Roll, J.P., Roll, R., and Velay, J.-L. (1991). Proprioception as a link between body space and extra-personal space. In J. Paillard (Ed.), Brain and space, pp. 112–132. Oxford: Oxford University Press. Rossetti, Y., Desmurget, M., and Prablanc, C. (1995). Vectorial coding of movement: Vision, proprioception or both? Journal of Neurophysiology, 74, 457–463. Rossetti and Pisella, this volume, Chapter 4. Ruch, T.C., Fulton, J.F., and German, W.J. (1938). Sensory discrimination in monkey, chimpanzee and man after lesions of the parietal lobe. Archives of Neurological Psychiatry, 39, 919–938. Rushworth, M.F.S., Nixon, P.D., and Passingham, R.E. (1997a). Parietal cortex and movement. I. Movement selection and reaching. Experimental Brain Research, 117, 292–310. Rushworth, M.F.S., Nixon, P.D., and Passingham, R.E. (1997b). Parietal cortex and movement. II. Spatial representation. Experimental Brain Research, 117, 311–323. Saadah, E.S.M. and Melzack, R. (1994). Phantom limb experiences in congential limb-de1cient adults. Cortex, 30, 479–485. Sakata H. and Taira M. (1994). Parietal control of hand action. Current Opinion in Neurobiology, 4, 847–856. Sakata, H., Takaoka, Y., Kawarasaki, A., and Shibutani, H. (1973). Somatosensory properties of neurons in the superior parietal cortex (area 5) of the rhesus monkey. Brain Research, 64, 85–102. Seal, J., Gross, C., and Bioulac, B. (1982). Activity of neurons in area 5 during a simple arm movement in monkeys before and after deafferentation of the trained limb. Brain Research, 250, 229–243. Shiffrar and Pinto, this volume, Chapter 19. Snyder, L.H., Batista, A.P., and Andersen, R.A. (1997). Coding of intention in the posterior parietal cortex. Nature, 386, 167–170. Soechting, J.F. (1982). Does position sense at the elbow re2ect a sense of elbow joint angle or one of limb orientation? Brain Research, 248, 392–395. Soechting, J.F. and Flanders, M. (1989). Sensorimotor representations for pointing to targets in three-dimensional space. Journal of Neurophysiology, 62, 582–594. Stark, L. and Bridgeman, B. (1983). The role of corollary discharge in space constancy. Perception and Psychophysics, 34, 371–380. Strick, P.L. and Kim, C.C. (1978). Input to primate motor cortex from posterior parietal cortex (area 5). I. Demonstration by retrograde transport. Brain Research, 157, 325–330. Tipper, S.P., Lortie, C., and Baylis, G.C. (1992). Selective reaching: Evidence for action-centered attention. Journal of Experimental Psychology: Human Perception and Performance, 18, 891–905. Tipper, S.P., Lloyd, D., Shorland, B., Dancer, C., Howard, L.A., and McGlone, F. (1998). Vision in2uences tactile perception without proprioceptive orienting. NeuroReport, 9, 1741–1744.



Ungerleider, L.G. and Mishkin, M. (1982). Two cortical visual systems. In D. Ingle, M.A. Goodale, and R.J. Mans1eld (Eds.), Analysis of visual behavior, pp. 549–586. Cambridge, MA: MIT Press. Vogt, B.A. and Pandya, D.N. (1978). Cortico-cortical connections of somatic sensory cortex (areas 3, 1 and 2) in the rhesus monkey. Journal of Comparative Neurology, 177, 179–192. Wallace, R.J. (1971). S-R compatibility and the idea of response code. Journal of Experimental Psychology, 88, 354–360. Welch, R.B. (1986). Adaptation of space perception. In K.R. Boff, L. Kaufman and J.P. Thomas (Eds.), Handbook of perception and human performance, pp. 24.1–24.44. New York: Wiley. Wolpert, D.M., Ghahramani, Z., and Jordan, M. I. (1995). Are arm trajectories planned in kinematic or dynamic coordinates? An adaptation study. Experimental Brain Research, 103, 460–470.

157


7 Action planning affects spatial localization Jerome Scott Jordan, Sonja Stork, Lothar Knuf, Dirk Kerzel, and Jochen Müsseler Abstract. When observers are asked to indicate the 1nal position of a moving stimulus, their localizations are reliably displaced beyond the 1nal position, in the direction the stimulus was traveling just prior to its offset. Recent experiments indicate that these localization errors depend on whether or not observers track the moving stimulus with eye-movements. If they track, there is a localization error; if not, the error reduces to zero. The present series of experiments investigated whether localization error might be due, in part, to the binding of the moving stimulus in an action plan. Experiment 1 utilized circular stimulus trajectories, and the eye tracking/ no-tracking discrepancy revealed in previous studies was replicated. Experiment 2 required central 1xation by all observers, and either the computer program (i.e. induction) or a button press by the observer (i.e. intention) produced the stimulus offset. The localizations made in the Intention condition were further in the direction of the planned action effect than those made in the Induction condition. Experiment 3 demonstrated these differences to be due to the intention to stop the stimulus, not the button press. And Experiment 4 revealed that action planning has its binding effect on the localization error for a duration that extends beyond the actual moment of action execution. In light of these data, an approach to perception–action coupling is proposed in which spatial perception and spatially directed action are modeled, not as input and output, respectively, but rather, as synergistically coupled control systems.

When observers are asked to indicate the 1nal location of an apparently moving, or moving stimulus, the indicated location is reliably displaced beyond the 1nal location, in the direction the target was traveling just prior to its offset (Finke, Freyd, and Shyi 1986; Freyd and Finke 1984; Hubbard 1995). In addition, the magnitude and direction of the displacement varies in a manner that is consistent with the laws of physics (i.e. velocity, friction, gravity; Hubbard 1995). Accounts of these errors are often conceptualized in terms of representational momentum—the notion that the dynamics of the external environment have been internalized into the dynamics of cognitive representational systems. Given that internal representations, just as external events, have dynamic properties that cannot simply be brought to a halt upon stimulus offset, dynamic representational transformations are assumed to continue for some time following stimulus offset. It is the momentum of these representations that is assumed to underlie the resulting localization error. Implicit in this account of localization error is the assumption that the actions produced by observers during stimulus movement do not in2uence the processes underlying the error. In short, action processes and representational momentum processes are assumed to be independent, and the localization error is described as a post-perceptual cognitive phenomenon. Contrary to this assumed independence, the purpose of the present paper is to present a series of experiments that test whether or not the actions produced in relation to a moving stimulus contribute to the spatial distortion manifested in the localization error. These experiments are motivated by the following: (1) data that indicate the localization error may, in part, be due to the action planning required to maintain an ongoing relationship between action and stimulus motion (i.e. action control), and (2) data that indicate that perception and action-planning share common mechanisms (i.e. common neural


Action planning affects spatial localization

mediation). Collectively, these data imply that the very act of planning an action in relation to a stimulus event serves to transform the processes underlying perceptual mappings of that stimulus event. In short, it implies that action planning in2uences the localization error.

7.1 Action control and localization error In representational momentum paradigms, observers are free to move their eyes. In fact, in most experiments no instruction is given in this regard, and it is assumed that eye-movements used to pursue and track the target do not contribute to the localization error. It has been demonstrated, however, that the eyes continue to drift in the direction of target motion if a pursued target, travelling on a linear trajectory, suddenly vanishes (Mitrani and Dimitrov 1978), and the magnitude of such drift varies directly with tracking velocity (Mitrani, Dimitrov, Yakimoff, and Mateeff 1979). In addition, static stimuli presented in the periphery are localized closer toward the fovea than they actually are ( foveal bias; e.g. Müsseler, Van der Heijden, Mahmud, Deubel, and Ertsey 1999; O’Regan 1984; Osaka 1977; Van der Heijden, Müsseler, and Bridgeman 1999). In light of these data, it may be the case that when a moving target suddenly disappears, the eyes overshoot the 1nal position of the stimulus, such that the fovea is shifted into the direction of motion. Subsequently, the foveal bias inherent in static localizations, coupled with the changing position of the fovea due to overshoot, causes the 1nal position of the target to be localized in the direction of the fovea’s motion (i.e. in the direction of the target’s motion). In short, it may be the case that the localization error is related to eye-movement control. To test this idea, Kerzel, Jordan, and Müsseler (in press) conducted a representational momentum experiment in which they asked observers to localize the 1nal position of a moving stimulus. Unlike other representational momentum experiments, however, they devised a condition in which observers were instructed to 1xate a stationary 1xation point during the presentation of the moving stimulus. This instruction, of course, prevented observers from making the smooth-pursuit movements observers normally make during such tasks. The results are depicted in Fig. 7.1. In the tracking condition, in which observers were allowed to track the moving stimulus, the traditional representational momentum effect was obtained. Localizations were displaced beyond the vanishing point, in the direction of stimulus motion, and the magnitude of the localization error varied directly with the velocity of the moving stimulus. In the 1xation condition, however, there was no displacement in the direction of stimulus motion. There was vertical displacement, probably due to the retinal eccentricity of the vanishing point (i.e. the 1xation stimulus was located 2° below the trajectory of the moving stimulus), but there was no horizontal localization error whatsoever. These data strongly imply that the localization errors reported in previous representational momentum experiments may have been due, in part, to the control of the eye movements necessary to track the moving stimulus. To be sure, arguments against an eye-movement account have been posed on many occasions (see Kerzel et al., in press, for a thorough review of these arguments). These arguments tend to treat the moving eye as a moving camera, however, and they do so by downplaying the fact that oculomotor tracking is a controlled action. Given the data of Kerzel et al., it seems this latter point is rather central to the localization error, and really cannot be downplayed. Oculomotor control requires planning, and this planning must (1) take into account anticipated future locations of the moving stimulus, and (2) be generated continuously in order to effectively control eye–target relationships. In light of these demands on eye-movement control, it may be the

159


160


Fig. 7.1 Mislocalization as a function of instruction (pursuit vs. 1xation) and velocity. The dark bars represent the Fixation condition, and the light bars, the Pursuit condition. Error bars represent standard errors between participants. Panel A: Positive values indicate errors in the direction of movement, negative values errors opposite to the direction of movement. Panel B: Positive values indicate errors above the 1nal position, negative values errors below the 1nal position.

case that the localization error is more due to momentum derived from action control than momentum derived from action-independent post-perceptual representations.

7.2 Action planning and perceptual mapping Another challenge to the idea that localization errors are action independent derives from data that reveal rather tight functional couplings between the planning aspect of action control and shifts in spatial perception. Classic research in visual attention, for example, indicates that roughly 50–100 ms after the presentation of a saccadic target, the threshold for the detection of events at the



target’s position is reduced (Bachmann 1999; Klein 1988; Posner 1980; Posner and Cohen 1984; Schneider and Deubel, this volume, Chapter 30; Wolff 1999). Such pre-saccadic shifts in detectibility thresholds constitute shifts in the spatial content of perception that are associated with the planning of an action. Some researchers even argue that these shifts constitute a necessary pre-condition of saccadic control (Rizollatti, Riggio, Dascola, and Umiltà, 1987; Wolff 1999). Further evidence of planning-perception coupling comes from experiments in which observers are asked to make judgments about the perceived location of a stimulus presented during the production of an action. Dassonville (1995), for example, asked observers to move their arm through the dark and localize the point at which their moving 1nger received a vibrotactile stimulus. Observers tended to localize the stimulus at locations beyond the point of stimulation. In other words, observers perceived the stimulus at locations to which they were planning to move their hand at the moment the stimulus was presented. Collectively, these data indicate a rather tight functional coupling between action planning and perceptual space. To be sure, this idea is not completely new. Both philosophers and psychologists have argued that actions are planned in terms of the distal effects they are to produce (i.e. in terms of distal perceptual space). Harless, for example (see Hommel 1998), referred to intentions, or action plans, as Effektbilder (effect images). James (1890/1950, p. 501) said, ‘. . . an anticipatory image . . . is the only psychic state which introspection lets us discern as the forerunner of our voluntary acts.’ And Hershberger (1976, 1987, 1998), in an attempt to explicate the idea that actions are planned in terms of their distal effects, referred to action plans as ‘afference-copies’ in order to contrast them with von Holst and Mittelstaedt’s (1950) concept, ‘efference copy’. This idea has recently received a more formal theoretical/empirical treatment in what is known as the theory of Common Coding (Prinz 1992, 1997). Basically, this theory assumes that (1) actions are planned in terms of their distal consequences, and (2) the planning of an action necessarily recruits, or rather presses into service, neural transformations that also mediate the perception of those distal consequences. Empirical support for this idea derives from both neurophysiological and psychophysical research. Several neurophysiological 1ndings of the last decade, for example, point to populations of neurons that seem to mediate both sensitivity to, and production of, distal events (i.e. they appear to be involved in both perception and action planning, respectively). Examples include the ‘visual-and-motor neurons’ (e.g. Taira, Mine, Georgopoulos, Murata, and Sakata 1990) found in monkey parietal cortex, and the ‘mirror neurons’ (e.g. di Pellegrino, Fadiga, Fogassi, Gallese, and Rizzolatti 1992) located in the premotor cortex. Additional neurophysiological support for common coding derives from research on neural mechanisms that accomplish coactivation of distributed brain areas (Roelfsema, Engel, König, and Singer 1997). Psychophysical support derives from studies in which participants are asked to identify the spatial value (i.e. left or right) of an arrow stimulus that is brie2y presented while participants plan either a left or right button press (for an overview see Müsseler 1999; Müsseler and Wühr, this volume, Chapter 25). These studies reveal that observers are better able to identify (i.e. perceive) the direction of the arrow stimulus if it is presented alone, versus in the midst of an action plan (see also comparable 1ndings by De Jong 1993; De Jong and Sweet 1994; Joliœur 1999). Further, if the arrow is presented in the midst of an action plan (i.e. it is presented while observers are planning a right or left button press), its direction is better identi1ed if it is opposite that of the planned action (e.g. left-pointing stimulus arrow presented during the planning of a right keypress). Common coding asserts these effects occur because the spatial content of planning the right or left button press becomes bound in the action plan and is, thus, less available for mediating

161


162


perception of the arrow’s direction (i.e. right or left). In short, the spatial dimension of planning one event interferes with the perception of another. Though these studies address the spatial relationship between planning one event and perceiving another, they do not address whether or not an action-plan involving a particular stimulus in2uences the perceived spatial location of that stimulus. This is, of course, the issue being addressed by the present paper. If spatial perception and action planning share common mediation, the perceived location of a stimulus should depend on whether or not the stimulus is bound in an action plan. In short, the localization error may be due to the action-planning aspect of action control. The data of Kerzel et al. (in press) seem to address this issue—localization errors were found in the Pursuit condition but not in the Fixation condition. One might assume these differences in localization were due to the binding of the moving stimulus in an action plan in the Pursuit condition (i.e. ‘track the moving stimulus’). It is not clear, however, whether the localization error resulted exclusively from the planning of the eye movements. If localization is in2uenced by action planning per se then localizations should vary as a function of action plans, regardless of the effector speci1ed in the action plan. We devised a series of experiments to address this issue. Speci1cally, observers were asked to indicate the perceived 1nal location of a target that moved on a circular trajectory around a central 1xation point. Circular trajectories were utilized, as opposed to linear trajectories, in order to control for the retinal eccentricity of the point at which the stimulus disappeared. Experiment 1 constituted a replication of the Kerzel et al. experiment. In Experiment 2, the offset of the moving stimulus was produced by either an observer-initiated button press (i.e. the Intention condition), or the computer program (i.e. the Induction condition). These two conditions were designed to test whether the relationship between action planning and localization error is speci1c to oculomotor control, or extends to action control in general. Experiment 3 was a replication of Experiment 2, save for a cue condition in which observers were instructed to press a button in response to the onset of the moving stimulus. This experiment was devised to clarify whether any localization differences between the Induction and Intention conditions in Experiment 2 were due to the fact that participants pressed a button in the Intention condition, yet did not do so in the Induction condition. Finally, Experiment 4 was devised to test just how long action planning has its binding effect upon perceptual space. To test this, we ran three versions of the Intention condition, each of which was programmed to produce a different degree of delay between the observer’s button press and the actual offset of the moving stimulus.

7.3 Experiment 1: oculomotor action plans The purpose of Experiment 1 was to determine whether a difference in oculomotor action plans (i.e. 1xation versus tracking), relative to the moving target, would produce differences in localization scores regarding the target’s 1nal position. This constituted a replication of Kerzel et al. (in press), save for the use of circular versus linear target trajectories.

7.3.1 Method 7.3.1.1 Participants Seven female and 1ve male students of the University of Munich who ranged in age from 21 to 32 years (mean age of 26 years) were paid to participate in the experiment. They reported normal or corrected-to-normal vision and were naive as to the purpose of the experiment.



7.3.1.2 Apparatus and stimuli The experiment was controlled by a Macintosh computer. The stimuli were presented on a 17 inch monitor with a refresh rate of 75 Hz and a luminance of approximately 40 cd/m2 with black-onwhite projection. The rest of the room was dimly lit. The participant’s head was placed on a chin and forehead rest 500 mm in front of the monitor. The moving stimulus was a dot, the size and luminance of which were 4.35 mm (0.5°)1 and 13 cd/ 2 m , respectively. On each trial, the dot traced out a trajectory that circled a 1xation cross at a radius of 48 mm (5.5°, cf. Fig. 7.2). The stimulus movement was induced by shifting the dot 0.54° or 2.15° clockwise with every vertical retrace of the monitor (13 ms per frame), resulting in two possible tangential velocities; 3.85 °/s (33.7 mm/s) and 15.4 °/s (134.6 mm/s). These target velocities were wellwithin the velocity range in which observers can accurately track a moving target (Robinson 1968). The movement started at the upper portion of the circle (in the range of 20° before and 20° after the 12 o’clock position). Movement length varied from 90° to 360° with absolute movement times of 2240 to 8960 ms for the slow velocity and 560 to 2250 ms for the fast velocity. An adjustment cursor, which was identical to the stimulus, appeared 500 ms after stimulus-offset at a random position on the circle. It could be moved either clockwise or counterclockwise along the circle’s edge by pressing a right or a left button, respectively. Each button press resulted in a 0.13° change in the adjustment cursor’s position. In order to accelerate the adjustment process, the adjustment cursor’s velocity accelerated if the button was pressed for a longer duration. Thus, a complete circle required approximately 1500 ms. Buttons were mounted on a 2at board in front of the participant. 7.3.1.3 Design and procedure The four combinations of two instructions (pursuit eye movement and 1xation) and two velocities were presented blockwise. The order was counterbalanced between participants. In the Pursuit condition, participants were instructed to follow the stimulus with their eyes until it vanished, while in the Fixation condition they were instructed to 1xate the 1xation cross during the presentation of the moving stimulus. Participants experienced 24 repetitions of each cell of the 2 × 2 within-subject design (i.e. 96 trials overall). The experiment lasted approximately 30 min, including training trials and short breaks.

Fig. 7.2 Stimulus con1guration used in the present experiments. The moving stimulus circled the central 1xation cross at a radius of 5.5°.

163


164


7.3.2 Results and discussion The localization score on every trial was computed as the difference between the computer-indicated and participant-indicated 1nal position of the stimulus with respect to movement direction. Positive values indicate localizations beyond the target’s 1nal position. Mean localization errors were computed separately for every participant and each condition. A 2 × 2 repeated measures analysis of variance (ANOVA) with the factors instruction (pursuit vs. 1xation) and velocity (3.75°/s vs. 15.4°/s) revealed a signi1cant difference due to instruction, F(1, 11) = 69.08, MSE = 8.34, p < 0.001. Figure 7.3 depicts these effects. In the Fixation condition, a post-hoc Scheffè test revealed signi1cant negative localization errors (15.4°/s: –4.95 mm, p < 0.01; 3.75°/s: –3.59 mm, p < .05), while in the Pursuit condition, in combination with the fast velocity, there was a signi1cant positive localization error of 4.20 mm, p < 0.01. Moreover, there was a signi1cant interaction between instruction and velocity, F(1, 11) = 9.91, MSE = 6.01, p = 0.009. The amount of error increased in both directions with faster velocity. Basically, Experiment 1 replicated the results from Kerzel et al. (in press). As can be seen in Fig. 7.3, the moving target’s vanishing point was localized signi1cantly further in the direction of target motion in the Pursuit versus the Fixation condition. In addition, faster-moving stimuli were localized further in the direction of target motion than slower-moving stimuli in the Pursuit condition, but not in the Fixation condition. However, in contrast to the results of Kerzel et al. (in press) the Fixation condition revealed a reliable negative localization error (i.e. an error in the direction opposite the movement direction of the target). Negative localization errors have been previously reported in tasks requiring localization of either (1) the initial target position (Actis Grosso, Stucchi, and Vicario 1996; Thornton 2001) or (2)—with an accompanying 2ash—the 1nal position (Müsseler, Stork, and Kerzel 2001). A possible explanation of this negative error is that, in the Fixation

Fig. 7.3 Mislocalization as a function of instruction (pursuit vs. 1xation) and velocity. The dark bars represent the Fixation condition, and the light bars, the Pursuit condition. Positive values indicate errors in the direction of movement, negative values errors opposite to the direction of movement. Error bars represent standard errors between participants.



condition, retinal stimulation during one refresh rate overlaps with the stimulation of the previous refresh rate(s). As a consequence, it is possible for stimulation to build up, simply as a function of the stroboscopic nature of stimulus presentation on a computer screen. Summation of stimulus information (cf. also the Bunson–Roscoe law) caused by stimulation during successive frames may occur at all positions on the stimulus trajectory, save the 1nal position. Given such summation, it may be the case that stimulation is less pronounced and consequently more often missed at the 1nal position. Preliminary results from our laboratory support this idea. In the present context, however, we are less concerned with negative localization error than we are with the differences in localization error between the Fixation and the Pursuit conditions. In short, the data indicate that the action control required in the pursuit condition gave rise to localization errors that were further in the direction of stimulus motion than those obtained in the Fixation condition. Experiment 2 was devised to determine whether or not it was the action planning aspect of action control that gave rise to these differences.

7.4 Experiment 2: effector-independent effects of action planning Given the results of the previous experiment, Experiment 2 was designed to determine whether the pattern of localizations is speci1c to oculomotor control. To test this, we repeated Experiment 1, and varied the action plans participants were to generate. In the Intention condition, participants 1xated the central 1xation cross throughout the presentation of the moving stimulus, yet were instructed to stop the stimulus’ motion via a button press. In the Induction condition, participants 1xated the central 1xation cross throughout the presentation of the moving stimulus, and the offset of the stimulus was caused by the computer program, not the participant. This is similar to the Fixation condition of Experiment 1. If the act of binding a stimulus in an action plan contributed to the localization differences revealed in Experiment 1, then localizations made in the Intention condition should differ from those made in the Induction condition. This is because the action plan generated in the Intention condition (i.e. stop the stimulus’ motion) necessarily entails the anticipated distal action effect, that is, the location of the target at the moment the button is pressed. This is not the case in the Induction condition. Due to common mediation, this anticipatory aspect of the Intention condition should alter perceptual space in an intention-relative manner, thus producing localization differences between the two conditions. To be sure, the pattern of results in Experiments 1 and 2 are not expected to be exactly the same. This is because of differences in the action plans required by the two experiments. In the Pursuit condition of Experiment 1, the task was to track the target via eye movements. In the Intention condition of the present experiment, the task was to stop the stimulus’ motion. Both tasks require action planning that takes into account anticipated future locations of the moving stimulus, but they differ in terms of the anticipation required. The Pursuit condition required continuous anticipation due to the need for continuous tracking. The Intention condition, on the other hand, only required anticipation and planning regarding the stimulus’ 1nal location (i.e. vanishing point). Thus, while the continuous anticipation required by the Pursuit condition gave rise to positive localization error, the rather discrete anticipation required of the Intention condition is not expected to give rise to positive localization error (i.e. the localizations should be more accurate).

165


166


7.4.1 Method 7.4.1.1 Participants Six female and four male students of the University of Munich who ranged in age from 21 to 34 years (mean age of 25.4 years) were paid to participate in the experiment. 7.4.1.2 Apparatus and stimuli All was the same as in Experiment 1, save for the target velocities and the presence of a response button. Given that the present experiment required participants to press a button to stop the stimulus motion in the Intention condition, we utilized the faster of the two velocities from Experiment 1 (i.e. 15.4°/s) as well as a new, faster velocity (30.8°/s, i.e. 269.3 mm/s). The faster velocity increased the salience of the action/effect relationship. In other words, faster targets approximated the more ecologically valid, natural relationship that exists between arm/1nger movements and moving visual targets (e.g. swatting at a 2y, de2ecting a rapidly moving projectile, catching a 2y ball). Stimulus movement started at a random position on the circular orbit and stimulus offset was either controlled by the computer or by a button press of the participant. Movement length varied from 90° to 360° with absolute movement times of 560 to 2240 ms for the slow velocity and 280 to 1120 ms for the fast velocity. 7.4.1.3 Design The two task conditions (Intention vs. Induction) were presented blockwise. Half the participants started with the Induction condition. The two velocity conditions varied randomly within each block. There were 10 blocks of 10 trials. Thus, each participant experienced 25 repetitions of each of the four unique experimental conditions. 7.4.1.4 Procedure In the Induction condition the offset of the moving stimulus was produced by the experimental program. In the Intention condition the participant was instructed to press a button with the right index 1nger in order to stop the target’s motion. Participants were instructed to stop the movement at an arbitrary point after the stimulus had moved 90°, yet before it had moved 360°. They were also instructed that over trials they should distribute the stop positions between 90° and 360° and should not choose recurrent salient positions (e.g. the 6 o’clock position). If a participant pressed the button too early or too late, an error message was presented, and the trial was repeated immediately. 7.4.2 Results and discussion Mean localization scores were computed separately for every participant and each condition. These scores were then entered into a 2 × 2 ANOVA with the factors instruction (Intention vs. Induction) and velocity (15.4°/s vs. 30.8°/s). As can be seen in Fig. 7.4, there was a signi1cant main effect due to instruction, F(1, 9) = 5.68, MSE = 8.75, p = 0.041, and a tendency towards an interaction F(1, 9) = 3.86, MSE = 11.72, p = 0.081. The Scheffé test revealed that the signi1cant effect of instruction was due to differences between the Fast-Induction condition and the Fast-Intention condition ( p < 0.05). The data are consistent with an action-planning account. The localization errors made in the Intention condition were signi1cantly further in the direction of stimulus motion than those made in the Induction condition; that is they were more accurate especially at the faster velocity. These data



Fig. 7.4 Mislocalization as a function of instruction (intention vs. induction) and velocity. The dark bars represent the Induction condition, and the light bars, the Intention condition. indicate that action planning exerted an in2uence on the perceived location of the stimulus even though the action used to attain the planned effect was a button press, not an eye movement. This 1nding is also telling, in that the spatial location of the action (i.e. the location of the button press) and the location of the event speci1ed in the action plan (i.e. the position of the moving stimulus) did not spatially overlap—the two were located at different spatial locations. These data support the assertion of the action-planning account that due to the common mediation of perception and action planning, the planning of an action recruits the transformations to be used in the perception of the planned distal event. In the present case, the binding of the target’s 1nal location within a buttonpress action plan (i.e. ‘stop the stimulus motion by pressing the button’) shifted the localization in the direction of the intended effect.

7.5 Experiment 3: action-independent effects of action planning Given the 1ndings thus far, one might claim that the localization differences revealed in Experiment 2 were due to the fact that participants produced a button-press action in the Intention condition, but not in the Induction condition. If it was action planning per se, not only action production, that was responsible for the differences, then different action plans that utilize the same action should produce different localization patterns. We devised an experiment to test this idea. In one condition (the Cue condition) participants pressed a button in response to the onset of the moving stimulus, while in another (the Intention condition) participants pressed a button in order to produce stimulus offset. If the action plan is truly critical to the localization error, then localizations made in the two conditions should not be the same. In the Cue condition, the action plan refers to the initial stimulus location, while in the Intention condition, it refers to the 1nal stimulus location. In both conditions, the offset of the moving stimulus was produced by the participant’s action (i.e. the button press), but in the Cue condition the 1nal position of the moving stimulus did not have to be bound in the participant’s button-press action plan. Rather, given the instructions, all a participant had to do was react to the onset of the moving stimulus. In short, the moving stimulus constituted an action cue, and it

167


168


was the initial position of the moving stimulus, not the 1nal position, that was relevant to, and thus, potentially bound within, the participant’s button-press action plan. If there are differences between the Cue condition and the Intention condition, however, one will not know if they are due to action planning or trajectory length, since the stimulus, simply due to instructions, traces out a larger trajectory in the Intention condition. Thus, a variant of the Induction condition of Experiment 2 was utilized in which the length of the trajectory was limited to a quarter circle. Trajectory length, therefore, was similar to that in the Cue condition, but the offset of the stimulus was produced by the computer program, not the participant.

7.5.1 Method 7.5.1.1 Participants Eight female and four male students of the University of Munich who ranged in age from 20 to 41 years (mean age of 27.2 years) were paid to participate in the experiment. 7.5.1.2 Apparatus and stimuli Stimulus presentation was the same as in Experiment 2, with the following exceptions. A Cue condition was added in which participants were instructed to stop the movement of the stimulus, via a button press, as soon as the moving stimulus appeared. The Intention condition remained unchanged. In order to control for trajectory length, a variant of the Induction condition was utilized in which the length of the trajectory was limited to a quarter circle. 7.5.1.3 Design and procedure The three task conditions (Intention vs. Induction vs. Cue) were presented blockwise and their order was counterbalanced between participants. Stimulus velocities were the same as in Experiment 2, and were randomized within blocks. Overall, participants experienced 150 trials. The procedure was the same as Experiment 2. The experiment lasted approximately 35 min, including a training block.

7.5.2 Results and discussion One participant had to be excluded from further analysis because, following the experiment, it was discovered she had followed the instructions incorrectly. Two separate ANOVAs were conducted on the Cue vs. Intention and Induction vs. Intention data. The Cue vs. Intention analysis revealed only a signi1cant main effect of instruction F(1, 10) = 5.95, MSE = 6.75, p = 0.035, that is, judgments were more accurate in the intention condition (Fig. 7.5). The Induction vs. Intention analysis revealed a signi1cant interaction, F(1, 10) = 6.81, MSE = 3.47, p = 0.026. Follow-up Scheffé tests revealed the interaction to be due to differences between the Induction-fast and slow conditions (p < .01) and the Induction-fast and Intention-fast conditions (p < .05). This constitutes, save for the predictability of the trajectory length in the Induction condition, a replication of Experiment 2. Localizations made in the Intention condition were more accurate than those made in the Cue condition. If this was due solely to trajectory length, then the Cue and Induction conditions should have expressed a similar relationship to the Intention condition. This was not the case. While the Intention/ Cue analysis revealed only an effect of instruction, the Intention/Induction analysis revealed an interaction. The differences between these patterns indicate that something other than trajectory length was



Fig. 7.5 Mislocalization as a function of instruction and velocity for the Cue–Intention data (top) and the Induction–Intention data (bottom). In the Intention–Cue graph, dotted bars represent the Cue condition, and light bars, the Intention condition. In the Induction–Intention graph, dark bars represent the Induction condition, and light bars, the Intention condition. responsible for the differences between the Intention and Cue conditions. Speci1cally, in the Induction condition, the moving stimulus was not bound in an action plan. Since the trajectory length was constant (i.e. a quarter turn) and the task conditions were blocked, the 1nal position of the moving stimulus, though not bound, may have nonetheless become predictable due to repetition, especially in the slower conditions. This may account for the differences between the Induction–fast and Induction– slow conditions. In the Cue condition, however, stimulus duration was similar to that in the Induction condition, but the initial position may have been bound in the button-press action plan. Thus, the perceived vanishing point appears to have been attracted to the location of the initial position. Collectively, these data support the following assertions: (1) the differences between the Intention and Cue conditions were not due to trajectory length, and (2) the localization differences revealed in Experiments 2 and 3 were due to differences in action planning, not action execution, per se.

169


170


7.6 Experiment 4: the duration of action-relative binding In all experiments reported so far, localizations of stimuli bound in an action plan were attracted to the location of the planned effect. In the present experiment we attempted to assess the duration of such binding. We did so by presenting observers three different versions of the Intention condition, each of which was programmed to produce a small degree of delay between the observer’s button press and the actual offset of the moving stimulus. If the action plan loses its binding impact on perceptual space immediately upon action completion, localizations of stimuli having delayed offsets should not be attracted to the intended location, and there is no reason to expect the error in such localizations should vary with changes in the offset delay. If the action plan still has a binding impact on perceptual space at the moment of the delayed stimulus offset, however, the localizations should be attracted to the intended offset location, and the localization errors should vary inversely with offset delays. This is because the localizations of stimuli entailing longer offset delays would entail the growing discrepancy between intended and actual offset location.

7.6.1 Method 7.6.1.1 Participants Six female and four male students from the University of Munich, ranging in age from 20 to 37 years (mean age of 27.5 years), were paid to participate in the experiment. 7.6.1.2 Apparatus and stimuli Stimulus presentation in the Induction condition was the same as in Experiment 2. In the Delay conditions, stimulus offset was brought about by a participant-produced button press, as in Experiments 2 and 3, plus a pre-programmed, post button-press delay. There were three levels of delay: 0, 53 ms (4 frames), and 107 ms (8 frames). 7.6.1.3 Design and procedure The two task conditions (Induction vs. Delay) were presented in blocks. Stimulus velocity varied randomly within both blocks, while delay also varied randomly within the delay block. Half the participants started with the Induction condition. The participants underwent 25 repetitions per cell of the 4 × 2 within-subjects design, for a total of 200 trials. The experiment lasted approximately 40 min, including training trials. The participants were not informed that delays were utilized in the present experiment. After the experiment participants were asked whether they had noticed the delays. 7.6.2 Results and discussion None of the participants reported noticing the different delay conditions. A tendency towards an instruction effect resulted from a 2 × 2 ANOVA with the factors instruction (induction vs. intention: delay 0) and velocity, F(1, 9) = 4.00, MSE = 14.44, p = 0.077. This result replicated the 1nding of the previous experiments. A 3 × 2 ANOVA with the factors delay (0, 53, vs. 107 ms) and velocity (15.4°/ s vs. 30.8°/s) revealed a signi1cant effect of delay, F(2, 18) = 7.40, MSE = 1.68, p = 0.005. As can be seen in Fig. 7.6, the localization error varied inversely with offset delay. Note, the displacement was



determined relative to the actual offset of the moving target (not the position intended by the button press); thus, decreasing localization values represents localization towards the intended stopping point. In sum, the data indicate that action planning has its binding impact on perceptual space for a duration that extends beyond the actual moment of action execution. Localization errors varied inversely with offset delays. This would not have been the case had the action plan lost its binding impact immediately upon action completion. Rather, the inverse relationship between localization error and offset delay indicates that the localizations were attracted toward the intended 1nal location at all levels of delay, thus causing the localization scores to decrease as the distance between the intended and actual offset location increased. To be sure, the methodology of the present experiment only extends the binding interval 107 ms beyond the moment of action execution. Thus, further research is needed to measure the size of the interval precisely.

7.7 General discussion When observers are asked to indicate the 1nal location of a moving stimulus, their localizations tend to be displaced beyond the actual 1nal location, in the direction of stimulus motion. Traditional accounts attribute this localization error to post-perceptual action-independent cognitive processes (Hubbard 1995). The present series of experiments tested this assumed independence between action processes and localization error because recent 1ndings indicate the error may be in2uenced by the action-planning processes involved in action control (Kerzel, Jordan, and Müsseler, in press). In addition, research indicates that action planning affects spatial localization because action planning is mediated by processes that also mediate the perception of stimulus location. In short, perception and action planning seem to share common mediation. This notion of common mediation has received formal theoretical/empirical treatment in what is known as the Theory of Common Coding (Prinz 1992, 1997). Speci1cally, the theory assumes the following: (1) actions are planned in terms of their intended distal effects, (2) action planning and perception, due to their inherently distal nature, share common neural mediation, and (3) action planning produces anticipatory recruitment (i.e. binding) of the transformations mediating the perception of the intended distal effect. If this

8

mislocalization in movement direction [mm]

6

induction

delay 1

delay 0

delay 2

4 2 0 –2 –4 –6 –8 15.4˚/s

30.8˚/s velocity

Fig. 7.6

Mislocalization as a function of instruction (intention vs. induction), velocity, and delay.

171


172


notion of common mediation is correct, then localization error is not independent of action processes. Rather, the perceived location of a stimulus should vary as a function of whether and how the stimulus is bound in an action plan. Localizations of stimulus location therefore, should be intention relative. We began our investigation with the same type of localization task used in traditional representational momentum paradigms, save for our use of circular versus linear stimulus trajectories. In representational momentum paradigms, observers usually track the target with their eyes and, in order to do this, anticipate the future positions of the target. Given the nature of oculomotor tracking (Mitrani and Dimitrov 1978; Mitrani et al. 1979), we assumed that the forward bias observed in representational momentum experiments re2ects a tendency to localize the target toward the anticipated locations inherent in the eye-movement action plan. Consequently, we predicted the bias would disappear in a 1xation condition in which observers were not allowed to pursue the target via eye movements. Indeed, localizations made in the Pursuit condition of Experiment 1 were further in the direction of stimulus motion than those made in the Fixation condition, and the magnitude of this difference increased with increases in target velocity. Both results replicate a recent 1nding of Kerzel et al. (in press), save for the use of circular stimulus trajectories. In the subsequent experiments, eye movements were suppressed, and observers’ action plans did not require continuous anticipation of future positions of the moving target. Rather, the intended effect was the offset of the target, and it was accomplished via a button press. In this situation, we also expected localizations to be attracted toward the intended action effect. That is, localizations were expected to be biased toward the anticipated offset location. Indeed, in Experiment 2 localizations made in the Intention condition were further in the direction of the actual stopping position of the target (i.e. the location of the intended action effect) than those made in the Induction condition, even though (1) the action speci1ed in the Intention condition involved a button press (i.e. not an eye movement), and (2) the action was not directed toward the moving stimulus. In the Induction condition, displacement varied with velocity in the same manner as Experiment 1. These 1ndings indicate that at least a portion of the transformation of perceptual space associated with action control is due to action planning itself, and not to the speci1c effector speci1ed in the action plan. Experiment 3 was a replication of Experiment 2, save for a Cue condition in which participants were instructed to press a button in response to the onset of the moving stimulus. This experiment was devised to test whether differences in localization discovered in Experiment 2 were simply due to the fact that participants pressed a button in the Intention condition. If so, there should have been no differences between the Cue and Intention conditions. If, however, localizations made in Experiment 2 were due to action planning, then localizations made in the Intention condition should have been closer to the (intended) offset position than those made in the Cue condition. This is because, in the Intention condition, the 1nal position of the moving stimulus was relevant to the action plan (i.e. press the button to stop the stimulus) while in the Cue condition, the initial position was relevant (i.e. press the button in response to the onset of the stimulus). In fact, Experiment 3 revealed that the differences in localization observed in Experiment 2 were due to action planning, not action execution. Finally, in Experiment 4 we realized three versions of the Intention condition, each of which was programmed to produce a small degree of delay between the participant’s button press and the actual offset of the moving stimulus. Our hypotheses were as follows: if the target moves beyond the intended stopping position, and participants have a tendency to localize the target toward the intended stopping position, localization scores should decrease with increasing delay between



button press and target offset (compared to the zero delay condition) as a larger distance between the intended stopping position and the actual stopping position is traversed. This hypothesis was con1rmed. Of course, if action planning serves to bind perceptual space, then it should also lose its binding impact some time after the planned action has been executed. However, the 54 and 107 ms delays introduced in Experiment 4 were probably too short to unbind stimuli from action planning—a conclusion consistent with the observation that none of the subjects noticed the delays at all. The present 1ndings are consistent with the notion that action planning affects spatial localization. The action plans generated in all the experiments transformed perceptual space in an intentionrelative fashion. Given this anticipatory, action-relative aspect of spatial localization, it seems dif1cult to sustain the representational momentum account and its assertion that localization error is due to action-independent post-perceptual cognitive momentum. In Experiments 1, 2, and 3, the error either disappeared or reversed (i.e. became negative) when observers were asked to generate action plans that did not involve (i.e. were not related to) the final position of the moving stimulus (i.e. the Fixation, Induction, and Cue conditions, respectively). This pattern of 1ndings indicates that the error comes and goes as a function of an observer’s intentional stance relative to the stimulus. If the intention is to stop a moving stimulus, there is no localization error because localizations are attracted toward the intended offset location. If the intention is to track a stimulus, there is localization error because continuous tracking requires continuous anticipation of future locations of the stimulus, and these anticipated locations are bound in the action plan because the intended distal effect is to stay on target. In short, localization error is not just related to action planning, it is dependent on it. Localization error, therefore, does not seem to be a property of post-perceptual cognitive processes. It seems, rather to be a property of the type of relationship one is attempting to maintain with a stimulus. Given the present data and their support of the idea that localization error is dependent upon action planning, it seems dif1cult to sustain the functionally orthogonal, input–output approach to perception and action control that lies at the theoretical core of the representational momentum account. If perceptual processes and action-planning processes do share common mediation, the two, by de1nition, are dependent and, therefore, cannot be realistically modeled as constituting input- and output-control processes that are processed independently. Such problems do not arise in the Common Coding account, for the account asserts that action plans specify distal effects, not behavioral outputs. In addition, these distal effect plans are assumed to be processed via resources that are also used to perceive distal effects. As a result, spatial perception and action-control are not modeled as being functionally orthogonal. They are not assumed to be processed independently on opposite sides of the system. Instead, one might model them as being synergistically coupled. According to this approach (cf. also Jordan 1998, 1999), spatial perception and action planning, due to their common mediation, constitute a distal-effect system that allows one to both specify and detect distal events. This distal-effect system, however, although it is involved in action planning, is not responsible for the effector control required of an action. Its role, rather, is to constrain effector control systems toward the attainment of a speci1ed distal effect. This implies it should be possible for action planning to produce transformations of perceptual space, regardless of the effector speci1ed in the plan. This is exactly what happened in the present experiments. The action plans speci1ed in Experiments 1 and 2 produced intention-relative shifts in spatial perception despite the fact Experiment 1 required eye movemets while Experiment 2 required 1nger movements. Given this de-coupling of the systems underlying action planning and the systems underlying effector control, one can see how spatial perception and action control are synergistically coupled. As one

173


174


engages in effector control, one produces changes in body–environment relationships, and these changes feed back into the distal-effect system. This perceptual feedback then allows the distal-effect system to assess whether or not the speci1ed distal effect (i.e. the action plan) has been attained. By being able to both specify the distal states toward which effector control systems are constrained, as well as detect the changes in distal states produced by effector control, the distal-effect system can be said to constitute a distal-effect control system. And as a result of the simultaneous mutual in2uence of effector control systems and distal-effect control systems, the two can be said to be synergistically coupled. Given this approach, spatial perception and action control are not coupled in an orthogonal input– output fashion. Rather, they are nested control systems that are coupled synergistically. Action planning affects spatial localization therefore, because action planning and spatial perception share common mediation and, as a result, constitute aspects of the same distal-effect control system. This system, however, does not control distal effects by engaging in effector control. Rather, it does so by specifying the distal effects toward which effector control should be constrained, while simultaneously being sensitive to the changes in body–environment relationship produced by effector control. In conclusion, this notion of synergistically coupled nested control systems may explain why it is possible to de-couple (dissociate) perceptual space and behavioral space (Bridgeman 1999, this volume, Chapter 5; Hansen and Skavenski 1985; Prof1tt, Bhalla, Gossweiler, and Midgett 1995; Rossetti and Pisella, this volume, Chapter 4) but not perceptual space and action-planning space (Haggard, Aschersleben, Gehrke, and Prinz, this volume, Chapter 13; Hershberger and Jordan 1992; Jordan 1999; Rieser and Pick, this volume, Chapter 8; Viviani, this volume, Chapter 21). In the former, the two versions of space belong to functionally distinct, yet synergistically yoked control systems, while in the latter, they belong to the same system.

Acknowledgements This research was initiated and conducted while JSJ was a Fellow at the Max Planck Institute for Psychological Research (supported by the Alexander von Humboldt Foundation and Saint Xavier University). The experiments were supported by a grant from the Deutsche Forschungsge meinschaft (DFG As 79/3) to JM. The authors would like to thank Wayne A. Hershberger, John Rieser, Wolfgang Prinz, and an anonymous reviewer for their comments on an earlier version of the manuscript.

Note 1. If the description contains the degree scale only, the unit refers to the corresponding angle in respect to the circle! If scales giving both degree and millimeter are used, the units refer to the corresponding angle in respect to the eye.

References Actis Grosso, R., Stucchi, N., and Vicario, G.B. (1996). On the length of trajectories for moving dots. Paper presented at the Proceedings of the Twelfth Annual Meeting of the International Society for Psychophysics, Fechner Day 96, Padua, Italy.



Bachmann, T. (1999). Twelve spatiotemporal phenomena and one explanation. In G. Aschersleben, J. Müsseler, and T. Bachmann (Eds.), Cognitive contributions to the perception of spatial and temporal events, pp. 173–206. North Holland: Elsevier. Bridgeman, B. (1999). Separate representations of visual space for perception and visually guided behavior. In G. Aschersleben, J. Müsseler, and T. Bachmann (Eds.), Cognitive contributions to the perception of spatial and temporal events, pp. 3–13. North Holland: Elsevier. Bridgeman, B. (2001). Attention and visually guided behavior in distinct systems. In this volume, Chapter 5. Dassonville, P. (1995). Haptic localisation and the internal representation of the hand in space. Experimental Brain Research, 106, 434–448. De Jong, R. (1993). Multiple bottlenecks in overlapping task performance. Journal of Experimental Psychology: Human Perception and Performance, 19, 965–908. De Jong, R. and Sweet, J.B. (1994). Preparatory strategies in overlapping-task performance. Perception and Psychophysics, 55, 142–151. di Pellegrino, G., Fadiga, L., Fogassi, V., Gallese, V., and Rizzolatti, G. (1992). Understanding motor events: A neurophysiological study. Experimental Brain Research, 91, 176–180. Finke, R.A., Freyd, J.J., and Shyi, G.C.-W. (1986). Implied velocity and acceleration induce transformation of visual memory. Journal of Experimental Psychology: General, 115, 175–188. Freyd, J.J. and Finke, R.A. (1984). Representational momentum. Journal of Experimental Psychology: Learning, Memory, and Cognition, 10, 126–132. Haggard, P., Aschersleben, G., Gehrke, J., and Prinz, W. (2001). Action, binding, and awareness. In this volume, Chapter 13. Hansen, R. and Skavenski, A.A. (1985). Accuracy of spatial localizations near the time of saccadic eye movements. Vision Research, 25, 1077–1082. Hershberger, W. (1976). Afference copy, the closed-loop analogue of von Holst’s efference copy. Cybernetics Forum, 8, 97–102. Hershberger, W. (1987). Sacccadic eye movements and the perception of visual direction. Perception and Psychophysics, 41, 35–44. Hershberger, W.A. (1998). Control systems with a priori intentions register environmental disturbances a posteriori. In J.S. Jordan (Ed.), Systems theories and a priori aspects of perception, pp. 3–23. Amsterdam: Elsevier. Hershberger, W.A. and Jordan, J.S. (1992). Visual direction constancy: Perceiving the visual direction of perisaccadic 2ashes. In E. Chekaluk (Ed.), The role of eye movements in perceptual processes, pp. 1–43. Amsterdam: Elsevier. Hommel, B. (1998). Perceiving one’s own action—and what it leads to. In J.S. Jordan (Ed.), Systems theories and a priori aspects of perception, pp. 143–179. Amsterdam: Elsevier. Hubbard, T.L. (1995). Environmental invariants in the representation of motion: Implied dynamics and representational momentum, gravity, friction, and centripetal force. Psychonomic Bulletin and Review, 2, 322–338. James, W. (1890/1950). The principles of psychology, Vol. 2. New York: Henry Holt. Jolicœur, P. (1999). Dual-task interference and visual encoding. Journal of Experimental Psychology: Human Perception and Performance, 25, 296–616. Jordan, J.S. (1998). Recasting Dewey’s critique of the re2ex-arc concept via a theory of anticipatory consciousness: Implications for theories of perception. New Ideas in Psychology, 16(3), 165–187. Jordan, J.S. (1999). Cognition and spatial perception: Production of output or control of input? In G. Aschersleben, J. Müsseler, and T. Bachmann (Eds.), Cognitive contributions to the perception of spatial and temporal events, pp. 69–90. North Holland: Elsevier. Kerzel, D., Jordan, J.S., and Müsseler, J. (in press). The role of perceptual anticipation in the localization of the 1nal position of a moving target. Journal of Experimental Psychology: Human Perception and Performance. Klein, R. (1988). Inhibitory tagging system facilitates visual search. Nature, 334, 430–431. Mitrani, L. and Dimitrov, G. (1978). Pursuit eye movements of a disappearing moving target. Vision Research, 18, 537–539. Mitrani, L., Dimitrov, G., Yakimoff, N., and Mateeff, S. (1979). Oculomotor and perceptual localization during smooth eye movements. Vision Research, 19(5), 609–12. Müsseler, J. (1999). How independent from action control is perception? An event-coding account for more equally ranked crosstalks. In G. Aschersleben, J. Müsseler, and T. Bachmann (Eds.), Cognitive contributions to the perception of spatial and temporal events, pp. 121–147. Amsterdam: Elsevier. Müsseler, J. and Wühr, P. (2001). Response-evoked interference in visual encoding. In this volume, Chapter 25.

175


176


Müsseler, J., Van der Heijden, A.H.C., Mahmud, S.H., Deubel, H., and Ertsey, S. (1999). Relative mislocalization of brie2y presented stimuli in the retinal periphery. Perception and Psychophysics, 61, 1646–1661. Müsseler, J., Stork, S., and Kerzel, D. (2001). Comparing mislocalizations in movement direction: The Fröhlich effect, the 2ash-lag effect and the representational momentum effect (submitted for publication). O’Regan, J.K. (1984). Retinal versus extraretinal in2uences in 2ash localization during saccadic eye movements in the presence of a visible background. Perception and Psychophysics, 36, 1–14. Osaka, N. (1977). Effect of refraction on perceived locus of a target in the peripheral visual 1eld. Journal of Psychology, 95, 59–62. Posner, M.I. (1980). Orienting of attention. Quarterly Journal of Experimental Psychology, 32, 3–25. Posner, M.I. and Cohen, Y. (1984). Components of visual orienting. In H. Bouma and Bouwhuis (Eds.), Control of language processes: attention and performance, Vol. 10, pp. 531–556. Hillsdale, NJ: Erlbaum. Prinz, W. (1992). Why don’t we perceive our brain states? European Journal of Cognitive Psychology, 4(1), 1–20. Prinz, W. (1997). Perception and action planning. European Journal of Cognitive Psychology, 9, 129–154. Prof1tt, D., Bhalla, M., Gossweiler, R., and Midgett, J. (1995). Perceiving geographical slant. Psychonomic Bulletin and Review, 2(4), 409–428. Rieser, J. and Pick, H. (2001). The perception and representation of human locomotion. In this volume, Chapter 8. Rizzolatti, G., Riggio, L., Dascola, I., and Umiltà, C. (1987). Reorienting attention across the horizontal and vertical meridians: Evidence in favor of a premotor theory of attention. Special Issue: Selective visual attention. Neuropsychologia, 25, 31–40. Robinson, D.A. (1968). Eye movement control in primates. Science, 184, 1219–1224. Roelfsema, P.R., Engel, A.K., König, P., and Singer, W. (1997). Visuomotor integration is associated with zero time-lag synchronization among cortical areas. Nature, 385, 157–161. Rossetti, Y. and Pisella, L. (2001). Several ‘vision for action’ systems: A guide to dissociating and integrating dorsal and ventral functions. In this volume, Chapter 4. Schneider, W. and Deubel, H. (2001). Selection-for-perception and selection-for-spatial-motor-action are coupled by visual attention. In this volume, Chapter 30. Taira, M., Mine, S., Georgopoulos, A.P., Murata, A., and Sakata, H. (1990). Parietal cortex neurons of the monkey related to the visual guidance of hand movement. Experimental Brain Research, 83, 29–36. Thornton, I.M. (2001). The onset repulsion effect. (submitted for publication). Van der Heijden, A.H.C., Müsseler, J., and Bridgeman, B. (1999). On the perception of positions. In Aschersleben, G., Bachmann, T., and Müsseler, J. (Eds.). Cognitive contributions to the perception of spatial and temporal events, pp. 19–37. Amsterdam: Elsevier. Viviani, P. (2001). Motor competence in the perception of dynamic events. In this volume, Chapter 21. von Holst, E. and Mittelstaedt, H. (1950). Das Reafferenzprinzip. Naturwissenschaften, 37, 464–476. Wolff, P. (1999). Space perception and intended action. In G. Aschersleben, J. Müsseler, and T. Bachmann (Eds.), Cognitive contributions to the perception of spatial and temporal events, pp. 43–63. North Holland: Elsevier.


8 The perception and representation of human locomotion John J. Rieser and Herbert L. Pick, Jr.

Abstract. Locomotion is both a class of actions and a class of perceptible events. As a class of actions it is embedded in its environmental context and participates in 1ght, 2ight, and many functional behaviors. As a class of perceptible events, it is central to dynamic spatial orientation, that is, keeping up-to-date on the changes in self-to-object distances and directions that occur during locomotion. This chapter is focused on locomotion and on spatial orientation when people walk without vision and without access to nonvisual information about their surroundings. The thesis is that even when walking without vision, people perceive their locomotion relative to the remembered environment as a frame of reference. This environment-centered perception of locomotion accounts for how they keep up-to-date on their changing spatial orientation relative to features of the remembered surroundings. And the resulting representation is used to steer their ongoing locomotion and control all other environmentally directed actions.

Locomotion is organized around the environment as a frame of reference, both as a class of actions and as a class of perceptible events. As a class of actions, people typically control their locomotion, so that navigation is safe and productive with respect to the relevant objects, events, and features of the surrounding environment. As a class of perceptible events, people tend to perceive their locomotion relative to the environment as a frame of reference. In other words, people tend to notice where they are standing and how they are facing relative to features of their surroundings. An implication of this is that the complementary aspect of the perception of locomotion is that people keep upto-date on the dynamic changes in spatial orientation that result from locomotion, and the resulting dynamic representation is a basis for steering locomotion and controlling other environmentally directed actions. When people walk with vision, they can steer their locomotion and control their actions relative to their goals and other visible features of their surroundings. However, people do not typically steer their locomotion relative to goals that are continuously in view, because, for example, they look away from their destination while walking, or the destination is occluded by obstacles, or because it is dark, or because they are blind. Nonetheless, even when walking without vision people steer their walking relative to remembered targets with relatively high precision across paths that vary in length and complexity (Loomis and Klatzky 1993; Rieser, Ashmead, Talor, and Youngquist 1990; Rieser, Guth, and Hill 1986; Steenhuis and Goodale 1988; Thompson 1983). This chapter is about how people steer locomotion when they walk without vision and without nonvisual forms of environmental feedback. When people locomote in situations where they can see and hear, locomotion results in optical and non-optical forms of environmental 2ow which specify the changes in their distances and directions


178


relative to features of the surroundings. The main thesis of this chapter is that when people locomote in situations where they cannot see or hear, they perceive their walking relative to the remembered surroundings. Motor information is used to update the changes in spatial orientation, as if they could see and hear the environmental 2ow. This dynamic representation of spatial orientation serves, in turn, as a basis for steering ongoing locomotion and as a basis for controlling the wide range of actions that are directed toward features of the remembered surroundings. From our point of view it is dif1cult to dissociate the actions of locomotion from the perception and representation of locomotion, since we believe they result from the same processes. The three major sections of this chapter are organized around three central claims. The 1rst claim is about the different frames of reference that can serve to organize perception and action. Whereas people can 2exibly perceive their locomotion relative to the body as a frame of reference, we hypothesize people tend to perceive it relative to relevant features of the surrounding environment. This tendency makes sense, since many adaptive behaviors are coordinated with features of the environment. In what situations, we ask, do people tend to perceive their own locomotion when walking without vision as a temporal sequence of limb positions, and in what situations do they perceive it relative to the surroundings as a frame of reference? The second claim is about the processes that account for how locomotion can be organized relative to the environment, especially when people walk without vision—how is it that the motor information while walking with or without vision is integrated with the remembered features of the surrounding environment? We hypothesize the efferent and many forms of afferent information are integrated via perceptual–motor learning processes and result in a uni1ed perception of locomotion and its complementary aspect, the dynamic representation of spatial orientation. And the third claim is about the generality of this uni1ed, dynamic representation. We hypothesize that it is the basis for steering locomotion with and without vision, and for controlling the forces and directions of all actions that are directed toward features of the remembered environment. According to our view, a single representation and a single set of processes mediate the control of ‘locomotion as action’, the perception of ‘locomotion as event’, and in addition it mediates the broad range of other actions that are directed toward features of the surrounding environment.

8.1 Actions and their multiple frames of reference Some actions are directed toward one’s own body surfaces and thus are coordinated with a limbcentered or body-centered frame of reference. Examples of this include scratching an itch, swatting mosquitoes, buttoning a shirt, and brushing one’s teeth. For each of these, the details of the motor actions needed to scratch, swat, button, or brush depend on the position of the targeted area of the body relative to the active limb that effects the action; they do not generally depend on the active limb’s position relative to features of the surroundings. However, most actions are directed toward objects or features of the surrounding environment, and thus are coordinated with an environmental frame of reference. This is the case for actions that involve very different systems of motor effectors and very different categories of functional goals— for example, using the hand–arm–trunk system to throw a football to a receiver, using the mouth– vocal tract system to communicate with a conversational partner, and using the legs–trunk system to jump a stream in the woods. In each case, the force and direction of the needed actions take into account the distance and direction of one or more objects or features of the surrounding environment—the receiver, the partner, and the stream in the examples above.


The perception and representation of human locomotion

People often initiate actions while they walk or run—for example, they throw a football while running to evade a tackler, they shout across the room to talk with one friend while running to catch up with another friend, and they jump across a stream while running to catch their pet dog. In each of these cases, the force and direction of the throw, the shout, and the jump depend on the person’s shifting location, and so the force and direction of the action need to be planned in anticipation of the person’s future location, that is, the location at the instant the action is initiated. This is the case whenever people are underway. This chapter is focused on situations that are the extreme case of this, namely where people walk and act without vision and without access to non-visual information about the target’s location. When walking without vision, people steer their walking relative to remembered targets. To accomplish this they need to integrate the motor information for the walking with their memory representations of their surroundings. Likewise, when throwing or communicating while walking, people need to coordinate the force of their throws and volume of their conversation with respect to their changing distance from the remembered targets in the surroundings. To do this they need to perceive their locomotion relative to features of the remembered surroundings and coordinate their throws and shouts with this representation. We focus on situations where people guide their actions while walking without vision. In our experiments people typically look at their surroundings from one point of observation, close their eyes, and guide their walking, turning, or whispering relative to features of the remembered surroundings while they are walking.

8.2 Locomotion and the frames of reference for perceiving locomotion Whether walking with vision or without vision, people typically control their walking relative to features of their immediate environment. Consider two strategies that could be used to accomplish this. According to the 1rst strategy, people would perceive their locomotion in body-centered terms as the temporal sequence of limb and body positions that occur during locomotion. In order to steer their locomotion relative to the surrounding environment, they would then need to integrate their body-centered perceptions with real-time input for the surroundings (when walking with vision or audition) or with their representation of the surroundings (when walking with their eyes closed). Although this is a plausible strategy, it seems relatively inef1cient because it hypothesizes that a body-centered stage of perception intervenes between the sensory inputs and the environmentcentered control that is needed. According to the second strategy, people would perceive their locomotion in environmentcentered terms. When walking with eyes open or with useful auditory information, this could be accomplished by attending to sights and sounds that directly specify one’s time to reach the target, or the target’s distance and direction. However, when walking without being able to see or hear, these environmental frame-of-reference cues are not available, and yet people steer their walking with considerable precision. Our hypothesis is that guiding locomotion without vision is highly similar to steering it with vision, and is based on perceptual–motor learning that occurs when walking while able to see and hear the environment. That is, when walking with visual and nonvisual environmental feedback, people learn the correlation of the temporal 2ow of motor information with the temporal 2ow of the environment relative to their shifting points of observation. Then while walking without vision, they draw on this knowledge in order to perceive their walking in environmental

179


180


terms. According to this strategy, walking without vision is generally steered the same way as walking with vision. These two alternatives for how self-locomotion is perceived, we argue, are similar to the classic distinction of the proximal versus distal stimulus in vision and in haptic perception. For example, Gibson distinguished these in his writings about the haptic perception of objects (Gibson 1962, 1963, 1966). His analysis recognized that the haptic identi1cation of objects depended on patterns of skin pressures and joint con1gurations. However, he rejected the idea that object perception depended on a stage of awareness of the sensations of the pressures and con1gurations. Rather he distinguished between ‘input of information’ and ‘input of sensory quality’ (Gibson 1966, p. 108). His observations of the haptic perception of sculptured objects that were nonsense forms indicated that often all ten 1ngers moved when people explored a particular object and that the patterns of exploration differed from moment to moment and from trial to trial. The patterns of cutaneous pressures and joint con1gurations (the input of sensory quality) were dif1cult to detect and describe. But the patterns of relations of one part of an object to another (the input of information) resulted in object identi1cation and it came relatively easily. Describing this in relation to judgments of weight and material composition, he suggested the ‘2ux of subjective sensations is so complex as to be unreportable’ (Gibson 1966, p. 127). Since that time, Lederman and Klatzky (1987, 1998) have described very systematically the various haptic movements observers use when exploring objects by touch in order to compare them on different features.

8.2.1 Proprioception and a body-centered frame of reference for perceiving self-locomotion What would it mean to perceive locomotion relative to the body as a frame of reference? Classically, proprioception is de1ned for a stationary person as the perception of static limb positions relative to other static limb positions, and for a moving person it is de1ned as the temporal series of limb positions relative to other limb positions (sometimes this is called kinesthesis). In order to perceive self-locomotion proprioceptively (that is, in body-centered terms), one could attend to the sequence of one leg in relation to the other, or of both legs in relation to the trunk, and these could be integrated over time as the intrinsic shape of the path walked. We have been able to think of relatively few situations that call for the proprioceptive perception and control of locomotion. As one example, consider the 1gure eights and other compulsory 1gures practiced by competitive 1gure skaters, which are de1ned as shapes irrespective of the shape’s position relative to the surroundings. As another example, consider the pliés and hand positions of classical ballet. What information could serve as the basis of body-centered perception of locomotion when walking without vision? When walking, information is transmitted from receptors associated with movements of joints and muscles that signal changes in limb position, and these are integrated over the space and time of locomotion to result in a record of the movements. In addition, the semicircular canals of the vestibular system signal angular accelerations and the otoliths signal linear accelerations and gravity. Since people tend to walk paths consisting of segments with relatively constant speeds separated by turns, it makes sense to suppose that the vestibular information on its own could signal the distances turned accurately but could not signal the distances translated accurately (Berthoz, Israel, Georges-Frances, Grasso, and Tsuzuku 1995). Joint information, muscle information, and in some cases deep pressure information could be used in conjunction with vestibular information to result in a uni1ed representation of the shape of one’s path (Guendry 1974); some believe that the vestibular inputs provide a necessary basis for this integration (Kornhuber 1974).



What, in principal, are the geometric properties of locomotion that could be perceived from such an integration, in a form like this that is disconnected from the surrounding environment? We limit this discussion to locomotion within the ground plane. In terms of perceiving the direction of locomotion, people could perceive themselves rotating to their left or right, and perceive themselves translating forward, backward, left, or right. In terms of perceiving distance and/or its derivatives of time, consider what could be perceived 1rst for rotational locomotion and then for translational locomotion. For rotational locomotion, there seems to be an intrinsic body-centered distance scale, namely the distance of a full turn. We do not know the origin of this scale. For example, we are not aware of anything about the neurophysiology of the vestibular canals that would identify a full turn as a special category, so perhaps its categorical quality re2ects perceptual–motor learning of the amount of motor activity that it takes to turn to spot a feature in the surroundings and continue turning in the same direction until returning to face the original feature. Whether or not it is learned, it emerges quite early, at least by the second year of life (Rieser and Heiman 1982). For translational locomotion, distances walked could be encoded in body-scaled units such as strides or eye heights. For translations and rotations alike, in the processing stream the body-centered perception of selflocomotion would initially be disembedded from the environment, since neither joints, muscles, or vestibular organs know anything about the surroundings.

8.2.2 Exteroception and an environmental frame of reference for perceiving self-locomotion An alternative strategy is that people perceive their locomotion relative to the environment as a frame of reference, directly in terms of their changing positions and facing directions relative to the seen, heard, or remembered surroundings as a frame of reference. Lee (1978) noted that the classical use of the term ‘exteroception’ referred to the perception of the environment and that the classic use of the term ‘proprioception’ referred to the perception of the body. He coined the term ‘exproprioceptive’ to label situations such as this, where actions are organized relative to the environment as a frame of reference. Consider what it would mean to perceive self-locomotion in environmental terms. Distances translated would be noticed in terms of one’s changing position relative to one or more of the features of the seen, heard, or remembered surroundings. Likewise, distances turned would be noticed as changes in angles to environmental features and changes in perspective views of objects and environmental features. We hypothesize that when people locomote without vision they can direct their attention outward toward the surrounding environment or inward toward their own body positions. In the next section we review some of the classical studies of the psychophysics of self-locomotion and show that they focus on proprioception. Then we present evidence from our laboratories indicating that when people walk with or without vision, they typically focus their attention on their changing orientation relative to the seen or remembered environment; they do not typically focus their attention on the sequences of body positions that occur while they walk.

8.2.3 Traditional studies of the psychophysics of self-locomotion emphasize perceiving in body-centered terms The perception of self-locomotion results from the integration of efferent and of multiple forms of afferent information. We are not aware of empirical demonstrations of the role played by efferent

181


182


information in the perception of self-locomotion. In classic studies, Mountcastle (1958) and Taub (1968) recorded the changes in proprioception and in motor control that occurred in patients with anesthetized limbs or with deafferented limbs. We are not aware of analogous studies of the perception of locomotion, and so we do not know what role, if any, that efference plays in the perception of self-locomotion. Intuitively, it seems plausible that efference plays a role. Consider, for example, the progressively diminished perceptions that might occur when one walks actively without vision to a predetermined goal, versus when one walks by grasping the upper arm of a sighted guide in order to follow the guide’s path to an unknown destination, versus being pushed in a wheelchair to an unknown destination. Multiple forms of afferent information contribute to the perception of locomotion and each has been studied in isolation from the others. The contributions of the vestibular organs have long been noted as contributing to the perception of turning even while sitting in a dark room (Wade 1998). The contributions of vision were noted and studied during the nineteenth century. For example, Helmholtz (1858) remarked on the moving train illusion (the illusion of self-movement when sitting in a stationary train while the train on the adjacent track begins to move); and Mach (1875) used an optokinetic drum to study circular vection. The contributions of the biomechanical system were noted more recently through the use of circular treadmills. Analogous to linear treadmills, on circular treadmills the 2oor rotates, and people step in place in order to compensate for the 2oor’s rotation. Bles (1981), Lackner (1988), and Garing (1999) asked people to close their eyes and step in place while walking on circular treadmills. All the subjects reported a powerful illusory perception that they were physically turning. Garing (1999) studied the psychophysics of the perception of rotational locomotion with vestibular input alone, visual input alone, and biomechanical input alone. In all three conditions the subjects stood atop a circular treadmill that was surrounded by a striped cylindrical curtain. The treadmill consisted of a waist high T-bar centered within a 122 cm disc. Subjects stood atop the disc while grasping the T-bar. When the T-bar turned (it was driven by an electric motor beneath the housing), subjects were asked to step around in a circle in order to keep the T-bar centered at their waists. When the T-bar was stationary and the disc rotated (it was driven by another electric motor beneath the disc), subjects were asked to step in place in order to compensate for the disc’s rotation. In the vestibular condition the subjects closed their eyes and grasped the T-bar (now freely spinning) while they were passively spun at 5 rpm; in the visual condition the subjects opened their eyes and stood still while a cylindrical curtain rotated around them at 5 rpm. And in the biomechanical condition the subjects grasped the stationary t-bar and stepped in place to compensate for the 2oor, which rotated at 5 rpm under their feet. In all three conditions, every subject reported a sense of self-rotation. Subjects varied in how soon they 1rst reported the sense of self-rotation in the different conditions, averaging 0 s, 4 s, and 1 s until the 1rst reports for the vestibular, visual, and biomechanical conditions, respectively. Classical studies of the psychophysics of locomotion assess locomotion in body-centered terms. The subjects were typically asked to judge proprioceptive qualities and quantities—for example, they were asked to say whether or not they were moving, to judge their direction of movement in body-centered terms (e.g. left, right, forward, backward), and to rate their speed or acceleration in terms of environment-free scales. Guendry’s (1974) chapter on the psychophysics of vestibular sensation is a powerful example of the body-centered approach to the perception of self-locomotion. In the studies covered in his review, the subjects sat in the dark in a rotating chair or on a linearly accelerating cart and were asked to say whether, in what direction, and at what rate they were moving



(in revolutions per minute or in meters per second). In none of the studies were subjects asked to say how their self-motions changed their distance or direction relative to features of the seen or remembered surrounding environment. The body-centered approach is also apparent in studies describing the psychophysics of the visually based perception of self-locomotion. In the important studies of visually based circular vection by Dichgans and Brandt (1972, 1978), the subjects typically stood while surrounded by a striped cylinder (an optokinetic drum) that rotated around them, or they viewed optical simulations of environmental rotation that were projected onto screens (Post 1988). When the information for cylinder rotation started, subjects typically 1rst reported that the cylinder appeared to be rotating (e.g. object motion), but then after a few seconds their perception shifted to self-motion, and they reported seeing themselves rotating within a stationary cylinder (Wertheim 1994). Again, people were asked to describe their self-motion in body-centered terms—to say whether they perceived themselves to be rotating and, if so, to say whether the rotation was clockwise or counterclockwise in direction and to estimate the speed in revolutions per minute. People were not asked to judge their turning in environmental terms, for example, by judging how far they turned relative to remembered features of their surroundings outside of the rotating drum. In studies of visually induced linear vection, subjects watch visual displays that show the radial or lamellar 2ow that speci1es translatory locomotion. The classical studies of visually induced linear vection also illustrate the body-centered approach, since the subjects were typically asked to say whether they perceived themselves to be moving forward and, if so, to judge the rate of movement (e.g. Berthoz, Pavard, and Young 1975; Lishman and Lee 1973; Johannson 1977; Telford and Frost 1993; Warren 1995). However, in recent studies (Giannopulu and Lepecq 1998; Israel 1999; Lepecq, Jouen, and Dubon 1993) the subjects were asked to judge their changing distances relative to a landmark that was out of sight in the remembered surroundings.

8.2.4 People’s self-reports of what they notice when walking without vision Our thesis is that the frame of reference for perceiving self-locomotion is 2exible and is determined by the perceiver’s goals. Since so many goals are embedded functionally in the surrounding environment, people tend to experience their self-locomotion in environmental terms. However, goals are sometimes independent of the particular environmental circumstances, and we suppose people can shift their attention onto their body surfaces or other possible frames of reference. To 1nd out about this, we collected systematic interview data, asking people what they noticed when they walked without vision with a sighted guide. At the start of each of three conditions, the tester and subject stood in a large open room and chatted about the doors, windows, and desks in the room as part of the conversation. Then subjects were asked to walk with the tester while they continued to chat. In one condition the subjects walked and chatted with their eyes open, in a second condition the subjects wore a blindfold and sound system while walking and chatting, and in a third condition the blindfolded subjects were asked to pay attention to the intrinsic shape of the path that they walked. Subjects were not given any additional instructions, and no one was informed that there was any particular purpose to walking and chatting. Ten subjects participated in each of the three conditions. The walk had six turns, and its shape can be represented as an ‘X enclosed in a square’. After the walk, the subjects were asked to tell about the walk in their own words. All ten subjects who walked with their eyes open described their walks exproprioceptively, in terms of the how they faced and where they turned relative to the features of the room. Nobody spontaneously reported the shape of

183


184


the walk or the number of turns. When they were asked, eight of the subjects were able to 1gure out the number of turns; none described anything like having walked anything like ‘an X in a square’. The results were very similar in the blindfolded condition where the subjects were not instructed to attend to the shape of the walk. Even in the condition where the blindfolded subjects were instructed to attend to the shape of the path they walked, although all of the subjects reported making six turns and seven reported crossing their own path (the crossing in the ‘X’), none reported walking along anything like an ‘X enclosed in a square’. The point of this is to show that people tend to perceive the distances they walk and turn in terms of features of the surrounding environment and that they do so even when they are not able to see or hear the features of their surroundings. Although the environment serves as the ‘natural’ frame of reference under these conditions, it is also clear that people can adjust their attention and notice body-centered features of their locomotion like the shape of the path, number of turns, and so forth. It may be the case that it is easier to talk about one’s locomotion in environmental terms than in body-centered terms. If so, then it might be that people tend to perceive their blindfolded walking relative to their bodies as a frame of reference, but then translate body-centered perceptions into environmental terms because it is easier to talk in environmental terms. The experimental studies of path integration that are described in the next section relate to this possibility.

8.2.5 Path integration when walking without vision is more accurate when people know the surrounding environment If people tend to perceive their locomotion in terms of their shifting distances and directions relative to features of the surroundings, we reasoned they should perceive their locomotion more accurately when walking in surroundings that were differentiated with nearby features and less accurately in surroundings that did not have distinguishable features. Rieser, Frymire, and Berry (1998; Frymire 1999) conducted an experiment to 1nd out whether this was the case. Blindfolded subjects were asked to walk with a sighted guide along a path and then turn to face their starting position at the end of the path. The paths had four legs and averaged 30 m in length; on a typical path the subjects walked 10 m straight ahead, turned right and walked 10 m, turned right again and walked 5 m, and 1nally turned right again and walked 5 m and stopped. After walking paths like this, subjects were asked to turn and face their remembered starting position, and errors in their facing directions were assessed with an electronic compass. We devised three conditions to evaluate whether people turned to face the remembered starting point of their path more accurately when they had a differentiated representation of their immediate surroundings than when they did not know their surroundings. Our preliminary observations were that when people attempted path integration tasks in situations where they did not know their surroundings, they reported that it is like walking on a large open homogeneous 1eld, a ‘Ganzfeld’ if you will. In the ‘virtual Ganz feld’ condition, subjects were equipped with a blindfold so they could not see and a sound system, so that they could converse easily but could not localize what they heard in space. Then they were guided via circuitous routes around campus for about ten minutes, while answering questions they were asked by their sighted guide. The purpose of the walk was to disorient the subjects, and it succeeded according to each subject. In the ‘actual surroundings’ condition, subjects looked at their actual surroundings at the beginning of each path integration trial. Then they put on the blindfold, and walked the path with the sighted guide. While walking, they were asked to keep several features of the surroundings in mind



as well as their starting point. And 1nally, in the ‘imagined surroundings’ condition, subjects were disoriented as in the ‘virtual Ganzfeld’ condition, but then they were asked to imagine they were standing in a familiar location and given time to generate an image of the locale. Having generated an image of the familiar location, they were instructed to keep the imagined landmarks in mind while walking the path and while turning to face the starting point. The errors 1t the predictions closely. The subjects averaged 20 deg of error when turning to face the starting position in the ‘actual surroundings’ condition, 35 deg in the ‘imagined surroundings’ condition, and 47 deg in the ‘virtual Ganzfeld’ condition. The thesis that people tend to perceive their locomotion in environment-centered terms led to this prediction. But on the other hand, if people tend to perceive their locomotion in body-centered terms, then one might have predicted better performance in the ‘virtual Ganzfeld’ condition than in the other two conditions. The reason for this is that people do not need to know anything about their surroundings in order to perform path integration tasks, and the instruction to focus attention on the remembered features of the surroundings while walking the to-be-integrated path might have placed an unhelpful load on working memory. In summary, when people walk without vision they generally describe their walks in environmental terms, as if they tend to perceive their walks in environmental terms. In addition, when they are asked to perceive a path that they walk in order to judge the direction back to their starting point, they do so more accurately when they have the chance to perceive their walk relative to the layout of landmarks in the remembered surroundings than when they do not know the layout of landmarks in their remembered surroundings.

8.3 Re-calibration of locomotion in environmental terms Recall the central thesis that perceptual–motor learning accounts for how people perceive their walking without vision in environmental terms. The basic idea is that while walking with vision people learn how the world looks (sounds or feels) as a function of variations in their walking. Then while walking without vision, since people already know the dynamic changes in how the world looks and sounds and feels, they draw on this knowledge to perceive their locomotion in environmentcentered terms, and thus, they represent the dynamic changes in their spatial orientation. If this hypothesis is correct, then manipulations of the relation between locomotor actions and environmental 2ow should lead to corresponding changes in spatial orientation and in the guidance of walking without vision (e.g. Pick, Rieser, Wagner, and Garing 1999; Rieser 1999; Rieser, Pick, Ashmead, and Garing 1995). We brie2y summarize our methods and central 1ndings below, for translational locomotion (generally forward walking) and for rotational locomotion (stepping in order to turn in place).

8.3.1 The recalibration of translational locomotion The methods consisted of pre-tests, a learning intervention, and post-tests (Rieser et al. 1995). During the pre- and post-tests, subjects stood in an open 1eld, viewed a target located about eight meters straight ahead, and then attempted to walk without vision to the remembered target location. During the learning intervention phase, subjects walked either in a biomechanically faster condition (the rate of walking was paired with a slower-than-normal rate of movement through the surroundings) or in a biomechanically slower condition (the rate of walking was paired with a faster-than-normal rate of movement through the surroundings).

185


186


The equipment used to arrange these conditions consisted of a motor-driven treadmill mounted on a trailer that was towed by a tractor. Subjects were asked to walk at one rate on the motor-driven treadmill while they were towed at a different rate behind the tractor. The motor-driven treadmill determined the rate of the biomechanical activity involved in the subject’s locomotion, while the speed of the tractor determined the environmental 2ow rate. After walking for ten minutes in the biomechanically slower condition, subjects tended to stop short of the remembered target during the post-tests, as if they perceived their rates of walking as faster than their actual rates in the scale of the surrounding environment. After walking for ten minutes in the biomechanically faster condition, on the other hand, they tended to walk past the target, as if they perceived their rates of walking as slower than their actual rates in the environmental scale. These experiments show that people perceive the scale of their own translational locomotion relative to the remembered surroundings as a frame of reference.

8.3.2 Recalibration of rotational locomotion Analogous pretest-intervention-posttest methods were used to change the calibration of rotational locomotion, but the circular treadmill described in an earlier section was used to change the calibration of turning in place instead of the tractor and linear treadmill used to change the calibration of forward walking (see Pick et al. 1999; Rieser et al. 1995). In this case during the pre- and post-tests, subjects stood in the laboratory, faced a target that was straight ahead of them, put on a blindfold, and then attempted to turn 360 deg in the clockwise or counterclockwise direction so they again faced a remembered target. During the learning intervention phase, subjects walked on the circular treadmill. In the biomechanically faster condition, the treadmill’s disc rotated at 5 revolutions per minute (rpm) while the T-bar rotated 5 rpm in the opposite direction. The result was that subjects physically turned at a rate of 5 rpm relative to the environment, but their feet and legs moved at a rate of 10 rpm. In the biomechanically slower condition, the treadmill’s disc rotated at 10 rpm while the handle rotated 5 rpm in the same direction. The result was that subjects physically turned at a rate of 10 rpm relative to the environment, but their feet and legs moved at a rate of 5 rpm. The results demonstrated a recalibration of people’s rotational locomotion. The post-tests after the biomechanically faster condition showed that subjects tended to turn about 90 deg too far during the post-tests, as if they perceived themselves to be turning at a slower rate relative to the remembered surroundings than they were actually turning. The post-tests after the biomechanically slower condition showed the opposite pattern. The subjects tended to stop about 50 deg short of the remembered target, as if they perceived themselves to be turning at a faster rate relative to the remembered surroundings than they were actually turning.

8.3.3 How are the changes in calibration organized? Are actions, including locomotion, organized as a single global system? Or are they organized more narrowly, for example around functional categories of action, or around speci1c limb systems or around speci1c muscle groups? To 1nd out about this we have probed the organization of locomotion by testing for transfer of the changes in calibration that we induced experimentally to other forms of locomotion that were not practiced during the recalibration phases of our experiments. Consider some of the alternative possible organizations. At one extreme, changes in the calibration of locomotion might transfer very broadly, so that the gain of all forms of locomotion and



forms of non-locomotor actions all increased or decreased in similar ways. In order to assess this possibility, Rieser et al. (1995) induced changes in the calibration of forward walking in the ‘biomechanically faster’ condition, so that the subjects all walked too far during the post-tests, and tested to see if the ‘gain’ induced in the calibration of their forward walking would transfer to their throwing a bean bag at a remembered target. The results showed that it did not transfer, indicating that the calibrations mapping actions onto the surrounding environment are not globally organized to encompass all actions. Another extreme possibility is that changes in the calibration of locomotion might be very speci1c and not transfer to any other actions or to any other forms of locomotion. To assess this, Rieser et al. (1995) again induced changes in the calibration of forward walking in the ‘biomechanically faster’ condition, so that the subjects all walked too far during the post-tests. And then we tested to 1nd out if the subjects would also walk too far when assessed with a very different gait, namely sidestepping. In order to sidestep to the target, subjects started by facing the target, turning so that it was 90 deg to their left or right, putting on their blindfold, and then moving their feet left (or right) to reach the remembered target. The step cycle and gait were quite different from forward walking, which consisted of a smooth and continuous series of steps. The cycle for sidestepping consisted of the subject moving his or her target-side foot one pace to the left (or right) toward the target, pausing in order to bring the other foot to meet the target-side foot, pausing again, and repeating the cycle. Despite the differences between forward walking and sidestepping in the muscle groups, gaits, and step cycles, each subject walked too far to the remembered target during the sidestepping post-tests, and the magnitude of the overshoot when sidestepping was virtually the same as the overshoot during the forward-walking post-tests. The results showed that the experimentally induced change in the calibration of forward walking did transfer to sidestepping. Thus, according to these 1ndings, actions are not globally organized nor are they anatomically organized at the level of particular muscle groups, step cycles, or gaits. An additional 1nding indicates that they are functionally organized—that is, actions that accomplish the same function are organized as a functional system, so that changes induced in the calibration of one method of accomplishing the function transfer to other methods of accomplishing that same function. The hypothesis of a functional organization is consistent with the positive transfer that was obtained for translational locomotion from forward walking to sidestepping. Recently, Berry (2000) put the hypothesis of a functional organization of rotational locomotion to a strong test. She induced changes in the calibration of rotational locomotion by asking subjects to walk on the circular treadmill in the ‘biomechanically faster’ condition, where their feet were stepping at a 10-rpm rate while they were physically turning relative to the surroundings at a 5-rpm rate. As expected, during the post-tests the subjects overshot the remembered target, turning too far by an average of about 90 deg. Berry wished to 1nd out whether the changes in rotational locomotion that were experimentally induced while people turned themselves by stepping with their feet and legs would transfer to turning themselves by their hands and arms. To assess turning by hands and arms, she asked subjects to stand on a swivel on the 2oor inside a round railing that was within reach. During the pre-tests and post-tests the subjects faced the target, put on a blindfold, and pushed themselves around by moving the hand over hand along the railing. Berry’s results were that the subjects turned too far by hand, averaging about 45 deg of overshoot. This indicates that the experimentally induced change in the calibration of turning by foot partially transferred to turning by hand. It is consistent with the hypothesis that locomotion is organized as a functional system.

187


188


8.4 Unified representations of dynamic spatial orientation People’s perception of locomotion is calibrated in environmental terms and our ongoing experiments indicate the calibrations are functionally organized. In this section we consider the thesis that the perception of locomotion in environmental terms results in a representation of the dynamic changes in spatial orientation that occur during locomotion. And that this representation is uni1ed and is used to guide a wide range of environmentally directed actions that people produce when they walk without vision. Loomis and his colleagues provided support for the analogous thesis for visual space perception (Loomis, Da-Silva, Philbeck, and Fukusima 1996; Philbeck, Loomis, and Beall 1997). By ‘uni1ed’ we mean that the representation of dynamic spatial orientation results from the perception of one’s own locomotion, and in addition, that the same representation mediates the guidance of locomotion and of other forms of environmentally directed actions. The representation likely re2ects parallel processing through dorsal, ventral, and other anatomical streams. But we hypothesize that different functions are not served by different representations of spatial orientation, and instead a single uni1ed representation of space accounts for both the perception of locomotion, the control of locomotion, and the control of other functional categories of spatially coordinated action. If the diversi1ed representation hypothesis is correct, then experimentally induced errors in the perception of locomotion should not lead to errors in the control of other environmentally directed actions. If, on the other hand, the uni1ed hypothesis is correct, then experimentally induced errors in the perception of locomotion should lead to predictable errors in the control of other environmentally directed actions. These predictions were investigated in two studies. In both studies, changes in the calibration of rotational locomotion were induced. The prediction in the 1rst study was that the resulting errors in the perception of rotational locomotion would transfer to errors in controlling locomotion while walking complex paths without vision. The prediction in the second study was that the resulting errors in the perception of rotational locomotion would cause errors in whispering in situations where people turned without vision and whispered to their remembered partners.

8.4.1 Changing the calibration of rotational locomotion alters the representation of dynamic spatial orientation when walking two-dimensional paths The changes induced in the calibration of rotational locomotion might only narrowly affect the perception of turning in place. Or alternatively, they might have more general effects, and in2uence the control of locomotion along any two-dimensional paths that include changes in heading. Wagner (1996) conducted a series of experiments to 1nd out. The experiments followed the methods described above by inducing a recalibration of rotational locomotion, and then by testing to see whether the recalibration transferred to tests of spatial orientation when walking complex twodimensional paths. Before and after the recalibration experience, the participants were tested by asking them to look at target objects in a familiar room, put on a blindfold, follow a sighted guide walking along a complex path, and then point at the locations of targets in the remembered room. Wagner’s results showed that the induced changes in the recalibration of rotational locomotion resulted in subjects underestimating their rates of turning by about 15%. Their errors in pointing at targets after walking two-dimensional routes were predicted closely by these 15% underestimates in turning. Wagner’s results reveal that the induced changes in the perception of rotational locomotion led to predictable changes in their dynamic representations of spatial orientation. They are consistent with the ‘unitary representation’ view of locomotion and not with the ‘diversi1ed representation’ view.



It is important to note that we have simpli1ed two sets of complicating details. One complication is that there are two recalibration processes, one operating like a sensory aftereffect and the other operating like a perceptual learning effect. They behave somewhat differently. The other detail is that the manifestations of recalibration are different when tested while turning in the same direction as testing in the recalibration experience and when tested in the opposite direction. Both of these complexities are examined in detail in Pick et al. (1999).

8.4.2 The calibration of speaking as a function of the speaker-to-listener facing direction Our hypothesis is that the dynamic representation of one’s spatial orientation during locomotion is unitary, so that the same representation mediates different actions. In order to test this hypothesis we induced changes in the calibration of rotational locomotion, and then tested to see if the resulting changes in spatial orientation when walking without vision in2uenced the vocal intensity of subjects when they attempted to speak to a remembered listener after turning without vision to a new facing direction. Skillful speakers adjust their vocal intensity to 1t their spatial orientation relative to their listeners. For example, we know that adults modify their vocal intensity with changes in the distance to their listener (Johnson, Pick, Siegel, Ciccarielli, and Garber 1981; Michael and Siegel 1998). Conversational effectiveness also depends on the speaker’s facing direction relative to the listener— in order to be heard, people need to talk more loudly when they face away from the listener than when they face directly toward the listener. We exploited this in the following experimental study of the calibration of vocal intensity as a function of the speaker’s facing direction relative to the listener (McMillan 1998; Rieser, McMillan, Berry, and Pick 1998). In order to assess the calibration of vocal communication, subjects were asked to whisper their telephone number, so that their conversational partner [standing 1m away] could just barely hear the number. Subjects were asked to do this after they turned to face away from the listener by 0, 180, and 360 deg. Their vocal productions were tape-recorded and scored to determine their vocal intensity, which was averaged across the duration of the utterance. In the ‘eyes open’ condition, people could see their facing direction relative to the listener during each trial. In the ‘eyes closed’ condition, subjects faced the listener, put on a blindfold, and then were turned by the experimenter into one of the facing directions. In the ‘eyes open’ condition, subjects could adjust their loudness according to their visible facing direction relative to the listener. But in the ‘eyes closed’ condition, subjects needed to perceive their turn in relation to the remembered listener, and adjust their loudness in conjunction with this representation of their spatial orientation. The experiment consisted of pre-tests of whispering as a function of the different facing directions in the ‘eyes open’ and ‘eyes closed’ conditions, the standard intervention to induce a change in rotational locomotion together with the tests to assess the recalibration as described earlier in this chapter, and 1nally whispering post-tests, which were the same as the whispering pre-tests. As expected, stepping on the circular treadmill caused people to recalibrate their rotational locomotion, so that they underestimated their degree of rotation by about 20% during the post-tests relative to the pre-tests. Also as expected, the recalibration of rotational locomotion did not, per se, lead to a change in the calibration of listening. That is, when the subjects could see their facing direction relative to the observer, their average vocal intensity during the post-tests was the same as their average during the pre-tests after turning 0.180, and 360 deg. The theoretically critical results involved the vocal intensities when people whispered after turning with their eyes closed. The ‘unitary representation’ hypothesis leads to this prediction:

189


190


the 20% underestimation error in the perception of rotational locomotion that was induced by the recalibration phase should result in a 20% error in the represented facing direction during the posttest trials, and this error in the representation should lead to predictable changes in vocal intensity as a function of the facing direction. Consider three speci1c predictions, for when subjects viewed the listener straight ahead of them, and then closed their eyes. First, after turning 0 deg with their eyes closed, the subjects should have an accurate representation of their facing directions, and thus there should be no pre-test to post-test change in their vocal intensity. The results con1rmed this. Second, after turning 180 deg with their eyes closed, the subjects faced directly away from the listener, and correspondingly, during the pre-tests their whispering was the loudest in this condition. During the post-tests, the subjects actually faced directly away from the listeners, but we predicted the recalibration would induce them to underestimate their turn by about 36 deg. If this is the case, then they should incorrectly represent themselves as facing about 144 deg away from the listener instead of their actual 180 deg. As predicted, the subjects whispered more softly during the posttests than the pre-tests in this condition, averaging a 5-decibel reduction in vocal intensity. And third, after turning 360 deg with their eyes closed, subjects faced directly toward the listener and, correspondingly, during the pre-tests their whispers were the softest in this condition. During the post-tests, we predicted that recalibration of rotational locomotion would induce them to underestimate their turn by about 72 deg. If this is correct, their representation would be of facing 288 deg (i.e. the 360 deg of their actual facing direction minus the 72 deg underestimation due to the recalibration) away from the listener (whereas they actually faced 0 deg away), and they should speak more loudly during the post-tests than the pre-tests. The results con1rmed this for all the subjects, whose vocal intensities averaged a 3-decibel increase during the post-tests relative to the pre-tests.

8.5 Summary and conclusions Locomotion causes changes in spatial orientation relative to the surrounding environment. When walking with vision or without it, people tend to perceive their locomotion relative to the surroundings as a frame of reference, and their perception serves to update their representation of their spatial orientation. We have argued that the resulting representation of spatial orientation is unitary—it re2ects the perception of locomotion, it mediates guiding ongoing locomotion, and it mediates the control of other environmentally directed actions that depend on spatial orientation. The chapter has summarized 1ve types of empirical evidence that support the argument. First, the study of people’s reports of conscious awareness while walking to follow a sighted guide without vision indicated that people tended to notice their walking in the environmental terms of their changing locations and facing directions relative to objects in the remembered surroundings. Second, the studies of path integration showed that people are better able to judge the starting point of a path walked without vision in conditions where they can relate the path to a remembered structured frame of reference than when they can only relate it to a ‘Ganzfeld’. Third, the studies of the recalibration of translational and of rotational locomotion showed that brief perceptual–motor learning experiences induce changes in the environmental scale of perception. And in addition, they showed the experimentally induced changes in calibration were functionally speci1c. Whereas the changes did not transfer to different functions (e.g. changes in the calibration of forward walking did not transfer to throwing a bean bag or to whispering to a conversational partner), the changes readily transferred to new gaits of forward walking and new limb systems of turning in place. Fourth, Wagner’s (1996) studies showed that whereas the recalibration of rotational locomotion did not



transfer to translational locomotion, it did instead lead to predictable errors in pointing at remembered targets while walking paths in the two-dimensional ground plane. And 1fth, the studies of whispering while turning without vision showed that the recalibration of rotational locomotion did not affect the loudness of whispering per se. But instead, the recalibration induced changes in the representation of spatial orientation, and the changed representation did in2uence how loudly the subjects whispered to their remembered partner while turning in place without vision. We close with four questions and answers in light of our work. First, is self-locomotion with or without vision perceived in body-centered or environment-centered terms? Our answer is that it can be perceived 2exibly in either way, depending on the perceiver’s immediate goals. Second, is self-locomotion an action to be controlled or is it an event to be perceived? Our answer is that it is both. Third, is the perception and perceptual representation of self-locomotion uni1ed or diversi1ed? Our answer is, it is uni1ed, such that errors induced in the perception of rotational locomotion also cause errors in walking paths in the two-dimensional ground plane and in the vocal intensity of whispering. And fourth, is the perception of self-locomotion and its complement, the representation of spatial orientation, ‘perception’ in the sense of responding to on-line input, or is it ‘memory’ in the sense of being mediated by memory representations? Our answer is it is both or it is neither. Gibson (1979) pointed out that the pick-up of information by the senses is typically distributed through time and through space. Like him we do not know where to draw the line between perceiving and remembering, and point out that during exploratory actions the information for perceiving objects and for perceiving the environment is always distributed through the space and time involved in the exploratory actions. We think the perception of self-locomotion when walking without vision and the complementary representation of the resulting changes in spatial orientation, are good examples of this dynamic feature of perception.

References Berry, D. (2000). Does recalibration of turning with the feet transfer to turning with the hands? Thesis submitted to the Faculty of the Graduate School of Vanderbilt University in partial fulfillment of the requirements for the Master’s Degree. Nashville, TN. Berthoz, A., Pavard, B., and Young, L.R. (1975). Perception of linear horizontal self-motion induced by peripheral vision (linear vection): Basic characteristics and visual vestibular interactions. Experimental Brain Research, 23, 471–489. Berthoz, A., Israel, I., Georges-Francois, P., and Grasso, R. (1995). Spatial memory of body linear displacement: What is being stored? Science, 269, 95–98. Bles, W. (1981). Stepping around: Circular vection and Coriolis effects. In J. Long and A. Baddeley (Eds.), Attention and performance IX, pp. 47–62. Hillsdale, NJ: Lawrence Erlbaum. Dichgans, J. and Brandt, T. (1972). Visual–vestibular interaction and motion perception. Bibiotheca Ophthalmologica, 82, 327–338. Dichgans, J. and Brandt, T. (1978). Visual–vestibular interactions: Effect on self-motion perception and postural control. In R. Held, H. Leibowitz, and H.L. Teuber (Eds.), Handbook of sensory physiology, Vol. 8: Perception. New York: Springer. Frymire, M. (1998). Path integration when walking without vision as a function of whether or not the surrounding environment is known. Unpublished senior thesis with honors in Cognitive Studies, Vanderbilt University. Garing, A.E. (1999). Intersensory integration in the perception of self-movement. Unpublished Ph.D. dissertation, Vanderbilt University.

191


192


Giannopulu, I. and Lepecq, J-C. (1998). Linear vection chronometry along spinal and saggital axes in erect man. Perception, 27, 363–372. Gibson, J.J. (1958). Visually controlled locomotion and visual orientation in animals. British Journal of Psychology, 49, 182–194. Gibson, J.J. (1962). Observations on active touch. Psychological Review, 69, 477–491. Gibson, J.J. (1966). The senses considered as perceptual systems. Boston: Houghton Mif2in. Gibson, J.J. (1979). The ecological approach to visual perception. Boston: Houghton Mif2in. Guendry, F.E. (1974). Psychophysics of vestibular sensation. In H.H. Kornhuber (Ed.), Vestibular system. Part 2: Psychophysics, applied aspects and general interpretations. Berlin: Springer-Verlag. Guth, D.A. and Rieser, J.J. (1997). Perception and the control of locomotion by blind and visually impaired pedestrians. In B. Blasch, W. Wiener, and R. Welch (Eds.), Handbook of orientation and mobility, pp. 9–39. New York: American Foundation for the Blind. Helmholtz, H. von (1866). Handbuch der physiologischen Optik, Leipzig: Voss. Johansson, G. (1977). Studies on visual perception of locomotion. Perception, 6, 365–376. Johnson, C., Pick, H.L., Siegel, G.M., Ciccarielli, C., and Garber, S.R. (1981). Effects of interpersonal distance on children’s vocal intensity. Child Development, 52, 721–723. Kornhuber, H.H. (ed.) (1974). Vestibular system. Part 2: Psychophysics applied aspects and general interpretations. Berlin: Springer-Verlag. Lackner, J.R. (1988). Some proprioceptive in2uences on the perceptual representation of body shape and orientation. Brain, 111, 281–297. Lederman, S.J. and Klatzky, R.L. (1987). Hand movements: A window into haptic object recognition. Cognitive Psychology, 19, 342–368. Lederman, S.J. and Klatzky, R.L. (1998). Relative availability of surface and object properties during early haptic processing. Journal of Experimental Psychology: Human Perception and Performance, 23, 1680–1707. Lee, D.N. (1978). The functions of vision. In H. Pick and E. Salzmann (Eds.), Modes of perceiving and processing information. Hillsdale, NJ: Erlbaum. Lee, D.N. and Lishman, J.R. (1974). Visual proprioceptive control of stance. Journal of Human Movement Studies, 1, 87–95. Lepecq, J.C., Jouen, F., and Dubon, D. (1993). The effect of linear vection on manual aiming at memorized directions of stationary targets. Perception, 22, 49–60. Lishman, J.R. and Lee, D.N. (1973). The autonomy of visual kinesthesis. Perception, 2, 287–294. Loomis, J.M., Klatzky, R.L., Golledge, R.G., Cicinelli, J.G. (1993). Nonvisual navigation by blind and sighted: Assessment of path integration ability. Journal of Experimental Psychology: General, 122, 73–91. Loomis, J.M., Da-Silva, J.A., Philbeck, J.W., and Fukusima, S.S. (1996). Visual perception of location and distance. Current Directions in Psychological Science, 5, 72–77. Loomis, J.M. and Philbeck, J.W. (1999). Is the anisotropy of perceived 3-D shape invariant across scale? Perception and Psychophysics, 61(3), 397–402. Mach, E. (1875). Grundlinien der Lehre von den Bewegungsemp1ndungen. Leipzig: Wilhelm Engelmann. McMillan, A. (1998). Changes in whispering as a function of the recalibration of rotational locomotion. Unpublished senior thesis with honors in Cognitive Studies, Vanderbilt University. Michael, D., Siegel, G.M., and Pick, H.L., Jr. (1995). Effects of distance on vocal intensity. Journal of Speech and Hearing Research, 38, 1176–1183. Mountcastle, V.B. (1958). Somatic functions of the nervous system. Annual Review of Physiology, 20, 471–508. Philbeck, J.W. and Loomis, J.M. (1997). Comparison of two indicators of perceived egocentric distance under full-cue and reduced-cue conditions. Journal of Experimental Psychology: Human Perception and Performance, 23(1), 72–85. Philbeck, J.W., Loomis, J.M., and Beall, A.C. (1997). Visually perceived location is an invariant in the control of action. Perception and Psychophysics, 59(4), 601–612. Pick, H.L., Jr., Yonas, A., and Rieser, J.J. (1979). Spatial reference systems in perceptual development. In M. Bornstein and W. Kessen (Eds.), Psychological development from infancy. Hillsdale, NJ: Erlbaum. Pick, H.L., Jr., Rieser, J.J., Wagner, D., and Garing, A.E. (1999). The recalibration of rotational locomotion. Journal of Experimental Psychology: Human Perception and Performance, 25, 1179–1188. Post, R.B. (1988). Circular vection is independent of stimulus eccentricity. Perception, 17, 737–744. Rieser, J.J., and Heiman, M.L. (1982). Spatial self-reference systems and shortest-route behavior in toddlers. Child Development, 53, 524–533.



Rieser, J.J., Guth, D.A., and Hill, E.W. (1986). Sensitivity to perspective structure while walking without vision. Perception, 15, 173–188. Rieser, J.J. (1989). Access to knowledge of spatial structure from novel points of observation. Journal of Experimental Psychology: Human Learning, Memory, and Cognition, 15, 1157–1165. Rieser, J.J., Ashmead, D.A., Talor, C., and Youngquist, G. (1990). Visual perception and the guidance of locomotion without vision to previously seen targets. Perception, 19, 675–689. Rieser, J.J., Pick, H.L., Jr., Ashmead, D.A., and Garing, A.E. (1995). The calibration of human locomotion and models of perceptual–motor organization. Journal of Experimental Psychology: Human Perception and Performance, 21, 480–497. Rieser, J.J., McMillan, A., Pick, H.L., and Berry, D.M. (1998). Changes in the vocal intensity of whispering without vision as a function of the recalibration of rotational locomotion. Presentation to the Psychonomics Society. Rieser, J.J. (1999). Dynamic spatial orientation and the coupling of representation and action. In R. Golledge (Ed.), Cognitive mapping and spatial behavior. Baltimore, MD: Johns Hopkins Press. Rieser, J.J., Frymire, M., and Berry, D.M. (1999). Path integration when walking without vision depends on the geometry of the remembered surroundings. Presentation to the Psychonomics Society. Steenhuis, R.E. and Goodale, M.A. (1988). The effects of time and distance on accuracy of target-directed locomotion: Does an accurate short-term memory for spatial location exist? Journal of Motor Behavior, 20, 399–415. Taub, E., Ellman, S., and Berman, A. (1966). Deafferentation in monkeys: Effect on conditioned grasp response. Science, 151, 593–594. Telford, L. and Frost, B.J. (1993). Factors affecting the onset and magnitude of linear vection. Perception and Psychophysics, 52(6), 682–692. Wade, N.J. (1998). Light and sight since antiquity. Perception, 27, 637–670. Wagner, D.G. (1996). Generalizing recalibration of locomotion to natural spatial updating. Unpublished Ph.D. Dissertation, University of Minnesota, MN. Warren, W.H. (1995). Self-motion: Visual perception and visual control. In Handbook of perception and cognition, Vol. 5: Perception of space and motion. New York: Academic Press. Wertheim, A.H. (1994). Motion perception during self-motion: The direct versus inferential controversy revisited. Behavioral and Brain Sciences, 17, 293–310.

193




II

Timing in perception and action




9 Perspectives on the timing of events and actions Introduction to Section II Jeff Summers

Time perception in terms of the temporal properties of environmental events and time production in terms of the execution of self-generated sequences of precisely timed movements are abilities that are essential for successful interactions with the world around us. Furthermore, coordinating our actions in time to coincide with an external event, that is, perception–action coupling, is a feature of many everyday activities, from pressing a key on a piano, to shaking someone’s hand or hitting a squash ball. Unfortunately the study of the perception of event structure (i.e. the motional and rhythmic properties of events) and the study of movement timing have progressed somewhat independently, leaving the crucial question of how perceptual variables map onto motor variables largely neglected. In this section four papers are presented dealing with different aspects of timing. The 1rst chapter, by Wing and Beek, presents a tutorial review of studies of movement timing. This aspect of timerelated behaviour has a long history in experimental psychology dating back to the late 1800s, and the paradigmatic task has involved repetitive key tapping in synchrony with or as a continuation of a sequence of auditory stimuli. The question of interest has been: How does the motor system compose sequences of precisely timed movements? Historically, cognitive models of movement timing have appealed to central interval timing mechanisms or clocks that emit a regular series of pulses with some variability to trigger movements. In recent years, strong criticism of internal timekeepers has come from proponents of a dynamical systems approach to movement behaviour. This approach emphasizes self-organizing processes within the motor system and sees timing as an emergent consequence of these processes rather than as being explicitly controlled. Initial support for dynamical models came from periodicities evident in cyclic repetitive movements, such as locomotion and bimanual movements. In their chapter Wing and Beek juxtapose the cognitive and dynamical approaches on a number of research issues within the area of movement timing. What becomes clear is that over the last 1fteen years or so the two approaches have pursued their research agendas independently and in parallel, focusing on different kinds of phenomena and levels of analysis. This is nicely illustrated in the research conducted by both groups on bimanual polyrhythmic tapping. Dynamical systems researchers have focused on movement processes, in particular the intrinsic constraints on the phasing between the limbs and the mechanisms underlying loss of stability and pattern switches. Cognitive researchers, in contrast, have been concerned more with the behavioural events (e.g. key


198


taps) produced by the movements and have attempted to uncover the underlying motor organization adopted by subjects through an analysis of the covariability among intertap intervals. Abernethy and Sparrow in 1992 predicted that the motor behaviour 1eld was entering a long period of bitter and intense con2ict from which one of the competing views would emerge as dominant. Others, in contrast, argued that the 1eld would move towards a reconciliation between the two approaches (e.g. Summers 1992). In the chapter written by Wing and Beek, two of the leading 1gures in the ‘opposing’ camps, the mood is clearly toward a merging of the approaches. Whether the merging is best realized by a uni1ed mathematical form (Pressing 1999), a two-level oscillator model (Peper, Beek, and Daffertshofer 2000), or a two-level organization incorporating both central timekeeping processes and trajectory coupling phenomena (Semjen 2001) remains to be seen. In the second chapter Aschersleben and colleagues investigate the role sensory feedback may play in the online correction of movements that have to be synchronized with an external stimulus. To address this issue the authors exploit the negative asynchrony observed when subjects are asked to tap the index 1nger to coincide with a predictable external stimulus, usually an isochronous sequence of auditory pacing signals. The asynchrony, noted over 100 years ago, refers to the 1nding that in this simple repetitive task subjects typically tap consistently ahead of the tones by some 30–50 milliseconds. One explanation for the anticipation error maintains that synchrony involves a matching, at a central representational level, between the sensory consequences of the tap and the auditory pacing signal. As it takes longer for the tactile and kinesthetic feedback from the tapping 1nger to reach the brain than the auditory feedback from the ear, the tap must be executed prior to the pacing signal for synchrony to be perceived at the central level. The authors present several lines of evidence in support of a model (the sensory accumulator model) in which the temporal asynchronies re2ect differences in processing times needed to generate the central representations, rather than pure nerve conduction time differences. Of particular interest is a series of experiments involving a deafferented subject, IW, to examine the effect of eliminating sensory feedback from the 1nger on the anticipation error. The studies not only show that IW has been able to substitute visual feedback for the missing kinesthetic feedback to spatially control his movements, but also that even in the absence of all feedback sources he was still able to produce a relatively stable series of taps coordinated with the pacing signal. To the authors, the latter remarkable feat is consistent with the notion of forward modelling in which the expected sensory consequences of an action (i.e. tap) can be used to time the action (ie. match with the perceived pacing signal) even when no actual feedback is available. At another level the chapter by Aschersleben and colleagues illustrates nicely the importance of multiple sources of information in interceptive timing tasks. That is, under the minimal information conditions that exist in the synchronization experiments, subjects’ performance is quite poor and determined mainly by sensory processing times. However, in everyday activities where predictive information can be gained from a variety of sources (e.g. optic 2ow), interceptive timing is remarkably accurate. For example, compare the 30–50 millisecond error in synchronized tapping with the 2–5 millisecond timing accuracy exhibited at bat/ball contact by expert table tennis players when executing a series of forehand drives towards a small target area across the net (Bootsma and van Wieringen 1990). The importance of information sources on synchronization performance is further illustrated in the chapter by Bruno Repp. Two experiments are described in which skilled musicians attempted to synchronize 1nger taps with click sequences that represented variations from an average expressive timing pro1le derived from an analysis of 115 expert performances of a musical excerpt. It should be noted that expressive timing patterns when represented as a click sequence are quite meaningless.


Perspectives on the timing of events and actions

The question of interest was whether adding musical context to the clicks would facilitate synchronization performance. Providing the click sequences with accompanying music immediately improved synchronization, particularly for the most typical pattern but also for other expressive timing patterns. These 1ndings in the music domain provide a clear illustration of the tight coupling between perception and action. Furthermore, both having accompanying music and having subjects merely imagine the music produced systematic deviations from regularity in the tapping of an isochronous click sequence. These involuntary effects provide further evidence of coupling between music perception, musical imagery, and movement timing. In the 1nal chapter, Haggard looks at the role of consciousness in the link between perception and action. While the debate over what consciousness is continues (e.g. Dennett 2001), Haggard approaches the issue by asking what is the function of consciousness when achieved in the generation of intentional action. Conscious representations are seen as contributing a constructive role by providing the necessary link between intention and action underlying the sense of agency. To construct the relation between intention and action, consciousness participates in a hypothetical neural process of efferent binding. Through efferent binding intentions, actions and the environmental sensory consequences of these actions are linked. To examine the binding between action and effect Libet et al. (1983) paradigm is resurrected in which subjects judge via the rotating hand of a clock the perceived time of occurrence of sensory (a pure tone) and motor (a key tap) events. Although one may question the ecological validity of the paradigm, strong support for the efferent binding hypothesis is obtained across two experiments, in the form of interchangeable attraction effects when stimuli and actions are associated in a causal context. It appears, therefore, that the processes underlying conscious awareness are similar for stimulus and motor representations. The interplay between perception and action is the central theme of the present volume, and the nature of the interface between environmental information and the control of movement is also the source of greatest division between the ecological and cognitive theoretical positions. In the timing domain, cognitivists have addressed the question of how timing in perception maps onto timing in action by assuming common representations and coding for environmental events and actions and a common mechanism underlying the perception and production of time (e.g. Ivry and Hazeltine 1995). Proponents of the ecological view, in contrast, expunge the notion of timing mechanisms and rely on the Gibsonian concept of affordance as the link in the direct coupling between perception and action. The direct perception perspective, however, has been hampered by a lack of consensus within the ecological community with regard to the nature and role of affordances (e.g. Michaels 2000; Stoffregen 2000) and the environmental information relevant to the control of movement (Summers 1998). As illustrated in the review by Wing and Beek, the study of movement timing has to a large extent ignored the perceptual side of the perception–action cycle. Implicit in most models of key tapping is the assumption that perceptual variables in2uence timing by modulating the internal timekeeping processes that pace movements (e.g. Repp, this volume, Chapter 12). Analyses of coordination dynamics (arguably a perspective within ecological psychology) also rarely refer to the affordance concept or to how perceptual variables map onto motor variables. Recently, however, a strong link has been demonstrated between coordination dynamics and perceptual dynamics (e.g. Zaal, Bingham, and Schmidt 2000). That is, the common measures of interlimb coordination, relative phase and phase variability, appear to be properties to which the perceptual system is also sensitive. Furthermore, the stability of judgements of the relative phasing of two circles moving rhythmically on a computer display exhibited an inverted U-shaped function of relative phase similar to that

199


200


obtained with rhythmic limb movements. Interestingly, highest variability in the perceptual judgements was found around 90° relative phase, even when there was no variability in the display. One, somewhat radical, interpretation of these results is that patterns of interlimb coordination are primarily perceptually determined rather than re2ecting physical constraints on limb movements imposed by coupled oscillator systems (see Zaal et al. 2000, for other interpretations). Bruno Repp’s 1nding that the perception of musical structure involuntarily affected synchronization with an isochronous click sequence is another illustration of the powerful in2uence of perceptual factors on movement timing. At the very least, these results suggest that future models of coordination dynamics will need to more explicitly model how perceptual variables interact with motor coordination variables. The remaining two chapters in the section provide support for the alternative view that perception–action coupling is a consequence of a common representational basis for perceptual events and actions. The sensory accumulator model proposed by Aschersleben and colleagues to account for the negative asynchronies in their tapping task, assumes that synchronization is achieved through the matching of central representations of clicks and taps. A further assumption of the model is that the representation or neural state is in terms of the sensory consequences of an event or action (i.e. somatosensory feedback or reafference). Finally, the generation of a representation involves the accumulation of sensory evidence (afferent neural signals) until a threshold is reached. According to this account the negative asynchrony between a tap and the auditory pacing signal is a consequence of different accumulation functions for auditory and tactile information. Support for the model has come from a variety of experiments in which factors (e.g. amount of sensory feedback) affecting the accumulation functions have been manipulated. The model proposes, therefore, a common representational coding for both environmental events and actions and a common mechanism (accumulation function) through which they are coordinated. Haggard and colleagues, using a different experimental paradigm, provide further support for a common coding for perceptual events and actions and for the hypothesis that the time of an action is determined by its consequences rather than antecedents. The general aim of the research reported in this section has been to elucidate the basic mechanisms underlying the timing of movements. In order to examine these mechanisms without confounding from other factors (e.g. environmental and musculo-skeletal constraints), researchers in the timing domain have typically chosen simple tasks in which the environmental events are con1ned to sequences of auditory signals (tones) and actions to simple key taps. An important issue, therefore, is the extent to which the timing mechanisms identi1ed for this class of tasks also underlie tasks involving more complex interactions between perception and action. There is some evidence, for example, to suggest that different timing mechanisms may predominate in the control of discrete versus continuous movements (Semjen 2001).

References Abernethy, B. and Sparrow, W.A. (1992). The rise and fall of dominant paradigms in motor behaviour research. In J.J. Summers (Ed.), Approaches to the study of motor control and learning, pp.3–46. Amsterdam: NorthHolland. Bootsma, R.J. and van Wieringen, P.W.C. (1990). Timing an attacking forehand drive in table tennis. Journal of Experimental Psychology: Human Perception and Performance, 16, 21–29.


Perspectives on the timing of events and actions

Dennett, D. (2001). Are we explaining consciousness yet? Cognition, 79, 221–237. Ivry, R.B. and Hazeltine, R.E. (1995). Perception and production of temporal intervals across a range of durations: Evidence for a common timing mechanism. Journal of Experimental Psychology: Human Perception and Performance, 21, 3–18. Libet, B., Gleason, C.A., Wright, E.W. and Pearl, D.K. (1983). Time of conscious intention to act in relation to onset of cerebral activity (readiness potential). Brain, 106, 623–642. Michaels, C.F. (2000). Information, perception, and action: What should ecological psychologists learn from Milner and Goodale (1995). Ecological Psychology, 12, 241–258. Peper, C.E., Beek, P.J., and Daffertshofer, A. (2000). Considerations regarding a comprehensive model of (poly)rhythmic movement. In P. Desain and L. Windsor (Eds.), Rhythm perception and production, pp. 35–50. Lisse: Swets and Zeitlinger. Pressing, J. (1999). The referential dynamics of cognition and action. Psychological Review, 106, 714–747. Semjen, A. (2001). On the timing basis of bimanual coordination in discrete and continuous tasks. Brain and Cognition, doi:10.1006/brcg.2001.1309. (Online reference.) Stoffregen, T.A. (2000). Affordances and events. Ecological Psychology, 12, 1–28. Summers, J.J. (1992). Movement behaviour: A 1eld in crisis? In J.J. Summers (Ed.), Approaches to the study of motor control and learning, pp. 551–562. Amsterdam: North-Holland. Summers, J.J. (1998). Has ecological psychology delivered what it promised? In J. Piek (Ed.), Motor behavior and human skill: A multidisciplinary approach, pp. 385–402. Champaign, IL: Human Kinetics. Zaal, F.T.J.M., Bingham, G.P., and Schmidt, R.C. (2000). Visual perception of mean relative phase and phase variability. Journal of Experimental Psychology: Human Perception and Performance, 26, 1209–1220.

201

aapc10.fm Page 202 Thursday, December 6, 2001 9:52 AM

10 Movement timing: a tutorial Alan M. Wing and Peter J. Beek

10.1 Introduction: two traditions in studying the timing of movement Two broad traditions underlie behavioural studies of movement timing. In one, which we will refer to as the information processing approach, time is considered a mental abstraction, applicable to, but represented independently of, any particular effector system. In this view our ability to carry out an action fast or slow, to write slow or fast, to speak with variation in rate, or to adjust the speed of the music we play, depends on central timing processes. It may be supposed that these depend on brain circuits that make contact with the motor system. However, they are functionally contained in that they do not require the action of any particular effector system. Rather they may be set to initiate movements at certain times, and these movements’ other parameters, such as force or direction, may be independently speci1ed. In the other perspective, which we will call the dynamical systems approach, timing is considered a by-product or emergent property of the organizational principles (i.e. dynamical equations of motion) governing a particular coordinated action. If an action has a characteristic timing, it is part and parcel of other movement dimensions of that action, such as frequency or its dynamical equivalent, stiffness. A repetitive activity such as handwriting may be carried out with regular timing but that is a consequence of a dynamical regime specifying a sequence of pen stroke directions and amplitudes under particular stiffness constraints. Time as such is not an explicitly controlled variable, but follows from dynamical equations of motion and their parameter settings. In this view the control of timing in the production of a musical rhythm may thus be said to follow from the effector system used to implement movement. In this tutorial review we consider the two approaches as they have been applied to repetitive timing behaviour. We 1rst describe a commonly used experimental paradigm, the continuation of a previously speci1ed interval. Then we review a number of performance measures. Next we detail two models as representative of the two approaches before considering how each has been applied to a set of studies illustrating a range of experimental paradigms. These ‘application areas’ include dual-task isochronous unpaced responding, synchronization, multi-effector, isochronous timing, and multi-effector, multifrequency timing. We conclude that we are witnessing a vibrant and active 1eld of behavioural research which has many and varied analytic tools. Although the two traditions have maintained largely separate development paths, our review suggests ways in which the two can be seen as complementary and deserving of more attention to bridges that might be built between them.


Movement timing: a tutorial

10.2 Paced and free tapping Our starting point is a paradigm developed over 100 years ago by an American psychologist working at Harvard University, L.T. Stevens. In his research into the ‘time sense’, Stevens asked the subject to tap a morse telegraph key in time with a metronome. Once the subject was following the beat, Stevens stopped the metronome while the subject continued to tap on his own. Using a smoked drum kymograph to record key depressions with an accuracy of one millisecond, Stevens’ interest was in the accuracy of the interresponse intervals produced in the continuation phase. He presented his data in the form of graphs with interval duration on the vertical axis and repetition along the horizontal axis; that is, a time time series. Fig. 10.1 shows three such time series with mean intervals 400, 600, 1000 ms. It will be observed that the time intervals produced re2ect the different metronome target times, but that there is some 2uctuation around the correct value, which is greater at longer intervals. Stevens’ use of a paradigm combining synchronization with continuation was an important methodological contribution to the study of timing, offering experimental control over the produced interval. The effectiveness of the paradigm testi1es to the adjustability of timing and also to the stability of timing, in the sense that the interval is maintained at approximately the right value well after synchronization stops. However, it may be noted that maintenance of the interval at the correct value is not a trivial task and indeed Stevens noted that the intervals 2uctuate more at slower rates. In fact, Stevens made two fundamental observations about the nature of the variability in timing.

Fig. 10.1 Paradigm for studying timing (Stevens 1886). (a) Trials are in two phases. An initial synchronization phase is followed by a continuation phase when the subject continues to tap at the same rate. (b) Data from three different trials with target interval of 400, 600, and 1000 ms shown as a time series with intervals on the y-axis shown in succession along the x-axis. (c) Variability increases with mean interval.

203


204


He observed that the variability had short- and long-term components. Short-term 2uctuations around the mean he attributed to motor limitation—‘the hand (or perhaps the will during the interval) cannot be accurately true’. Long-term drift around the target he suggested re2ected ‘rhythmic variation of the standard carried in the mind’. Although these quotes may sound somewhat quaint to modern-day academics, the insight that variability in timing may be partitioned into components related to timing control structures on the one hand, and the effector system on the other, is essential, and still plays a key role in the theory and experimental study of timed motor behaviour.

10.3 Linear models and measures of performance Making inferences about the timing control structures that may underlie performance in tasks such as Stevens’ paradigm depends on accurate characterization of the behaviour being studied. Stevens’ observations on variability were based on visual inspection. However, we now have more sophisticated statistical tools at our disposal to characterize the nature of variation and these are reviewed under two headings. We 1rst consider those that apply when behaviour is stable based on the concept of a general linear model. That is, it is assumed that the behaviour of interest, in the present case the series of time intervals, is generated by a process that can be expressed as a weighted set of additive components plus a random element which re2ects a residual indeterminacy in the system. This may be characterized in terms of a linear or polynomial regression (a deterministic trend) plus random noise with mean zero. A key assumption in using regression methods to characterize the time series is that the noise term is stationary; its distributional properties do not evolve with time. Linear models have also been developed where the successive observations are related, not through a deterministic trend resulting in mean shift, but in local dependence. In autoregressive models, the deviation of the current observation from the mean (y) is proportionally related to previous observations’ deviations from the mean plus a random term (x). A 1rst-order autoregressive process assumes just one proportionality term. Thus: y ( j) = a*y ( j − 1) + x ( j) −1 < a < 1. Another class of model is the moving average, in which the deviation of the current observation from the mean is given by a random term plus a weighted sum of previous random terms. Thus, for a 1rst-order moving average process: y (j) = b*x (j − 1) + x (j)

–1 < b < 1.

Such models may be distinguished in terms of their autocorrelation function. For a series of observations, the autocorrelation at lag one is de1ned as the average of products of successive deviations from the mean taken over the whole series (the autocovariance) divided by the variance of the series. At lag two the products involve deviations two steps, rather than one step, apart. In general, the autocorrelation at lag k is given by: A(k) = [Av {y (j)*y ( j − k)}]/var(y)

k = 1, N.

The value of the autocorrelation is bounded by plus and minus one. At lag zero, the autocovariance in the numerator is equal to the denominator so the autocorrelation is 1. Low-order autoregressive



models give rise to autocorrelation functions with nonzero values persisting at long lags (the values decrease with increase in lag k). Low-order moving average models have nonzero autocorrelation only at low lags, the transition to zero autocorrelation being determined by the order of the moving average process. An alternative representation of the form of variability in stationary processes is the power spectrum. The time series is represented in terms of the relative power of a set of sinusoidal components. Rapid 2uctuations involving closely-spaced observations are represented by sinusoidal terms of short period (high frequency). Fluctuations around the mean taking place over observations spaced further apart (at a longer lag) are represented by sinusoidal terms of longer period (lower frequency). Random variation is represented by equal spread of power across all frequencies, sometimes referred to as white or Gaussian noise. It will be observed that the use of the spectrum to characterize the patterning of values in a time time series (time values ordered by occurrence) departs from another use of the power spectrum in which some measure, such as amplitude of movement, is observed as a continuous function of time. In such cases the form of the power spectrum constitutes a description of the movement trajectory. In repetitive tapping, a signi1cant proportion of the power is concentrated at the frequency corresponding to the reciprocal of the mean interval being produced.

10.4 Nonlinear models and measures of performance The assumption is often made in psychology that behaviour is stable over time. If repeated observations do not yield identical results, it is assumed that the process is subject to random variation but that long-term predictability improves on average with the availability of more observations. In the dynamical systems approach to the study of timing the interest is in the evolution of performance with time. Characterization of a dynamical system is a matter of describing its evolution path, or trajectory. This may be to compare observed properties with those predicted from assumed algebra or the inverse, to identify the underlying model which provides a suitable phenomenological description of the observed data in the sense of a formal analogy (cf. Beek, Peper, and Stegeman 1995). Stability and changes in stability due to variations in internal or external parameters form a special entry point for constructing dynamical models of behaviour. Of particular interest are those behavioural organizations that are resistant to small, transient perturbations. Such stable organizations are called attractors and are de1ned mathematically as regions in phase space to which trajectories converge (see Fig. 10.2). Phase space is a multidimensional space whose axes represent system variables, excluding time, such as effector position and velocity, so that a trajectory in phase space depicts the evolution in time from a particular set of starting conditions. (For these and other useful de1nitions see Gregson and Pressing 2000.) In the theory of dynamical systems, as it has evolved to date, four types of attractors are recognized. These are: (1) the point attractor or stable 1xed point (nearby trajectories converge onto a point; e.g. the equilibrium point of a damped pendulum), (2) the periodic attractor or limit cycle (trajectories converge onto a closed orbit; e.g. the periodic oscillations of a damped pendulum with an escapement to sustain the oscillations), (3) the quasi-periodic attractor (trajectories converge onto a non-closed orbit de1ned by a number (≥2) of basic frequencies that do not relate as integers; e.g. an externally driven limit cycle oscillator), and (4) the chaotic attractor (trajectories converge onto a non-closed orbit that cannot be meaningfully decomposed into a number of basic frequencies

205


206


x(t)

y(t)

2.00 1.00 0.00 – 1.00 – 2.00

2.00 1.00

t

0.00

y(t) 2.00 1.00 0.00 – 1.00 – 2.00

– 1.00 – 2.00

– 2.00 – 1.00 0.00 1.00

2.00

t

x(t)

Fig. 10.2 Example of phase plane trajectory (phase portrait) of the Rayleigh oscillator for different initial conditions (x(t) represents position and y(t) velocity). Even if motion is started outside the normal range of interest the trajectory rapidly converges onto the characteristic limit cycle of the oscillator. because all frequencies are present; e.g. a dripping faucet or a metal pendulum attracted by three magnets positioned in a triangle underneath it). A remarkable property of the latter kind of dynamically stable organization is that the resulting erratic behaviour appears to be noise to the observer, at least at 1rst glance, and is equally unpredictable in the long run, even though it is intrinsically deterministic. Therefore, chaos in the technical sense of dynamical systems theory is also called deterministic or classical chaos. Necessary conditions for the occurrence of chaos in time-continuous dynamical systems are that the evolution laws describing the relations between the state variables are nonlinear and that the number of relevant state variables is three or more. Thus, when variability is studied from the perspective of dynamical systems theory, the assumption of a linear model with superimposed random variability (stochastic noise) is typically avoided, leaving open the possibility that the observed variability is the result of deterministic chaos. Traditional statistical methods, such as analysis of variance, linear time series analysis or frequency partition into energy spectra derive from a general linear model with superimposed additive random noise and an underlying assumption of stability. The characterization of a dynamical system trajectory exceeds the bounds of this model. In the absence of analytic solutions to nonlinear dynamics, investigators may need to draw on qualitative methods for studying a dynamical system’s topology (as recognized by Poincaré almost a century ago) and quantitative methods for studying its dimensionality and 2ow characteristics. A common issue is to determine whether, in the 1rst place, a dataset is low-dimensional and possibly chaotic, or, instead, random in the sense of being stochastic and of high dimensionality. One approach in this case is to determine the Lyapunov exponent. This measure describes the exponential rate of convergence/divergence of two initially close points in phase space along relevant dimensions of the attractor. One issue for such an analysis is that a large number of observations are required, certainly in the hundreds if not thousands. Producing a large number of interresponse intervals in a laboratory context can be quite tedious and inattention or fatigue may develop. Thus the conditions required for analysing nonstationarity may, themselves, change performance from being stationary into an unstable process subject to drift.



10.5 Two approaches to modelling timing We now turn to consider two contrasting approaches to studying variability in timing. One, 1rst set out by Wing and Kristofferson (1973), follows Stevens in addressing the nature of variability in repetitive tapping with one hand. This approach concerns steady-state, stable performance and assumes simple, noninteracting component processes. Movements are treated in terms of discrete events or responses that demarcate psychologically signi1cant time intervals. A second approach, proposed by Haken, Kelso, and Bunz (1985), applies to two-hand responding that is synchronized with a metronome whose speed is progressively changed over a series of blocks. It is concerned with progressive changes (which can include qualitative changes or phase transitions) in relative phase of these movements of the two hands due to nonlinear interactions of component oscillatory processes. In this approach, movements are represented as continuous functions of time.

10.6 Wing–Kristofferson (WK) model The fundamental idea in the WK model is that variability in tapping re2ects two independent additive sources of variance, one arising in an adjustable central timekeeper and the other in delays associated with motor implementation (see Fig. 10.3). Wing and Kristofferson (1973) proposed a hierarchical arrangement in which responses, triggered by the central timekeeper at the end of each internally generated interval Cj, are subject to delays in motor implementation Mj before the occurrence of observable responses. If, over successive responses, j = 1, 2, . . . N, the Cj and Mj are statistically independent Mj, and both are random variables, the Ij have variance: var(I) = var(C) + 2var(M). The model predicts dependence between adjacent Ij, Ij + 1 with lag one autocovariance: acov(I(1)) = −var(M) This results in the following prediction for the autocorrelation at lag k: acorr[I(k)] = acov(I(k))/var (I) = −1/{2 + (var(C)/var (M))} k = 1 0 k > 1. From the last relation it can be seen that the WK model predicts that the lag one autocorrelation (the correlation between adjacent pairs of intervals taken through the sequence of intervals) should be bounded by zero and minus one-half. The autocorrelation at higher lags is predicted to be equal to zero, acorr[I(k)] = 0, k > 1. This has generally been reported to be the case. However, a number of studies have reported a proportion of estimates outside this range. This may represent statistical 2uctuation and/or bias in estimators associated with relatively short sequences (see Vorberg and Wing 1996). From a control point of view the WK model is very simple—it has no feedback loop to correct errors. Indeed, for this reason, it is inadequate as an account of synchronized tapping with a metronome, a point to which we return later. However, the model does embody the form of the short-term

207


208


Fig. 10.3 Wing–Kristofferson (WK) timing model. Variable timekeeper intervals subject to random motor implementation delays result in interresponse intervals that are negatively autocorrelated at lag 1. variability noted by Stevens in predicting the negative correlation of successive intervals (Wing and Kristofferson 1973; Vorberg and Wing 1996). In fact, this is an example of a more speci1c case, that random delays superimposed on a periodic appointment stream result in patterns of short–long alternation of intervals occurring more often than would be expected from a random series (Govier and Lewis 1967). In the more general case of the WK model the magnitude of the negative correlation produced by the two-level model depends on the relative amount of variability in timekeeper and motor delays. According to the WK model, the variability of the observed interresponse intervals is equal to the sum of the variability of the timekeeper process and twice the variance of the motor delays. The covariance of successive intervals is equal to the variance of the motor delays. Two equations expressing two observable quantities in terms of two unknowns can be solved to give the unknowns—in this case giving the variances of the underlying component processes, in terms of the observable interresponse interval measures. Thus, the covariation between successive intervals can be used to estimate variance of motor delays. Then, by subtraction of the motor delay variance from the interresponse interval variance, it is possible to estimate the variance of the timekeeper intervals. This approach was taken, for example, in an experiment in which subjects produced on each trial a series of responses with target interval selected from a range 290 to 540 ms (Wing 1980). It was found that timekeeper variance increased linearly with mean, whereas estimates of motor delay variance were relatively constant. Thus, at longer intervals, variability re2ected the timer, while at shorter intervals the motor implementation delays were relatively more important (and so the correlation between successive intervals was more negative at short intervals). The WK model of timing has been extended to provide a psychological account of rhythm (Fig. 10.4). Western music is frequently organized into rhythmic 1gures that follow hierarchical rules. Thus bars are subdivided into beats which may be further subdivided into simple fractions of the beat. This led Vorberg and Hambuch (1978, 1984) to suggest that the production of rhythms may involve separate timers at each level of the hierarchy. Assuming variability at each level is independent of variability at other levels leads to a prediction of negative correlation, not only for adjacent interresponse intervals, but also for some nonadjacent intervals. Moreover, intervals between the repetition of any particular response in successive cycles of the rhythm will have variability related



Fig. 10.4 Vorberg and Hambuch extended the WK model to include multiple timekeepers nested in a hierarchy. The cycle interval de1ned at the highest level of the hierarchy (bold line) has the lowest variability. to the position of that response in the hierarchy. Both predictions received support in an analysis of two-handed synchronous responding (Vorberg and Hambuch 1984) which partialled out characteristics of the motor delays by using covariation between left- and right-hand interresponse intervals to estimate multilevel timer properties. However, in this analysis some cases of positive rather than negative covariation in the timekeeper were found and these are not predicted by the basic hierarchical timer model. One possible account of such positive covariation in interresponse intervals is that it re2ects propagation of a rate parameter through successive layers of the timer hierarchy (Vorberg and Wing 1996). In music, speeding up or slowing down does not affect the fundamental structure of a rhythm. Thus it is reasonable to suppose that rhythms are speci1ed in terms of ratios of intervals (e.g. 1:2) and not by the absolute durations of those elements (e.g. 150 ms with 300 ms). However, operation of the timekeeper at each level in the hierarchy does ultimately require the duration be speci1ed. Thus, the model assumes that, before each cycle of a rhythm is produced, a preparatory process propagates a rate parameter down through the hierarchy and the multiplicative nature of this process can introduce overall positive correlations, albeit still modulated by negative correlations re2ecting the hierarchy. This model has been successfully applied to data from skilled musicians performing rhythm production tasks (Krampe et al. 2000).

10.7 Haken–Kelso–Bunz (HKB) model The starting point for the HKB model was the observation of Kelso (1984) of an abrupt, spontaneous transition in the phase coordination between the index 1ngers of the hands when making

209


210


Fig. 10.5 Phase transitions in bilateral index 1nger oscillations. x1 and x2 represent the 1nger displacements. Finger trajectories as a function of time (above) and relative phase as a function of frequency (below) show antiphase 1nger movements switch to inphase movements.

simultaneous cyclic extension/2exion movements at a common frequency (see Fig. 10.5). A transition from one phase coordination to another occurs as frequency increases. If the movements start in antiphase (involving nonhomologous muscles) they may switch to moving in phase (homologous muscles). HKB showed that this behaviour could be modelled in terms of a so-called potential V(φ) which they de1ned by dφ/dt = −dV/dφ so that it re2ects the rate of change of relative phase. In order to represent the fact that antiphase movements can be produced in a relatively stable manner at lower frequencies but not at higher frequencies, and given the periodic nature of the system under study, HKB suggested the potential be de1ned as the sum of two cosines: V(φ) = −acos(φ) − bcos(2φ) The parameters a,b specify a family of functions with extrema (i.e. minima or maxima) at 0 and 180 deg. If the ratio b/a, the so-called bifurcation parameter, is larger than 0.25 (with a and b both larger than 0), the extrema at 0 and 180 deg are minima (Fig. 10.6). At these minima a change in the value of relative phase φ results in a rate of change of relative phase that tends to restore relative phase to the value before the change. Hence these minima represent points of stable relative phase. In the HKB model it is assumed that the ratio b/a decreases with frequency so that, ultimately, when b/a becomes smaller than 0.25, the minimum at 180 deg disappears. In this approach, relative phase, φ, is termed a collective variable or order parameter (capturing system states in terms of



Fig. 10.6 Haken–Kelso–Bunz (HKB) potential V describing the loss of stability of antiphase coordination with increasing frequency. As the control parameter frequency increases the ratio b/a decreases leaving only the stable region at φ = 0. (Adapted from Daffertshofer 1997.)

coordination) and frequency is a control parameter (affecting the dynamics of the order parameter without actually specifying the dynamics). In order to account for the description of phase switching behaviour by the potential, HKB proposed a model in which movements of each hand are described in terms of two oscillators subject to a pair of functions that couple the state of each effector to that of the other. They modelled each 1nger as a nonlinearly damped oscillator with a linear negative damping term for the insertion of energy and a nonlinear energy dissipation term—HKB considered x2(dx/dt) (Van der Pol) and (dx/dt)3 (Rayleigh)—which make the oscillations self-sustaining. In the phase plane (with dx/dt plotted against x) the trajectory is a limit cycle. Kinematic studies (e.g. Kay, Kelso, Saltzman, and Schöner 1987) have provided support for a limit-cycle model of 1nger motion involving both a Van der Pol and a Rayleigh term. Thus, increasing frequency results in oscillations with smaller amplitudes (a consequence of the Rayleigh term) and higher peak velocities (a consequence of the Van der Pol term). Using, for the sake of convenience, a limit-cycle model of the 1nger movements with only a Rayleigh term (that is, a Rayleigh oscillator), HKB analysed what coupling functions linking the states of one effector to those of the other, and vice versa, would produce the desired theoretical order parameter equation. Although many different mathematical forms of the coupling function are possible, HKB considered two speci1c cases. In one, the coupling is determined by time derivatives of the state variables, in the other, by time-delayed values of the state variables. Recall that, in the HKB account, the relative phases 0 and 180 deg are stable, at least at low frequencies. That is to say that a change in relative phase results in restorative rates of change of

211


212


Fig. 10.7 The stability of a minimum in a potential V as measured by the standard deviation SD (a, b, c) or the relaxation time τrel (d, e, f). The SD is smaller for a steep minimum (a) than for a shallower minimum (b). If the shape of the potential is varied by manipulating a control parameter, the SD exhibits the corresponding change in stability of the stationary state (c). Similarly, in a steep minimum, the system relaxes more quickly from a small perturbation of given size ε (d) than in the case of a shallower potential (e). If the shape of the potential is varied by manipulating a control parameter, τrel also re2ects the induced change in the stability of the stationary state (f). (Figure adapted from Kelso et al. 1994.) relative phase. If phase is perturbed (e.g. the movement of one hand is brie2y arrested), relative phase should subsequently return to its pre-perturbation value. However, the potential landscape changes with frequency. At higher frequencies, the local minimum at 180 deg gets shallower and eventually disappears, so that 0 deg remains as the only stable region. At this point spontaneous transitions from 180 to 0 deg relative phase would be expected. Prior to this point, changes in the potential should be evident in a decreased stability with longer settling (or ‘relaxation’) time after perturbation at higher frequencies (see Fig. 10.7). This has been observed to be the case by Scholz, Kelso, and Schöner (1987). Biological systems are invariably prone to random 2uctuations, or noise, and Schöner, Haken, and Kelso (1986) considered the effect of adding random noise to the equation of motion of the order parameter, relative phase (i.e. dφ/dt = −dV/dφ). This extension to the HKB model predicts that, at suf1ciently low frequencies, the variability of relative phase is lower for in-phase than for antiphase movements, and that the variability of relative phase of the two coordination modes increases differentially with increasing movement frequency. In addition, this stochastic model predicts a strong increase of the variability of relative phase in the antiphase mode



just prior to the phase transition. These critical 2uctuations are a consequence of the 2attening of the potential well corresponding to the 180 deg phase relation and have been observed by Schöner et al. (1986).

10.8 Application area 1: dual task isochronous free responding The regularity of tapping, or more precisely, the variance, has been used as an index of mental load (Michon 1966). It is therefore interesting to review two studies of attention and timing, one from the information processing and one from the dynamical systems perspective. Sergent, Hellige, and Cherry (1993) analysed the effects of concurrent anagram solving on timing in terms of the twolevel timing model. They were interested to know whether the cognitive task would have differential effects on estimates of timer and motor delay. Figure 10.8 shows that, compared to no secondary task baseline, the secondary task increased var(C), leaving var(M) unchanged. One possible reason for selective central interference is that memory processes for timing are affected by the concurrent tasks (Inhoff and Bisiacchi 1990; Saito and Ishio 1998). A possible account of memory processes in central timing was provided by Gibbon, Church, and Meck (1984), based on ideas of Creelman (1962) and Treisman (1963). Gibbon et al. assumed that timekeeping is based on pacemaker pulses gated into an accumulator with a count being compared with a target value maintained in a reference memory to determine when a response should be made. From this perspective, impaired timing during simultaneous performance of another task might result from disturbances to reference memory or disruption of the gating process. Yamanishi, Kawato, and Suzuki (1979) took an approach to the effects of attention on repetitive 1nger tapping more in line with the dynamical systems perspective. They were interested in determining whether oscillatory 1nger movements in tapping would exhibit local instability which varied as a function of a contrasting interfering secondary task. They reasoned that varying the degree of cognitive complexity of a probe task to include speaking, remembering a visual decision, or responding with a 1nger of the other hand while tapping might be used to identify the level of the oscillator circuit subserving 1nger tapping. If interference was equal in all cases, then the oscillator

Fig. 10.8 A concurrent secondary task (anagram solution) increases var(C) not var(M) compared to baseline conditions without secondary task. (Adapted from Sergent et al. 1993.)

213


214


circuits include cognitive functions such as visual perception and memory, whereas if interference was restricted, say, to the 1nger movement condition, it would suggest the oscillator involved overlapping motor processes but was insulated from the cognitive processes. Yamanishi et al. used phase transition curves (PTC) to describe the nature of the interactions. They showed that the tendency to reset the tapping was greatest in the case of the motor task. In contrast the other tasks had relatively little effect, leading to the conclusion that the oscillator includes the motor system. Such coupling between the hands, in this case between a continuous oscillator in one hand and a discrete movement in the other, anticipated the assumptions of the HKB model. It is interesting to note that PTCs provide an alternative to Scholz et al.’s (1987) method of estimating the relative stability of the oscillator using relaxation time.

10.9 Application area 2: synchronization The WK model is an open-loop model and makes no provision for processing feedback about the times of responses. Yet synchronization, as in the Stevens paradigm, is often used to de1ne the target interval and subjects typically experience no dif1culty tapping in phase. How might the timekeeper be adjusted to keep responses in phase with an external pacing stimulus? One possible approach was outlined by Vorberg and Wing (1996) and developed by Semjen, Schulze, and Vorberg (2000). It assumes that the current timekeeper interval is adjusted as a 1xed proportion of the asynchronies between the previous pacing stimuli and associated responses. The correction is linear in taking a 1xed proportion of either just the immediately preceding response (1rst order) or the response before that as well (second order). A(n + 1) = (1 − α)A(n) − βA( n − 1) + T + C(n) + D(n + 1) − D(n). This results in stable performance as long as β lies in the range plus/minus one and α lies between −β and 2 + β. Such correction has profound effects on the predicted variance and correlations of the asynchronies and interresponse intervals. Figure 10.9 shows changes with α on the x-axis for different values of β. As α increases, asynchrony variance (Fig. 10.9a) decreases but interresponse interval variance (Fig. 10.9d) increases. The group average estimates of the model parameters estimated using numerical methods by Semjen et al. (2000) are shown in Fig. 10.10. Synchronization data are shown with 1lled circles, continuation data with open circles. The values for the correction factors (shown at the bottom) show a cross-over with second-order correction being more important at high rates and decreasing at lower rates. A similar 1nding of greater second-order correction at higher response rates has also been reported by Pressing and Jolley-Rogers (1997) using spectral methods. One interpretation is that correction is a time-demanding process and, at higher rates, there is insuf1cient time to apply the correction within the period of the next timer interval. In such cases it seems reasonable to assume some degree of carry-over of the correction to the next interval. The HKB model places strong emphasis on the frequency of limb movements as the experimental control parameter. As frequency increases, antiphase coordination becomes less stable and a phase transition to in-phase coordination is often observed. A metronome is commonly used to effect the necessary frequency change and it is interesting to examine the relation between metronome and



1400

1.0 (a)

(b)

1200

0.5

1000 800

0.0

600 400

– 0.5

200 0 0.2

0.4

0.6

0.8 1.0 var (An)

1.2

1.4

– 1.0 0.2 1.6 1.8

1.0

1400

(c)

0.4

0.6

0.8 1.0 1.2 ρ ( An1, An+1)

1.4

1.6

1.8

(d)

1200 0.5

1000 800

0.0

600 400

– 0.5

200 – 1.0 0.2

0.4

0.6

0.8 1.0 1.2 ρ ( A n1, A n+2)

1.4

1.6

1.8

1.0

0 0.2

0.4

0.6

0.8 1.0 var (In)

1.2

1.4

1.6 1.8

1.0 (f)

(e) 0.5

0.5

0.0

0.0

– 0.5

– 0.5

– 1.0 0.2 0.4

0.6 0.8 1.0 1.2 1.4 ρ (In1, In+1)

1.6

1.8

– 1.0 0.2 0.4

0.6 0.8 1.0 1.2 1.4 ρ (In1, In+2)

1.6

1.8

Fig. 10.9 Linear phase correction in synchronization. Effects on variance and autocorrelation of asynchrony (a,b,c) and interresponse interval (d,e,f) of varying the 1rst- and second-order correction parameters (α horizontal axis, β positive and negative above and below the solid line). Observe that var(A) is minimized with β = 0, α < 1, var(I) is minimized with α = β = 0. (Semjen et al. 2000.)

215


216


Fig. 10.10 Estimates of motor implementation and timekeeper standard deviation (s.d.) in synchronization (filled circles) and free (open circles) responding (above, middle) and of correction factors α (triangles) and β (squares) in synchronization (below). (Semjen et al. 2000.) hand movement. For example, as frequency increases does the antiphase hand movement tend to move in phase with the metronome? Kelso et al. (1990) have shown that, with one hand responding in a synchronization task, it does. However, in addition to phase transitions, cases of progressive drift in phase (phase wandering) are observed. The original formulation of the HKB models



Fig. 10.11 Skewing of the HKB-potential due to adding an asymmetry term Ωφ. If Ω = 0 the potential is left-right symmetric (a). With increasing Ω, the degree of skewedness increases, resulting eventually in loss of the stable solution near π (b, c). Black balls represent stable states, the white ball the lost stable state. assumed symmetry of the coupling functions between the two effectors. Clearly, in synchronization, the coupling is quite asymmetric, with the metronome in2uencing the participant rather than vice versa. Kelso et al. suggested a modi1cation to the potential landscape description: V(φ) = Ωφ − acos(φ) − bcos(2φ) where Ω represents the difference in uncoupled frequencies of the components (i.e. the frequencies when oscillating alone). When there is no difference in the uncoupled frequencies, that is, Ω = 0, the symmetric-coordination law of the original HKB model is recovered However, for a given movement frequency, if Ω is increased (Ω > 0), the stable 1xed points will drift away from their initial value (see Fig. 10.11). Moreover, if Ω is increased further, transitions from (more or less) antiphase to (more or less) in-phase may be observed or even a complete loss of synchronization (depending on the value of b/a). A third prediction is that the probability of maintaining a stable state in the asymmetric case is lower than in the symmetric case. Some support for this extension of the HKB model in terms of directions of phase transitions and pre-transition variability was obtained by Kelso et al. (1990). An assumption of the HKB model is that the frequency-induced phase transitions in rhythmically coordinated movements result from the coupling function describing their interaction. HKB considered two forms, one based on time derivatives, the other on time delays. Both give rise to the same potential landscape and both postulate that the phase transition from antiphase to in-phase coordination with increasing frequency is mediated by a frequency-induced reduction in movement amplitude in the component oscillators. The difference between the two forms of the model is that in the time-derivative model the decrease of coordinative stability with increasing frequency is solely associated with the decrease in movement amplitude, whereas according to the time delays

217


218


version there is also an effect of movement frequency per se. The latter is a general effect that does not account for the differential loss of stability of the in-phase and antiphase modes of coordination with increasing frequency. However, it could be capitalized upon to provide an alternative explanation for the occurrence of frequency-induced phase transitions in the absence of changes in amplitude (see below). Peper and Beek (1998a) set out to test the HKB model assumption that the frequency-induced phase transitions in coordinated rhythmic movements are mediated by a drop in amplitude. They examined the effects of restricting movement amplitude in a task that involved the synchronization of arm movement with an oscillating visual target. In case the subjects started making antiphase movements with the targets (i.e. movements with a direction opposite to that of the target), switching to in-phase movement occurred in the majority of the trials as movement frequency increased. However, no effects of movement amplitude on pattern stability (operationalized as variability of relative phase and switching frequency) were observed. This 1nding is evidence against the kind of amplitude coupling postulated in both versions of the HKB model. The occurrence of a phase transition, however, may be reconciled with the time-delays version of the model because in this formulation increasing frequency leads to an overall decrease in coordinative stability. Thus, at a certain critical frequency the stability provided by the antiphase coordination may become too small to resist the stochastic 2uctuations in the system, resulting in a noise-induced phase transition from antiphase to in-phase coordination. In this sense, the time delays version provides a better account of the data than the time derivatives model. It has also more intuitive appeal in that the time delays postulated in the model are more readily identi1ed with neural transmission delays, such as those related to feedback processing.

10.10 Application area 3: multi-effector isochronous tapping In this section we consider application of the WK and HKB approaches to movements involving two effectors each moving at the same frequency. We 1rst consider the information processing approach applied to two-hand simultaneous responding by Helmuth and Ivry (1996). They asked subjects to tap simultaneously with one or two hands and found less variability in timing with two hands. Application of the WK model showed this was due to reduction in timekeeper variance. Interestingly, the bene1t also occurred with hand and foot. Helmuth and Ivry suggested that the effect was due to combining the output of two separate timing systems. For example, if onehanded timing involves setting a threshold level on noisy accrual of activation (e.g. Gibbon et al. 1984), with two hands two separate accrual processes might be combined so that the 1rst crossing of either threshold is used to trigger both hands. This would result in statistical reduction of variance of the time interval as observed. However, it should be noted that such a process would also lead to advancing of the expected time to threshold or a bias to shorter time intervals, and this has not been reported to be the case. In contrast an averaging approach, with response triggered at the midpoint of the time dictated by each timing signal, results in lowered variability without shift in mean interval. Helmuth and Ivry suggested such averaging might be achieved by summation of two separate integrative processes against a threshold normalized to re2ect the increase in number of inputs. Interestingly, Ivry and Hazeltine (1999) observed the two-hand variance reduction in a callosotomy patient, suggesting that integration involves subcortical integration of each hemisphere’s input. Wing, Church, and Gentner (1989) studied alternate hand tapping, involving antiphase movements of the two hands. They noted that intervals of the same average duration produced between



hands were more variable than the same interval occurring within hands. They also found adjacent between-hand intervals (e.g. LR, RL) exhibited correlations more negative than the limit of minus one-half predicted by the WK model. Results of a simulation study suggested that this might be accounted for by two coupled WK timing systems with left and right clock intervals kept in appropriate relative phase by corrections involving preceding produced intervals—in essence a linear phase correction model. The original formulation of the HKB model assumed symmetry of the coupling functions between the two effectors. However, we have already seen that phase transition phenomena occur in synchronizing with an oscillating visual target where it is reasonable to suppose the coupling between pacing stimulus and effector is asymmetric (Kelso et al. 1990). Asymmetry in the potential due to unequal uncoupled frequencies might also be expected to apply to situations in which movements involve different limbs. Kelso and Jeka (1992) showed this accounted for coordination dynamics between arm and leg, and Jeka and Kelso (1995) showed how the 1nding applies to changes in coordination dynamics when the arm is weighted to make its uncoupled frequency more similar to the leg.

10.11 Application area 4: multi-effector, multifrequency tapping With the two hands tapping at the same rate, at certain phase relations intermediate between alternation and synchrony, simple rhythmic patterns such as (1:2) or (1:3) may be de1ned by the betweenhand intervals. More complex rhythmic patterns may be produced if the two hands tap regularly but with different periods that are not in simple integer relation to each other. Thus, for example, if the hands start in synchrony but one hand produces two intervals while the other produces three in the same time (e.g. the left hand taps at 300 ms intervals and the right hand at 200 ms), a between-hand interval pattern of (2:1:1:2) results. Three against four produces a pattern (3:1:2:2:1:3). Given such polyrhythms involve periodic responding by each hand, it is interesting to ask whether control involves parallel independent timing by each hand (but with a link to keep in phase at the beginning of each cycle). When such parallel control is contrasted with serial integrated control in which a single timer is responsible for the between-hand intervals, the pattern of covariances observed between component intervals (see Fig. 10.12) rejects the parallel model (Jagacinski, Marshburn, Klapp, and Jones 1988; Summers, Rosenbaum, Burns, and Ford 1993b) even after extensive practice (Summers, Ford, and Todd 1993a; Summers and Kennedy 1992; Klapp, Nelson, and Jagacinski 1998). However, an analysis of highly skilled keyboard performance has recently produced evidence for parallel timing when overall response rate is high (Krampe et al. 2000). If the above account of polyrhythm were correct, the deviations from the mean intervals produced within each hand should be random. Engbert et al. (1997) examined this assumption using a nonlinear time series analysis which provides visualization for departures from randomness. Successive intervals in each hand’s contribution to 12 cycles of a polyrhythm were marked as 0 or 1 depending on whether they fell within a window around the correct interval proportion (Fig. 10.13). They found that at slower response rates, the departures were indeed random. However, at faster response rates there were systematic departures from the random pattern. Engbert et al. provided a theoretical account of the phase transition which yielded at least a qualitative match to the data. The model was based on a nonlinear dependence in successive within-hand intervals plus a between-hand adjustment at the end of every rhythm cycle.

219


220


Fig. 10.12 Schematic of serial integrated and parallel timing models of polyrhythmic performance. (a) The former predicts I4 positively correlated with I5 and the latter positively correlated I3 with I1. (b) The results support the former (redrawn from Jagacinski et al. 1988.)

Fig. 10.13 Patterning of 2uctuations over 140 polyrhythm trials (3 in LH against 4 in RH) (Engbert et al. 1997). (a) The symbolic analysis (for each trial, and each of 36 LH and 48 RH response intervals, empty rectangle indicates interval near target, 1lled rectangle if not) with trials sorted by overall cycle duration. Dark and light regions indicate departures from target intervals are not random but change systematically with overall rate. (b) Estimates of entropy con1rm departures from random patterns suggested by the symbolic analysis.



Engbert et al. (1997) showed qualitative departures from strict timing of polyrhythm performance. From the dynamical systems perspective, it is also interesting to note that polyrhythm performance is liable to exhibit phase transitions as frequency increases, with more complex patterns switching to simpler patterns (Peper, Beek, and van Wieringen 1991, 1995). For instance, if a skilled drummer performs a complex polyrhythm, such as 3:8 or 5:8, and the cycle frequency of the fast hand, as speci1ed by an auditory metronome, is gradually increased, abrupt, spontaneous transitions to simpler (poly)rhythms, such as 2:3, 1:3, 1:2, and 1:1, occur at some critical frequency. If the cycle frequency of the hands is further increased, a second transition (e.g. from 2:3 to 1:2) or even a third may be observed (e.g. from 1:2 to 1:1). To account for such phase transitions, Haken, Peper, Beek, and Daffertshofer (1996) extended the HKB model to polyrhythms. The HKB model concerned the loss of stability of relative phase between two limbs moving at the same frequency, whereas the HPBD extension described the loss of stability of the ratio of two different frequencies produced by the two limbs. However, the logic of the HPBD model was similar in explaining phase transitions (between different frequency ratios) as a result of a frequency-induced drop in amplitude in the movements of the two effectors. In the HPBD model each stable frequency ratio was identi1ed with a speci1c coupling term for the interaction between the hands. These coupling terms described the interaction between the hands in terms of their normalized amplitudes (i.e. values between 0 and 1) raised to certain powers, with more complex rhythms requiring higher powers than simpler rhythms. According to the HPBD model, as the frequencies of the hands increase and their amplitudes decrease, the more complex rhythms lose their stability earlier than the simpler rhythms. This is because the coupling coef1cients associated with these rhythms decrease more rapidly (due to the higher powers) than those associated with the simpler rhythms. The coupling between the component oscillators in the HPBD model was originally formulated in terms of time derivatives of the state of the other oscillator, which resulted in an exclusive dependence of pattern stability on amplitude. As an alternative, Peper and Beek (1998b) demonstrated that it is also possible to determine a coupling function based on time-delayed values of the state of the other oscillator. (It will be recalled that a similar development of time derivative and time delay forms of coupling function was described by Haken et al. 1985 of the two hands moving at the same frequency.) As stated before, such time delays may be identi1ed with the neural processes that underlie the interaction between the hands, such as the neurophysiological delays associated with the use of kinesthetic feedback. This leads to an alternative HPBD model with the same coupling terms as in the original but with an additional, overall reciprocal dependence of coupling strength on the frequencies of the hands. According to this alternative model, the degree of interaction between the hands depends not only on the amplitudes of the oscillations but also on the frequencies at which the hands move. To test the two versions of the HPBD model, Peper and Beek (1998b) conducted an experiment aimed at dissociating the effects of frequency and amplitude on interlimb coupling in stable, steadystate production of the 2:3 polyrhythm. Subjects tapped this polyrhythm at 1ve different tempos under three amplitude conditions and performed unimanual control trials in which they had to tap at the same frequencies as produced in the bimanual trials. Subsequently, the degree of coupling between the hands in the bimanual trial was assessed by comparing the degree of harmonicity of the movements of the hands in bimanual compared to unimanual conditions. Harmonicity was operationalized as the relative contribution to the power spectrum (here the spectrum refers to the movement trajectory rather than the earlier noted use of the spectrum to characterize periodicity in a train of time intervals) of the evolving phase in the oscillation pattern. Using this measure, no signi1cant effects of

221


222


amplitude were observed, whereas the strength of interaction between the hands decreased with increasing tempo (i.e. movement frequency). Again, this result favours the time-delays version over the time-derivatives version of the HPBD model and, as noted by Peper and Beek (1998b), is suggestive that the interaction between limbs may re2ect kinesthetic in2uences with the time delay re2ecting associated neural conduction delays.

10.12 A merging of the two approaches Shaping of stochastic 2uctuations by deterministic equations of motion plays an essential role in the dynamical systems approach to accounting for the loss of stability of coordinated movement patterns. However, very little work has been done in which these stochastic 2uctuations are used as a window into the control structure of timed motor behaviour. In contrast, showing that the covariance structure of timed motor responses may be exploited to uncover the underlying timekeeper organization has been a major contribution of the information processing approach. Recently, however, an initial attempt has been made to account for empirically observed patterns of temporal variability from the perspective of dynamical systems theory. Speci1cally, Daffertshofer (1998)—following an earlier suggestion of Schöner (1994)—examined both analytically and numerically the minimal conditions under which limit cycle models with noise consistently produce a negative lag-1 serial correlation (with a value between 0 and −0.5) between consecutive periods of oscillation. Contrary to earlier intuitions, he showed that a single limit cycle oscillator that is stochastically forced by (additive or multiplicative) white or by coloured noise cannot produce the desired period correlation but predominantly results in phase diffusion. In order to obtain reliable negative correlations, it is necessary either to introduce two conveniently placed noise sources (as in the WK model), or to add a second oscillator that is coupled either unidirectionally (i.e. an external forcing function) or bidirectionally to the limit cycle oscillator of interest, thus stabilizing its phase. In the latter system a single noise source is suf1cient to obtain the sought-after negative lag-1 serial correlation. This is interesting because, in this case, the property is not simply the result of certain statistical properties but rather the consequence of deterministic interactions between two coupled oscillators with a single noise source. In order to incorporate Daffertshofer’s conclusions as well as to account for the previously described results that the stability of the coordination between two rhythmically moving limbs is unaffected by their amplitudes, Beek, Peper, and Daffertshofer (in press; see also Peper, Beek, and Daffertshofer 2000) proposed a new dynamical model for interlimb coordination. This comprises a two-tiered structure that resembles the Wing and Kristofferson (1973) model as extended by Vorberg and Hambuch (1984) for application to bilateral responding (see also Wing 1982; Wing et al. 1989). Thus it represents timing in terms of two levels, one neural, one effector, and there is a divergence at the effector level. However, in the new model the elements at each level are composed of oscillators (see Fig. 10.14) and the two lower-level effector oscillators are driven by two nonlinearly coupled limit cycle oscillators at the neural level. Because of their nonlinear coupling, the limit cycles of the neural level oscillators can exhibit phase transitions between phase locked and/or frequency locked states as the frequency is increased. However, this effect is not necessarily mediated by the amplitudes of the oscillators, which may be set to 1 (representing a 1xed level of neural activity). The driving signals from each of the coupled neural limit cycle oscillators are transferred to a peripheral oscillator having certain physical properties, such as mass, stiffness, and damping. These physical properties lead to a particular amplitude response to the driving signals, which may or may



forcing

feedback

coupling (HKB)

neural level

effector level

Fig. 10.14 Neural and effector levels of oscillators are assumed in the two-tiered oscillator model of Beek et al. (in press) and Peper et al. (2000).

not be characterized by a resonance peak. In either case the response of the peripheral oscillators may entail a drop in amplitude over a large range of frequencies. This frequency-induced drop in amplitude is viewed as a purely peripheral effect that has no consequence for the stability properties of the observed patterns of coordination. These follow from the neural limit cycles and their interactions, although kinesthetic feedback signals related to the phasing of the peripheral oscillators may play into this neural control structure. Due to its two-tiered structure, the proposed model is capable of providing an account for the differential effects of frequency (tempo) and amplitude on pattern stability. However, it also provides a good starting point for explaining the temporal variability characteristics of timed motor responses in that it satis1es the minimal requirements for a dynamical model of negative lag-1 serial correlations in the periods.

10.13 Summary and conclusions In this review we have considered two approaches to variability in timing. The information processing model or time series approach is based on linear autoregressive moving average (ARMA) modelling of stable discrete behaviour (Pressing 1999). This approach lends itself to hierarchical modelling with subdivision of noise sources through explicit consideration of dependence in the dataset. The lowest level in the hierarchy is then the motor process. Thus the information processing approach does not reject motor systems, rather it avoids detailing all aspects of movement timing. The dynamical systems approach, based on nonlinear oscillators, is represented in terms of differential equations. The latter provide a full description of spatial trajectory and in that sense fully model the movement system. A major achievement of the dynamical systems perspective is that various aspects of classic physical system phase transitions (or bifurcations) have been shown to occur in repetitive coordinated human behaviour. These transitions, which are not obviously explained by motor program approaches, include features such as discontinuity, hysteresis, critical 2uctuations, and critical slowing. Scienti1c explanation often proceeds by setting two competing views against each other to determine which is right and which is wrong. This is enshrined in inferential statistics by hypothesis testing in which the goal is to prove the null hypothesis wrong and so support the alternative. However,

223


224


behaviour is multifaceted and context-dependent. So, while one theory may be supported when one set of aspects of behaviour or contexts is considered, the other theory may receive support when the focus shifts to other aspects of behaviour in other contexts. Hence our understanding of behaviour may advance not by proving one right and the other wrong, but rather by characterizing why one theory works in one situation, the other (or others) in another situation. With two different approaches to timing, we would argue that, rather than asking which is right or which is better, it is more useful to think what the relation is between the theories (see also Heuer 1996). While the two approaches were originally drawn up to account for different aspects of behaviour, they are now increasingly being applied to the same or related aspects of behaviour. So we consider it important to actively seek aspects of each approach that can be used in a combined approach that might be more powerful. Pressing (1999) for example has argued for a general approach, referential behaviour theory, based on an underlying principle of homeostasis and drawing on the language of control theory, which can be applied to both information processing and dynamical systems approaches to timing. However, we acknowledge that there is a risk of losing simplicity and comprehensibility if the relation between the approaches is not carefully examined. Thus we believe development will be through contrasting and comparing as well as seeking to combine the two approaches.

Acknowledgement We thank Andreas Daffertshofer for comments on an earlier version of this paper and for assistance with the 1gures. AMW was supported by a grant from the MRC.

References Beek, P.J., Peper, C.E., and Daffertshofer, A. (in press). Modeling rhythmic interlimb coordination: Beyond the Haken–Kelso–Bunz model. Brain and Cognition. Beek, P.J., Peper, C.E., and Stegeman, D.F. (1995). Dynamical models of movement coordination. Human Movement Science, 14, 573–608. Creelman, C.D. (1962). Human discrimination of auditory duration. Journal of the Acoustical Society of America, 34, 582–593. Daffertshofer, A. (1997). Nichtgleichgewichtsphasenübergänge in der Menschlichen Motorik und Erweiterung des synergetischen Computers. Aachen: Shaker Verlag. Daffertshofer, A. (1998). Effects of noise on the phase dynamics of non-linear oscillators. Physical Review E, 58, 327–338 Engbert, R., Scheffczyk, C., Krampe, R.T., Rosenblum, M., Kurths, J., and Kliegl, R. (1997). Tempo-induced transitions in polyrhythmic hand movements. Physical Review E, 56, 5823–5833. Gibbon, J., Church, R.M., and Meck, W.H. (1984). Scalar timing in memory. In J. Gibbon and L. Allan (Eds.), Timing and time perception. Annals of the New York Academy of Sciences, 423, 52–77. Govier, L.J. and Lewis, T. (1967). Estimation of the dispersion parameter of an (A,B) process. In R. Cruon (Ed.), Queuing theory: Recent developments and applications. Amsterdam: Elsevier. Gregson, R.G. and Pressing, J. (2000). Dynamic modeling. In L.G. Tassinary, J.T. Caccioppo, and G. Berntson (Eds.), Principles of psychophysiology: Physical, social, and inferential elements. Cambridge: Cambridge University Press. Haken, H., Kelso, J.A.S., and Bunz, H. (1985). A theoretical model of phase transitions in human hand movements. Biological Cybernetics, 51, 347–356. Haken, H.H., Peper, C.E., Beek, P.J., and Daffertshofer, A. (1996). A model for phase transitions in human hand movements during multifrequency tapping. Physica D, 90, 179–196.



Helmuth, L.L. and Ivry, R.B. (1996). When two hands are better than one: Reduced timing variability during bimanual movements. Journal of Experimental Psychology: Human Perception and Performance, 22, 278–293. Heuer, H. (1996). In H. Heuer and S. Keele (Eds.), Human motor performance. New York: Academic Press. Inhoff, A.W. and Bisiacchi, P. (1990). Unimanual tapping during concurrent articulation: Generalized and lateralized effects of memory encoding upon the rate and variability of concurrent 1nger tapping. Brain and Cognition, 6, 24–40. Ivry, R.B. and Hazeltine, E. (1999). Subcortical locus of temporal coupling in the bimanual movements of a callosotomy patient. Human Movement Science, 18, 345–375. Jagacinski, R.J., Marshburn, E., Klapp, S.T., and Jones, M.R. (1988). Tests of parallel versus integrated structure in polyrhythmic tapping. Journal of Motor Behavior, 20, 416–442. Jeka, J.J. and Kelso, J.A.S. (1995). Manipulating symmetry in the coordination dynamics of human movement. Journal of Experimental Psychology: Human Perception and Performance, 21, 360–374. Kay, B.A., Kelso, J.A.S., Saltzman, E.L., and Schöner, G. (1987). Space–time behavior of single and bimanual rhythmical movements: Data and limit cycle model. Journal of Experimental Psychology: Human Perception and Performance, 13, 178–192. Kelso, J.A.S. (1984). Phase transitions and critical behavior in human bimanual coordination. American Journal of Physiology, 15, 1000–1004. Kelso, J.A.S., DelColle, J.D., and Schöner, G. (1990). Action-perception as a pattern formation process. In M. Jeannerod (Ed.), Attention and performance XII, pp. 139–169. Hillsdale, NJ: Lawrence Erlbaum. Kelso, J.A.S., Ding, M., and Schöner, G. (1994). Dynamic pattern formation: A primer. In L.B. Smith and E. Thelen (Eds.), A dynamic systems approach to development, pp. 13–50. Cambridge, MA: MIT Press. Kelso, J.A.S. and Jeka, J.J. (1992). Symmetry breaking dynamics of human multilimb coordination. Journal of Experimental Psychology: Human Perception and Performance, 18, 645–668. Klapp, S.T., Nelson, J.M., and Jagacinski, R.J. (1998). Can people tap concurrent bimanual rhythms independently? Journal of Motor Behavior, 30, 301–322. Krampe, R.T., Kliegl, R., Mayr, U., Engbert, R., and Vorberg, D. (2000). The fast and slow of skilled bimanual rhythm production: Parallel vs. integrated timing. Journal of Experimental Psychology: Human Perception and Performance, 26, 206–233. Michon, J.A. (1966). Tapping regularity as a measure of perceptual motor load. Ergonomics, 9, 401–412. Peper, C.E. and Beek, P.J. (1998a). Are frequency-induced transitions in rhythmic coordination mediated by a drop in amplitude? Biological Cybernetics, 79, 291–300. Peper, C.E. and Beek, P.J. (1998b). Distinguishing between the effects of frequency and amplitude on interlimb coupling in tapping a 2:3 polyrhythm. Experimental Brain Research, 118, 78–92. Peper, C.E., Beek, P.J., and Daffertshofer, A. (2000). Considerations regarding a comprehensive model of (poly)rhythmic movements. In P. Desain and L. Windsor (Eds.), Rhythm perception and production, pp. 35–49. Lisse: Swets and Zeitlinger. Peper, C.E., Beek, P.J., and Wieringen, P.C.W. van (1991). Bifurcations in bimanual tapping: In search of Farey principles. In J. Requin and G.E. Stelmach (Eds.), Tutorials in motor neuroscience, pp. 413–431. Dordrecht: Kluwer. Peper, C.E., Beek, P.J., and Wieringen, P.C.W. van (1995). Frequency-induced phase transitions in bimanual tapping. Biological Cybernetics, 73, 301–309. Pressing, J. (1999). The referential dynamics of cognition and action. Psychological Review, 106, 714–747. Pressing, J. and Jolley-Rogers, G. (1997). Spectral properties of human cognition and skill. Biological Cybernetics, 76, 339–347. Rosenbaum, D.A. and Patashnik, O. (1980). Time to time in the human motor system. In R.S. Nickerson (Ed.), Attention and performance VIII. Hillsdale, NJ: Erlbaum. Saito, S. and Ishio, A. (1998). Rhythmic information in working memory: Effects of concurrent articulation on reproduction of rhythms. Japanese Psychological Research, 40, 10–18. Scholz, J.P., Kelso, J.A.S., and Schöner, G. (1987). Nonequilibrium phase transitions in coordinated biological motion: Critical slowing down and switching time. Physics Letters A, 123, 390–394. Schöner, G. (1994). From interlimb coordination to trajectory formation: Common dynamical principles. In S.P. Swinnen, J. Massion, H. Heuer and P. Casaer (Eds.), Interlimb coordination: Neural, dynamical, and cognitive constraints, pp. 339–368. San Diego, CA: Academic Press. Schöner, G., Haken, H., and Kelso, J.A.S. (1986). A stochastic theory of phase transitions in human hand movements. Biological Cybernetics, 53, 247–257.

225


226


Semjen, A., Schulze, H.-H., and Vorberg, D. (2000). Timing precision in continuation and synchronization tapping Psychological Research, 63, 137–147. Sergent, V., Hellige, J.B., and Cherry, B. (1993). Effects of responding hand and concurrent verbal processing on time-keeping and motor-implementation processes. Brain and Cognition, 23, 243–262. Shaffer, L.H. (1982). Rhythm and timing in skill. Psychological Review, 89, 109–122. Stevens, L.T. (1886). On the time sense. Mind, 11, 393–404. Summers, J.J. and Kennedy, T.M. (1992). Strategies in the production of 5:3 polyrhythm. Human Movement Science, 11, 101–112. Summers, J.J., Ford, S.K., and Todd, J.A. (1993a). Practice effects on the coordination of the two hands in a bimanual tapping task. Human Movement Science, 12, 111–133. Summers, J.J., Rosenbaum, D.A., Burns, B.D., and Ford, S.K. (1993b). Production of polyrhythms. Journal of Experimental Psychology: Human Perception and Performance, 19, 416–428. Treisman, M. (1963). Temporal discrimination and the indifference interval: Implications for a model of the internal clock. Psychological Monographs, 77, 13, Whole No. 576. Vorberg, D. and Hambuch, R. (1978). On the temporal control of rhythmic performance. In J. Requin (Ed.), Attention and performance VII, pp. 535–555. Hillsdale, NJ: Erlbaum. Vorberg, D. and Hambuch, R. (1984). Timing of two-handed rhythmic performance. In J. Gibbon and L. Allan (Eds.), Timing and time perception. Annals of the New York Academy of Sciences, 423, 390–406. Vorberg, D. and Wing, A.M. (1996). Modeling variablity and dependence in timing. In H. Heuer and S. Keele (Eds.), Handbook of perception and action, Vol. 2, pp. 181–262. London, New York: Academic Press. Wing, A.M. (1980). The long and short of timing in response sequences. In G.E. Stelmach and J. Requin (Eds.), Tutorials in motor behaviour, pp. 469–486. Amsterdam: North-Holland. Wing, A.M. (1982). Timing and coordination of repetitive bimanual movements. Quarterly Journal of Experimental Psychology, 34, 339–348. Wing, A.M, Church, R.M., and Gentner, D.R. (1989). Variability in the timing of responses during repetitive tapping with alternate hands. Psychological Research, 51, 28–37. Wing, A.M. and Kristofferson, A.B. (1973). Response delays and the timing of discrete motor responses. Perception and Psychophysics, 14, 5–12. Yamanishi, T., Kawato, M., and Suzuki, R. (1980). Two coupled oscillators as model for the coordinated 1nger tapping by both hands. Biological Cybernetics, 37, 221–225.


11 Timing mechanisms in sensorimotor synchronization Gisa Aschersleben, Prisca Stenneken, Jonathan Cole, and Wolfgang Prinz Abstract. This study examines the in2uence of sensory feedback on the timing of simple repetitive movements in a sensorimotor synchronization task. Subjects were instructed to synchronize 1nger taps with an isochronous sequence of auditory signals. Although this is an easy task, a systematic error is commonly observed: taps precede clicks by several tens of milliseconds. One explanation proposed for ‘negative asynchrony’ is based on the idea that synchrony is established at the level of central representation (and not at the level of external events) and that the timing of an action is determined by the (anticipated) action effect. To test this hypothesis the sensory feedback available from the tap as well as its temporal characteristics were manipulated, and evidence supporting the hypothesis was obtained. After reviewing these 1ndings we report new evidence obtained from a deafferented subject who suffers from a complete loss of tactile and kinesthetic afferences from below the neck. In three experiments we studied his performance (and compared it with a group of age-matched control subjects) under conditions with differing amounts of feedback. In the 1rst experiment, in which all information about the tapping movement was excluded, the deafferented subject nevertheless maintained a stable phase relationship between the pacing signal and his movements but with a large negative asynchrony. In the second experiment, an auditory feedback tone was provided each time the subject touched the key. This manipulation led to clear improvement in his performance; however, he was only able to tap in exact synchrony in the third experiment, when he was allowed to visually monitor his tapping movements as well. These results demonstrate the important role of sensory feedback in the timing of movements. Furthermore, the 1ndings suggest an internal prediction of the movement’s sensory consequences, as expressed in the account of internal forward models.

11.1 Introduction The importance of sensory feedback in the control of movements has been shown in numerous studies. One important function of feedback is the online correction of movements, for example in pointing. Seeing the hand as it approaches a visible target is an instance of closed-loop control. Under openloop conditions, when feedback is unavailable, no corrective movements are possible, leading to a degradation in performance. Besides visual feedback, sensory reafferent information from proprioception and touch plays an important role in the temporal and spatial control of movements. The present study is concerned with the role of sensory feedback in the timing of movements. To analyze the timing of movements in the absence of other confounds, it is reasonable to study simple, repetitive tasks, in which subjects are required to accompany a predictable stimulus with a simple movement. In the synchronization task, subjects tap with, for example, the right index 1nger on a key at a given rate (tapping task). The beat is presented by a metronome emitting clicks. The interval between the 1nger touching the key and the presentation of the pacing signal is calculated as the dependent variable. Though this is an easy task, subjects are generally not able to perform the movement in exact synchrony with the clicks and a systematic error is observed: taps usually precede clicks by several tens of milliseconds—the ‘negative asynchrony’. This effect was described


228


more than a century ago (e.g. Dunlap 1910; Johnson 1898; Miyake 1902) and has been replicated ever since in many studies (e.g. Aschersleben and Prinz 1995, 1997; Fraisse 1980; Kolers and Brewster 1985; Mates, Müller, Radil, and Pöppel 1994; Repp 2000; Thaut, Tian, and Azimi-Sadjadi 1998; Vos, Mates, and van Kruysbergen 1995; Wohlschläger and Koch 2000; for an overview, see Aschersleben, in press). The size of the negative asynchrony depends to a great extent on the experimental conditions, though there are large interindividual differences as well. One important factor in2uencing the asynchrony is musical experience. Highly trained musicians exhibit a much smaller asynchrony than musically untrained persons. Ludwig (1992) asked 27 students at the Music Academy in Munich to synchronize their taps to an auditory pacing signal under controlled feedback conditions (visual and auditory feedback from the tap was eliminated). Their mean asynchrony was 14ms (SD between subjects: 13 ms) compared with 50 ms (SD between subjects: 23 ms) in an age-matched cohort of musically untrained persons. However, all subjects revealed a negative asynchrony, so that even the highly musically trained subjects did not tap in exact synchrony with the metronome. One reason for the reduced asynchrony in musicians might be the huge amount of training with different kinds of synchronization tasks during individual and ensemble playing. To analyse the in2uence of training on the asynchrony, Aschersleben (2000, 2001) asked musically untrained subjects to tap for ten sessions (45 min each; about 10,000 taps in total) with and without knowledge of results about their asynchronies. If people tap without this feedback, no change in asynchrony was observed as a function of practice. Only when knowledge of results was provided did the subjects alter their performance and become able, after ten sessions, to tap in exact physical synchrony. However, in this condition subjects reported that they had to subjectively delay their tap to produce the required objective synchrony.

11.1.1 An explanatory account: the role of sensory feedback Although the negative asynchrony has been known for more than a hundred years, the underlying mechanisms are still not completely understood. Some recent accounts are based on the assumption that synchrony is established at a central level at which both events and actions are represented in terms of their sensory consequences (Aschersleben, in press; Aschersleben and Prinz 1995, 1997; Aschersleben, Gehrke, and Prinz in press; Gehrke 1996; Prinz 1990, 1997; for comparable ideas see Fraisse 1980; Mates 1994; Paillard 1949). If so, the temporal delays involved in perceiving the click and the tap become crucial. Because of differences in (central and/or peripheral) processing times, the temporal delay between actual and perceived click is likely to be shorter than the delay between actual and perceived tap, so that the actual tap must precede the actual click to result in synchrony between the perceived events. As a consequence, the negative asynchrony between click onset and overt tap is observed. There are two versions of this hypothesis (the nerve-conduction hypothesis and the sensory accumulator model) that differ in the levels at which the asynchrony is caused. According to the nerveconduction hypothesis (also known as the Paillard–Fraisse hypothesis), differences in nerve-conduction time for the sensory information about the click (from ear to brain) and the tap (from 1nger to brain) are responsible for the negative asynchrony (Aschersleben 1994; Aschersleben and Prinz 1995, 1997; Fraisse 1980; Paillard 1949). Thus it is assumed that the negative asynchrony originates in peripheral processes. In contrast, the sensory accumulator model assumes the crucial factor to be the central processing times involved in generating central representations of the peripheral events. According to that


Timing mechanisms in sensorimotor synchronization

model, these central representations unfold in time and cannot be considered as punctuate events. For both clicks and taps the unfolding of the sensory evidence is captured by an accumulation function. It is assumed that the points at which the accumulation functions reach their respective thresholds determine the times at which the two events are perceived. The sensory accumulator model posits that the sensory evidence for clicks and taps gets accumulated in the same functional domain and that the same threshold applies to both of them. It is further assumed that to achieve perceived synchrony, the two accumulation functions need to hit the threshold at the same point. Given these two assumptions, it is the steepness of the two accumulation functions that determines to what extent the onsets of the two physical events must be offset to achieve perceived synchrony of the two mental events. For instance, if the auditory pacing signal used in the synchronization task has a steeper accumulation function than the tap (because the temporal resolution is much higher in the auditory than in the tactile system), the tap onset needs to precede the click onset to make sure that both functions hit the threshold at the same point (Aschersleben et al. in press; Gehrke 1996). Although the two hypotheses differ in the processes assumed to be crucial, both rely on the assumption that synchrony is established at a central level at which both events and actions are represented. Therefore, we will refer to them as representational models (see Aschersleben, in press). This view differs from models assuming that the cognitive system is able to generate coincidence between external events in a veridical manner. These models are based on the idea of an asymmetrical error tolerance, that is, the cognitive system is assumed to tolerate (small) negative errors (taps preceding clicks) whereas positive errors (taps following clicks) are corrected almost immediately. Results from psychophysical experiments support this view by showing that taps following clicks are detected with larger probability than taps preceding clicks (Koch 1999; for overviews see Aschersleben, in press; Müller, Aschersleben, Koch, Freund, and Prinz 1999). We will not elaborate on these models because the empirical part of this chapter aims at further testing the representational models. Another important feature of the two hypotheses described above is their emphasis on the role of sensory (re)afferent signals in the timing of tapping movements. An obvious way to test this assumption is to manipulate the feedback coming from the tap so as to change the temporal delay between actual and perceived tap. The longer this delay, the more the actual tap must precede the actual click to result in coincidence of the two perceived events, and the more pronounced the negative asynchrony should be. This prediction has received empirical support in several experiments, which will now be summarized.

11.1.2 Empirical support for the role of sensory feedback Experiments designed to manipulate the delay between the tap and its central representation have delayed the intrinsic (somatosensory) feedback, added extra, extrinsic feedback components, and intensi1ed or eliminated (part of) the intrinsic feedback. A simple way to manipulate the delay between the tap and its central representation is to make participants tap with different body parts. Experiments in which subjects were asked to tap with the hand or foot (under the assumption that the temporal delay between actual and perceived tap increases with the ‘neural distance’ between effector and brain) have shown that the negative asynchrony is more pronounced with foot than with hand tapping (Aschersleben and Prinz 1995; Billon, Bard, Fleury, Blouin, and Teasdale 1996a; Fraisse 1980). Corresponding results have been reported

229


230


for self-initiated tapping with hand and foot, that is, conditions in which the absolute timing of the (simultaneously performed) hand and foot tap was determined by the subject (Bard, Paillard, Teasdale, Fleury, and Lajoie 1991; Bard et al. 1992; Billon et al. 1996a; Paillard 1949). The increased asynchrony in foot tapping compared with hand tapping was independent of the body side involved and of whether the two effectors performed the tapping movement separately or simultaneously (Aschersleben and Prinz 1995; Billon et al. 1996a). To study the in2uence of additional auditory feedback on the timing of the tap, some studies presented an auditory signal each time the subject touched the key. Under the assumption that the central representation of the tap is an integrated percept of all feedback components (‘joint event code’), a reduced negative asynchrony was expected because the additional auditory feedback was thought to arrive prior to the tactile/kinesthetic feedback. Any integration of these two parts should lead to a reduced negative asynchrony (Aschersleben and Prinz 1995, 1997; for a similar assumption, see Fraisse, Oléron, and Paillard 1958).1 This effect was indeed found in a number of such experiments (e.g. Aschersleben and Prinz 1995, 1997; Mates and Aschersleben 2000; Mates, Radil, and Pöppel 1992; O’Boyle and Clarke 1996; Repp, 2001). An extension of this approach can be found in studies on delayed auditory feedback in which a delay is introduced between the touch of the key and the presentation of the corresponding feedback tone. If the delay is short enough for the subjects not to be aware of this manipulation (less than 100 ms), an increase in the negative asynchrony with increasing delay is observed (Aschersleben and Prinz 1997; Mates and Aschersleben 2000; see also Fraisse et al. 1958). Moreover, if there is a joint event code resulting from all feedback components, then a linear relationship between asynchrony and delay is expected. This is because if there is a constant weighting between the feedback components, a linear shift in the timing of one component should result in a linear shift in the timing of the joint event code and, thus, a linear shift in the observed asynchrony. Corresponding experiments con1rm this prediction (Aschersleben and Prinz 1997). Another way to manipulate the delay between the tap and its central representation is to intensify tactile/kinesthetic feedback from the 1nger (Aschersleben et al. in press; Gehrke 1995). According to the threshold model outlined earlier, this should lead to a steeper accumulation function and thus reduce the negative asynchrony. One way to test this prediction is to vary the amplitude of 1nger movement. A large amplitude leads to an increase in force and, thus, to an increase in tactile stimulation. In addition, larger 1nger movements are performed with higher velocity, which increases kinesthetic feedback. Altogether, a larger movement amplitude intensi1es sensory stimulation at the 1nger, which should result in a reduced asynchrony. Indeed, when subjects tapped with large movement amplitudes the negative asynchrony was signi1cantly smaller than in conditions with small movement amplitudes (Aschersleben et al. in press; Gehrke 1995). Finally, another way of testing the hypothesis that sensory feedback is important in determining the timing of movements is to eliminate feedback altogether, either by applying local anesthesia to healthy subjects or by studying subjects with pathological loss of sensory feedback. In both kinds of studies it is important to make sure that the efferent nerves are unimpaired. Aschersleben, Gehrke, and Prinz (2001) studied the in2uence of local anesthesia of the index 1nger on the timing of taps. Local anesthesia suppresses tactile reafferent information without disturbing the reafferent discharge of the joint and muscle receptors. It affected neither the maximum tapping rate nor the timing of synchronized 1nger movements without a response key (tapping in the air like a conductor), indicating that the efferent nerve 1bers and the transmission of kinesthetic feedback information were unimpaired. However, under conditions of standard 1nger tapping the



negative asynchrony was signi1cantly increased, demonstrating an important in2uence of tactile feedback on the timing of the taps. Only few pathological cases meet the criterion of complete sensory deafferentation because complete sensory loss with unimpaired efferent pathways and cortical motor structures is very rare. Bard et al. (1992) studied a woman who is almost completely deafferented (for a description, see Cooke, Brown, Forget, and Lamarre 1985). The task was to initiate, simultaneously, ipsilateral 1nger extension and heel raising in two conditions: (1) simple reaction time (movements triggered by an external signal), and (2) self-initiated movement. In the reactive condition, the deafferented subject showed a behavior similar to healthy controls, with 1nger movement preceding heel raising, but her performance differed in the self-initiated condition. In healthy subjects, a lead of the foot movement is observed, which supports the idea of synchronized afferent feedback. However, the deafferented subject did not show this effect, which suggests that she had to rely on synchronization of her motor commands. Billon, Semjen, Cole, and Gauthier (1996b) studied another deafferented subject who lacked proprioceptive and tactile sensitivity below the neck. They asked their subject to produce sequences of periodic 1nger taps involving a complex pattern of accentuation in synchrony with a metronome. They found that, without visual and auditory feedback, the deafferented man did not lose correct phasing between the taps and the clicks of the pacing signal. That is, at least for the required sequence of 35 taps, he was able to produce a regular sequence of synchronized taps without any feedback available. However, his mean synchronization error increased and so did the force of the taps and the amplitude of the tapping movements. In addition, the deafferented subject did not show an effect observed in the control subjects, namely a delay in the initiation of the accentuated taps. This result indicated that movement-related feedback plays a prominent role in the temporal control of the tapping movements (Billon et al. 1996b). The aim of the present experiments was to investigate the effect of loss of sensory feedback on the timing of tapping movements in greater detail by studying IW, who had a complete loss of tactile and kinesthetic afferent 1bers from below the neck. We asked IW to tap with hand and foot in synchrony with an auditory metronome under different feedback conditions. We wondered whether IW would be able to perform the synchronization task without auditory and visual control, that is, without any feedback at all about his own movements (Experiment 1). The report by Billon et al. (1996b), who had studied the same deafferented subject IW, gives a 1rst hint that he can keep his taps in phase with an acoustic metronome simply by listening to the metronome sounds. We also wanted to compare IW’s performance to that of healthy control subjects. To study the role of sensory feedback in greater detail, we then manipulated the availability of auditory feedback and visual control in Experiments 2 and 3.

11.2 Experiment 1 In this experiment, we asked IW to tap either with the hand or with the foot. However, we excluded auditory as well as visual information about the tapping movement. IW’s performance under these conditions was compared with that of an age-matched group of healthy controls. From the perspective of the representational models we would expect that IW is unable to perform the synchronization task under conditions in which neither proprioceptive nor auditory or visual feedback about the tapping movement is available. However, the results from the Billon et al. study (1996b) indicate that IW can follow an auditory pacing signal with his taps.

231


232


11.2.1 Method 11.2.1.1 Subjects A control group of 14 healthy subjects (6 female, 8 male, between 42 and 53 years of age, mean age 46.8 years; all right-handed) and a deafferented male subject, IW (age 47 years, left-handed), participated. At the age of 19 years, IW suffered a purely sensory neuronopathy with acute onset. This led to a total loss of kinesthetic and tactile sensitivity for the whole body below the neck. Clinical tests revealed an absence of large myelinated 1ber function. Sensory nerve action potentials and cutaneous muscular reflexes are absent. Temperature and deep pressure sensations and perception of muscle fatigue are still present, suggesting a signi1cant sparing of small myelinated and unmyelinated 1bers. Motor conduction velocities are normal. Clinical rehabilitation extended over three years. All movements require sustained attention and visual control. IW is able to move by visually monitoring his actions, but he is unable to carry out simultaneous motor tasks such as maintaining a precision grip while walking, because of a limited span of attention (for details, see Cole 1995; Cole and Paillard 1995; Cole and Sedgwick 1992). 11.2.1.2 Apparatus and stimuli The subjects were seated at a table and were asked to tap with the dominant index 1nger or foot on a silent electrical contact switch mounted on a wooden board. To eliminate visual feedback the key was placed behind a wooden plate that obstructed the subject’s view (see Fig. 11.1). The auditory pacing signal (1000 Hz, 80 dB[A], duration 10ms, interstimulus interval 600 ms) was presented binaurally through headphones (audio-technical ATH–A5). To mask other external sounds, continuous white noise (20 dB[A]) was added. The stimuli were produced by a personal computer (Compaq Presario 1630) via a SoundBlaster-compatible sound card. The computer controlled the experimental procedure and registered the onset of keypresses (with a resolution of 1ms).

Fig. 11.1 Illustration of the experimental set-up in conditions without visual control.



11.2.1.3 Procedure The hand and foot tapping conditions were presented blockwise, each block consisting of five trials. Each trial presented 50 pacing signals. Instructions required the subjects to start tapping as soon as they picked up the beat (usually within the 1rst three signals) and then to tap along as precisely as possible. At the beginning the subjects performed some taps with the hand and the foot without any pacing signal to get a ‘feeling’ for the required movement. We knew from previous studies with IW that in the absence of visual feedback he would lose contact with the response key. Therefore we 1xed his hand (at the wrist and the other 1ngers) and his foot (at the heel) with Velcro straps on the wooden part of the keyboard. However, this manipulation did not interfere with his natural 1nger and foot tapping movements. To avoid fatigue and feedback via muscle fatigue, short breaks between trials and longer breaks between blocks were introduced. 11.2.2 Results Data analysis started with the fourth signal in each trial. The initial taps were not included because about three signals were required for the subject to pick up the beat. Hence, the means reported below always refer to the taps accompanying the remaining 47 signals in each trial. The means of the asynchronies between tap onsets and click onsets were computed for each trial. Negative values indicate that taps preceded clicks. Trials were dropped from the analysis if they contained fewer than 25 taps or when the standard deviation exceeded a pre-set criterion of 100ms. For the control subjects 2.1% of the trials had to be rejected, for IW one single trial.

Fig. 11.2 Averages and standard errors of the mean asynchronies for the deafferented subject IW and the age-matched control subjects, for hand and foot tapping without auditory feedback or visual control (Exp. 1). The standard errors for control subjects represent between-subject variability in mean asynchrony. Fig. 11.2 shows the mean asynchronies for hand and foot tapping for IW and for the control subjects. IW had large mean asynchronies for both hand and foot tapping (−103 ms and −146 ms). Although the mean difference between hand and foot tapping amounted to 43ms, it was not signi1cant, t(8)=1.80,

233


234


p > 0.10. This was due to large variability between trials: the trial mean asynchrony ranged from −145ms to −56ms in hand tapping and from –186ms to –93ms in foot tapping. The control subjects, who had tactile information about the tap onset and kinesthetic information about the 1nger movement available, also showed clear negative asynchronies under both effector conditions (hand:−28 ms, foot:−59 ms). However, a one-sample t-test procedure testing whether IW’s asynchrony differed from the average asynchronies of the control subjects revealed that their asynchronies were much smaller than IW’s, t(13) = 12.10, p < 0.001. In addition, a signi1cant difference between hand and foot was observed in the control subjects (31 ms), as expected, t(13) = 4.96, p < 0.001. The mean standard deviations of the asynchronies within trials for IW (hand: 27 ms; foot: 36 ms) were similar to those of the control subjects (hand: 23 ms; foot: 27 ms).

11.2.3 Discussion In this 1rst experiment, hand and foot tapping were studied under conditions without any auditory or visual information about the tapping movement. Healthy controls had to rely on their tactile and kinesthetic feedback from the 1nger movement to control their taps, whereas the deafferented subject IW was deprived of all feedback information about the spatial and temporal characteristics of his movements. Nevertheless, he coped with the task rather well. The results for the control subjects revealed a clearly negative asynchrony in both the hand and the foot-tapping condition, with the asynchrony for the foot being even more pronounced. These results are in accordance with those reported in the literature (Aschersleben and Prinz 1995; Billon et al. 1996a; Fraisse 1980). Even if we quantitatively compare the results of the age-matched controls in the present study (mean age 47 years) with the results from the usually studied college students (mean age about 27 years), there is no substantial difference (for a similar result, see Stenneken, Aschersleben, Cole, and Prinz in press). The mean asynchrony for hand tapping in college students is typically between −50ms and −30ms (e.g. Aschersleben and Prinz 1995, 1997; Billon et al. 1996a; Fraisse 1980; Mates and Aschersleben 2000; Peters 1989), and that for foot tapping is between −90 ms and −50 ms (Aschersleben and Prinz 1995; Billon et al. 1996a; Fraisse 1980). The mean asynchronies for the present age-matched control group fall well within these ranges. Therefore, we conclude that age was not an important factor. More interestingly, the deafferented subject IW was able to coordinate his taps with the pacing signal (as with indicated in the study by Billon et al. 1996b) and, moreover, he showed a negative asynchrony as well. Compared control subjects, IW performed the synchronization task with rather large asynchronies. Though interindividual differences are well known, IW’s performance was still outside the range of control subjects. The mean asynchronies for individual subjects in the control group ranged from −65 ms to + 1 ms for hand tapping and from −116 ms to 0 ms for foot tapping. How was IW able to perform the synchronization task? Although he cannot rely on any kind of feedback to time and to control the 1nger or foot movements, his performance in the synchronization task was close to normal. The variability of the asynchronies within trials indicates that the timing of the movements was far from being random. On the contrary, performance was as stable as in the control subjects. However, variability between trials was rather high. For both hand and foot tapping the range in the corresponding trials was about 90ms, whereas the mean range in the control subjects was about 25 ms. This may suggest that IW made an estimate of the delay between his motor command and the tap at the beginning of each trial and then tried to synchronize the click with this simulation of the tap. To be more precise, we assume that he internally simulated the



central representation of the tap, which is usually based on the sensory feedback, and compared the timing of this internally generated tap representation with the timing of the click representation. We come back to this interpretation in the General Discussion. However, there are at least two possible alternative explanations. One hypothesis is that IW reacted to the previous signal, rather than anticipating the upcoming one. Then, his mean reaction time would amount to 500ms for the hand and 450ms for the foot condition, which is rather long. In a simple reaction time task, Stenneken et al. (in press) observed mean reaction times less than 300 ms in IW. Another argument against this interpretation is the fact that in reaction time tasks, the hand is clearly faster than the foot. This has been reported not only for healthy control subjects (e.g. Bard et al. 1991, 1992; Paillard 1949; Seashore and Seashore 1941; Stenneken et al. in press) but for deafferented subjects as well (Bard et al. 1992; Stenneken et al. in press). The second alternative explanation is that IW just maintained the initial phase he started within a given trial. This strategy would imply that IW performs his taps without any error correction. Tapping without error correction, however, would lead to considerable drifts within each trial.2 Moreover, the pattern of asynchronies at the beginning of each trial did not differ systematically from what is usually observed in control subject, that is, in some trials he kept the initial value whereas in other trials he started at a larger (or smaller) asynchrony and then ‘tuned in’.

11.3 Experiment 2 In the second experiment we studied the in2uence of auditory feedback on synchronization performance. We asked our subjects again to tap either with the hand or with the foot, but each time the subject touched the key an auditory feedback signal was presented via headphones. Based on earlier 1ndings (Aschersleben and Prinz 1995, 1997; Mates and Aschersleben 2000; Mates et al. 1992; O’Boyle and Clarke 1996; Repp, 2001), we expected a reduction in the negative asynchrony for both conditions (hand and foot tapping) in the control subjects. However, the hand–foot difference should persist in reduced form (see Aschersleben and Prinz 1995). This pattern of results is consistent with the assumption that, when auditory feedback is available, each tap is represented by a late tactile/kinesthetic and an early auditory feedback code which are integrated into a joint event code that is ‘dated’ somewhere between the two codes it consists of. This model also makes a clear prediction for the deafferented subject. As central and peripheral processing times should be the same for both auditory stimuli (pacing signal and auditory feedback signal) and no other source of feedback is available, the mean negative asynchrony should disappear if the corresponding representations (perceived click and perceived tap) are synchronized at a central level.

11.3.1 Method Subjects, stimuli, and procedure were the same as in Experiment 1, except that an auditory feedback signal (2000Hz, 60 dB[A], duration 10ms) was presented binaurally through headphones each time the 1nger/foot touched the response key. Feedback was clearly distinguishable from the 1000-Hz pacing signal.

235


236


11.3.2 Results and discussion Data analysis was identical to that in the 1rst experiment. According to our pre-set criteria, 3.6% of the trials of the control subjects were rejected whereas in the deafferented subject IW no trial had to be rejected. Figure 11.3 shows the mean asynchronies for hand and foot tapping for IW and for the control subjects. Contrary to our expectations, the asynchrony between tap and click was not reduced in the control subjects, relative to Experiment 1 (Fig. 11.2), but the difference in asynchrony between hand and foot tapping remained (19ms), as expected, t(13) = 2.97, p = 0.01. A 2 × 2 ANOVA comparing the results of Experiments 1 and 2 only revealed a signi1cant main effect of effector, F(1, 13) = 21.26, p < 0.001. For the control subjects in the present study, synchronizing the auditory feedback tone with the pacing signal seemed a fairly dif1cult task. First, the number of trials that had to be eliminated (because of large variability) increased compared with Experiment 1. Second, even after the elimination of these trials the mean variability within trials was clearly higher than in Experiment 1 (33 ms vs. 25ms). Earlier experiments with college-age students did not show such an increase in variability (e.g. Aschersleben and Prinz 1995, 1997). The age of the subjects may have played a role here. The deafferented subject, however, showed clearly reduced asynchronies as a consequence of auditory feedback (hand tapping:−37 ms, foot tapping:−35ms). However, the negative asynchrony did not disappear as expected. A 2 × 2 ANOVA comparing IW’s results from Experiments 1 and 2 revealed only a highly signi1cant effect of feedback F(1,4) = 181.38, p < 0.001. The main effect of effector and the effector × feedback interaction clearly failed to reach signi1cance (p > 0.18). In addition, the difference in asynchrony between hand and foot tapping disappeared both statistically and numerically (2ms), t(8) = 0.21, p > 0.20. The results of the deafferented subject indicate that he used the available feedback to time his actions and, moreover, that the auditory feedback clearly improved his performance. The persisting negative asynchronies may derive from the fact that this subject very much relies on visual control of movements. He told us that performing a movement without visual control is a very hard, atten-

Fig. 11.3 Averages and standard errors of the mean asynchronies for the deafferented subject IW and the age-matched control subjects, for hand and foot tapping with auditory feedback (Exp. 2).



tion-demanding task for him that requires huge effort. Therefore, performing the tapping movement without visual control created a kind of dual-task situation and, as a consequence, it may well be that for IW the situation we studied in the present experiment was not comparable to the same situation in control subjects. This idea was tested in Experiment 3.

11.4 Experiment 3 In this experiment, we allowed subjects to monitor their movements visually. If the hypothesis is correct that the remaining asynchrony in the deafferented subject was due to a strong demand of the spatial control of the movement, the asynchrony should disappear in the present experiment. For the control subjects, however, visual monitoring should not make a big difference because they mainly rely on proprioceptive feedback in the spatial control of their movements. Support for these predictions can be found in the study by Billon et al. (1996b). They compared a condition without auditory and visual feedback with a condition that offered ‘natural’ feedback, that is, subjects were allowed to visually monitor their movements and could hear the touch of the key. Billon et al. found no difference between these two conditions in the asynchronies of the control subjects. However, the deafferented subject showed a negative asynchrony only in the no-feedback condition. In fact, under natural feedback he even showed a small positive asynchrony.

11.4.1 Method Subjects, stimuli, and procedure were the same as in Experiment 2, except that subjects were able and instructed to visually monitor their 1nger or foot movements during tapping.

11.4.2 Results and discussion According to our pre-set criteria, in the control subjects and in the deafferented subject IW no trial had to be rejected. Fig. 11.4 shows the mean asynchronies for hand and foot tapping for IW and for

Fig. 11.4 Averages and standard errors of the mean asynchronies for the deafferented subject IW and the age-matched control subjects, for hand and foot tapping with auditory feedback and visual control (Exp. 3).

237


238


the control subjects. As expected, the control subjects still showed negative asynchronies similar to those in the previous two experiments. Moreover, the asynchrony between tap and click was not reduced in the control subjects, relative to Experiment 2 (Fig. 11.3), and the difference in asynchrony between hand and foot tapping remained (17ms), t(13) = 4.82, p < 0.001. A 2 × 2 ANOVA comparing the results of Experiments 2 and 3 only revealed a signi1cant main effect of effector, F(1,13) = 19.57, p < 0.001. These results clearly indicate that visual monitoring of the 1nger and foot movements had no in2uence on the timing of the taps. In the deafferented subject the average asynchrony between click and tap disappeared completely, t(9) = 0.77, p > 0.20, and there was again no signi1cant difference in asynchrony between hand and foot tapping (hand tapping: −11 ms, foot tapping: + 4 ms), t(4) = 1.63, p > 0.10. The results of this experiment 1nally showed the expected pattern. If IW is allowed to monitor his movements visually and gets auditory feedback about the tap onset, he is able to tap in exact synchrony with the auditory pacing signal. This is exactly what was predicted from the representational models. In this situation the time to establish a central representation of the auditory pacing signal should be identical to establishing a central representation of the tap, which is, under these conditions, only represented by the auditory feedback signal. Although he probably has some central engrams of tapping, which could interact with an auditory feedback, he apparently did not rely on them when presented with external ‘veridical’ feedback.

11.5 General discussion In the present chapter we examined the in2uence of sensory feedback on the timing of simple repetitive movements in a sensorimotor synchronization task. To account for the usually observed lead of the tap (negative asynchrony), we proposed that it is the central representations of tap and click that are synchronized, which are derived from sensory feedback. We then reviewed studies, in which the sensory feedback available from the tap, as well as its temporal characteristics, was manipulated. Going to the limits of such manipulations, we presented a study with a completely deafferented subject. We asked this subject, IW, to perform three synchronization tasks, which differed in the amount of feedback available from the tapping movement (hand and foot tapping). His performance was compared with the results of an age-matched control group. In principle, inferring the role of sensory feedback by comparing IW and control subjects is problematical, because IW has spent more than 25 years developing compensatory strategies to control his movements. A more appropriate test of deafferentation would have been to observe a deafferented patient soon after the onset of his neuropathy and before the start of the rehabilitation. However, IW’s remaining de1cits do serve to highlight where sensory feedback remains necessary. Contrary to our expectations, no in2uence of the feedback manipulation was found in the control subjects. A summary 3 × 2 ANOVA comparing the results of the three experiments revealed neither a signi1cant effect of feedback nor of the feedback × effector interaction. Only the main effect of effector reached signi1cance, F(1, 13) = 27.77, p < 0.001, which replicates the standard 1nding that foot taps occur earlier than hand taps. Although additional auditory feedback has been found to reduce the amount of asynchrony in previous studies, this 1nding was not replicated here, perhaps because of large variability in the age-matched group. In addition, presence or absence of visual control of the tapping movement also had no effect on the asynchrony in control subjects. This result is crucial for the comparison with the results obtained in the deafferented subject, and it is in line with the 1ndings reported by Billon et al. (1996b). They compared conditions with and without



‘natural’ feedback (seeing and hearing the tapping 1nger) and also found no asynchrony difference in the usual tapping task with normal subjects. For the deafferented subject, a corresponding summary ANOVA indicated a highly signi1cant effect of feedback, F(2, 8) = 241.99, p < 0.001, whereas the main effect of effector and the interaction between the two factors were far from being signi1cant (p-values > 0.20). With increasing amount of feedback, the timing of IW’s taps became more precise and the negative asynchrony disappeared. This interesting pattern of results leaves us with at least three questions. First, how was the deafferented subject able to tap at all in the absence of any feedback (Experiment 1)? Second, why did he produce (rather large) negative asynchronies in that condition? Third, why did the asynchrony not disappear in the condition with auditory feedback (Experiment 2), as predicted by the representational models? This prediction was only con1rmed in Experiment 3, when the deafferented subject was allowed to visually monitor his movements. Thus, we need to discuss the role of visual control in more detail. Let us start with the last question and then proceed to the 1rst two questions, which are, from our theoretical point of view, closely related. According to the representational models, which emphasize differences in peripheral and central processing times between the external events (click and tap) and their corresponding central representations, the asynchrony between click and tap should disappear when the processing times are identical. In normal subjects, this condition is hard to create. A recent study by Müller and colleagues (2001) apparently succeeded by applying a tactile pacing signal to the left index 1nger, which resulted in a disappearance of the negative asynchrony.3 With the deafferented subject we had the chance to study a situation in which processing times between click and tap could really be matched by using comparable auditory signals for both the pacing stimuli and the feedback from the taps. But, as Experiment 2 indicated, the asynchrony did not disappear completely. This was only the case when IW was allowed to visually monitor his movements. This visual monitoring had basically no effect on the timing of the taps in the healthy control subjects. What caused this fundamental difference between a deafferented subject and healthy control subjects in the use of visual control? In deafferented subjects like IW there is a permanent requirement for visual monitoring of their movements. This has also been reported for other deafferented subjects (see, e.g. Cole and Paillard 1995; Sanes, Mauritz, Dalakas, and Evarts 1985). Even in situations in which visual control is possible, each movement requires mental control, that is, visual feedback can only be used with concentration and attentional effort. For example, IW is not able to maintain a precision grip while walking. Studies using the dual-task paradigm have shown that without proprioception signi1cant mental resources are necessary to monitor movements (Ingram et al. 2000; Lajoie et al. 1996). Of course, the situation is even worse if visual control is excluded as in our 1rst two experiments. The presentation of auditory feedback in our second experiment gave only discrete temporal information about the movement at a certain point in time—it just indicated when the 1nger (or toe) hit the key. In contrast, control subjects received continuous feedback about the timing and the spatial parameters of their 1nger and toe movements via proprioception. This information was missing in the deafferented subject. We would like to suggest that IW requires visual feedback to control the spatial component of movement; that is, that visual feedback replaces the missing proprioceptive information. When IW could not visually monitor his actions, he had to produce and time the motor command from memory without the possibility of correcting it on the basis of proprioception, which led to a decrease in performance. Interestingly, however, IW was able to produce stable performance even in the absence of any sensory feedback (Experiment 1). As already discussed, the high variability between trials and the

239


240


relatively low variability within trials suggested that for each single sequence of taps IW determined in advance the movement parameters for the taps and then reproduced these movements rather consistently (however, still with some variability due to variability in the production of the motor command). Such an interpretation is supported by the fact that before a new trial could be started he 1rst made some 1nger taps under visual control, then put his head behind the screen (to eliminate visual feedback); only then could the trial be started. This suggests that during the following sequence he reproduced the motor command from memory in a way he had just determined under visual control. Although this strategy is devoid of peripheral feedback because feedback was excluded, IW’s ability to produce a sequence of 50 taps without considerable drifts indicates the presence of an errorcorrection process. But what was this process based on if there was no feedback available? One possible interpretation is that the deafferented subject internally predicted the time of his tap and then compared the time of the click with the time of the predicted tap. Or, to be more precise, he internally simulated the central representation of the tap which is usually based on the sensory feedback and compared the timing of this internally generated tap representation with the timing of the click representation. The idea of internal generation of sensory consequences of actions is clearly in line with so-called forward models (e.g. Wolpert 1997; Wolpert, Ghahramani, and Jordan 1995). These models assume that ef1cient motor control requires the representation of different states of our cognitive system, such as the current, the desired, and the predicted states. Inverse modeling is used to derive the actions required to move from the current state to the desired state, while forward modeling is used to derive the predicted state if these actions were to be performed. The forward model predicts the sensory consequences of an action, that is, it allows for anticipating reafferences (sensory effects) on the basis of efference copy (motor out2ow). Discrepancies between the desired and the actual outcome of motor commands are then the basis for error correction. These forward models have been used, for example, to explain abnormalities in the perception and control of actions such as in schizophrenia (e.g. Frith 1992), but the concept can be easily applied to the control of actions in deafferented subjects. In our deafferented subject, the forward model would predict the sensory consequences of the tap which then can be used to control and time the action even under conditions in which no actual feedback is available. At this point, it is important to recall that IW has a long experience in controlling movements via visual feedback only and is very used to his condition, unlike a control subject or someone who has just lost feedback due to some accident. Thus, it might be that during rehabilitation the original representation of the proprioceptive feedback was replaced by visual feedback. As a consequence, IW developed a way to internally generate tap-related information in situations in which no feedback is available. The basic idea would then be that he times his motor command such that the internally simulated feedback from the tap coincides with the perceived click. Both the relevance of visual information and the internal generation of movement consequences are supported by brain-imaging data of the subject IW (Athwal, Cole, Wolpert, Frith, and Frackowiak 1999). During PET scan a sequential 1nger–thumb opposition task of one hand was applied. The 1nger-opposition movement and a resting state were both combined with two types of visual feedback (vision of moving 1nger on a video screen vs. vision of resting 1nger). IW showed a pattern of activation that differed from that obtained in control subjects. In the movement condition with veridical visual feedback contralateral [previsual] areas were activated more than in controls, indicating the requirement of visual control. Moreover, contralateral activation of the inferior parietal cortex in conditions with a mismatch between produced and observed state, and bilateral cere-



bellar activation in all experimental conditions are in line with a feedforward control of movements (see also, Blakemore, Wolpert, and Frith 1998). Such an interpretation can explain IW’s relatively stable performance within trials because the internally generated feedback might not be as variable as sensory feedback (since additional ‘noise’ may arise in peripheral processing). In addition, the interpretation offers an explanation for IW’s negative asynchrony. If the original representation of the proprioceptive feedback was replaced by visual feedback, which then is internally generated in situations in which no feedback is available, then the timing of this generated feedback should be comparable to the original feedback. However, at the same time the withdrawal of visual control created a kind of dual-task situation for IW (because visual monitoring of the spatial components of the movement was no longer possible), resulting in rather large asynchronies.

11.6 Conclusions In demonstrating the important contribution of sensory feedback to the timing of actions, we have gained further support for the general idea that actions are timed and controlled by their anticipated effects (e.g. their sensory consequences). This idea, according to which actions are more effective if they are planned in terms of their intended outcome rather than in terms of their proximal effects, has been proposed by several authors (e.g. Greenwald 1970; Hommel, Müsseler, Aschersleben, and Prinz, in press; James 1890; Prinz 1990, 1997). Empirical support for such an approach not only comes from our studies in sensorimotor synchronization but also from studies on bimanual coordination (Drewing and Aschersleben 2001; Drewing, Hennings, and Aschersleben in press), on compatibility effects, and on sequence learning (see, e.g. Elsner and Hommel 2001; Kunde 2001; the contributions by Hazeltine this volume, Chapter 33; Müsseler and Wühr, this volume, Chapter 25; and Ziessler and Nattkemper, this volume, Chapter 32).

Acknowledgments The authors express their gratitude to IW for his participation in the experiment and his great patience in this matter. We wish to thank Bruno Repp, an anonymous reviewer, and Bernhard Hommel for their helpful criticism, suggestions, and comments on an earlier draft. We also wish to thank Frank Miedreich for programming and Renate Tschakert for her support in data collection. This research was partially supported by a grant from the Deutsche Forschungsgemeinschaft to the 1rst author. Requests for reprints should be sent to Gisa Aschersleben, Max-Planck-Institut für Psychologische Forschung, Postfach 34 01 21, D-80098 München, Germany.

Notes 1. This idea is closely related to an assumption proposed by Rieser and Pick (this volume, Chapter 8). They propose that sensory information resulting from different sensory channels gets integrated and results in a unitary representation of space that is used for locomotion. 2. Even if only a small deviation from the target interval is introduced (e.g. the intertap interval is set at 599 ms instead of 600 ms), the error in the asynchrony would accumulate (in our example to a sum of 50 ms after 50 taps), resulting in a clear drift.

241


242


3. When the tactile pacing signal was applied to the big toe, no asynchrony in 1nger tapping was observed either, but this condition led to large intra- and interindividual variabilities, showing that the task was rather dif1cult for the subjects to perform. This 1nding as well as others revealed in experiments with manipulations on the stimulus side cannot easily be explained by the representational models. They suggest that other factors in addition to the feedback from the tap are involved in the origin of the negative asynchrony (for overviews, see Aschersleben, in press; O’Boyle 1997).

References Aschersleben, G. (1994). Afferente Informationen und die Synchronisation von Ereignissen. Frankfurt: Lang. Aschersleben, G. (2000). Knowledge of results and the timing of actions (Paper No. 1/2000). Munich: Max Planck Institute for Psychological Research. Aschersleben, G. (2001). Effects of training on the timing of simple repetitive movements. Manuscript submitted for publication. Aschersleben, G. (in press). Temporal control of movements in sensorimotor synchronization. Brain and Cognition. Aschersleben, G. and Prinz, W. (1995). Synchronizing actions with events: The role of sensory information. Perception and Psychophysics, 57, 305–317. Aschersleben, G. and Prinz, W. (1997). Delayed auditory feedback in synchronization. Journal of Motor Behavior, 29, 35–46. Aschersleben, G., Gehrke, J., and Prinz, W. (2001). Tapping with peripheral nerve block: A role for tactile feedback in the timing of movements. Experimental Brain Research, 136, 331–339. Aschersleben, G., Gehrke, J., and Prinz, W. (in press). A psychophysical approach to action timing. In C. Kaernbach, E. Schröger, and H. Müller (Eds.), Psychophysics beyond sensation: Laws and invariants in human cognition. Hillsdale, NJ: Erlbaum. Athwal, B., Cole, J.D., Wolpert, D.M., Frith, C., and Frackowiak, R.S.J. (1999). A PET study of motor control in a ‘deafferented’ subject. Journal of Physiology, 518P, 65P. Bard, C., Paillard, J., Teasdale, N., Fleury, M., and Lajoie, Y. (1991). Self-induced versus reactive triggering of synchronous hand and heel movement in young and old subjects. In J. Requin and G.E. Stelmach (Eds.), Tutorials in motor neuroscience, pp. 189–196. Amsterdam: Kluwer. Bard, C., Paillard, J., Lajoie, Y., Fleury, M., Teasdale, N., Forget, R., and Lamarre, Y. (1992). Role of the afferent information in the timing of motor commands: A comparative study with a deafferent patient. Neuropsychologia, 30, 201–206. Billon, M., Bard, C., Fleury, M., Blouin, J., and Teasdale, N. (1996a). Simultaneity of two effectors in synchronization with a periodic external signal. Human Movement Science, 15, 25–38. Billon, M., Semjen, A., Cole, J., and Gauthier, G. (1996b). The role of sensory information in the production of periodic 1nger-tapping sequences. Experimental Brain Research, 110, 117–130. Blakemore, S.J., Wolpert, D.M., and Frith, C. (1998). Central cancellation of self-produced tickle sensation. Nature Neuroscience, 1, 635–640. Cole, J. (1995). Pride and a daily marathon. Cambridge, MA: MIT Press. Cole, J. and Paillard, J. (1995). Living without touch and peripheral information about body position and movement: Studies with deafferented subjects. In J.L. Bermudez and A.J. Marcel (Eds.), The body and the self, pp. 245–266. Cambridge, MA: MIT Press. Cole, J.D. and Sedgwick, E.M. (1992). The perceptions of force and of movement in a man without large myelinated sensory afferents below the neck. Journal of Physiology, 449, 503–515. Cooke, J.D., Brown, S., Forget, R., and Lamarre, Y. (1985). Initial agonist burst duration changes with movement amplitude in a deafferented patient. Experimental Brain Research, 60, 184–187. Drewing, K. and Aschersleben, G. (2001). Reduced timing variability during bimanual coupling: A role for sensory information. Manuscript submitted for publication. Drewing, K., Hennings, M., and Aschersleben, G. (in press). The contribution of tactile reafferences to increased temporal regularity during simple bimanual 1nger tapping. Psychological Research. Dunlap, K. (1910). Reactions on rhythmic stimuli, with attempt to synchronize. Psychological Review, 17, 399–416. Elsner, B. and Hommel, B. (2001). Effect anticipation and action control. Journal of Experimental Psychology: Human Perception and Performance, 27, 229–240.



Fraisse, P. (1980). Les synchronizations sensori-motrices aux rythmes. In J. Requin (Ed.), Anticipation et comportement, pp. 233–257. Paris: Centre National. Fraisse, P., Oléron, G., and Paillard, J. (1958). Sur les repères sensoriels qui permettent de contrôler les mouvements d’accompagnement de stimuli périodiques. [On the sensory reference points that allow for controlling movements accompanying periodic stimuli]. L’Année Psychologique, 58, 322–338. Frith, C.D. (1992). The cognitive neuropsychology of schizophrenia. Hove, UK: Erlbaum. Gehrke, J. (1995). Sensorimotor synchronization: The intensity of afferent feedback affects the timing of movements (Paper 15/1995). Munich: Max Planck Institute for Psychological Research. Gehrke, J. (1996). Afferente Informationsverarbeitung und die Synchronisation von Ereignissen. Dissertation at the Ludwig Maximilians University, Munich, Germany. Greenwald, A. (1970). Sensory feedback mechanisms in performance control: With special reference to the ideomotor mechanism. Psychological Review, 77, 73–99. Hazeltine, E. This volume, Chapter 33. Hommel, B., Müsseler, J., Aschersleben, G., and Prinz, W. (in press). The theory of event coding (TEC): A framework for perception and action planning. Behavioral and Brain Sciences. Ingram, H.A., van Donkelaar, P., Cole, J., Vercher, J.L., Gauthier, G.M., and Miall, R.C. (2000). The role of proprioception and attention in a visuomotor adaptation task. Experimental Brain Research, 132, 114–126. James, W. (1890). The principles of psychology. New York: Macmillan. Johnson, W.S. (1898). Researches in practice and habit. Studies from the Yale Psychology Laboratory, 6, 51–105. Koch, R. (1999). Detection of asynchrony between click and tap (Paper 1/1999). Munich: Max Planck Institute for Psychological Research. Kolers, P.A. and Brewster, J.M. (1985). Rhythms and responses. Journal of Experimental Psychology: Human Perception and Performance, 11, 150–167. Kunde, W. (2001). Response-effect compatibility in manual choice reaction tasks. Journal of Experimental Psychology: Human Perception and Performance, 27, 387–394. Lajoie, Y., Teasdale, N., Cole, J.D., Burnett, M., Bard, C., Fleury, M., Forget, R., Paillard, J., and Lamarre, Y. (1996). Gait of a deafferented subject without large myelinated sensory 1bers below the neck. Neurology, 47, 109–115. Ludwig, C. (1992). Experiment zur Synchronisation akustischer Führungssignale. Seminararbeit, Ludwig Maximilians University, Munich, Germany. Mates, J. (1994). A model of synchronization of motor acts to a stimulus sequence. I. Timing and error corrections. Biological Cybernetics, 70, 463–473. Mates, J. and Aschersleben, G. (2000). Sensorimotor synchronization: The in2uence of temporally displaced auditory feedback. Acta Psychologica, 104, 29–44. Mates, J., Müller, U., Radil, T., and Pöppel, E. (1994). Temporal integration in sensorimotor synchronization. Journal of Cognitive Neuroscience, 6, 332–340. Mates, J., Radil, T., and Pöppel, E. (1992). Cooperative tapping: time control under different feedback conditions. Perception and Psychophysics, 52, 691–704. Miedreich, F. (2000). Zeitliche Steuerung von Handlungen. Empirischer Test des Wing–Kristofferson Modells. Aachen: Shaker Verlag. Miyake, I. (1902). Researches on rhythmic action. Studies from the Yale Psychology Laboratory, 10, 1–48. Müller, K., Aschersleben, G., Koch, R., Freund, H.-J., and Prinz, W. (1999). Action timing in an isochronous tapping task: Evidence from behavioral studies and neuroimaging. In G. Aschersleben, T. Bachmann, and J. Müsseler (Eds.), Cognitive contributions to the perception of spatial and temporal events, pp. 233–250. Amsterdam: Elsevier. Müller, K., Aschersleben, G., Schmitz, F., Schnitzler, A., Freund, H.-J., and Prinz, W. (2001). Modality-speci1c central control units in sensorimotor synchronization. Manuscript submitted for publication. Müsseler, J. and Wühr, P. This volume, Chapter 25. O’Boyle, D.J. (1997). On the human neuropsychology of timing of simple, repetitive movements. In C.M. Bradshaw and E. Szabadi (Eds.), Time and behaviour. Psychological and neuro-behavioural analyses, pp. 459–515. Amsterdam: Elsevier. O’Boyle, D.J. and Clarke, V.L. (1996). On the source of the negative synchronization error during temporaltracking performance. Brain Research Association Abstracts, 13, 40. Paillard, J. (1949). Quelques données psychophysiologiques relatives au déclenchement de la commande motrice. [Some psychophysiological data relating to the triggering of motor commands]. L’Année Psychologique, 48, 28–47.

243


244


Peters, M. (1989). The relationship between variability of intertap intervals and interval duration. Psychological Research, 51, 38–42. Prinz, W. (1990). A common-coding approach to perception and action. In O. Neumann and W. Prinz (Eds.), Relationships between perception and action: Current approaches, pp. 167–201. Berlin: Springer-Verlag. Prinz, W. (1997). Perception and action planning. European Journal of Cognitive Psychology, 9, 129–154. Repp, B.H. (2000). Compensation for subliminal timing perturbations in perceptual-motor synchronization. Psychological Research, 63, 106–128. Repp, B.H. (2001). Phase correction, phase resetting, and phase shifts after subliminal timing perturbations in sensorimotor synchronization. Journal of Experimental Psychology: Human Perception and Performance, 27, 600–621. Rieser, J. and Pick, H.L. This volume, Chapter 8. Sanes, J.N., Mauritz, K.-H., Dalakas, M.C., and Evarts, E.V. (1985). Motor control in humans with large-1ber sensory neuropathy. Human Neurobiology, 4, 101–114. Seashore, S.H. and Seashore, R.H. (1941). Individual differences in simple auditory reaction times of hands, feet and jaws. Journal of Experimental Psychology, 29, 342–345. Stenneken, P., Aschersleben, G., Cole, J., and Prinz, W. (in press). Self-induced versus reactive triggering of synchronous movements. A comparative study with a deafferented patient. Psychological Research. Thaut, M.H., Tian, B., and Azimi-Sadjadi, M.R. (1998). Rhythmic 1nger tapping to cosine-wave modulated metronome sequences: Evidence of subliminal entrainment. Human Movement Science, 17, 839–836. Vos, P.G., Mates, J., and van Kruysbergen, N.W. (1995). The perceptual centre of a stimulus as the cue for synchronization to a metronome. Quarterly Journal of Experimental Psychology, 48, 1024–1040. Wohlschläger, A. and Koch, R. (2000). Synchronization error: An error in time perception. In P. Desain and L. Windsor (Eds.), Rhythm perception and production, pp. 115–127. Lisse: Swets. Wolpert, D.M. (1997). Computational approaches to motor control. Trends in Cognitive Sciences, 1, 209–216. Wolpert, D.M., Ghahramani, Z., and Jordan, M.I. (1995). An internal model for sensorimotor integration. Science, 269(5232), 1880–1882. Zießler, M. and Nattkemper, D. This volume, Chapter 32.


12 The embodiment of musical structure: effects of musical context on sensorimotor synchronization with complex timing patterns Bruno H. Repp Abstract. Two experiments demonstrate that musical context facilitates sensorimotor synchronization with complex timing patterns that are compatible with the musical structure. Several very different timing patterns were derived from an analysis of expressive performances of a musical excerpt. A random pattern (Exp. 1) or phase-shifted versions of the musical patterns (Exp. 2) served as comparisons, and an isochronous pattern served as practice. Musically trained participants 1rst attempted repeatedly to synchronize their 1nger taps with click sequences instantiating these timing patterns. Subsequent repetitions of the click sequences were accompanied by the identically timed music, and 1nally, the music disappeared and was only to be imagined in synchrony with the clicks. Compared with the random or phase-shifted patterns, synchronization accuracy for the musical patterns improved as soon as the music was introduced, especially when the pattern was highly typical. This relative improvement was reduced or absent when the music was merely imagined. Nevertheless, both musical context and imagery systematically modulated the timing of 1nger taps in synchronization with strictly isochronous click sequences. Thus perception or imagination of musical structure can involuntarily affect the timing of concurrent action, presumably by modulating the timekeeping processes that pace the motor behavior. This study also demonstrates that radically different timing patterns are compatible with the same musical structure, as they seem to be in expert artistic performance.

12.1 Introduction There is an intimate relationship between music and the human body (see, e.g. Clarke 1993a; Iyer 1998; Pierce and Pierce 1989; Repp 1993; Shove and Repp 1995). Music is produced by moving various extremities across musical instruments, or by engaging the mouth, lungs, and vocal tract. These moving parts of the body are attached to (or embedded in) the trunk which provides structural support and often participates by swaying along. In most cultures, listeners participate in music by dancing, clapping, tapping, or rocking in synchrony with its rhythm. Only in the Western tradition of serious art music, overt movement is proscribed for audiences in concert halls, but listeners still feel a readiness to move, or imagine themselves moving along with the music, or speak about being moved by the music. Thus there is a very close relation between music perception and action, particularly with regard to rhythm and timing. Human music performances are distinguished from machine renditions (unless they successfully simulate human performance) by the presence of many subtle features that originate in the musicians’ movements. Clynes (1983) has referred to these features as ‘expressive microstructure’ which conveys ‘living qualities’. One of these features is expressive timing. It consists in systematic deviations


246


from temporal regularity which signify to a listener that the music was not produced by a machine but by a thinking, feeling, and moving being. Expressive timing originates from three sources (cf. Penel and Drake 1998, 1999): (1) biomechanical constraints in technically dif1cult passages; (2) obligatory perceptual–motor patterns related primarily to rhythm and rhythmic grouping; and (3) intentional communication of structural or emotional aspects of the music. The present study is mainly concerned with the second of these factors, only incidentally with the third, and not at all with the 1rst. Recent research has produced considerable evidence that a particular musical structure is often associated with a particular expressive timing pattern. This most typical pattern corresponds to the average timing pattern of a large sample of human performances. It is representative of many individual performances (Repp 1998a) and is judged to be aesthetically pleasing (Repp 1997). When musicians are requested to play with perfectly regular timing (as speci1ed in a musical score) or in synchrony with a metronome (Repp 1999c), or when they try to create a perceptually regular performance on a computer by adjusting successive temporal intervals (Penel 2000), they nevertheless produce small but systematic timing variations whose pattern resembles that of the typical expressive timing pattern (Behne and Wetekam 1993; Drake and Palmer 1993; Palmer 1989; Penel and Drake 1998; Repp 1999a,c). A complementary pattern of perceptual biases is observed when listeners are asked to detect local deviations from perfect regularity in a musical passage (Repp 1992b, 1998b,c,d, 1999b,c). These 1ndings suggest that there is a level of subconscious and obligatory timing variation upon which larger intentional expressive variations are superimposed. The obligatory variations seem to be linked to the lowest level of rhythmic grouping in the music, whereas intentional expressive timing re2ects several hierarchical levels of grouping (Penel and Drake, 1998) as well as possibly other factors (meter, melodic contour, harmony, etc.). The similarity of the pattern of obligatory variations to the typical expressive timing pattern may be explained by the fact that they share the lowest level of grouping, which accounts for much of the timing variation (Penel 2000). However, intentional expressive timing does not always follow the most typical pattern. The timing patterns produced by experienced concert artists sometimes represent quite radical departures from the norm (Repp 1992a, 1998a). While such highly individual timing patterns may sound strange on 1rst hearing, the fact that they were produced by outstanding musicians indicates that they are not arbitrary or inappropriate to the musical structure. Nevertheless, it seems that these patterns are not strongly implied by the musical structure, if at all. It appears that creative performers must overcome a natural tendency to produce the most typical timing pattern (Repp 2000b). Penel and Drake (1998) have argued that typical timing is a form of motor compensation for perceptual timing distortions caused by rhythmic grouping. If so, then the typical timing pattern must always be present underlyingly, even if it is overridden by different intentions. Alternatively, the typical timing pattern may be regarded as a natural strategy for representing rhythmic groups in action, a strategy that in turn causes perceptual biases via a motor–perceptual interaction (Repp 1998d; Viviani and Stucchi 1992). Perhaps, then, the typical (obligatory) timing pattern is a consequence of carrying out grouped actions on a musical instrument. However, Repp (1999a,b,c) eliminated this factor by asking participants (including non-pianists and even non-musicians) to tap with the index 1nger in synchrony with piano music that was reproduced under computer control in a perfectly regular fashion. The tap-tone asynchronies and inter-tap intervals were still found to exhibit systematic deviations from regularity that tended to be positively correlated with the typical expressive timing pro1le. Thus, perception of musical structure exerted an in2uence even on a concomitant action pattern that had


The embodiment of musical structure

no structure of its own. The correlation between the obtained timing pattern and the typical expressive timing pattern was relatively small; this may have been due in part to an additional process of automatic error correction in synchronization (Mates 1994; Pressing 1998; Repp 2000a; Vorberg and Wing 1996), which counteracted the emergence of the typical timing pattern. The tentative conclusion from these results, therefore, was that a musical structure tends to induce a tendency towards the typical timing pattern in concurrent motor activity. It may be predicted, then, that this tendency to move expressively should facilitate the synchronization of movements with music that exhibits the typical (intentional) expressive timing pattern, even though that pattern shows much larger deviations from regularity than the obligatory timing variations induced by the music, which are generally below the perceptual detection threshold. This prediction has been investigated previously by asking pianists to tap their index 1nger in synchrony with (1) one of their own previously recorded expressive performances, (2) a computer-generated version that exhibited the typical timing pattern (the average timing pattern of a large number of human performances), and (3) a sequence of clicks that instantiated the typical timing pattern, while participants imagined the music in synchrony with the clicks (Repp 1999a).1 The pianists were quite successful in all three tasks (though not as accurate as in tapping to an isochronous sequence). Moreover, their synchronization was as accurate with the clicks as with the music itself, which suggested that musical imagery could effectively substitute for the musical sound. However, one shortcoming of that study was that it included no other conditions that the pianists’ synchronization accuracy could be compared with. For example, it was not determined how well they could synchronize with the clicks without imagining the music, or with music having expressive timing patterns other than the most typical one, or with non-musical timing patterns of comparable average tempo and variability. Thus it was not clear whether synchronization with the most typical timing pattern in music was better than with other possible timing patterns, or indeed whether the relatively good synchronization performance had anything to do with music at all. It was the purpose of the present study to make these additional comparisons. Two similar experiments were conducted to address 1ve hypotheses or predictions. One hypothesis was that synchronization with music exhibiting a typical expressive timing pattern would be more accurate than synchronization with music exhibiting a less typical (but still structurally appropriate) timing pattern, because the former pattern is more strongly implied by the musical structure than the latter. To that end, several timing patterns of different typicality, derived from an extensive performance analysis (Repp 1998a), were used. Another hypothesis was that synchronization with even the less typical musical timing patterns would be more accurate than synchronization with an arbitrary or structurally inappropriate timing pattern, imposed on the same music. To test this prediction, synchronization with the musical patterns was compared to synchronization with a random pattern (Exp. 1) or with phase-shifted versions of the musical patterns (Exp. 2). A third hypothesis was that the differences just predicted would also emerge, though perhaps be smaller in magnitude, when the music was merely imagined in synchrony with a click sequence instantiating the timing patterns. (This click sequence also accompanied the music when music was present.) A fourth hypothesis was that timing patterns derived from expressive music performance might be easier to synchronize with than arbitrary timing patterns even in the absence of real or imagined music, simply because musical patterns are more regular. Moreover, musical timing patterns may differ from each other in their degree of regularity (i.e. periodicity or predictability), and hence in how dif1cult they are to learn and predict in repeated presentations.2 Therefore, synchronization accuracy was also assessed in a condition in which music was neither present nor imagined (i.e. where

247


248


the timing pattern was carried only by a click sequence). This condition provided a crucial baseline for interpreting the 1ndings in the music and imagery conditions, and it necessitates an important quali1cation of the 1rst three hypotheses. Speci1cally, their predictions are that synchronization with musical timing patterns should be selectively facilitated when music is present or imagined, compared with a condition in which music is neither present nor imagined. This selective facilitation should be largest for the most typical timing pattern and smaller for the less typical musical patterns. There should be no facilitation and possibly even interference for arbitrary or structurally inappropriate timing patterns. Viewed from an ANOVA perspective, the effects of primary interest in this study thus were interactions between condition and pattern type, not main effects. Finally, a 1fth hypothesis was that synchronization accuracy would improve as a function of repeated presentation of the same timing pattern, but more so for musical patterns than for structurally inappropriate patterns (and most clearly for the most typical pattern) when music is present or imagined. Thus, an interaction between pattern type and trial number was also predicted. To get used to the synchronization task and the three experimental conditions (clicks only, clicks plus music, clicks plus imagined music), participants 1rst tapped in time with an isochronous pattern. This made it possible to address another interesting issue in passing, as it were. As mentioned earlier, tapping in synchrony with isochronous music leads to systematic deviations from regularity in the timing of the taps (Repp 1999a,b,c). One question was whether that 1nding would be replicated when the music merely accompanies an isochronous click sequence that participants try to synchronize with. Even more interesting, however, was the question of whether similar systematic deviations from regularity would be evident when the music was merely imagined in synchrony with the isochronous click sequence. A previous attempt to determine this (Repp 1999a) led to unclear results, perhaps because the instructions had not suf1ciently emphasized musical imagery. If a signi1cant effect of musical imagery were found in this very simple synchronization task, this would constitute convincing evidence of the reality of musical imagery and provide further proof of a close connection between music perception and action.

12.2 Experiment 1 12.2.1 Methods 12.2.1.1 Materials The timing patterns were derived from an analysis of 115 expert performances of the opening (bars 1–5) of Frédéric Chopin’s Etude in E major, op. 10, No. 3 (Repp 1998a). A computer-generated score of this music is shown on top of Fig. 12.1. The second half of the original bar 5 was condensed into a chord to give maximal closure to the excerpt, as heard in the experiment. Below the musical score and vertically aligned with it, Fig. 12.1(a) shows the most typical expressive timing pattern (or timing pro1le) for this excerpt (T0). This is the average pro1le of the 115 performances whose timing was measured from digitized acoustic recordings. It is equivalent to the 1rst unrotated principal component obtained in a principal components analysis of the performance timing pro1les, a component which accounted for 61% of the variance. The graph depicts tone interonset intervals (IOIs) as a function of metrical (score) position, with 8 sixteenth-note subdivisions per bar. The initial upbeat IOI, corresponding to an eighth note in the score, has been excluded from all graphs and statistics; its average duration was 1122 ms. All other IOIs represent nominal sixteenth-



Fig. 12.1 (top) A computer-generated score of the opening of Etude in E major, op. 10, No. 3, by Frédéric Chopin. (a) The most typical expressive timing pro1le (T0) for this music. (b), (c), (d) Mutually uncorrelated timing pro1les (T1, T2, T4) representing principal components of the timing patterns observed in expert performances. (e) An arbitrary timing pattern (R1) obtained by randomizing the inter-onset intervals (IOIs) of T1. Solid circles indicate IOIs initiated by melody notes, open circles those initiated by accompaniment notes only.

note intervals. IOIs initiated by melody tones (among other tones) are shown as 1lled circles, those initiated only by accompaniment tones as open circles. The melody, in the highest voice, is divided into six rhythmic groups (runs of 1lled circles in the graph), each ending with a sustained tone during which the accompaniment in the other voices continues. It can be seen that the T0 pattern

249


250


includes ritardandi (1nal slowing) within each of the melodic segments, as well as a lengthening of the 1nal IOI in bar 3 (which is the initial IOI of the longest melodic group) and sometimes of the 1nal IOI of an accompaniment passage immediately preceding a melodic group (the initial IOIs in bars 2, 3, and 5). The T0 pattern was not used in Experiment 1 because of a concern that its correlation with the other patterns, especially T1, might lead to carry-over effects of pattern learning. However, it was used in Experiment 2. Three additional musical timing pro1les (T1, T2, T4) were used in Experiment 1 and are shown in Fig. 12.1(b), (c), and (d). They represent the 1rst, second, and fourth Varimax-rotated principal components of the timing patterns of the 115 expert performances (Repp, 1998a) and respectively accounted for 31%, 17%, and 11% of the variance.3 Thus, T1 was more typical of expert performance than were T2 or T4, and this was also re2ected in their respective correlations with T0 (see Table 12.1), which may serve as indices of typicality. Being principal components, these three pro1les were mutually uncorrelated. Originally vectors of standard scores, they were converted into IOIs by multiplying them with the average within-performance standard deviation (80 ms) and adding them to the grand average IOI duration of the 115 performances (533 ms). Thus they all had the same basic tempo and degree of timing modulation. A fourth pattern, R1, was generated by randomly scrambling the IOI durations of the T1 pattern (Fig. 12.1(e)). As can be seen in Table 12.1, the typicality of R1 was even lower than that of T4. The R1 pro1le correlated with the three musical pro1les 0.21, −0.18, and −0.04, respectively (all n.s.). The duration of the initial upbeat IOI (not shown) was 1000 ms in all four patterns used in Experiment 1. The four timing patterns also differed in complexity or regularity. For example, T1 is characterized by strong ritardandi within all melodic groups, but it lacks the other timing features seen in T0, and this results in a very clear periodicity. By contrast, T2 shows a striking accelerando in the melodic group of bar 2 and to a lesser degree also in bars 1 and 5, but not at all in bars 3 and 4, which makes this pattern more complex than T1. T4 shows pronounced between-group ritardandi that exceed the within-group ritardandi, as well as a lengthening of the 1nal IOI in bar 3. It seems to be of intermediate complexity. The random pattern, of course, is the most complex pattern. To quantify these intuitions, an index of the degree of pattern periodicity was computed in the form of the lag-8 autocorrelation (ac8), which assesses the average similarity of timing from one bar to the next. A measure of relative pattern complexity was then obtained by subtracting ac8 from 1. These complexity indices are shown in Table 12.1. Furthermore, Table 12.1 includes the lag-1 autocorrelations (ac1) of the four patterns, which will be referred to later. In addition to the four timing patterns, an isochronous sequence with constant IOIs of 500 ms (except for an initial 1000-ms IOI) was presented. Each of these 1ve timing patterns was imposed

Table 12.1 Typicality indices (i.e. correlations with T0), complexity indices (i.e. 1 − ac8; see text for explanation), and lag-1 autocorrelations (ac1) for the four timing patterns used in Exp. 1 Pattern

Typicality

Complexity

T1 T2 T4 R1

0.67 0.46 0.36 0.20

0.31 0.73 0.58 0.90

ac1 0.29 0.59 0.15 – 0.25



on a series of what will informally be called ‘clicks’. Each click was in fact a high-pitched tone (C8, MIDI pitch 108, fundamental frequency 4,168 Hz) produced on a Roland RD-250 s digital piano, with a nominal duration of 20 ms. The tones had sharp onsets followed by a rapid decay and a longer soft ringing. Each click sequence comprised 38 identical sounds. When the click sequence was accompanied by the music, the music had exactly the same timing pattern in terms of its top-line tones (the highest tones in all sixteenth-note positions). The clicks coincided (within 1 ms) with the onsets of these top-line tones and were clearly audible above the music. The precise methods for synthesizing the music performances are described in Repp (2000b).

12.2.1.2 Participants Twelve undergraduate students from Yale University were paid to participate. All had advanced musical training, which was a necessary requirement in a study of expressive timing and musical imagery. Three of them were pianists, and the others, several of whom also played the piano, were players of string instruments in the Yale Symphony Orchestra. 12.2.1.3 Procedure Participants were 1rst instructed in the use of the response device, a Fatar Studio 37 MIDI controller (a silent three-octave piano keyboard). They were instructed to hold the controller on their lap, to keep their index 1nger in contact with a self-chosen white key, to release the key fully before pressing it again, to start tapping with the second click in each sequence, to stay in synchrony with the clicks at all times, and not to count the clicks. The response key moved about 10 mm from its resting position to the (cushioned) bottom position, but the electronic contact occurred before the lowest position was reached, which added a small negative constant to the tap-tone asynchronies. The response key did not make any audible sound unless it was struck very hard, so that participants generally had to gauge their synchronization errors cross-modally. The keypresses were registered by a MAX patch running on a Macintosh Quadra 660AV computer, which also controlled the playback of the sequences on a Roland RD-250s digital piano (‘Piano 1’ sound).4 Participants sat in front of the computer monitor, which displayed the trial number, and listened binaurally over Sennheiser HD540 II earphones. The three conditions (clicks only, clicks with accompanying music, and clicks with imagined music—referred to in the following as ‘clicks’, ‘music’, and ‘imagery’) were presented in the same order to all participants, constituting three successive parts of the experimental session. Within each condition, all timing patterns were presented in the same order. Each condition started with the isochronous sequence, but the order of the other four sequences was varied across participants, according to three different 4 × 4 Latin squares. Each timing pattern was presented 10 times in succession, without any preceding practice trials. The participants’ task was to tap in synchrony with each pattern to the best of their ability, and to try to predict the pattern with their taps from the second trial on. In the music condition, the instruction was to tap in synchrony with the clicks and not to pay any special attention to the music. In the imagery task, participants were told to imagine the music in synchrony with the clicks and to be sure not to make an extra tap at the end, since this would indicate that they had not imagined the music correctly. A copy of the musical score (Fig. 12.1, top) was in view throughout the music and imagery conditions, propped up below the computer monitor. There were 3 seconds of silence between trials, short breaks between timing patterns, and longer breaks between conditions.

251


252


12.2.1.4 Analysis Three different measures of synchronization accuracy were used. One was the standard deviation of the asynchronies (sda). This measure was useful because all sequences used had the same average tempo (i.e. mean IOI duration) and the same average timing modulation (i.e. standard deviation of IOIs). If participants were able to perfectly predict a timing pattern with their taps, then the standard deviation of the asynchronies would be equal to that found with an isochronous sequence. Of course, in view of the complexity of the patterns, prediction was not expected to be perfect in any condition. The other two measures of synchronization accuracy were correlational. One was the lag-0 cross-correlation (r0) between the inter-tap intervals (ITIs) and the click IOIs. If the taps predict the sequence timing pattern accurately, then r0 will be high. The other measure was in a way the converse of r0. Michon (1967) 1rst demonstrated that attempts to synchronize with an auditory sequence whose temporal intervals vary unpredictably result in ITIs that echo the sequence IOIs at a lag of one. (See also Hary and Moore 1985, 1987; Schulze 1992.) This temporal tracking behavior seems to be the consequence of an automatic error-correction process that tries (unsuccessfully) to minimize the synchronization error. It results in a high lag-1 cross-correlation (r1) between ITIs and IOIs, which thus is a measure of the participant’s inability to predict the temporal pattern. Thaut, Tian, and Azimi-Sadjadi (1998) found that tracking occurred even with sequences that were modulated in a regular, periodic fashion, but this may have been due to the small size of the modulations. When larger modulations of a regular, meaningful, or familiar nature are imposed on a stimulus sequence, the participant’s taps will tend to predict the sequence timing, which reduces r1 and increases r0 (Michon, 1967). However, it is problematic to rely on the raw values of r0 and r1. Each of these correlations has a theoretical lower limit that depends on the temporal structure of the sequence. In fact, it seems that both correlations have the same lower limit, namely the lag-1 autocorrelation (ac1) of the sequence timing pattern: when prediction (r0) is optimal, r1 will approach ac1 because the sequence of ITIs is similar to the sequence of IOIs. When tracking (r1) is maximal, r0 will approach ac1 because the ITIs echo the IOIs at a lag of 1. Therefore, a correction was applied to both r0 and r1, in order to take into account the fact that different timing patterns have different ac1 values (see Table 12.1). The prediction index (r0*) thus was computed as (r0 − ac1)/(1 − ac1), and the tracking index (r1*) was computed as (r1 − ac1)/(1 − ac1). Both indices had a theoretical range from near zero to 1. 12.2.2 Results and discussion Because of space restrictions, only the results for one of the three indices of synchronization accuracy, the prediction index (r0*), will be reported in detail. In general, the results for the sda index were similar, whereas those for the tracking index (r1*) were less clear, suggesting that, despite a strong negative relationship with r0*, r1* captures somewhat different aspects of synchronization behavior. The results for r0* are shown as a function of trial number in Fig. 12.2. Rather than comparing the results among all three experimental conditions at once, three separate repeated-measures ANOVAs were conducted, each of which compared two conditions. The 1xed variables in each ANOVA were condition (2), pattern (4), and trial (10). Within each ANOVA, separate comparisons were carried out between each musical pattern and the R1 pattern, which served as the baseline. Additional twoway ANOVAs were conducted on each individual condition. The main effect of pattern was highly signi1cant in all three two-way ANOVAs, F(3, 33) > 7.8, p < 0.0005. Overall, performance tended to be best for the T1 pattern, followed by T4, R1, and T2. It



Fig. 12.2 Average prediction indices (r0*) as a function of trial number for four timing patterns in the three conditions of Experiment 1. (a) Click condition. (b) Music condition. (c) Imagery condition.

was surprising that T2 yielded poorer performance than R1, but T2 happened to be the musical pattern with the highest ac1 coef1cient (Table 12.1), so that its r0 coef1cient was most affected by the correction that turned it into r0*. It is possible that this correction was too extreme, as it did not take into account automatic error correction in tracking. The better performance with T1 and T4 is consistent with the lower complexity of these patterns. The main effect of trial was also highly signi1cant in all analyses, F(9, 99) > 8.7, p < 0.0001, due to gradual improvement within conditions. The Pattern × Trial interaction reached signi1cance in the music condition, F(27, 297) = 1.6, p < 0.03, and in the imagery condition, F(27, 297) = 1.9, p < 0.008, but not in the click condition, F(27, 297) = 0.6. These interactions are dif1cult to interpret,

253


254


Fig. 12.3 Prediction indices (r0*), averaged across trials within conditions, as a function of condition in Experiment 1. (a) Click and music conditions. (b) Click and imagery conditions. however. More rapid improvement for the more typical patterns, as hypothesized in the Introduction, was not evident. Rather, all patterns seemed to improve at about the same rate. The Condition × Trial interaction was signi1cant when comparing the music and imagery conditions, F(9, 99) = 3.0, p < 0.004, and also for clicks vs. music, F(9, 99) = 2.2, p < 0.04, but not for clicks vs. imagery, F(9, 99) = 1.7, p < 0.10. These interactions were due to somewhat greater improvement for all patterns within the music condition than within the other two conditions. The main effect of condition was highly signi1cant in the two ANOVAs involving the click condition, F(1, 11) > 43, p < 0.0001, but not in the music vs. imagery comparison. Performance in these latter two conditions did not differ, but was substantially better than in the click condition. This suggests that pattern prediction was improved by both the presence and imagery of music, but the improvement could also have been due to general pattern learning, as observed within conditions. Therefore, the Pattern × Condition interaction was the crucial statistic. That interaction was signi1cant in all three ANOVAs: clicks vs. music, F(3, 33) = 3.2, p < 0.04; clicks vs. imagery, F(3, 33) = 4.1, p < 0.02; and music vs. imagery, F(3, 33) = 4.9, p < 0.007. The corresponding data, averaged over trials, are shown in Fig. 12.3, which focuses on the two interactions of primary interest. Individual comparisons of each musical pattern with R1 in the clicks vs. music analysis (Fig. 12.3(a)) con1rmed that the presence of music selectively improved prediction performance for T1, F(1, 11) = 6.6, p < 0.03, and for T4, F(1, 11) = 12.3, p < 0.005, but not signi1cantly for T2, F(1, 11) = 3.6, p < 0.09. An interesting aspect of these data is that the selective advantage for the musical patterns was already present in the 1rst trial of the music condition (see Fig. 12.2). In the comparison of the click and imagery conditions (Fig. 12.3(b)), a selective facilitation relative to R1 was evident only for T4, F(1, 11) = 7.5, p < 0.02. The signi1cant Pattern × Condition interaction in the music vs. imagery ANOVA was mainy due to T2, for which prediction performance was worse in the imagery than in the music condition.

12.2.2.1 Average timing pro1les Figure 12.4 shows the results for the isochronous sequence, which is represented by the horizontal dotted line (IOI = 500 ms). The ITIs are shown as data points with double standard errors (roughly, 95% con1dence intervals). In the click condition (Fig. 12.4(a)), the ITIs closely matched the



sequence IOIs from the fourth IOI on. The initial three ITIs re2ect a ‘tuning in’ to the sequence (see also Fraisse 1966; Repp 1999b; Semjen, Vorberg, and Schulze 1998): Despite the constant sequence tempo from trial to trial, the 1rst tap tended to occur too late, so that the following ITIs had to be shortened to achieve synchrony; however, there were also substantial individual differences in that respect, as re2ected in the large standard errors. The pattern of tap timing from the fourth ITI on did not show any signi1cant deviation from uniformity in a one-way ANOVA with the independent variable of position (33), F(32, 352) = 1.1. By contrast, the tap timing pro1le in the music condition (Fig. 12.4(b)) did show signi1cant variation from the fourth ITI on, F(32, 352) = 14.7, p < 0.0001, and also showed a different pattern of the initial three ITIs. Moreover, the pattern of systematic deviations from regularity was quite similar to that obtained in a previous study (Repp 1999b: Exp. 3) with the

Fig. 12.4 Average inter-tap interval (ITI) pro1les (with double standard errors) in the three conditions of Experiment 1. (a) Click condition. (b) Music condition. (c) Imagery condition.

255


256


same musical excerpt, but without superimposed clicks: the correlation was 0.76 (p < 0.001), or 0.86 if the initial three ITIs are included. Thus the earlier results for synchronization with music alone were replicated, even though the present task required synchronization with clicks that were merely accompanied by music. It appears that the effect of the musical structure on tap timing is unavoidable (see also Repp 1998b: Exp. 3). The most interesting and novel 1nding, however, is that this systematic tap timing pattern persisted in attenuated form in the imagery condition (Fig. 12.4(c)). Here there was again a signi1cant deviation from uniformity from the fourth ITI on, F(32, 352) = 9.2, p < 0.0001, and the pattern correlated 0.84 with that in the music condition (Fig. 12.4(b)), or 0.91 if the initial three ITIs are included. Thus musical imagery had a signi1cant effect on motor timing in synchronization with a perfectly isochronous click sequence.5

12.2.2.2 Summary In terms of the 1ve hypotheses outlined in the Introduction, the results may be summarized as follows. The 1rst hypothesis was that more typical musical timing patterns would be synchronized with more accurately than less typical timing patterns when music is actually present. The predicted rank order of T1 > T2 > T4 was only partially con1rmed, due to an unexpectedly (perhaps artifactually) low prediction index for T2. The second hypothesis was that all three musical patterns would be synchronized with more accurately than the R1 pattern when music was present. This was true for T1 and T4 but not for T2, for the same reason as before. The third hypothesis was that the 1rst two predictions would also hold in the imagery condition, though perhaps less clearly. Indeed, the results in the imagery condition were similar to those in the music condition, only less pronounced. The fourth hypothesis was that there would be signi1cant differences among the patterns already in the click condition, due to differences in pattern complexity. Signi1cant differences were indeed obtained, but they did not reflect differences in pattern complexity in a straightforward way. Consideration of these differences led to quali1ed predictions with respect to the 1rst three hypotheses. One prediction was that synchronization with typical musical patterns should be selectively facilitated compared to less typical patterns in the music and imagery conditions. This prediction received little support. The second and most important prediction was that, in comparing the click and music conditions, synchronization with musical patterns should be selectively facilitated compared with the random pattern in the music condition. This prediction received substantial support. The third hypothesis, that the same would be true in the comparison of the click and imagery conditions, received only weak support. Finally, the 1fth hypothesis, that pattern learning would be faster for musical than for random patterns when music is present or imagined, was not supported. Instead, it appeared that music facilitated the learning of all patterns to some extent.

12.3 Experiment 2 Experiment 1 provided reasonable evidence that synchronization with complex timing patterns derived from music performance is facilitated when the appropriate music is heard or imagined, relative to a condition in which the music is neither heard nor imagined. The results were not as strong as expected, however, and this may be attributed to a methodological weakness having to do with the R1 pattern. In hindsight, it was not a good idea to employ only a single random pattern for comparison; it would have been better to use a different random pattern for each participant. Accidentally, R1 had some features in common with T1, namely long IOIs at the ends of several melodic groups



(see Fig. 12.1). Thus, this pattern was not as inappropriate to the music as it could have been and may actually have received some slight facilitation from the musical context. Experiment 2 took a different approach. Instead of constructing arbitrary timing patterns for comparison with the musical patterns, a phase-shifted version of each musical timing pattern (a method employed previously by Clarke 1993b, in an imitation study) was constructed to serve as its speci1c comparison. Without a musical context, the phase shift had little signi1cance, but once the music was present or imagined, the original patterns were properly aligned with the musical structure whereas the phase-shifted patterns were not. Thus the prediction was that synchronization with each musical pattern would be selectively facilitated relative to its phase-shifted version in both the music and imagery conditions, but not in the click condition. Indeed, it was considered possible that musical context would even impair synchronization with phase-shifted patterns, relative to the click condition. Experiment 1 provided only limited support for the hypothesis that the degree of facilitation of synchronization with musical patterns in musical contexts would be positively related to the typicality of these patterns in music performance, in the form of an advantage of T1 over T2 and T4. However, the experiment did not include the most typical musical timing pattern, T0, for which the greatest amount of facilitation should be expected. Experiment 2 included this pattern as well, at the risk of some carry-over of learning between it and the fairly similar T1 pattern (see Fig. 12.1). Another methodological change concerned the arrangement of the three experimental conditions. In Experiment 1, all timing patterns were presented in one condition before being presented in the next one. The main advantage of this design was that participants did not hear the music until after the click condition. A possible disadvantage was the temporal separation of the music and imagery conditions, which may have weakened the strength of the musical imagery. In Experiment 2, the design was blocked by timing pattern instead. For each timing pattern, an unbroken series of trials was presented, in the course of which the three conditions followed each other in the same 1xed order as previously. This design had the advantage of revealing the transitions between the three conditions more clearly, but the disadvantage that participants might feel tempted to imagine the music during the click condition, despite instructions that discouraged this strategy. The new design was motivated by the intriguing observation in Experiment 1 that the selective advantage for the musical patterns seemed to be present on the very 1rst trial in the music condition. In Experiment 2, the immediacy of such contextual effects could be observed more directly, without any intervening breaks.

12.3.1 Methods 12.3.1.1 Materials The materials were the same as in Experiment 1, except for the following differences. The R1 pattern was no longer employed. Instead, there were four musical patterns (T0, T1, T2, T4) and a phase-shifted version of each (T0′, T1′, T2′, T4′). The phase-shifted patterns were obtained by moving the 1rst two IOIs (following the initial 1000-ms ‘upbeat’ IOI) to the end of the pattern. Thus the phase shift amounted to one-eighth note, or −90 degrees relative to the metrical cycle de1ned by the musical bars.6 Table 12.2 shows that the phase-shifted versions were all atypical of expressive performance, with one (T4′) actually contradicting the most typical pattern. However, the complexity and ac1 indices were only slightly affected by the IOI manipulation. When the music accompanied

257


258


Table 12.2 Typicality indices (i.e. correlations with T0), complexity indices (i.e. 1 − ac8; see text for explanation), and lag-1 autocorrelations (ac1) for the eight timing patterns used in Exp. 2 Pattern

Typicality

Complexity

ac1

T0 T0′ T1 T1′ T2 T2′ T4 T4′

1.00 –0.17 0.67 –0.03 0.46 –0.15 0.36 –0.54

0.53 0.31 0.31 0.14 0.73 0.62 0.58 0.60

0.19 0.29 0.29 0.35 0.59 0.59 0.15 0.15

the clicks, it started and stopped with the click sequence and followed the same timing pattern. An isochronous pattern was also included, mainly for practice but also to replicate the intriguing effect of musical imagery on tap timing (Fig. 12.4(c)).

12.3.1.2 Participants Twelve musically trained Yale undergraduates were paid for their participation. Nine of them were players of string instruments in the Yale Symphony Orchestra. Five of them had participated in Experiment 1, but since one year had elapsed between experiments, no carry-over of learning was expected. The remaining three participants had less advanced musical training but instead had considerable practice in synchronization tasks. 12.3.1.3 Procedure Each temporal pattern was repeated 20 times, with 3 s of silence between repetitions. Trials 1–8 constituted the click condition; the 1rst two of these trials were considered practice and were not analyzed. Trials 9–14 constituted the music condition, and trials 15–20 the imagery condition. Participants were urged not to imagine the music during the initial 8 trials; otherwise, the instructions were the same as in Experiment 1. The isochronous pattern was always presented 1rst, and the remaining 8 patterns were presented in an order that was counterbalanced across participants according to 1.5 Latin squares, constructed so that original and phase-shifted patterns alternated and the other three patterns intervened between the original and phase-shifted versions of the same pattern. The musical score was in view throughout the experiment. 12.3.2 Results and discussion The data were again analyzed in terms of the three indices of synchronization accuracy, but only the results for r0* are reported here. The results in terms of sda were similar, whereas r1* again yielded a less clear picture. The ANOVAs were largely analogous to those in Experiment 1 but included the variable of version (2) in addition to pattern, condition, and trial. Figure 12.5 shows that the difference between the original and phase-shifted versions of each timing pattern increased substantially in favor of the original version in the music condition relative to the click condition, and decreased again in the imagery condition. The Condition × Version



Fig. 12.5 Average prediction indices (r0*) as a function of trial number for eight timing patterns in Experiment 2. (a) T0 and T0′. (b) T1 and T1′. (c) T2 and T2′. (d) T4 and T4′.

interaction was highly signi1cant when comparing the click and music conditions, F(1, 11) = 24.0, p < 0.0006, and also when comparing the music and imagery condition, F(1, 11) = 36.6, p < 0.0002, but not when comparing the click and imagery conditions, F(1, 11) = 1.5, p < 0.25. The triple interaction with pattern was not signi1cant, indicating that the Condition × Version interaction was similar for all four patterns. The main effect of version in favor of the original patterns was signi1cant not only in the comparisons involving the music condition but also in the comparison of the click and imagery conditions, F(1, 11) = 5.0, p < 0.05; however, it did not change signi1cantly between these two conditions, nor did it interact signi1cantly with pattern in any condition. Prediction performance increased across trials in the music condition, F(5, 55) = 7.9, p < 0.0001, but not in the click and imagery conditions, where the main effect of trials was nonsigni1cant. This was also re2ected in a signi1cant Condition × Trials interaction for clicks vs. music, F(5, 55) = 2.9, p < 0.03, and for music vs. imagery, F(5, 55) = 5.8, p < 0.0003, but not for clicks vs. imagery. In the music condition, there was also a Pattern × Trials interaction, F(15, 165) = 2.4, p < 0.004. The largest improvement over trials was shown by T0/ T0′ and the smallest by T2 /T2′. Note that original and phase-shifted versions improved at the same rate; the Version × Trials interaction was nonsigni1cant.

259


260


12.3.2.1 Average timing pro1les The average ITI pro1les for the isochronous sequence were extremely similar to those of Experiment 1 (Fig. 12.4) and therefore are not shown separately. Apart from the initial three ITIs (which were omitted in the statistical analyses), there was no signi1cant deviation from uniformity in the click condition, F(32, 352) = 1.2, p < 0.23, whereas there were highly signi1cant deviations in both the music condition, F(32, 352) = 9.9, p < 0.0001, and the imagery condition, F(32, 352) = 4.8, p < 0.0001. The pattern of the deviations was highly similar in these two conditions, r(31) = 0.81, p < 0.0001, although it was less pronounced in the imagery condition, and the ITI pro1les also correlated highly with those obtained in Experiment 1, r(31) = 0.93 and 0.80, p < 0.0001, for the music and imagery conditions, respectively. These results indicate that participants were imagining the music correctly in the imagery condition, even though they had heard it only 6 times at that point in the experiment. One curious result worth noting is that the (nonsigni1cant) pattern of deviations from regularity in the click condition exhibited some resemblance to the (signi1cant) patterns obtained in the music and imagery conditions, r(31) = 0.52 and 0.50, p < 0.01, respectively. This had not been the case in Experiment 1. In neither experiment had the participants yet heard the music. However, the participants in Experiment 2 knew which music they would be hearing subsequently and had the musical score in front of them. Thus it is possible that some participants imagined the music spontaneously from the notation (cf. Brodsky, Henik, Rubinstein, and Zorman 1998), especially since emphatic instructions not to imagine the music during the click condition were given only after the isochronous practice sequence.7 12.3.2.2 Summary The results of Experiment 2 largely con1rm those of Experiment 1. The 1rst hypothesis, predicting that more typical original timing patterns would be synchronized with more accurately than less typical original timing patterns when music was actually present, received some support in that performance for T0 and T1 was better than for T2 and T4. The second hypothesis, that all original patterns would be synchronized with more accurately than their phase-shifted versions when music was present, was strongly supported. The third hypothesis, that the 1rst two predictions would also hold in the imagery condition, was supported in the case of the second prediction only. The fourth hypothesis, that there would be signi1cant differences among the patterns in the click condition, was supported, but not for the reason originally envisioned: differences in pattern complexity did not seem to play an important role. The quali1ed predictions of the 1rst two hypotheses, which take differences among patterns in the click condition into account, were strongly con1rmed in that more typical patterns bene1ted more from musical context than less typical (original) patterns, and especially in that original patterns bene1ted more than phase-shifted patterns. However, the quali1ed third hypothesis, concerning imagery, was not supported by the results of Experiment 2, which did not demonstrate a selective bene1t of musical imagery for musically appropriate patterns. The 1fth hypothesis, that pattern learning would be faster for original than for phase-shifted patterns when music is present or imagined, was not supported. Instead, in agreement with Experiment 1, the results suggested that audible music facilitates the learning of timing patterns regardless of their appropriateness.

12.4 General discussion The present study investigated the ability of musically trained participants to synchronize a simple motor activity with complex timing patterns derived from expressive timing in music performance.



These patterns were of a kind not previously investigated in pattern learning or synchronization tasks: they were neither isochronous nor rhythmic nor random (except in one case), but are best described as semi-regular or quasi-periodic in various degrees. Their regularities derived from their original association with a musical structure. In the click condition, especially in Experiment 1 (where the participants were unaware that music would be introduced later) but also in Experiment 2 (to the extent that the participants followed instructions to refrain from musical imagery), the question of interest was whether the regularities inherent in the timing patterns would help participants to learn and predict the timing variations to some extent. The participants’ success in this task was limited, which is not surprising in view of the small number of trials (10 in Exp. 1, 8 in Exp. 2). Their synchronization performance was characterized primarily by tracking, which, as Michon (1967) and others have shown, is the characteristic response to unpredictable temporal patterns in a synchronization task. Only in Experiment 1 was there evidence for improvement across trials within the click condition. However, this improvement did not differ among timing patterns, which suggests a general learning effect that was independent of pattern structure. Nevertheless, there were signi1cant differences among patterns from the very beginning in the click condition. For example, the T1 and T4 patterns exhibited larger r0* indices than T2 in Experiment 1 (Fig. 12.2(a)), and T4 was more predictable than all other patterns in Experiment 2 (Fig. 12.5). The reasons for these differences are not well understood at present. Differences in pattern complexity, de1ned here as the degree of periodicity, did not seem to be the only cause. The music condition was the primary focus of interest here. The main hypothesis was that complex timing patterns derived from expressive music performance, which are quite meaningless when carried by a click sequence, would suddenly gain meaning and structural support when they are appropriately instantiated by the accompanying music, and that this would automatically facilitate pattern prediction in synchronization. Synchronization with random or phase-shifted timing patterns, by contrast, was not expected to bene1t from the musical context. These predictions received strong con1rmation in both experiments. In Experiment 2, all four original patterns were shown to bene1t much more from the musical context than their phase-shifted versions. In fact, three of the phaseshifted patterns seemed to suffer interference from the music, at least on the 1rst music trial (see Fig. 12.5). These effects evidently derive from the relative compatibility of the timing patterns with the musical structure, particularly with the rhythmic grouping in the melody (cf. Clarke 1985, 1993b). Auditory perception of musical structure primes certain action patterns that are expressive of that structure, and timing is the most important characteristic of these action patterns. Shin and Ivry (1999), in a recent study, proposed a similar explanation for their 1nding that incidental learning of arbitrary temporal patterns occurred only when these patterns were systematically associated with a constant action pattern, in their case spatial hand movements in response to visual stimuli. (One of their manipulations also involved a phase shift of the temporal pattern relative to the spatial pattern.) Timing is a property of actions or events. In the case of music, appropriate actions are implied by the sound structure which de1nes compatibility with regard to timing. Previous research has demonstrated that the most typical timing pattern, T0, has a privileged relation to the musical structure: it is representative of many individual performances (Repp 1998a); it is observed when pianists try to play in strict time (Repp 1999a,c); it is aesthetically appealing (Repp 1997); it biases imitation of expressive timing (Repp 2000b); and perception of timing in music exhibits strong biases whose pattern closely resembles T0 (Repp 1998b,c,d) and which have

261


262


been attributed to basic auditory and/or motor grouping processes (Penel and Drake 1998; Repp 1998d). Therefore, it was of special interest to see whether timing patterns other than T0 would be selectively facilitated by the music in the present task. There was clear evidence for facilitation of T1 in both experiments, but that pattern is moderately correlated with T0 and hence fairly typical as well. By contrast, the T2 and T4 patterns are of low typicality, although they do resemble the expressive timing patterns of some outstanding pianists. Nevertheless, selective facilitation of these patterns by the music did occur. This seems to be the 1rst demonstration, other than by the original pianists’ performances themselves, that radically different timing patterns can be compatible with the same musical structure, as hypothesized by Repp (1998a). The differences between the original timing patterns and their phase-shifted versions can also be viewed in terms of relative typicality. The seemingly equal bene1t bestowed by musical context on the four original patterns relative to their phase-shifted versions may be a consequence of the fact that the lower relative typicality of the phase-shifted versions varied in parallel with the higher relative typicality of the original patterns (see Table 12.2). An interesting and somewhat unexpected 1nding was that synchronization performance improved more during the music condition than during the preceding click condition or the following imagery condition, and that this improvement occurred regardless of pattern typicality. It appears that the musical context provided a structural framework that facilitated pattern learning, regardless of the appropriateness of the pattern. In other words, the temporal pattern could be ‘pegged to’ the musical structure, which served as a memory aid. This process presumably also accounts for musicians’ ability to reproduce structurally inappropriate timing patterns reasonably well in an imitation task (Clarke 1993b; Repp 2000b). The present study also addressed the question of whether musical imagery can have behavioral effects similar to those of music actually heard. In the present context, musical imagery refers to the generation of auditory and/or motor images from a memory representation of recently heard music. Basically, this amounts to an ‘inner singing’ of the melody, perhaps with accompaniment notes 1lled in where there are no melody note onsets. How vivid or detailed the participants’ imagery was is not known. What is clear from the results, however, is that imagery was not an effective substitute for hearing the music. Evidence for a bene1t due to imagined music was weak in Experiment 1 and effectively absent in Experiment 2. This is a somewhat disappointing result, but it may simply indicate that the participants’ imagery was not strong enough. In Repp’s (1999a) study, skilled pianists who had played the Chopin Etude excerpt earlier in the same experimental session were capable of equally accurate synchronization performance in music and imagery conditions; their synchronization performance was also much better overall as that of the present participants. Thus, more experienced or more practiced individuals may well show a clearer bene1t of musical imagery. In Experiment 2, a tendency of some participants to imagine the music during the click condition may have worked against 1nding a relative bene1t in the imagery condition. Indeed, 45% of the trials in the click condition did not exhibit an extra tap at the end, which indicates that the end of the sequence had not come as a surprise. However, strategies other than outright imagery (e.g. counting, grouping, or memory for local temporal pattern features near the end) could also have been responsible. By contrast, 96% of the imagery trials ended without an extra (or missing) tap, which suggests that the music was imagined correctly, though perhaps not vividly enough, in synchrony with the clicks. Evidence that musical imagery occurred also comes from the results with isochronous sequences. Here imagery induced systematic deviations from regularity in the 1nger taps, similar to those that



are observed in tapping to isochronous music (Repp 1999a,b) or, as also demonstrated here, to isochronous clicks accompanied by isochronous music. The deviations induced by imagery were smaller than those induced by real music, which again shows that imagery was less effective than hearing the actual sound. However, the 1nding that musical imagery can have involuntary effects on motor timing is theoretically interesting. It suggests a close connection between musical imagery and movement timing, just as there is a close connection between music perception and movement timing. The pattern of systematic deviations from regularity in tapping may represent a combination of expressive tendencies and automatic error correction, which is required to maintain synchronization. This issue is in need of further research, however. Automatic error correction is also responsible for the tracking tendency which dominated synchronization performance, especially in Experiment 2. Tracking is the consequence of unsuccessful synchronization, where each large asynchrony is partially corrected on the next tap while simultaneously a new large asynchrony may arise from the unpredicted time of occurrence of the next tone. The underlying mechanism is likely to be phase correction (Mates 1994; Pressing 1998; Vorberg and Wing 1996), which is an obligatory process that commonly occurs without awareness (Repp 2000a). A second error-correction mechanism hypothesized to underlie synchronization performance, timekeeper period correction (Mates 1994), probably does not play any important role in tracking as long as the average tempo of the sequence is constant, as it was in the present experiments. However, the period correction mechanism may well be responsible for the prediction of a learned pattern. In other words, remembered aspects of timing patterns as well as perceived or imagined musical structure may in2uence tap timing via intentional or unintentional modulations of the timekeeper period. Period correction may in part be a top-down mechanism which mediates temporal expectations and governs intentional temporal control, whereas phase correction is largely bottom-up and input-driven (Repp, 2001). If this interpretation is correct, then the error correction mechanisms that have been identi1ed in simple synchronization tasks may have broader implications for temporal pattern learning, motor control, and perception. Indeed, Large and Jones (1999) have proposed a perceptual model of beat tracking that incorporates analogous mechanisms. The possible parallels between error correction processes in perception and production of timing warrant further study.

Acknowledgments This research was supported by NIH grant MH-51230. I am grateful to Paul Buechler and Steve Garrett for assistance. Address correspondence to Bruno H. Repp, Haskins Laboratories, 270 Crown Street, New Haven, CT 06511–6695 (e-mail: [email protected]).

Notes 1. Throughout this article, the term ‘clicks’ is used to refer to what was actually a series of very high-pitched digital piano tones (see Methods). 2. The term prediction rather than anticipation is used to avoid confusion with the anticipation tendency (i.e. taps precede sequence events) commonly observed in synchronization tasks (see, e.g. Aschersleben and Prinz 1995). As the terms are used here, prediction is pattern-speci1c whereas anticipation is not. 3. The third principal component was of little interest because it mainly consisted of a greatly lengthened IOI following the initial downbeat (cf. Fig. 12.1(a)).

263


264


4. A MAX patch is a program written in the graphic MAX programming environment. Due to a peculiarity of that software, real-time durations were 2.4% shorter than speci1ed or recorded, and than reported here. 5. Figures comparing the IOIs and average ITIs for the four modulated timing patterns may be found in an electronic appendix to this article on the author’s web page: <www.haskins.yale.edu/haskins/STAFF/repp.html> 6. A graphic example of T0 and T0′ is included in the electronic appendix. 7. For a 1gure illustrating the average ITI pro1les for the T0 and T0′ timing patterns, see the electronic appendix.

References Aschersleben, G. and Prinz, W. (1995). Synchronizing actions with events: The role of sensory information. Perception and Psychophysics, 57, 305–317. Behne, K.-E. and Wetekam, B. (1993). Musikpsychologische Interpretationsforschung: Individualität und Intention. Musikpsychologie, 10, 24–37. Brodsky, W., Henik, A., Rubinstein, B., and Zorman, M. (1998). Demonstrating inner hearing among highlytrained expert musicians. In S.W. Yi (Ed.), Proceedings of the Fifth International Conference on Music Perception and Cognition, pp. 237–242. Seoul, Korea: Western Music Research Institute. Clarke, E.F. (1985). Structure and expression in rhythmic performance. In P. Howell, I. Cross, and R. West (Eds.), Musical structure and cognition, pp. 209–236. London: Academic Press. Clarke, E.F. (1993a). Generativity, mimesis and the human body in music performance. Contemporary Music Review, 9, 207–219. Clarke, E.F. (1993b). Imitating and evaluating real and transformed musical performances. Music Perception, 10, 317–342. Clynes, M. (1983). Expressive microstructure in music, linked to living qualities. In J. Sundberg (Ed.), Studies of music performance, pp. 76–181. Stockholm: Royal Swedish Academy of Music. Drake, C. and Palmer, C. (1993). Accent structures in music performance. Music Perception, 10, 343–378. Fraisse, P. (1966). L’anticipation de stimulus rythmiques: Vitesse d’établissement et précision de la synchronisation. L’Année Psychologique, 66, 15–36. Hary, D. and Moore, G.P. (1985). Temporal tracking and synchronization strategies. Human Neurobiology, 4, 73–77. Hary, D. and Moore, G.P. (1987). Synchronizing human movement with an external clock source. Biological Cybernetics, 56, 305–311. Iyer, V.S. (1998). Microstructures of feel, macrostructures of sound: Embodied cognition in West African and African-American musics. Unpublished doctoral dissertation, University of California, Berkeley. Large, E.W. and Jones, M.R. (1999). The dynamics of attending: How we track time-varying events. Psychological Review, 106, 119–159. Mates, J. (1994). A model of synchronization of motor acts to a stimulus sequence. I. Timing and error corrections. Biological Cybernetics, 70, 463–473. Michon, J.A. (1967). Timing in temporal tracking. Assen, NL: van Gorcum. Palmer, C. (1989). Mapping musical thought to musical performance. Journal of Experimental Psychology: Human Perception and Performance, 15, 331–346. Penel, A. (2000). Variations temporelles dans l’interpretation musicale: Processus perceptifs et cognitifs. Unpublished Ph.D. dissertation, University of Paris 6. Penel, A. and Drake, C. (1998). Sources of timing variations in music performance: A psychological segmentation model. Psychological Research, 61, 12–32. Penel, A. and Drake, C. (1999). Seeking ‘one’ explanation for expressive timing. In S.W. Yi (Ed.), Music, mind, and science, pp. 271–297. Seoul, Korea: Seoul National University Press. Pierce, A. and Pierce, R. (1989). Expressive movement: Posture and action in daily life, sports, and the performing arts. New York: Plenum Press. Pressing, J. (1998). Error correction processes in temporal pattern production. Journal of Mathematical Psychology, 42, 63–101. Repp, B.H. (1992a). Diversity and commonality in music performance: An analysis of timing microstructure in Schumann’s ‘Träumerei’. Journal of the Acoustical Society of America, 92, 2546–2568.



Repp, B.H. (1992b). Probing the cognitive representation of musical time: Structural constraints on the perception of timing perturbations. Cognition, 44, 241–281. Repp, B.H. (1993). Music as motion: A synopsis of Alexander Truslit’s ‘Gestaltung und Bewegung in der Musik’ (1938). Psychology of Music, 21, 48–72. Repp, B.H. (1997). The aesthetic quality of a quantitatively average music performance: Two preliminary experiments. Music Perception, 14, 419–444. Repp, B.H. (1998a). A microcosm of musical expression: I. Quantitative analysis of pianists’ timing in the initial measures of Chopin’s Etude in E major. Journal of the Acoustical Society of America, 104, 1085–1100. Repp, B.H. (1998b). Obligatory ‘expectations’ of expressive timing induced by perception of musical structure. Psychological Research, 61, 33–43. Repp, B.H. (1998c). The detectability of local deviations from a typical expressive timing pattern. Music Perception, 15, 265–290. Repp, B.H. (1998d). Variations on a theme by Chopin: Relations between perception and production of deviations from isochrony in music. Journal of Experimental Psychology: Human Perception and Performance, 24, 791–811. Repp, B.H. (1999a). Control of expressive and metronomic timing in pianists. Journal of Motor Behavior, 31, 145–164. Repp, B.H. (1999b). Detecting deviations from metronomic timing in music: Effects of perceptual structure on the mental timekeeper. Perception and Psychophysics, 61, 529–548. Repp, B.H. (1999c). Relationships between performance timing, perception of timing perturbations, and perceptual–motor synchronization in two Chopin preludes. Australian Journal of Psychology, 51, 188–203. Repp, B.H. (2000a). Compensation for subliminal timing perturbations in perceptual–motor synchronization. Psychological Research, 63, 106–128. Repp, B.H. (2000b). Pattern typicality and dimensional interactions in pianists’ imitation of expressive timing and dynamics. Music Perception, 18, 173–211. Repp, B.H. (2001). Processes underlying adaptation to tempo changes in sensorimotor synchronization. Human Movement Science, 20, 277–312. Schulze, H.-H. (1992). The error correction model for the tracking of a random metronome: Statistical properties and an empirical test. In F. Macar, V. Pouthas, and W.J. Friedman (Eds.), Time, action, and cognition, pp. 275–286. Dordrecht, The Netherlands: Kluwer. Semjen, A., Vorberg, D., and Schulze, H.-H. (1998). Getting synchronized with the metronome: Comparisons between phase and period correction. Psychological Research, 61, 44–55. Shin, J.C. and Ivry, R.B. (1999). Concurrent temporal and spatial learning in a serial reaction time task. Poster presented at the Annual Meeting of the Psychonomic Society, Los Angeles, CA. Shove, P. and Repp, B.H. (1995). Musical motion and performance: Theoretical and empirical perspectives. In J. Rink (Ed.), The practice of performance, pp. 55–83. Cambridge, UK: Cambridge University Press. Thaut, M.H., Tian, B., and Azimi-Sadjadi, M.R. (1998). Rhythmic 1nger tapping to cosine-wave modulated metronome sequences: Evidence of subliminal entrainment. Human Movement Science, 17, 839–863. Viviani, P. and Stucchi, N. (1992). Motor–perceptual interactions. In G.E. Stelmach and J. Requin (Eds.), Tutorials in motor behavior II, pp. 229–248. Amsterdam: Elsevier. Vorberg, D. and Wing, A. (1996). Modeling variability and dependence in timing. In H. Heuer and S. W. Keele (Eds.), Handbook of perception and action, Vol. 2, pp. 181–262. London: Academic Press.

265


13 Action, binding, and awareness Patrick Haggard, Gisa Aschersleben, Jörg Gehrke, and Wolfgang Prinz Abstract. This chapter focuses on the process of intentional action, on our conscious awareness of some events occurring during that process and on the chronometry of these conscious states. We 1rst introduce the concept of efferent binding: a hypothesized neural process which links representations of intentions to act to representations of the action itself, and 1nally to representations of the external consequences of action. We then describe two experiments investigating the perceived times of actions and of associated stimulus events. Our results provide evidence for an efferent binding process which influences conscious awareness, and which amounts to a common principle for conscious coding of perception and action.

13.1 Introduction Intentional action is fundamental to human existence. We all believe we have the ability to do what we want. We safeguard this ability by quite careful cultural formulations within our legal system, and we consider highly disabling any pathological condition in which people cannot act to realize their wishes and desires. At these more social levels, intentional action typically implies some very complex behaviour such as publishing a newspaper, walking in the mountains, or crossing a geopolitical frontier. However, at its most basic level, the problem of intentional action can be reduced to the problem of how ‘I’ generate movements of my muscles. The key components of intentional action are as follows. First, a movement must occur. For example, simply thinking about doing something could not constitute an intentional action, whereas doing the same thing might. Second, the behaviour must be generated, in the sense that it must be produced by a goal-directed thought. There must be an identi1able link between a mental state (i.e. the intention), and the behaviour subsequently performed to achieve that goal. Third, the generation of behaviour must some how come from within me. The mental state that ultimately gives rise to the behaviour must be my mental state. This connection between intentional action and ‘I’ was expressed very de1nitively in the earliest psychological studies of volition: La conscience de l’action doit donc être considerée comme . . . la forme d’intervention du moi phénomenal dans la vie psychique. (Michotte and Prum, 1910)

Intentional actions, then, are things that I do. Thus reflex behaviours, such as the doctor produces by tapping on my tendons, are not intentional actions, because I have not caused them. We are left then with the key problem of intentional action:—how do ‘I’ manage to make my body move? This question has traditionally been the province of philosophers, and only recently has experimental evidence about neural and mental function been seen as relevant to answering it. The brain processes


Action, binding, and awareness

underlying intentional action have been reviewed elsewhere (Frith et al. 2000) and will not be discussed further here. Instead, this paper addresses a more psychological question about the relationship between intentional action and conscious awareness. As stated above, the position that the self or ‘I’ must be the generator of intentional actions immediately raises the questions of whether intentions are conscious, and of how the primary consciousness of intentions is related to the secondary self-consciousness of ‘I’. Perhaps the best-known position, and certainly the one that most clearly underlies lay belief, in Western cultures, is the Cartesian position of the conscious thinking self generating actions from her or his conscious free will. On this model, I consciously decide to do something, and that conscious state is suf1cient to lead to appropriate muscular movement to realize my goal. The key role of consciousness in the question of intentional action arises through the necessary involvement of the self or ‘I’. The portion of the chain leading from intention to action could in principle operate without any of the participating representations reaching conscious awareness, in the sense of being accessible to verbal report. Indeed, this appears to be the case in Alien or Anarchic Hand Syndrome. Della Sala, Marchetti, and Spinnler (1991) described a patient who exhibited a right anarchic hand following a combined callosal and mesial frontal lesion. On being given a cup of hot tea, the patient announced that she would not drink it yet, but would wait for it to cool. Nevertheless, the right anarchic hand reached out for the hot tea in an apparently well-formed goal-directed action, which the patient had to resist by restraining her right hand with her left. In this case, it seems that the action of the right hand may well be intentional, because the right hand’s movement was caused by a mental representation of the action to pick up the cup. On the other hand, the intention was not conscious: the patient’s decision and verbal report were not to pick up the cup. It seems possible then, at least in pathology, that unconscious intentions exist. Della Sala et al.’s case demonstrates the possibility of unconscious intention, but does not meet the full conditions for intentional action described above. This is because the unconscious intention that moves the right hand does not belong to the patient’s ‘I’. The right hand’s action is owned, but the source of the action (the intention) is not owned by the ‘I’. (We are grateful to Tony Marcel for this form of words.) That is why this particular movement has such a strange phenomenology for the patient. These cases are, of course, the exception. Intentions need not always be conscious, but normal intentional action requires both 1rst-order consciousness (consciousness of intention) and second-order consciousness (self-consciousness, ‘I’). The dominant neuroscienti1c account of consciousness in intentional action is the central monitoring account of Frith (1992). This account neatly links 1rst and second-order consciousness through the process of attribution. When an event occurs in the environment, it is important to know whether I caused it, or whether it is an external event. By monitoring my intentions, and comparing the predicted consequences of my intentions to the stream of perceptual events, I can distinguish internally generated events from external events. The central monitoring process therefore plays a crucial role in separating perceptual awareness from the self-conscious states surrounding willed action.

13.2 The contribution of Libet Given that intentional actions can involve both conscious and nonconscious states, we can ask: what does consciousness add to them? Any discussion of this question must begin with the work of Libet. The most influential study in this area has been that of Libet, Gleason, Wright, and Pearl (1983). Their study combined the electrophysiological measurement of neural precursors of intentional

267


268


Fig. 13.1 Set-up for a typical experiment based on the method of Libet et al. (1983). Figure taken from Haggard and Eimer (1999).

action with a psychophysical method for measuring subjects’ conscious awareness of intention. A schematic of the arrangement used in our replication of Libet et al.’s study (Haggard and Eimer 1999) is shown in Fig. 13.1, but the description below refers to the original experiment of Libet et al. Briefly, subjects viewed a slowly rotating clock hand on a screen. The clock hand rotated once every 2560 ms. Subjects were instructed to make ‘freely voluntary acts’, that is, discrete movements of the right hand, at a time of their own choosing. Some random interval after the subject performed each intentional action, the clock hand ceased to rotate, and the subject was asked to report the position of the clock hand at which they had 1rst become consciously aware of the intention to produce the action. At the same time, Libet and colleagues measured the readiness potential over the motor cortical areas as in index of neural initiation of these intentional actions. The experimenter can then compare the subjective time of events (based on reports of clock positions) with the objective times of the corresponding events (based on recordings of EEG and muscle activity) to calculate a judgement error. A number of methodological points require special mention. First, since the intertrial variability of such judgements is high, a mean of several trials is typically used. Second, the subjects are given no feedback about their judgement error at any point in the task, since to do so would presumably rapidly reduce judgement errors to zero. Third, and most importantly, Libet and colleagues asked their subjects to use the rotating clock as an external metric for judging the time of their conscious intentions without letting the clock itself control the generation of their actions. Libet et al.’s results describe the temporal relations between preparation of intentional action, and conscious awareness of intentions. Briefly, readiness potential recordings showed that subjects began the neural preparation of their action at least 700 ms before the 1rst onset of muscle activity. In contrast, subjects’ verbal reports using the clock hand position suggested that they became aware of their



intentions to act only some 200 ms before the muscle became active. Therefore, during a period of at least 500 ms, the brain is processing the generation of an action of which ‘I’ am as yet unaware. Since backwards causation is impossible, it follows that neither ‘I’ nor my conscious intentions can be the cause of my actions. For Libet, the causation is in fact the other way round: my readiness potential causes my conscious intention (but see Haggard and Eimer 1999, for a revision of this view). Libet et al.’s results appear to represent a major dif1culty for the traditional Cartesian concept of conscious free will, and pose a real problem for the generative role of consciousness in intentional action. While Libet et al.’s result has been criticized on several counts (see the replies to Libet’s (1985) target article in Behavioural and Brain Sciences for a selection, including Breitmeyer 1985; Bridgeman 1985; Rugg 1985) the basic result showing the temporal order of neural and conscious events appears to hold (Haggard and Eimer 1999).

13.3 Generative versus constructive accounts of conscious intention What then is the role of consciousness in the process of generating action? We suggest that consciousness has at least two qualitatively distinct contributions to the process of intentional action. The 1rst, which has dominated almost all previous enquiries since Descartes, is the generative process whereby conscious states activate the sequence of events that ultimately generates muscular contraction. Libet et al.’s account, for example, can be seen as an experimental disproof of this generative role of consciousness. This problem is essentially a philosophical one, and it is not clear how a reductive neuroscientist (i.e. someone who ultimately believes that mental states are brain states) can usefully engage with the question. The second role that consciousness may play in intentional action has received far less scienti1c attention. We will call it the constructive role of consciousness in intentional action. On the constructive view, consciousness provides the background set of conditions against which intentional action can take place. Without the experience of conscious intention, ‘I’ would never acquire the concept of intentional action. Therefore, although consciousness would not be necessary to make any particular intention give rise to its associative action, consciousness may be necessary for us to acquire and retain the ability to make the process of intentional action work. In particular, consciousness may be crucially related to a sense of agency: if we did not formulate conscious intentions, and did not consciously represent the actions that those intentions produce, and the consequences of those actions for ourselves, then we would not produce any intentional actions. Conscious awareness of our intentions, and of the bodily and environmental consequences they cause, is required to construct the possibility of intentional action. The constructive view of consciousness implies a difference between goal-directed behaviours and true intentional actions on the basis of this background mental history. A nonconscious organism may be capable of goal-directed action, but only if the organism produced the behaviour from a mental background of conscious representation of its own internal and external states would we admit that the goal-directed behaviour also satis1ed the criteria for an intentional action. In summary, the generative and constructive views of conscious intention differ in the roles of consciousness that they emphasize. The generative role emphasizes how conscious states can have causal power over the material body, while the constructive role emphasizes how conscious representations contribute

269


270


to the sense of ‘I’ as an agent. This sense of agency is in turn required for us to bother making any intentional actions in the 1rst place.

13.4 Efferent binding How then does consciousness construct this relation between intention and action which is required to make agency possible? One simple possibility is that consciousness participates in a process which we will call efferent binding. This is a hypothetical brain process akin to the perceptual binding process that occurs in visual object perception (Engel et al. 1999). Efferent binding would associate intentions with representations of the actions that they produce, and with perceptual representations of the environmental consequences of these actions. Efferent binding is the way that we learn the relationship between our intentions and their results. If the intention does not produce the desired result, then the relationship between intention and action needs adjusted on future attempts. Motor learning therefore requires the efferent binding process. Efferent binding between intention, action, and consequence can take place entirely unconsciously. We have already seen that an unconscious intention may exist. Similarly, actions, in the sense of body movement, and sensory representation of their consequences can certainly occur without conscious awareness. An unconscious process such as Pavlovian conditioning could produce an association between these representations. Therefore efferent binding, in the sense of some association between representations of actions and their consequences, might occur without any conscious awareness either of the binding process or of any of the bound events. We suggest that efferent binding has both nonconscious and conscious elements. Nonconscious association between action and consequence representations occurs in many learning and performance situations. Several human behavioural studies show that representations of actions and of effects are integrated (for a selection, see Stoet and Hommel 2001; Ziessler and Nattkemper 2001; Hazeltine 2001; Elsner and Hommel, in press). Further, operant learning in animals constitutes a whole 1eld of psychology in which animals associate effects with their own actions (Dickinson 1980). At a more physiological level, the widely accepted reafference principle of action control (von Holst and Mittelstaedt 1950) requires matching afferent information about the effects of action with efferent information about motor commands. However, we here focus speci1cally on the conscious consequences of the hypothesized efferent binding process. In many circumstances the binding between action and effect seems to be highly relevant to consciousness. First, we are often conscious of events when efferent binding fails: the moment we realize that we meant to press one button, but actually pressed another can be phenomenologically very vivid. Moreover we are often conscious of intentional action, and external consequence representations which are successfully bound. An example occurs in motor learning. Sometimes we suddenly ‘feel’ the relation between an action and an effect, for example when learning to use the clutch in a new car. Consciousness and efferent binding are therefore intimately related, even if the former is not necessary for the latter. This chapter concerns solely the conscious aspects of the binding process: thus from now on we will use the term efferent binding to refer solely to the associations between conscious representations of actions and consequences. We believe that a principled account of when associations between actions and effects influence awareness, and when they do not is a priority for future research, as this important question has generally been neglected in associationist psychology.



In addition, the efferent binding process is important in making sense of our relation to the world around us. Speci1cally, efferent binding is required to answer the attribution question ‘did I do that?’ As outlined above, when an event occurs in the outside world, it is important to distinguish between whether it is a consequence of my own action, or an unexpected external event which I need to process and possibly to respond to. That is, the mind must be able to attribute actions to agents, and particularly to the self. Most previous research in this area has focused on the concept of and internal forward model (Kawato and Wolpert 1998), which predicts the consequences of our intentions, and attempts to cancel these predicted consequences against perceptions of actual external events. The best-known example is the classic outflow theory of the stability of the visual world (Sherrington 1898). In this, the visual world remains stable even when we move our eyes, because the actual retinal slips induced by eye movement can be cancelled against the retinal slips predicted from the motor command or intention to produce a speci1c eye movement. More recently, Blakemore et al. (1999) have shown a similar efferent binding process to be important in cutaneous sensation. They have argued that one cannot tickle oneself because the proprioceptive afference is cancelled by the predicted sensory consequences of the efference associated with the tickling movement. When somebody else tickles one, in contrast, the proprioceptive input is not cancelled in this way. The above example is important because it demonstrates that the efferent binding process has important implications for our conscious awareness of physical stimuli. We know from everyday experience that the phenomenology of tickling is very strong indeed. Therefore, the success or otherwise of efferent binding may be very important for the conscious awareness we have of our own actions and of external events. More speculatively, it seems likely that efferent binding may play a key role in constructing self-consciousness. To represent ‘I’ as a conscious agent may depend on binding my conscious intentions, to my actions and to their effects. There is a clear relation between the efferent binding process and Frith’s central monitor (see above, and Frith 1992). We suggest that efferent binding is a speci1c mental process with conscious consequences that occurs when the monitor detects a match between an external event and an intention. The two representations are then bound together, and tagged as an intentional action belonging to ‘I’.

13.5 Experimental measures of efferent binding The conscious aspects of efferent binding have rarely been studied experimentally. In this paper we use the perceived time of events as a useful experimental method for studying binding of conscious representations. Speci1cally, where the representation of an intention, of an action, and of the environmental sensory consequences of the action are bound together, we would expect the perceived times of those three separate events to be attracted towards each other. Several strands of logic underlie this claim. First, as Hume (1750) originally demonstrated, association of percepts across time is a crucial element in providing the idea of causation in general, and of free will in particular. Perceptual attraction seems a natural consequence of such association: certainly it is more so than perceptual repulsion. Second, just as reductions in reaction time with learning are often attributed to strengthening of associations between stimulus and response representations, by the same argument reductions in the interval between percepts is evidence for association between the conscious elements of those representations. Third, perceptual attraction has some clear similarities to learning to predict. When an animal learns an assocation between two events, the response to the unconditioned stimulus is temporally shifted back towards the predictive conditioned stimulus (e.g. Yeo, Lobo, and Baum 1997). We suggest that temporal shifts in percepts may also reflect association or binding between conscious states. Fourth,

271


272


perceptual attraction 1ts well with the function of efferent binding of matching conscious states across time into a coherent representation of intentional action. While perceptual attraction does not imply complete fusion of the several percepts involved, it would help to provide a uni1ed conscious experience of intentional action. Such a uni1ed experience would be required to develop a coherent sense of agency. Finally, we add that temporal attraction effects may be merely one class of evidence for efferent binding. Other classes of evidence may exist. Investigation of attraction between spatial representations in willed action would be a particularly fruitful area for future research. However, any such other evidence would be characterized by attraction effects rather than repulsion effects. On the other hand, temporal attraction effects between percepts are highly consistent with the function of consciousness of unifying experience across space and time (Kant 1781, 1963). We can therefore compare the perceived times of events which occur within an intentional action context with the perceived time of the same events occurring alone. If binding occurs, the bound representations should be attracted towards each other. Attraction effects on perceived time of occurrence of events are thus a useful experimental sign of the efferent binding process. Note that efferent binding does not imply that all of the bound components (intention, action, and consequence) come to form a single indivisible perceptual unit. They may indeed remain perceptually quite distinct. The process of binding will, however, strengthen associations between these representation so that they will influence each other. This strengthening of association might bring the events closer together in consciousness but should not push them further apart. We now report two experiments investigating attraction effects in simple manual actions. These experiments suggest that: (1) an efferent binding process exists which integrates the conscious representations of actions and of events; (2) this efferent binding process is speci1c to actions, rather than being an instance of general perceptual attraction; (3) efferent binding forms part of a wider process integrating our actions with the causes and effects in which they occur.

13.6 Experiment 1: perceived time of stimuli and of actions Our 1rst experiment focused on attraction effects between stimuli and actions. We reasoned that a binding process should produce attraction between the perceived times of these events. We wanted to measure the direction of these attraction effects; do actions tend to be attracted towards their consequences or, alternatively, do consequences tend to be attracted towards the actions that cause them? Second, we wished to compare whether the efferent binding process was comparable between intentional actions and other forms of sensorimotor association, such as simple reactions to imperative stimuli. We therefore used the method of Libet et al. (1983) in which subjects judge the perceived time of sensory or motor events. In our experiment, subjects judged the perceived time of onset of any of 6 events according to condition. The 1rst two conditions were control conditions, in which only one event occurred per trial, and subjects judged the position of the clock at which they perceived the event onset to occur. In the 1rst condition the event was a 1 kHz pure tone of 100 ms duration. In the second condition, the event was an intentional key press, which subjects made with the index 1nger of their right hand at a time of their own free choice. The remaining conditions required them to judge the perceived time of either a similar pure tone stimulus, or of a similar keypress action, when these events occurred in a sensorimotor context. The sensorimotor contexts were of two kinds: either simple reactions, or intentional operant actions. The simple reaction time (SRT) task was studied in conditions 3 and 4. In condition 3, the 1 kHz pure tone occurred at a random latency from the start of the trial, and subjects responded to it as quickly as possible with a right manual keypress. They then judged the time



of occurrence of the pure tone. In condition 4, subjects again responded to the auditory tone, but this time they used the clock hand to judge the onset of their keypress response. The 5th and 6th conditions used an intentional operant task, again combining the same auditory stimulus and manual keypress action. In condition 5, subjects made an intentional right index 1nger keypress at a time of their own choice. This was followed after 200 ms by a 1-kHz pure tone. The choice of operant interval was made to be similar to subjects’ predicted reaction times in the SRT task. Subjects used the clock hand to judge the perceived time of the manual action. In condition 6, subjects again made intentional operant actions, which again elicited the auditory stimulus at a latency of 200 ms. In this condition, however, they used the clock hand to judge the perceived time of the auditory stimulus. In summary, the conditions differed according to the sensorimotor context in which the events occurred (single, reactive, or operant), and the event judged (stimulus or action). In each condition, the subject’s judgement allowed us to calculate a judgement error, de1ned as the difference between the actual time of occurrence of the judged event (stimulus or action, according to condition) and the perceived time of its occurrence. Following convention, a negative judgement error is used for anticipatory awareness of events (the subject thought the event happened before it really did), and a positive judgement error is used for delayed awareness (the subject thought the event happened after it really did). Table 13.1 shows the sequence of events. S denotes the stimulus, A denotes the manual action, and subscript j denotes the event judged. In other respects, the experiments were similar to others reported previously (Haggard and Eimer 1999; Haggard and Magno 1999; Haggard, Newman, and Magno 1999). Briefly, subjects sat comfortably facing a computer screen on which a small clock face was shown. The clock had a single hand which rotated at a speed of 2560 ms per revolution. The clock was marked with a conventional 5-minute visual scale, though subjects were encouraged to report intermediate values to the maximum precision possible. Subjects began each trial at a time of their own choosing by pressing a key. This began the revolution of the clock. In conditions requiring intentional action (i.e. conditions 2, 5, and 6) subjects made an intentional keypress with their right index 1nger at a time of their own choosing. Subjects were instructed not to act in a stereotyped way, to avoid acting at a 1xed latency after the start of the trial, to avoid choosing to act at speci1c predecided positions of the clock hand, and to ensure that the clock rotated at least once prior to their action. In the conditions where the auditory stimulus was the 1rst event (i.e. conditions 1, 3, and 4) the auditory stimulus occurred at a random latency from the start of each trial.

Table 13.1 Experiment 1: design and results Condition

Mean Judgement Error (ms)

1

Sj

Judge time of a beep

– 30

2

Aj

Judge time of a willed action

–9

3

SjA

Judge time of a beep to which subject responds

– 19

4

SAj

Judge time of response to a beep

– 57

5

AjS

AjS Judge time of a willed operant action

6

ASj

Judge time of beep elicited by willed operant action

1 – 71

273


274


Twenty-1ve subjects from the subject pool at the Max Planck Institute for Psychological Research participated in this experiment. Each condition was performed in a separate block of the experiment. Each subject performed the blocks in a different random order. Each block contained 70 trials. A small (never more than 6%) number of trials had to be discarded on some blocks due to technical failures or subjects’ failure to follow instructions. Inspection of the raw judgement error data showed a number of subjects who produced exceptionally high standard deviations of judgement error in most conditions. These standard deviations were as high as 1 s in some instances, implying that subjects were sometimes not even sure of which side of the clock face the clock hand was on when the judged event occurred! This group of subjects was readily identi1able both in the data table, and, more objectively, by a cluster analysis. Both methods identi1ed the same group of 7 subjects as having unusually high standard deviations. The standard deviation of several perceptual estimates is related to the perceptual saliency of the judged event. Events with a distinct and vivid phenomenology should produce low trial-to-trial variability in perceived time judgements, whereas events with indistinct phenomenology should produce larger trial-to-trial variabilities. It seems, then, that for some people the phenomenology of the sensorimotor events in this experiment was too vague for genuine judgement. Alternatively, these subjects could have been very bad at temporal judgement. We concluded that these subjects could not perform the task, and we therefore excluded their data. Importantly, this decision was made on the basis of the standard deviation of judgement errors across trials, rather than the mean judgement errors.

13.6.1 Results and discussion The mean judgement errors from the 18 remaining subjects in each condition are shown in Table 13.1. The mean estimates for the action-only and stimulus-only conditions are comparable with previous reports. In particular, actions are perceived to occur rather earlier than they actually do (cf. Libet et al. 1983; Haggard 1999). More importantly, the conjunction of stimuli with actions in the operant and reactive conditions produced clear evidence of attraction effects. In condition 4 (SAj) an action performed in response to a stimulus was perceived to occur earlier than an action which is performed by itself. This suggests that the action percept is attracted by or bound to the stimulus percept. Similarly, in condition 6 (ASj), a stimulus which follows an operant action is perceived to happen earlier than a stimulus which occurs alone. The stimulus percept appears to be attracted by the operant action which caused it. Even more importantly, these attraction effects appear to be asymmetric. In SRT condition 3 (SjA), the perceived time of the imperative stimulus is only very slightly later than the perceived time of a stimulus occurring alone. That is, the stimulus percept is only minimally attracted by the action which follows it. In operance (condition 5, AjS) the action is perceived to occur only slightly later than the same action occurring alone without a subsequent stimulus. Therefore, the action percept is only minimally attracted by the subsequent stimulus. To study these attraction effects statistically, we treated each subject’s mean estimate in the stimulus-only (Sj) and action-only (Aj) conditions as baseline control values. We assumed that these estimates would vary with the sensory transmission of each subject, with the salience of the sensory or motor phenomenology for each subject, and with the particular division of attention between clock and sensory motor event generated by each subject. Because of such factors, the speci1c numerical estimates obtained for the perceived times of events in the Libet task are not informative by themselves (Haggard and Eimer 1999). However, these sensory and attentional factors can be assumed constant across conditions within a single subject. Therefore, the difference between two



Libet estimates of the same physical event obtained under different task circumstances may be informative. We therefore calculated the change in judgement error, or perceptual shift, for stimuli and actions in the operant and reactive conditions, by subtracting the appropriate baseline values obtained from each subject in the stimulus-only and action-only conditions. That is, we subtracted each subject’s judgement error in judging the perceived time of a stimulus presented alone (condition 1: Sj) from the perceived time of the physically identical stimulus presented in a simple reaction context (condition 3: SjA), and from the perceived time of the physically identical stimulus presented in an operant context (condition 6: ASj). Likewise, we subtracted each subject’s estimate of the perceived time of an action occurring alone (condition 2: Aj) from the perceived time of a physically similar action occurring in a simple reaction context (condition 4: SAj) or in an operant context (condition 5: AjS). These perceptual shifts are shown in Fig. 13.2.

Fig. 13.2 Perceptual shifts in Experiment 1 categorized by task.

The data of Fig. 13.2 were subjected to factorial ANOVA, using two within-subject factors. These were the type of event judged (stimulus or action) and task context (reactive or operant). This analysis showed no signi1cant effect of event type (F(1, 17) = 0.174, NS), no signi1cant effect of task context (F(1, 17) = 0.208 NS), but a highly signi1cant interaction between the two factors (F(1, 17) = 11.238, p = 0.004). This interaction arose because SRT tasks produced large attraction effects on action, but small attraction effects on stimuli. Operance produced large attraction effects on stimuli and small attraction effects on actions. Put another way, the cause, or 1rst event to occur in each sequence was not substantially shifted towards the effect or second event, but the second event was substantially shifted towards the 1rst. To emphasize the importance of causal status, we factorialized the same data in another way, using a factor of cause or effect, and a second factor of judged event type. This analysis is shown in Fig. 13.3, which is informationally equivalent to Fig. 13.2.

275


276


Fig. 13.3 Perceptual shifts in Experiment 1 categorized by causal status. Data is the same as in Fig. 13.2. This analysis shows only a signi1cant main effect of cause versus effect. Therefore, we conclude that the causal status of an event is highly relevant to the time at which it is perceived to occur. A process with the characteristics of efferent binding attracts sensory and motor percepts in action contexts. The main effect of causation applied to the data in Fig. 13.3 does not strictly test the hypothesis that effects are attracted more than causes. This is because efferent binding would produce attraction effects in opposite directions for causes and for effects. In efferent binding causes should be attracted towards effects whereas effects should be attracted towards causes. To test for differences in attraction effects, rather than perceptual shifts in general, it is therefore necessary to invert the sign of the shift for the causes, to accommodate the fact that attraction effects operate in different directions on the two classes of causal event. Statistical testing of the data after this transformation showed only a main effect of causation (F(1, 17) = 4.957, p = 0.04) therefore, the stronger hypothesis, that effects are attracted more than causes, is sustained. Thus, the efferent binding process apparent in our data appears to be asymmetric. Finally, we calculated the mean reaction time in the SRT task (conditions 3 and 4). This was 208 ms. Recall that the action–stimulus interval in the operant conditions was 1xed at 200 ms in this experiment. Thus, the operant and SRT tasks had very similar temporal extents.

13.7 Experiment 2: attraction effects in causation and in sequences In our discussion of the frst experiment we suggested that attraction effects are a sign of a process of efferent binding. We also claimed that effects are attracted towards causes more than causes are attracted towards effects. However, our 1rst experiment confounded the causal relation between



stimulus and action with the temporal relation between stimulus and action. On the basis of our 1rst experiment alone, we cannot distinguish whether effects are bound to causes, whether the 1rst percept of a sequence is bound to the second. The effects seen in Experiment 1 could also re2ect recency effects in memory for event times. Experiment 2 aimed to resolve this ambiguity by extending the design or Experiment 1 to include sequences of 2 stimuli, or 2 actions. When two external stimuli occur with a 1xed interstimulus interval, there is no reason why they should be represented as causally related, and no reason why they should be subject to any efferent binding process in conscious awareness. The same applies for two independent yet successive actions. We call such arrangements sequence contexts, as opposed to causal contexts. Note that this argument does not deny the possibility of some perceptual attraction between sequences of actions. It merely denies that any attraction effect between sequences could arise from the efferent binding process. Experiment 2 therefore compared attraction effects in sequence contexts and in causal contexts. There were ten experimental conditions. These are shown in Table 13.2, using the same notation as Table 13.1. The 1rst six conditions were identical to those in Experiment 1. In conditions 7 and 8 the subject judged the onset times of the 1rst and second of two successive auditory stimuli, respectively. In conditions 8 and 9 subjects judged the time of the 1rst or second of a pair of successive keypresses, respectively. In most respects, the apparatus, design, and analysis of the data resembled that in Experiment 1, so we will only mention here the speci1c respects in which Experiment 2 differed. Experiment 2 was based on a desired interval between the two events of 250 ms, rather than the 200 ms used in Experiment 1. Thus, for example, the operant delay used in conditions 5 and 6 was 250 ms, as was the inter-onset interval between the two auditory stimuli in conditions 7 and 8. In conditions 9 and 10, we trained subjects to produce key presses separated by an interval of 250 ms by giving subjects eight practice trials before each of the appropriate blocks of the experiment. In the practice trials, subjects heard pairs of tones separated by a 250 ms interval. They were asked to tap in synchrony with these tones, and to attend precisely to the interval between them. No feedback about their performance was given. In view of the increased number of judgement conditions in Experiment 2, we presented a prompt on the screen at the end of the trial. When the Libet clock 1nally ceased to rotate at the end of each trial, a single line of text appeared on the screen asking the subject to judge the clock position at the onset of ‘the 1rst beep’, ‘the second keypress’, or whatever was appropriate for the condition. During the trials, all keypress responses were made with the right index 1nger on the F9 key of the keyboard. The number of trials performed in each condition was reduced to 40 to ensure the experiment was not excessively long. Each condition was studied in a separate block of the experiment. Sixteen subjects participated in the experiment. All were right-handed, with normal or corrected to normal vision, and with no history of neurological impairment. Most were students at University College London. Eight random orders of the ten different conditions were used, with one subject experiencing each random order in the forward direction, and a second subject experiencing the same random order in the reverse direction. The mean judgement error and its standard deviation across trials were computed as for Experiment 1. No evidence was found in this experiment of a subset of subjects with egregiously high standard deviations. The mean judgement errors for the ten conditions of Experiment 2 are shown in Table 13.2, together with the standard deviation across subjects. As in Experiment 1, we subtracted the mean judgement error for each subject in the control conditions Sj and Aj from their judgement errors in

277

Experiment 2: design and results

Condition

Task

Event judged

Event position

Context

Mean judgement error (ms)

S

1

Single

−4

Standard deviation (ms)

1

Sj

Judge time of a beep

46

2

Aj

Judge time of a willed action

A

1

Single

4

48

3

SjA

Judge time of a beep to which subject responds

S

1

Causal

25

52

4

SAj

Judge time of response to a beep

A

2

Causal

−49

124

5

AjS

Judge time of a willed operant action

A

1

Causal

23

59

6

ASj

Judge time of beep elicited by willed operant action

S

2

Causal

−46

67

7

SjS

Judge time of 1rst of two beeps

S

1

Sequential

8

54

8

SSj

Judge time of second of two beeps

S

2

Sequential

17

78

9

AjA

Judge time of 1rst of two keypress actions

A

1

Sequential

−14

53

10

AAj

Judge time of second of two keypress actions

A

2

Sequential

22

46


Table 13.2



the other conditions. The resulting data set represents the perceptual shift in the perceived time of stimulus and of action in the causal and sequential conditions. These can be arranged as a 2 by 2 by 2 factorial with the following three factors: (1) the event judged in each condition (stimulus or action); (2) the position of the judged event within the trial (1rst or second); and (3) the type of context (causal or sequential). The mapping of the various conditions onto the cells of this factorial design is given in Table 13.2. For example, the judgement of a stimulus in an SRT task, as previously performed in Experiment 1, is now represented as judgement of a stimulus in the 1rst position of a causal context. Judging the time of the second of two successive actions is now represented as judging an action in position 2 of a sequential context.

13.7.1 Results and discussion The mean perceptual shifts were subjected to a repeated measures ANOVA using the factorial arrangement described above. This revealed a nearly signi1cant main effect of context (F(1, 15) = 4.214, p = 0.058), due to perceptual shifts being more anticipatory for causal contexts (mean = – 12 ms) than for sequential contexts (mean = 8 ms). The main effect of position also showed a trend toward signi1cance (F(1, 15) =3.575, p = 0.078), due to an overall attraction effect. The 1rst event in each pairing showed a perceptual shift towards delay (mean = 10 ms) while the second event in each pairing showed a perceptual shift towards anticipation (mean = – 15 ms). These results are consistent with a weak overall tendency towards mutual attraction between the events studied. Most importantly, however, a highly signi1cant interaction between the type of context and the position of the judged event was found (F (1, 15) = 18.330, p = 0.001). This interaction can be seen clearly in Fig. 13.4, which shows that the attraction effect is marked for the causal contexts, but is absent, and indeed is replaced by repulsion, for the sequential context. Finally, this effect did not interact with the event judged. Both stimuli and actions showed attraction effects in causal contexts. Neither stimuli nor actions showed attraction effects in sequential context.

Fig. 13.4 Perceptual shifts in Experiment 2.

279


280


A planned comparison between the unsigned attraction effects for causes and effects showed the same direction as Experiment 1, but failed to achieve statistical signi1cance for Experiment 2. Thus, while Experiment 2 replicates the 1nding of signi1cant attraction effects, the stronger assertion that these apply asymmetrically to causes and effects was not supported, and requires further investigation. We performed an additional analysis of the reaction times in the SRTs (conditions 3, 4) of Experiment 2, and of the interkey press interval in the action sequences (conditions 9, 10). The mean interval was 254 ms, very close to the operant and interstimulus intervals of 250 ms, and there were no signi1cant differences in the value of this interval across conditions. Therefore differences between conditions in the perceived time of events cannot arise from differences in the actual time of events.

13.8 General discussion We begin by briefly restating our results. When a stimulus elicits an action (SRT), or when an action produces a stimulus (operance), the perceived time of the events shifts as a result of the sensorimotor context in which they occur. These shifts represent attraction effects between the percepts of stimuli and of movements. They are consistent with an efferent binding process linking conscious representations of stimulus events and of actions. The effects were comparable across stimulus and action percepts, and across SRT and operant tasks. Therefore, the underlying binding process appears to be quite general. Experiment 2 further showed that the effects occur only when stimuli and actions are linked in a causal context, and do not occur in mere repetitive sequences of stimuli and actions. Thus our effects can be attributed to a speci1c process governing conscious representation of interactions between the subject and the external world, rather than a general process of time or sequence perception. Second, we take our results as an interesting validation of the method of Libet et al. (1983). This method has previously been criticized quite harshly, as discussed in the Introduction. Our Experiment 2 replicated the effects seen in Experiment 1 in a different laboratory and a different population of subjects. Moreover, the numerical values obtained for perceptual shifts in the two experiments are quite close. We have emphasized above that single numerical estimates in the Libet task are not informative. However, the present study suggests that differences between Libet estimates may be replicable, and may provide a useful method for studying perceived time of events. The Libet method has a strong advantage over other psychophysical timing methods such as the Temporal Order Judgement (Sternberg and Knoll 1973). Temporal Order Judgement requires presenting two events, and varying the interstimulus interval to 1nd the value at which the two onsets appear simultaneous. In intentional action, the event to be judged is internally generated by the subject. Therefore, the experimenter would not reliably be able to time a reference stimulus to occur just before it (though just after is of course possible). In the Libet task, in contrast, the timing reference is constantly present, in the form of the rotating clock.

13.9 Artefactual explanations of attraction effects The Libet method has been criticized many times. In particular, it involves cross-modal matching, and the numerical estimates it yields will depend on the division of attention between the clock and



the judged event, due to the prior entry phenomenon (Sternberg and Knoll 1973). These criticisms have been considered in detail elsewhere (Haggard 1999). Such general criticisms are mitigated by using differences between subjects’ estimates across conditions, as in the present experiment, and are not discussed further here. We now deal with speci1c criticisms of the method as applied to the context conditions studied here. Experiment 2 showed that attraction effects exist between stimuli and actions in causal contexts, such as SRT and operance, but are absent or replaced by repulsion when stimuli or actions merely form a sequence. This contrast rules out several possible artefactual interpretations of our results. First, our attraction effects cannot merely be examples of a general process of perceptual con2ation or perceptual centres. Morton, Marcus, and Frankish (1976) coined the term ‘perceptual centre’ or ‘P-centre’ to refer to the fact that the perceived onset of a speech sound is typically rather later than its physical onset. This suggests that judgements about discrete properties of a temporally extended stimulus are made with reference to an abstract point equivalent to the stimulus’ centre of perceptual gravity, at least in the case of speech stimuli. A P-centre hypothesis would predict equal attraction in sequence contexts, yet this was not observed in Experiment 2. Second, comparison with the sequence conditions shows that attraction effects are not simply due to predictability or expectancy. In the SS sequence conditions, for example, the 1rst beep perfectly predicted the occurrence and time of the second beep, yet the percept of the second beep was not attracted towards the 1rst. Third, the absence of attraction effects in the sequence conditions rules out any explanation based on refractory periods. Perceptual refractoriness might in principle mean that the percept of the second event was adjusted while the 1rst event was processed. However, classical accounts of refractoriness (see Jolicoeur, Oriet, and Tombu 2001) would predict a delay rather than an advance in the processing of the second event. This should produce a delay rather than an advance in the perceived time of the second event. In addition, any effect of refractoriness on the conscious awareness of the second event should apply equally to both causal and sequential context. Finally, there is one possible feature of our study which could produce an artefactual explanation of the attraction effect of an effect towards its cause. In causal contexts, the 1rst and second events occur in different perceptual streams, whereas in sequential contexts they occur in the same stream. Thus, in ASj and SAj conditions, subjects switch attention to a new perceptual stream for the purposes of judgement. A sceptic could argue that this switch is responsible for the attraction effect. We believe this sceptical explanation can be resisted for a number of reasons. First, this switching effect could simulate attraction of the second event by the 1rst, but cannot explain the attraction of the 1rst event by the second observed in our data for Experiment 2. Thus, the overall conclusion of efferent binding is not undermined. Second, switching attention is a time-consuming process, and should therefore produce a delay in the second percept, whereas our data show an advance. The sceptic would need to posit that subjective time is shortened so as to compensate for attention-switching delays, to explain our data. Third, in recent preliminary results (Haggard and Clark, unpublished data) we observed that the perceptual attraction characteristic of operant action is replaced by a repulsion effect if the ‘action’ is an involuntary muscle twitch produced by transcranial magnetic stimulation, rather than a voluntary contraction of the same muscles. In this last case, the modalities of the physical events are identical in both conditions, yet the percepts differ dramatically. Simple temporal effects of shifting attention cannot account for this pattern of results.

281


282


13.10 Generality of the present results Efferent binding should occur whenever we interact with our environment. Therefore, if the present results truly reflect a conscious consequence of the binding process, they should apply rather generally to a range of situations. First, there is a clear generality internal to the results presented here: the binding process applies equally to stimulus and action percepts, and applies equally to SRT and to operance. Therefore, we speculate that the effects reported here reflect the operation of a basic process constructing our awareness of interactions between the self and the world. Do the effects reported here also generalize to other tasks? While few studies have considered implications of efferent binding for awareness, we believe one study, in particular, provides evidence which converges with our own. Deubel, Irwin, and Schneider (1999) asked subjects to make either voluntary or relexive saccades, and to estimate their direction of gaze at the time of a brief test stimulus. They found that the perceived direction of gaze shifts up to 250 ms before the saccade itself. This single numerical value recalls Libet et al.’s (1983) observation that the perceived time of a manual action precedes the onset of muscle activity. More importantly, this anticipatory awareness is greatest for test stimuli presented at the location of the saccade target. We believe their results can be interpreted as an oculomotor efferent binding effect. In this case, the conscious representation of the saccade is bound with the visual consequence of the saccade, i.e. the visual stimulation at the target location. The association between a saccade and what is seen when the saccade 1nishes is much more direct than the arbitrary associations between keypresses and auditory signals we have studied here. The targetspeci1city of Deubel et al.’s effect is consistent with our view (see above) that conscious awareness reflects efferent binding of speci1c causal associations between our actions and their consequences. This study has reported effects of efferent binding on conscious representations, but it seems these effects may also generalize to performance. Ziessler and Nattkemper (2001) provide an elegant series of RT experiments, from which they conclude that ‘the planning of goal-directed actions always includes the anticipation of possible action effects’. They speculate that this could occur either by standard rules of associationist learning, or by a ‘presetting’ of the cognitive system. This ‘presetting’, in our view, is remarkably similar to the classical concepts of conscious intention and conscious volition. Ziessler and Nattkemper’s results can be interpreted as evidence for an efferent binding between nonconscious representations of action plans, of actions, and of their effects. We suggest that comparison of efferent binding processes in performance and in conscious awareness of the same events may prove a fruitful vein for future research. Finally, two directions of research are still required to ensure the generality of these results. First, we have studied a narrow range of inter-event intervals in largely arbitrary stimulus–action pairings. We plan to investigate how the compatibility between events and the time interval between events inluence efferent binding in future research. A second interesting aspect of generalization involves the extension of efferent binding to awareness of other people’s actions. Do actions of others show the same binding effects as my own, due to an inference about others’ intentions? If they do, this would suggest that both action production and action understanding activate a common conscious mechanism, perhaps paralleling similar nonconscious mechanisms underlying imitation performance (see Bekkering and Wohlschläger 2001).

13.11 Common coding Here we relate our results to the common-coding view (Prinz 1992). This asserts that a common form of mental representation exists for both external stimuli and our responses to them. The



common-coding view was developed to account for stimulus–response mapping tasks (see, e.g. Stoet and Hommel 2001; Ziessler and Nattkemper 2001), and has not been as extensively applied to intentional actions. However, we believe one aspect of our results is highly consistent with a common-coding view, while another is clearly not. First, we found that attraction effects apply interchangeably to stimulus and action representations. That is, both stimulus and motor representations are subject to a single efferent binding process. This represents a strong sense in which the brain operations underlying conscious awareness are no different for stimulus and for action codes. On the other hand, a central claim of the common-coding hypothesis has been that actions are represented in terms of their environmental effects (Prinz 1992). This is not borne out in our data. In Experiment 1, we found that operant actions were perceptually stable, whereas the percept of their auditory effects was labile. We believe this result is in the opposite direction from that predicted by the common-coding hypothesis.

13.12 Summary and conclusions Our work can be summarized with a model of efferent binding (see Fig. 13.5). The neural events underlying action are shown as a causal chain on the left-hand side of the model. Our interest has been in relating them to the conscious awareness of intention, action, and effect: these conscious representations are shown on the right-hand side. Our key point is that the neural events do not map one-to-one onto conscious states. That is, arrows connecting neural events to conscious events in the model are not all horizontal, but form a web. For example, previous studies have shown that our awareness of intention is not a simple consequence of neural preparation, as Libet et al. (1983) proposed. Rather, information about the speci1c movement to be performed contributes signi1cantly to the content of our conscious intentions (Haggard and Eimer 1999; Haggard and Magno 1999). Similarly, information about preparation of movement contributes to the percept of the movement itself (Haggard et al. 1999). Thus, the mappings between neural and conscious events are many-to-many, rather than one-to-one. We suggest that the efferent binding of conscious representation is an expression of this many-to-many mapping. It generates a strong association between representations of intention and representations of the actions they produce: this binding is conceptualized by the thin solid boxes within the conscious events section of the framework.

Fig. 13.5

A model of efferent binding. See text for discussion.

283


284


In this paper, we have studied the conscious representation of action and effect, rather than intention and action. Here, we have shown evidence for a second aspect of efferent binding, which associates actions with the stimuli that cause them, or that they cause. This further form of efferent binding between conscious representations equates to the lower solid box in the model of Fig. 13.5. In the context of the experimental paradigms used here, ‘causation’ involves a relation between one stimulus and one action. In future research we will test the speci1city of these binding effects, by investigating whether, for example, a stimulus which is not caused by the subject’s operant action is less subject to efferent binding than one that is so caused. In our introduction, we distinguished between a generative and a constructive role of consciousness in voluntary action. We have shown that a speci1c mental operation occurs during voluntary action, to pull together in time the conscious representations of the physical events that occur. That is, the conscious events are compressed in time relative to the neural events with which they are linked. We suggest that this temporal uni1cation of voluntary actions in a relatively restricted zone of conscious experience forms part of how the human mind constructs the strong association between intentions, actions, and consequences that underlie the self, and the sense of self-agency.

Acknowledgements This research was supported by MRC, BBSRC, DAAD, and MPG. We are grateful to Marisa Taylor-Clarke, Sam Clark, Rob van Beers, and Andy Wohlschläger for help and comments.

References Bekkering, H. and Wohlschläger, A. (2001). Action perception and imitation. This volume, Chapter 15. Blakemore, S.J., Frith, C.D., and Wolpert, D.M. (1999). Spatio-temporal prediction modulates the perception of self-produced stimuli. Journal of Cognitive Neuroscience, 11, 551–559. Breitmeyer, B. (1985). Problems with the psychophysics of intention. Behavioral and Brain Sciences, 8, 539. Bridgeman, B. (1985). Free will and the functions of consciousness. Behavioral and Brain Sciences, 8, 540. Della Sala, S., Marchetti, C., and Spinnler, H. (1991). Right-sided anarchic (alien) hand: a longitudinal study. Neuropsychologia, 29, 1113–1127. Deubel, H., Irwin, D.E., and Schneider, W.X. (1999). The subjective direction of gaze shifts long before the saccade. In W. Becker, H. Deubel, and T. Mergner (Eds.), Current oculomotor research: Physiological and psychological aspects. New York: Kluwer. Dickinson, A. (1980). Contemporary animal learning theory. Cambridge: CUP. Elsner, B. and Hommel, B. (in press). Effect anticipation and action control. Journal of Experimental Psychology: Human Perception and Performance. Engel, A.K., Fries, P., König, P., Brecht, M., and Singer, W. (1999). Temporal binding, binocular rivalry, and consciousness. Consciousness and Cognition, 8, 128–151. Frith, C. (1992). The cognitive neuropsychology of schizophrenia. Hove/Hillsdale: Erlbaum. Haggard, P. (1999). Perceived timing of self-initiated actions. In G. Aschersleben, T. Bachmann, and J. Müsseler (Eds.), Cognitive contributions to the perception of spatial and temporal events. Amsterdam: NorthHolland. Haggard, P. (in press). Conscious awareness of intention and of action. To appear in J. Roessler and N. Eilan (Eds.), Agency and self-awareness: Issues in philosophy and psychology. Oxford: Oxford University Press. Haggard, P. and Clark, S. (unpublished data). Voluntary action and conscious awareness. Manuscript in preparation.



Haggard, P. and Eimer, M. (1999). On the relation between brain potentials and the awareness of voluntary movements. Experimental Brain Research, 126, 128–133. Haggard, P. and Magno, E. (1999). Localising awareness of action with Transcranial Magnetic Stimulation. Experimental Brain Research, 127, 102–107. Haggard, P., Newman, C., and Magno, E. (1999). On the perceived time of voluntary action. British Journal of Psychology, 90, 291–303. Hazeltine, E. (2001). The representational nature of sequence learning: Evidence for goal-based codes. This volume, Chapter 33. Hume, D. (1750). Philosophical essays concerning human understanding. London: Millar. Jolicoeur, P., Oriet, C., and Tombu, M. (2001). From perception to action: Making the connection. This volume, Chapter 28. Kant, I. (1781/1963). Critique of pure reason. London: Macmillan. Kawato, M. and Wolpert, D.M. (1998). Internal models for motor control. In Novartis Foundation Symposium, 218, 291–304. Libet, B. (1985). Unconscious cerebral initiative and the role of conscious will in voluntary action. Behavioral and Brain Sciences, 8, 529–566. Libet, B., Gleason, C.A., Wright, E.W., and Pearl D.K. (1983). Time of conscious intention to act in relation to onset of cerebral activity (readiness-potential): The unconscious initiation of a freely voluntary act. Brain, 106, 623–642. Michotte, A. and Prum, E. (1910). Etude expérimentale sur le choix volontaire. Annales de Psychologie, 10, 194–279. Morton, J., Marcus, S.M., and Frankish, C.R. (1976). Perceptual centres (P-centres). Psychological Review, 83, 405–408. Prinz, W. (1992). Why don’t we perceive our brain states? European Journal of Cognitive Psychology, 4, 1–20. Rugg, M.D. (1985). Are the origins of any mental process available to introspection? Behavioral and Brain Sciences, 8, 552. Sherrington, C.S. (1898). Further note on the sensory nerves of muscles. Proceedings of the Royal Society, B62, 120–121. Sternberg, S. and Knoll, R.L. (1973). The perception of temporal order: Fundamental issues and a general model. In S. Kornblum (Ed.), Attention and Performance IV. New York: Academic Press. Stoet, G. and Hommel, B. (2001). Interaction between feature binding in perception and action. This volume, Chapter 26. von Holst, E. and Mittelstaedt, E. (1950). Das Reafferenz-Prinzip. Naturwissenschaft, 37, 464–476. Yeo, C.H., Lobo, D.H., and Baum, A. (1997). Acquisition of a new-latency conditioned nictitating membrane response—major, but not complete, dependence on the ipsilateral cerebellum. Learning and Memory, 3, 557–577. Ziessler, M. and Nattkemper, D. (2001). Effect anticipation in action planning. This volume, Chapter 32.

285




III Action perception and imitation




14 Processing mechanisms and neural structures involved in the recognition and production of actions Introduction to Section III Raffaella Ida Rumiati

14.1 Introduction Interest in the recognition and imitation of actions has grown considerably in the last ten years among neuroscientists, as testi1ed by the very ample review of related theories and empirical studies made available by Bekkering and Wohlschläger (this volume, Chapter 15). First, the authors stress the importance of imitation in the 1eld of social and developmental psychology. Second, they provide the reader with several conceptualizations of imitation and, by comparing the research on humans and non-human primates or on other species in this 1eld, they try to answer the question of who can and who cannot imitate. Finally, after discussing the most in2uential theories of imitation, Bekkering and Wohlschläger illustrate their own view on the mechanisms involved in imitation. In short, their theory of goal-directed imitation holds that, when humans imitate an action, for instance touching an ear, they map the goal of the action irrespective of the effector used for performing the movement (e.g. the left or right hand), and of the movement path (e.g. ipsi- or contralateral). In addition to that tutorial, the papers included in this section represent a substantial contribution to the understanding of both the neural structures (Gallese, Fadiga, Fogassi, and Rizzolatti; Jellema and Perrett) and of the processes underlying the ability to recognize and to reproduce actions (Castiello, Lusher, Mari, Edwards, and Humphreys; Shiffrar and Pinto).

14.2 Is there a common system for recognition and production of actions or are there two representations underlying these two processes? With regard to the studies on the neurophysiology of recognition and action, two views seem to be dominant in Gallese et al.’s and in Jellema and Perrett’s chapters. The one favored by the authors of the 1rst chapter holds that perception and action rely on a common system (Di Pellegrino, Fadiga, Fogassi, Gallese, and Rizzolatti 1992; Gallese, Fadiga, Fogassi and Rizzolatti 1996). Jellema and Perrett’s 1ndings, that most neurons in the upper bank of the superior temporal sulcus (STSa) of the monkeys’ brain code both form and space, do not 1t the original theory of the visual system


290


proposed by Ungerleider and Mishkin (1982). These authors proposed that the visual system is organized in two parallel processing pathways—the ventral stream, dealing with shape and identity of the stimulus (the ‘what’ stream), and the dorsal stream, coding the spatial characteristics of the stimulus (the ‘where’ stream). This dichotomy has been reformulated by Milner and Goodale (1995). In their revised model, the identity and the location of an object are processed by both pathways but for different purposes. Thus, the ventral stream serves visual recognition (‘what’), while the dorsal (‘how’) stream sustains the visual control of action. We shall now brie2y consider the chapter by Gallese et al. and that by Jellema and Perrett in turn. Rizzolatti and colleagues found that in the premotor area F5 of the monkey’s brain there are many bimodal neurons that discharge during a monkey’s goal-directed movements, and also when a monkey observes similar movements executed by another monkey (Di Pellegrino et al. 1992) or by an experimenter (Gallese et al. 1996). Given such properties, these cells were called ‘mirror neurons’. By means of a single-unit recording technique, Gallese et al. (this volume, Chapter 17) discovered that a considerable percentage of neurons in the anterior part of the inferior parietal lobule (area PF) of one macaque monkey’s brain had visual and motor properties. This means that, like the neurons previously described in F5, several PF neurons 1red when the monkey observed other individuals performing actions such as grasping and reaching, as well as when the monkey itself performed the same movements. Besides the discovery of mirror neurons in area PF, there are two more 1ndings in Gallese et al.’s study that deserve attention. First, a subset of the mirror neurons in area PF matched observed hand actions (input) to mouth responses of the monkey (output). As an alternative to the ontogenetic interpretation provided by the authors, I propose that this particular type of response may be explained in terms of the goal-directed theory of imitation proposed by Bekkering, Wohlschläger, and Gattis (2000; for an extensive presentation of this theory see also Bekkering and Wohlschläger, this volume, Chapter 15). According to Bekkering and colleagues’ view, when a subject imitates an observed action s/he reproduces the goal of the action (e.g. reaching for the ear/object), whereas the means are ignored (e.g. the effector employed or the trajectory of the reaching movement). The second interesting 1nding is that the hand performing the observed action in2uenced the discharge intensity. In particular, the monkey studied by Gallese et al. showed a preference for the experimenter’s left hand. Although this observation was made on a single animal, recording only from its left hemisphere, there is some ground for speculating that the mirror system may be lateralized. The mirror system observed in monkeys seems to exist in humans, too, as argued both in a study by Fadiga, Fogassi, Pavesi, and Rizzolatti (1995), who used Transcranial Magnetic Stimulation (see Gallese et al., this volume, Chapter 17), and in several brain-imaging studies. The tasks employed in these studies were quite different from each other. Participants were required to observe a model grasping three-dimensional objects (Grafton, Arbib, Fadiga, and Rizzolatti 1996; Rizzolatti, Fadiga, Matelli, Bettinardi, Perani, and Fazio 1996b), or to manipulate simple objects (Binkofsky et al. 1999). In the study carried out by Iacoboni et al. (1999), the observation and the imitation of 1nger movements were employed and brain activation patterns similar to those observed in other studies were obtained. The common neural network activated both during action recognition and during action production includes sectors of the Broca’s area in the premotor cortex (Brodmann areas 44– 45, corresponding to the F5 area in monkeys) and the inferior parietal lobule (Brodmann area 40, corresponding to PF area in monkeys). In a PET study, Decety et al. (1997) contrasted two cognitive strategies—the observation of actions for later recognition vs. for later imitation—and two types of stimuli, meaningful vs. meaningless


Processing mechanisms and neural structures involved in the recognition and production of actions

actions. Irrespective of the type of actions, the observation of actions for later recognition enhanced the activity in the ventral pathway bilaterally, whereas the observation for later imitation led to the activation of the dorsal pathway, bilaterally as well. The authors proposed that while the ventral structures sustain the semantic object processing and the recognition of actions (see also Rizzolatti et al. 1996b), the dorsal pathway could be necessary for generating visuomotor transformations. In sum, the neural circuit supporting both recognition and production of actions documented in the studies reviewed above consists of premotor, parietal, and temporal structures. Ultimately, the mirror system has also been attributed a social function in that it allows an individual to understand the actions of other individuals (Gallese and Goldman 1998). Similar to the PF mirror-like neurons (i.e. neurons with visual but devoid of motor properties) are those found in STSa of the macaque monkey where Jellema and Perrett (this volume, Chapter 18) recorded. The most interesting information reviewed in their paper is that STSa neurons enable the viewer to understand social signals. One kind of social signal coded by STSa cells is where another animal is directing its attention. This information is extracted from the visual cues characterizing the face and the body of the agent, as well as from their movements. In other words, the neurons in this region of the monkey’s brain seem to signal where in the environment someone else is looking. How can this be achieved? To start with, the cells build up discrete descriptions of seen eyes, head, and body; subsequently the outputs of these lower-level descriptions are integrated into a hierarchical scheme to form a somewhat more conceptually abstract tuning. Interestingly, preliminary data reported by the authors suggest that the spatial coding may be widespread in the temporal lobe, a suggestion which is at variance with what was originally predicted by the strict ‘what–where’ dichotomy put forward by Ungerleider and Mishkin (1982). According to these authors, the spatial information should be processed by the dorsal stream, whereas the region where Perrett and colleagues record, the STS, lies between the dorsal and ventral streams, with its anterior sections belonging anatomically to the ventral stream. Therefore Jellema and Perrett (this volume, Chapter 18) argue that the assignment of the STS to either the dorsal or the ventral stream is pointless and suggest considering the functions of the cells instead.

14.3 Psychophysical evidence supporting the view of common structures subserving action recognition and production A variety of experimental paradigms have been employed to investigate the mechanisms underlying the performance of recognition and imitation of action tasks. An extensive review can be found in the chapters by Shiffrar and Pinto and by Castiello et al. in this volume. In particular, Shiffrar and Pinto provide us with psychophysical evidence supporting Rizzolatti and colleagues’ view (Di Pellegrino et al. 1992; Rizzolatti et al. 1996a) according to which both motor responses and recognition processes rely upon the same neural circuit. Shiffrar and Pinto argue that their 1ndings do not 1t the theory of Milner and Goodale (1995), whereby motor responses and recognition processes require different visual representations. In distinguishing human versus non-human movements, Shiffrar and Pinto propose that human movements are processed based on an internal model while the non-human ones are not. Thus, movements that are consistent with an observer’s internal model of possible movements are analyzed by mechanisms underlying action perception, whereas inconsistent movements may be analyzed by mechanisms underlying the perception of physical events. In addition, a PET study (Stevens, Fonlupt, Shiffrar, and Decety 2000) indicated a signi1cant bilateral activity in the

291


292


premotor cortex when the observers saw two-frame apparent motion sequences. However, when these same picture pairs were presented more rapidly, participants perceived impossible paths of human movements, and the selective activity in the premotor cortex was no longer observed. Related to Shiffrar and Pinto’s approach is that followed by Castiello et al. (this volume, Chapter 16). In four experiments the authors tested whether the observation of grasping movements performed by a human actor or by a robot equally primed motor responses of normal participants. They found that motor priming only occurred when participants are exposed to human grasping. Castiello et al. also studied three levels of priming. The 1rst simply re2ects whether the participant observed a human or a robot arm. The remaining two levels reveal that priming effects depend on model kinematics. Overall, these 1ndings are in agreement with what is known from neurophysiology and developmental psychology. Gallese et al. have (1996) clearly shown that the mirror neurons were speci1cally activated when the actions observed by the monkey involved the interaction between a hand of the agent and an object. However, they remained silent when, for instance, the agent used a tool (e.g. pliers) in order to grasp an object. Woodward (1998, 1999) demonstrated how small children attend preferably to an object that is grasped by a human but not when the object is grasped by a mechanical device. Overall this section of the book offers a competent, exhaustive overview of the theoretical and empirical issues related to the recognition and the production of actions in the context of imitation. The predictions and the discussions of the 1ndings contained in these papers are based on a shared psychophysical and neurophysiological knowledge of the research on actions. This is undoubtedly a successful example of an interdisciplinary approach to the study of a cognitive domain.

Acknowledgements I would like to thank Alessio Toraldo and Wolfgang Prinz for their useful comments. This article was supported by a Co1nanziamento MURST (2000–02) awarded to Tim Shallice and to the author.

References Bekkering, H., Wohlschläger, A., and Gattis, M. (2000). Imitation of gestures in children is goal-directed. Quarterly Journal of Experimental Psychology, 53A, 153–164. Binkofski, F., Buccino, G., Stephan, K.M., Rizzolatti, G., Seitz, R.J., and Freund, H.J. (1999). A parieto-premotor network for object manipulation: Evidence from neuroimaging. Experimental Brain Research, 128, 210– 213. Craighero, L., Fadiga, L., Rizzolatti, G., and Umiltà, C.A. (1998). Visuomotor priming. Visual Cognition, 5, 109–125. Decety, J., Grèzes, J., Costes, N., Perani, D., Jeannerod, M., Procyk, E., Grassi, F., and Fazio, F. (1997). Brain activity during observation of actions: In2uence of action content and subject’s strategy. Brain, 120, 1763–1777. Di Pellegrino, G., Fadiga, L., Fogassi, L., Gallese, V., and Rizzolatti, G. (1992). Understanding motor events: A neurophysiological study. Experimental Brain Research, 91, 176–180. Fadiga, L., Fogassi, L., Pavesi, G., and Rizzolatti, G. (1995). Motor facilitation during action observation: A magnetic stimulation study. Journal of Neurophysiology, 73, 2608–2611. Gallese, V. and Goldman, A. (1998). Mirror neurons and the simulation theory of mind-reading. Trends in Cognitive Science, 2, 32–36. Gallese, V., Fadiga, L., Fogassi, L., and Rizzolatti, G. (1996). Action recognition in the premotor cortex. Brain, 119, 593–609.


Processing mechanisms and neural structures involved in the recognition and production of actions

Grafton, S.T., Arbib, M.A., Fadiga, L., and Rizzolatti, G. (1996). Localization of grasp representations in humans by position emission tomography. Experimental Brain Research, 112, 103–111. Iacoboni, M., Woods, R.P., Brass, M., Bekkering, H., Mazziotta, J.C., and Rizzolatti, G. (1999). Cortical mechanisms of human imitation. Science, 286, 2526–2528. Milner, A.D. and Goodale, M.A. (1995). The visual brain in action. Oxford: Oxford University Press. Rizzolatti, G., Fadiga, L., Gallese, V., and Fogassi, L. (1996a). Premotor cortex and the recognition of motor action. Cognitive Brain Research, 3, 131–141. Rizzolatti, G., Fadiga, L., Matelli, M., Bettinardi, V., Perani, D., and Fazio, F. (1996b). Localization of cortical areas responsive to the observation of hand grasping movements in human: A PET study. Experimental Brain Research, 111, 246–252. Stevens, J., Fonlupt, P., Shiffrar, M., and Decety, J. (2000). New aspects of motion perception: Selective neural encoding for apparent human movements. Neuroreport, 11, 109–115. Ungerleider, L.G. and Mishkin, M. (1982). Two visual systems. In D.J. Ingle, M.A. Goodale, and R.J.W. Mans1eld (Eds.), Analysis of visual behaviour, pp. 549–586. Cambridge, MA: MIT Press. Woodward, A.L. (1998). Infants selectively encode the goal object of an actor’s reach. Cognition, 69, 1–34. Woodward, A.L. (1999). Infants’ ability to distinguish between purposeful and nonpurposeful behaviours. Infant Behaviour and Development, 22(2), 145–165.

293


15 Action perception and imitation: a tutorial Harold Bekkering and Andreas Wohlschläger Abstract. Currently, imitation, or performing an act after perceiving it, is in the focus of attention of researchers from many different disciplines. Although this tutorial attempts to provide some interdisciplinary background, it will concentrate on possible cognitive mechanisms that underlie imitation performance in human beings. First, the importance of imitation in the 1eld of social and developmental psychology will be stressed. Then, some important notions are introduced about what imitation is and who can and cannot imitate. In the second part of this tutorial, some of the currently most widely cited theories of imitation will be described. The third section gives an overview of the major 1ndings that have led to a new view on the mechanisms underlying imitation—the so-called goal-directed theory of imitation. The concluding remarks start with a discussion about the confusion of the goal concept and a taxonomical suggestion is made to capture the different meanings of the word ‘goal’. Finally, a functional model is proposed that describes the sensorimotor processes that transform action goals into the execution of movements, and vice versa, the transformation from movement perception into the recognition of action goals.

15.1 Introduction Imitation, or performing an act after perceiving it, is currently in the focus of attention of researchers from many different disciplines. Regarding human beings, there is little doubt about the fact that we to some extent do learn to interact with the surrounding world by means of imitation. An introduction to some in2uential theories on the importance of imitation in 1elds like social and developmental psychology will be offered in the section below. However, a more complex question is whether non-human primates or other species learn by imitation. Therefore, comparative research will be overviewed in the sections about what is imitation and who can imitate. In the second part of this tutorial, some of the currently most widely cited theories of imitation will be described. These ideas will be contrasted in the third section with our own recently developed view about the mechanisms underlying imitation, in the so-called goal-directed theory of imitation. For a more detailed description of the functional mechanisms involved in action perception, we should like to direct the reader’s attention to the chapters by Shiffrar and Pinto (Chapter 19) and by Castiello et al. (Chapter 16) in this volume. Neurophysiological support for the existence of separate mechanisms for action perception can be found in the chapter by Jellema and Perrett (Chapter 18). In addition, Gallese et al., Chapter 17, describe possible neurophysiological structures involved in action perception and action execution in the monkey brain. A nice aside to this chapter regarding possible human neurophysiological mechanisms in imitation is provided by a recent fMRI-study of Iacoboni et al. (1999). From the neuropsychological 1eld, many recent insights into the functional control of imitative behavior are derived from studies of patients with apraxia (see Leiguarda and Marsden 2000, for a recent review).


Action perception and imitation: a tutorial

15.1.1 The in2uence of imitation research on social and developmental psychology By watching a model’s behavior, an observer can learn how to do something he or she did not know before. In fact, many cultural traditions, for instance the very complex Japanese tea ceremony, can only be explained by assuming imitative patterns as a way of inducting the child into adult ways. Providing models is not only a means of speeding up what might otherwise be a very tedious learning process. Rather, in some cases it is absolutely essential, for alternative-learning procedures would entail too high a risk. Learning to drive a car by trial and error would cost too much in terms of upended pedestrians (Bandura and Walters 1963). Aristotle recognized the importance of imitation, or learning to perform an act from a modeled example. He referred to imitative capacities as an important manifestation of intelligence: ‘Imitation is natural to man from childhood, one of his advantage over the lower animals being this, that he is the most imitative creature of the world, and learns at first by imitation’ (Aristotle, in McKeon 1941, p. 448b). Ever since then, imitation has been a major topic of interest, particularly within psychology. Within developmental psychology, Piaget’s ideas (e.g. 1975) are very in2uential to this day. In contrast to other views, he claimed that imitation cannot be seen as a mental activity of a lower order and has to be regarded as closely connected to the general cognitive development of children. He pointed out that the resulting growth in the capacity for imitation is a vital prerequisite for many further aspects of intellectual development including the acquisition of language (see below). During the 1rst few months infants are only capable of pseudo-imitation. If the father does something the baby did just a moment before (such as babbling) the baby is likely to resume this activity. This observation can be seen as an extension of the circular-reaction phenomenon—the sensory feedback from each behavior. For example, the child scratches an object and then gropes at it, scratches and gropes again, primes its own immediate recurrence. The difference is that in pseudo-imitation, the cycle is reactivated from the outside. In other words, the infant treats the father’s ‘dada’ as if it were a ‘dada’ of his or her own. In this view, imitation becomes more genuine with increasing age. From about four months on, infants can imitate actions they did not perform themselves just moments before, but they can only do so if their parent’s action leads to sights or sounds similar to those the infants encounter when they perform that action themselves. Examples are hitting the table or squeezing a pillow. What the infant sees when she watches her parents’ hands is similar to what she sees when her own hand goes through the same motions. Therefore, Piaget suggested that, at this age, imitation is restricted to movements that are visible to the imitator (however, see the Meltzoff studies described below for some striking contra-evidence). For movements like sticking out the tongue, the sensory–motor schemas need to be well developed to enable a correspondence of the organ end states of the infant’s own body to that of the other (only possible from nine months on). In general, Piaget makes the notion that imitation can only take place if the observer has a welldeveloped comprehension of what the model is doing, together with a schema that allows the translation of a desired perceptual outcome into motor patterns that bring it about, a process that starts in early development but continues into adulthood. Imitation is only possible when the appropriate schemas are already formed. Another very in2uential thought on the importance of imitation derives from social psychology. First, social learning theorists pointed out that imitative learning is not a species of instrumental conditioning. In particular, examples of imitation where the observer does not copy the model’s actions at the time he sees them (learning without performance) and where he imitates even though

295


296


he neither receives a reward himself nor sees the model receive one (learning without reinforcement, see also Gleitman 1981, p. 499) argue against typical instrumental-conditioning explanations. Second, a number of experiments have been conducted which show that the performance of an observed act depends in part upon the characteristics of the model. Not surprisingly, subjects are more likely to imitate people they like, respect, and regard as competent (e.g. Lefkowitz, Blake, and Mouton 1955). Finally, performance of an observed act will also depend upon the consequences that seem to befall the model. In an in2uential study, several groups of nursery-school children were shown a film that featured an adult and a large plastic ‘Bobo-doll’. The adult walked over to the doll and ordered it to get out of the way. When the doll did not comply, the adult punched and kicked it around the room, stressing her attacks with phrases such as ‘Right on the nose, boom, boom’. One group of subjects saw the film up to this point but no further. Another group saw a 1nal segment in which this behavior was shown to come to a bad end. A second adult arrived on the scene, called the aggressor a big, bad bully, spanked her, and threatened further spankings if she should ever do such bad things again. After seeing the films, all children were brought into a room that contained various toys, including a Bobo-doll. They were left alone, but filmed by a video camera. The children who never saw the bad ending of the film, imitated the model’s aggressive acts with the Bobo-doll. In contrast, the children who had seen the model’s punishment behaved much more pacifically (Bandura and Mischel 1965).

15.1.2 What is imitation? However, many issues about imitation have not yet been answered. For instance, how can imitation been answered be best de1ned, or to put it simply: ‘What is imitation?’ Also, is imitation restricted to humans, to primates, to mammals, to birds—in other words: ‘Who can imitate?’ The notion of ‘imitation’ has probably been under debate for as long as the concept has existed. Thorndike’s (1898) pragmatic de1nition of ‘learning to do an act from seeing it done’ focused on the key role observation plays in imitation without specifying any details about which aspects of the model are imitated or how imitation is achieved by the imitator. Somewhat later, Spence (1937) pointed out that what people typically refer to as imitation might be stimulus enhancement. That is, seeing some act done in a particular place, or with some particular object, has the effect of increasing the observer’s probability of going to that place or interacting with that object. In a more powerful formulation, this tendency would be specifc to cases where the conspeci1c obtains valued rewards by its action. As a consequence, this narrowing of attention ensures that the individual’s subsequent behavior becomes concentrated on the key variables of the action, and it is likely that many observations formerly considered to be imitation are thereby explained away. The view that stimuli can be seen as reinforcers for social learning has more recently also been called observational conditioning (Mineka, Cook, and Keir 1984). More recently, Tomasello (1990) introduced the concept of emulation. Whereas stimulus enhancement changes the salience of certain stimuli in the environment, emulation changes the salience of certain goals. In emulation (Köhler 1976; Tomasello 1990), the purpose of the goal towards which the demonstrator is striving is made overt as a result of his actions, and so becomes a goal for the observer, too. The observer attempts to ‘reproduce the completed goal . . . by whatever means it may devise’ (Tomasello 1990, p. 284). Lately, the de1nition of emulation has been broadened and refers to the possibility of learning about the physical situation as an indirect consequence of another’s behavior. In other words, many things may be learned from what happens to objects in the



environment as a result of an individual’s actions, quite apart from learning the actions themselves, for instance, the strength, brittleness, weight, what an object is made of or contains, and so forth. In emulation, actions of equivalent ultimate effect on the environment are replicated without the particular nuances or techniques of the behavior copied (see also Call and Carpenter, in press). Thus, in emulation the observer learns something about the environment, but nothing about the behavior of the model directly. Stimulus enhancement and emulation both refer to environmental learning processes, which might even take place without a model being present, as in the case of an apple falling from a tree. An experimental technique often used to discover whether the action performed depends on, or copies, the action observed is to compare the probability of the occurrence of an observed action in relationship to baseline. Imitation is operationally de1ned as a signifcant elevation in the frequency of an observed action over the normal probability of its occurrence (e.g. Byrne and Russon 1998). Important improvements to this basic technique have been developed. In animal research, for instance, two groups of animals are typically used, each seeing the same problem solved by a conspeci1c (the demonstrator) but in different ways. Importantly, the groups can then be compared in the frequencies of performing each technique. Imitation is subsequently de1ned as a signifcant frequency divergence between the groups in frequencies of using the two observed actions (see also Whiten 1998; Whiten and Ham 1992). In developmental psychology, typically an adult in front of an infant repeatedly performs one of a set of several different target gestures. Imitation is then de1ned as the selective increase in the frequency of making one gesture, and not the other gesture, compared with baseline performance. There now is considerable support among developmental and comparative psychologists that by this criterion the ability to imitate has been detected. For example, signi1cantly more infant tongue protrusion after observing adult tongue protrusion than after observing adult mouth opening has been reported and vice versa (Meltzoff and Moore 1977, 1983). Some researchers (e.g. Byrne and Russon 1998; Bekkering and Prinz, in press), however, have argued that it is hard to decide whether an increase in the performance of a specific action is a consequence of the stimulus events generated by a conspeci1c’s action—and should therefore not be called imitation—or whether it is a consequence of observing a conspeci1c’s action—and can be called imitation. An alternative explanation for the latter kind of imitation could be response facilitation. Therefore, Byrne and Russon (1998) bring into account the concept of novelty. However, de1ning novelty as a necessary condition for imitation would require full access to the behavioral history of the imitator. More important, the concept of novelty ignores the view of most (developmental) psychologists that new information is constantly being integrated into pre-existing programs to enable the animal to become adapted to new circumstances. The new information does not replace old information; rather, it is intimately woven into programs by a mutual process of assimilation and accommodation (e.g. Piaget 1975). Assimilation refers to the process of incorporating an internal operative act through which environmental data are acquired. Accommodation is the outwardly directed process of adapting an overall structure to a speci1c case. In this sense, it always includes an element of novelty, but it is an already present structure that becomes differentiated through observational learning. From this perspective, it is difficult to see how the definition of novelty could be reconciled with such a dynamic and mutual process of learning (see also Huber 1998). In summary, several conceptualizations of imitation have been formulated, each of them explaining how certain aspects of social learning can occur, without necessarily involving imitation. Our conclusion about what imitation might be will be delayed to the conclusion section.

297


298


15.1.3 Who can imitate? In his famous monograph, Thorndike (1898) presented the results of experiments with cats, dogs, and chicks that needed to escape from a box to get food by manipulating some simple mechanism (e.g. by pulling down a loop of wire, depressing a lever, or turning a button). No evidence whatsoever had been found that these animals could have learnt these acts from seeing their conspeci1c’s doing them. Thorndike’s behaviorist conclusion therefore was that the responses are ‘formed gradually by the stamping in of successes due to trial and error, and are not able to be formed by imitation or by one’s being put through them’ (1898, p. 553). This led Thorndike to state that associations in animals are not homologous with anything in the human association mechanisms. Comparative psychologists seem to differ fundamentally in their opinions on the question of whether non-human primates can imitate or not. In an in2uential paper, Byrne and Russon (1998) present a considerable amount of data to stress the point that gorillas, orangutans, and chimpanzees use a goal hierarchy when, for instance, eating different kinds of leaves. Is this imitation? Great apes, as for instance gorillas, from the 1rst days of life de1nitely know from their mother what parts of a plant are edible. In addition, the stimulus enhancement would tend to focus a young gorilla’s attention on the growing plant as a potential object to investigate. The interesting question is how the young gorilla 1rst acquires the elaborate sequence of coordinated actions that converts, say, nettle plants to edible mouthfuls. According to Byrne and Russon, the key concept is novelty, which lies in the arrangement of the acts. Thus the skill learnt by imitation is to arrange some basic repertoire of actions into novel and complex patterns rather than to learn new basic actions. The evidence they present suggests that young gorillas have functional control over hierarchical structures of actions comparable to their adult models. In some case studies of orangutans (e.g. Russon 1996; Russon and Galdikas 1993), they found imitation behavior of complex object–object relations. For instance, orangutans were observed stealing soap and laundry by canoe, weeding paths, or (unsuccessfully) trying to light a 1re. However, Tomasello, Savage-Rumbaugh, and Kruger (1993b) observed that mother-reared chimpanzees are much poorer at imitatively learning novel actions on objects than enculturated chimpanzees. Thus, they conclude that the enculturation process is crucial in observing imitation instead of emulation (for more details, see the theoretical section below). In another study, Nagell, Olguin, and Tomasello (1993) reported two studies in which chimpanzees and young children observed a human demonstrator using a tool that resembled a rake in order to retrieve an out-of-reach reward (food or a toy for chimpanzees and children, respectively). The demonstrator began with the rake in one of two positions, with either the teeth or the crossbar down. When the rake began in a teeth-down position, the experimenter 2ipped the rake so that the crossbar was down, and then used the crossbar to drag the object within reach. When the rake began in a crossbar-down position, the experimenter simply dragged the object within reach, again using the crossbar of the rake. A similar rake was provided for human and chimpanzee observers, always resting in a teeth-down position. The question of interest was not simply whether children and chimpanzees used the rake to obtain the reward, but how they used it, and whether the way they used it was in2uenced by the demonstrator’s behavior. Children who observed the demonstrator 2ipping the rake to a crossbar-down position before beginning to pull, were more likely to do the same and to use the crossbar-down rake to drag the object within reach, as compared with children who had not observed the 2ipping action. Children who observed the demonstrator pulling but not 2ipping used the rake in the teeth-down position and simply pulled. In contrast, chimpanzees 2ipped and pulled or pulled only with equal likelihood in both



observer conditions. In other words, while both children and chimpanzees learned by observation to use the tool to obtain the reward, the demonstrator’s behavior in2uenced the behavioral strategy employed by children but not that of the chimpanzees. Nagell et al. (1993) concluded that chimpanzees attended to the end result of the task (obtaining a reward) and to the functional relations involved in the task (obtaining the reward by using the rake) but failed to adopt the strategy used by the human model. Tomasello and colleagues named such behavior ‘emulation learning’ (e.g. Tomasello and Call 1997). Most scientists seem to agree that the behavior Tomasello and colleagues call emulation, such as the behaviors of the chimpanzees above, does not count as imitation. The imitation capacity of songbirds, on the other hand, is well established. The most widely accepted hypothesis of vocal imitation in birds states that vocal learning involves two steps: (1) an auditory memory is laid down, and then (2) vocal output is modi1ed until the auditory feedback it generates matches the model. It is also known that the pathways involved in song production respond to sound, an observation that blurs the demarcation between what an auditory and what a motor circuit is (see, for a review, Nottebohm 1991). For instance, in a well-controlled study, it was found that male zebra 1nches (Taeniopygia guttata) master the imitation of a song model 80 to 90 days after hatching and retain it with little change for the rest of their lives (e.g. Lombardino and Nottebohm 2000). Interestingly, a juvenile male zebra 1nch, kept singly with its father, develops a fairly complete imitation of the father’s song. The imitation is less complete when other male siblings are present, possibly because, as imitation commences, model abundance increases (Tchernichovski Lints, Mitra, and Nottebohm 1999). Recently, Doupe and Kuhl (1999) have argued that there are numerous parallels between human speech and birdsong. Both humans and songbirds learn their complex vocalizations early in life, exhibiting a strong dependence on hearing the adults they will imitate as well as themselves as they practice, and both show a waning of this dependence as they mature. Innate predispositions for perceiving and learning the correct sounds exist in both groups, although more evidence of innate descriptions of species-speci1c signals exists in songbirds, where numerous species of vocal learners have been compared. Humans also share with songbirds an early phase of learning that is primarily perceptual, and then serves to guide later vocal production. Both humans and songbirds have evolved a complex hierarchy of specialized forebrain areas in which motor and auditory centers interact closely, and which control the lower vocal motor areas also found in non-learners. In both these vocal learners, however, the way auditory feedback is processed during vocalization in these brain areas is surprisingly unclear. Finally, humans and songbirds have similar critical periods for vocal learning, with a much greater ability to learn early in life. In both groups, the capacity for late vocal learning may be decreased by the act of learning itself as well as by biological factors, such as the hormones of puberty. Although some features of birdsong and speech are clearly not analogous, such as the capacity of language for meaning, abstraction, and 2exible associations, there are striking similarities in how sensory experience is internalized and used to shape vocal outputs, and how learning is enhanced during a critical period of development. Empirical evidence for vocal imitation in infants derives from a study by Kuhl and Meltzoff (1996). They examined developmental changes in infants’ vocalizations in response to adults’ vowels at 12, 16, and 20 weeks of age. Vocal imitation was documented: Infants listening to a particular vowel produced vocalizations resembling that vowel. Another piece of evidence of vocal imitation derives from a study by Poulson, Kymissis, Reeve, Andreators, and Reeve (1991). Three infants, aged 9 to 13 months, and their parents participated in 2 to 4 experimental sessions per week for 2 to 4 months. During each 20-min session, the parent presented vocal models for the infant to imitate. During the model-alone condition, no social praise

299


300


was programmed for infant imitation. During the model-and-praise condition, the parents for infant imitation provided social praise on training trials, but not on probe trials. All three infants showed systematic increases in matching during training trials following the introduction of the model-andpraise condition. Although matching during probe trials was not directly reinforced, probe-trial responding increased systematically with training-trial responding. Furthermore, non-matching infant vocalizations did not increase systematically with the introduction of the model-and-praise procedure. Together, these 1ndings provide a demonstration of generalized vocal imitation in infants. Nevertheless, a critical note on vocal imitation derives from the study of Siegel, Cooper, Morgan, and Brenneise-Sarshad (1990). Children between 9 and 12 months of age were studied to determine if they would spontaneously imitate either the average fundamental frequency or the fundamental frequency contour of their speaking partners. In the 1rst experiment, children were recorded at home as they interacted with their fathers and mothers. Acoustic analyses failed to reveal any tendency on the part of the infants to adjust vocal pitch, amplitude, or duration to those of their speaking partners. In a second experiment, children were recorded while interacting with their parents in a laboratory setting. Again, there were no indications that the children imitated the vocal patterns of their speaking parents. Can infants imitate beyond speech? The studies of Meltzoff and Moore (see the 1997 paper for a recent overview) suggest that neonates can imitate facial gestures, such as tongue protrusion, which they cannot see themselves perform. In the original study (Meltzoff and Moore 1977), mouth-opening and tongue-protrusion gestures were shown to 3-week-old infants while they were engaged in the competing motor activity of sucking on a paci1er. The adult terminated the gestural demonstration, assumed a neutral face, and only then removed the paci1er. Three-week-old infants differentially performed both gestures despite the fact that the adult was no longer showing them. If one contemplates seriously the hypothesis that imitation is not just a certain kind of stimulus enhancement or response facilitation, but rather re2ects a behavioral acquisition through observation, a valid argument would be to show that deferred imitation can span a long delay of at least several hours or even days. In the 1994 study by Meltzoff and Moore, 6-week-old infants saw a person performing a speci1c gesture on Day 1, and then after a 24-hour retention interval they saw the same adult in a neutral pose. Different groups of infants saw different gestures on day 1, and they all saw the same neutral pose on day 2. Strikingly, the infants differentially performed the gestures they saw the day before. In another study, Meltzoff (1988) investigated imitation performance after a oneweek delay in 14-month-old children on six object-oriented actions. One of the six actions was a novel behavior that had a zero probability of occurrence in spontaneous play. In the imitation condition, infants observed the demonstration but were not allowed to touch the objects to prevent them attempting any immediate imitation. The results showed that infants in the imitation conditions produced signi1cantly more of the target actions than infants in control groups, who were not exposed to the modeling. Interestingly, there was also a strong evidence for the imitation of the novel act. Together, these studies were path-breaking for the notion that young infants are already able to learn and memorize a behavior through observation. Can young infants imitate novel actions? In the Meltzoff and Moore (1994) paper they also presented the 6-week-old infants with novel gestures such as tongue-protrusion-to-the-side (however, for a critical note on imitation in newborns, see Anisfeld 1996). They observed that the infants initially performed a small tongue movement with no lateral component. However, after some trials, through a kind of correction process, the baby’s tongue matches the organ end state of the adult’s tongue and thus results in a novel behavior that was not initially present.



15.2 Theories of imitation In the following section, three in2uential theories on imitation will be addressed: (1) The active intermodal mapping (AIM) theory by Meltzoff and Moore; (2) the social enculturated theory of imitation by Tomasello; and (3) the program and action level theory of imitation by Byrne and Russon.

15.2.1 The active intermodal mapping (AIM) theory of imitation The key claim of the active intermodal mapping theory of Meltzoff and Moore (for a recent elaborated version, see Meltzoff and Moore 1997) is that imitation is a matching-to-target process. The model is based on a series of experiments in which newborns seeing particular facial gestures are able to produce matching motor output, or in other words an intermodal mapping of visual input onto proprioceptive output. The active nature of the matching process is captured by the proprioceptive feedback loop. The loop allows infants’ motor performance to be evaluated against the seen target and serves as a basis for correction. The major components of the 1997 version of the model are (a) the perceptual system’s functions that provide the perception of the infant’s own body and the external world, (b) the supramodal representational system that allows comparison between the organ relations of an external target and the current position of the infant’s own body, and (c) the action system that executes a goal-directed act as long as a mismatch between the organ relations of the perceived model and the infant’s self exists. The organ relations provide the common framework in which the acts of the self and the other are registered. ‘Tongue-to-lips’, which is an organ relation, would be a description that cuts across modality of perception and could describe both the target and the self. A match indicates that the motor act seen and the motor act done are equivalent. This recognition of the equivalence of acts is speculated to be the starting ground for infants’ apprehension that the other is, in some primitive sense, ‘like me’ (Meltzoff and Moore 1998). The goal-directed component was added to the model after 1nding the correction process in the tongueto-the-side experiment described above.

15.2.2 The social enculturated theory of imitation Tomasello, Kruger, and Ratner (1993a) identify three strict criteria to specify imitational learning: (a) the imitated behavior should be novel for the imitator, (b) it should reproduce the behavioral strategies of the model, and (c) it should share the goal. Behaviors not satisfying these criteria are not considered true imitation. Although quite some evidence has been reported about imitation or at least action mimicking in infants and for instance songbirds, Tomasello et al. (1993b) exclude the possibility that wild animals display true imitative behavior (see also Whiten 1998, for a critical note on non-human primate imitation). For instance, non-human primates who have received no special treatment from humans do not seem to imitate novel actions on objects (see Section 15.1.3). On the contrary, enculturated chimpanzees do seem to have the ability to imitate. Tomasello et al. (1993b) propose that what is developing on chimpanzees as a result of their enculturation are not imitative abilities per se, but rather more fundamental cognitive skills. In their view, the most important ones are social interactions in which there is a joint focus of attention on some third entity such as an object (Savage-Rumbaugh 1990). Human enculturators encourage and structure such interactions in a way that adult chimpanzees in their natural environment do not. This scaffolding and intentional

301


302


instruction serves to ‘socialize the attention’ (Tomasello et al. 1993b, p. 1702) of the chimpanzee much in the same way that human children have their attention socialized by adults (Vygotsky 1978). They also argue that these broadly de1ned skills of social cognition might be a prerequisite for the acquisition of language (see also Tomasello 1992). In the typical environment of their species, young chimpanzees have very little chance to interact with others around objects, and when they do, the other chimpanzee does not attempt to direct their attention or to teach them a behavior. In human-like cultural environments, young chimpanzees must adapt to the complex triadic interactions around objects that constitute the majority of social interactions. As a result, speci1c social–cognitive abilities might emerge. Two alternatives theories need to be considered though. First, the mother-reared chimpanzees probably had less experience with human artifacts than the enculturated chimpanzees. However, this is experimentally rendered unlikely by the observation that without demonstrations, that is, in the free play and teaching trials, the mother-raised chimpanzees performed the target actions as often as the enculturated chimpanzees. Second, the understanding of what the chimpanzees were supposed to do in the experimental session needs to be looked at. The typical ‘Do what I do’ format was adopted for this study (for a discussion, see Whiten and Ham 1992). However, it is possible that the mother-reared chimpanzees still did not understand in the same way as the other chimpanzees what they were supposed to do.

15.2.3 Program-level and action-level imitation In an in2uential paper on imitation, Byrne and Russon (1998) advocate the view that voluntary behavior is organized hierarchically and, therefore, imitation can occur at various levels. A clear distinction can be made between the ‘action level’, a rather detailed and linear speci1cation of sequential acts, and the ‘program level’, a broader description of subroutine structure and the hierarchical layout of a behavioral ‘program’. At the bottom level of the hierarchy, the precise details of the manual actions and individual uses are probably learned without imitation. That is, each animal was found to have a different preferred set of functionally equivalent variants (Byrne and Byrne 1993), presumably a sign of trial-and-error learning. At a slightly coarser level, for instance the pattern of hand preferences (which is very strong in every animal), no evidence for imitation has been found either. That is, the hand preference of an offspring correlates neither with that of the mother nor the silverback male. Yet, when the overall form of the process was investigated, the order of action sequences was rather 1xed. And although environmental constraints can partially explain this 1xed pattern, learning by individual experience alone seems to be highly unlikely. Program-level imitation is de1ned as copying the structural organization of a complex process (including the sequence of stages, subroutine structure, and bimanual coordination), by observation of the behavior of another individual, while the implementation at the action level might arise from individual learning. Imitation at the program level, then, would consist of copying from a model’s action a novel arrangement of elements that already exist in the repertoire. Thus to imitate, the individual must have a mental apparatus that allows to assemble hierarchical frameworks, to organize the goal structure, and to maintain the goal structure while its detailed enactment is built. Byrne and Russon (1998) suggest that the everyday use of imitation is closer to the program level than to the action level. Interestingly, this notion is in sharp contrast to other notions which have stressed that ‘true imitation’ is only evident when a novel act is added as an unmodi1ed whole to an individual’s motor repertoire (Tomasello 1990; Whiten and Ham 1992).



15.3 Goal-directed imitation Recently, a new view on the representation that mediates perception and action in imitation has been proposed (Bekkering, Wohlschläger, and Gattis 2000; Gattis, Bekkering, and Wohlschläger, in press; Wohlschläger, Gattis, and Bekkering, submitted). This view postulates: 1. That behaviors are not simply replicated as uni1ed, non-decomposed motor patterns. Rather, imitation involves 1rst a cognitive decomposition of the motor patterns into their constituent components, and second a reconstruction of the action pattern from these components. 2. The decomposition–reconstruction process is guided by an interpretation of the motor pattern as a goal-directed behavior. Thus, the constituent elements in the mediating representation involve goals rather than motor segments. 3. It is assumed that these goals are organized hierarchically, with some of the encoded goals being dominant over others. The hierarchy of goal aspects follows the functionality of actions. Ends (objects and treatments) are more important than means (effectors and movement paths). 4. The reconstruction of the motor pattern from its analyzed goals in participants is subject to capacity limitations; only a few goal aspects are selected. 5. Finally, imitation follows the ideomotor principle. The selected goals elicit the motor program with which they are most strongly associated. This motor program does not necessarily lead to matching movements. Evidence in favor of the goal-directed theory of imitation was recently found in some of our studies (Bekkering et al. 2000; Gleißner, Meltzoff, and Bekkering 2000; Wohlschläger and Bekkering, submitted; for further discussion, see also Gattis et al., in press, and Bekkering and Prinz, in press). In an imitational setting, it was observed that young children always moved to the correct goal, such as an object or a particular ear to reach for, but widely ignored the agent (a particular hand to move with), or the movement path (ipsi- or contralateral to the object, see Fig. 15.1 for the gestures we used). This led us to assume that the action recognition process was strongly affected by the observed action effects. We proposed that imitation entails representing an observed behavior as a set of goals (possible action effects), which then automatically activates the motor program that is most strongly associated with these action effects. Goals may include objects (say, a particular ear), agents (a particular hand), a movement path (crossing the body or moving parallel to the body), or salient features (crossing the arms). We also proposed that these goals are represented hierarchically, with some goals dominating over others. When processing capacity is limited so that multiple goals compete, goals higher in the hierarchy are reproduced at the expense of goals lower in the hierarchy. Our results suggested that objects occupy the top of this hierarchy—children always grasped the correct ear, but in cases of errors used the wrong hand and the wrong movement path. The most common error was the so-called contra–ipsi error. In this case, although the adult model used the contralateral hand, children quite frequently touched the correct ear with the ipsilateral hand, the motor program most strongly associated with touching the ear. Because young children have dif1culty in processing multiple elements and relations, failures to reproduce all the goal aspects of an action are more likely than in adults. This proposal predicts that children’s imitation errors are malleable, depending on the number of goals identi1ed in the task as a whole. We tested this prediction in several additional experiments. One experiment limited the

303


304


Fig. 15.1

The six hand gestures used in Bekkering 2000.

movements to only one ear, thereby eliminating the necessity for children to specify the goal object. Nine children with a mean age of 4:4 years copied the movements of a model who always touched her right ear (randomly with either the left or right hand), or who always touched her left ear (again randomly using the left or right hand). In this circumstance, children made virtually no errors, grasping the ear contralaterally whenever the model did so. Eliminating the necessity of specifying the goal object thus enabled children to reproduce other goals in the imitative act, such as using the correct hand and the correct movement path. A further experiment compared the presence and absence of objects in the action set while keeping the total number of modeled gestures constant. Children (32 children, mean age 4:4 years) sat at a desk across from a model who made four unimanual gestures similar to those described above, but performed on a desk rather than at the ears. Half of the children saw four dots on the table, two in front of the model and two in front of the child. The model covered her dots ipsi- and contralaterally, sometimes with her right hand and sometimes with her left hand. Children were encouraged to copy the model, and directed their own actions at two corresponding dots. No dots were placed on the table for the other half of the children, and the model and child instead directed their actions at locations on the table. Children in the dot condition produced the same error pattern already observed in the hand-to-ear task, substituting ipsi- for contralateral gestures. In contrast, children in the no-dot condition who saw the identical movements directed at locations in space rather than dots produced signi1cantly fewer contralateral errors. We concluded that manipulating the presence or absence of a physical object had effectively manipulated the necessity of specifying objects as goals. Despite the fact that the movements of both model and child were identical in both conditions, removing the dots from the table eliminated the object goal, and allowed children to reproduce other goals in the imitative act, such as the agent and the movement. Further research has explored the question of how those goals are speci1ed and organized. Using a similar paradigm to the one described above, Gleißner et al. (2000) manipulated whether the



On

Near Contralateral

Ipsilateral

Contralateral

Unimanual

Ipsilateral

Gesture # 2

Gesture # 7

Gesture # 8

Gesture # 11

Gesture # 12

Bimanual

Gesture # 1

Gesture # 5

Gesture # 6

Fig. 15.2 Some of the gestures as used in Gleißner et al. 2000. gestures were directed at locations on the body, or locations near the body (see Fig. 15.2). A model performed ipsilateral and contralateral movements with her right or left hand or with both hands. She either touched a body part (an ear or a knee), or directed her movement at a location in space near the body part. Three-year-olds imitated less accurately when the model’s actions were directed at locations on the body than when her actions were directed at locations near the body. These results con1rmed the proposal of Bekkering et al. (2000) that objects (such as an ear or a knee) are high in the goal hierarchy and are able to displace other goals, such as agent and movement. Also, these results provide evidence for the view that perceiving unambiguous action effects automatically activates the motor program that is most strongly associated with this action effect, thereby largely ignoring the motor output observed. Whether the body part was visible or not (knee versus ear) did not signi1cantly in2uence imitative behavior, suggesting that visual feedback does not play an important role in specifying goals in gesture imitation. Wohlschläger et al. (submitted) compared other salient features—an open versus closed hand— against goals previously investigated. In an action similar to one of the conditions used by Gleißner et al., a model reached to the location in space near her left or right ear, using the left or right hand so that gestures were sometimes ipsilateral and sometimes contralateral. Simultaneously, the model either made a 1st or opened her hand, with the palm facing the child. We reasoned that the open versus closed hand introduced a new gesture goal, and wished to explore whether that goal would now displace the object goal, just as the object goal displaced other goals such as agents and movements. This was indeed the case. Children reproduced the open versus closed hand of the model every time, but now frequently performed the gesture on the contralateral different side of the head. In addition, since the con1guration of the hand and its position relative to the head were completed at the same time, it is clear that the goal is not simply de1ned by the end state.

305


306


15.3.1 Further evidence for goal-directed imitation 15.3.1.1 The presence or absence of an end-state goal There are several other studies providing additional support for the idea that goals or action effects are inferred from observing actions. For example, it was found that 16- to 24-month-old children imitated enabling sequences of events more accurately than arbitrary event sequences (Bauer and Mandler 1989; Bauer and Travis 1993). The novel–arbitrary sequences involved novel actions with simple objects, such as putting a sticker on a chalkboard, leaning the board against an easel, and drawing on the board with chalk. Novel–enabling sequences also involved novel actions with objects, with the difference that actions in the novel–enabling sequence enabled other actions in the sequence, and ultimately led to a salient novel event, such as a toy frog ‘jumping’ into the air. The frog-jump sequence, for example, involved putting a wooden board on a wedge-shaped block to form a lever, placing a toy frog on one end of the board, and hitting the other end of the board, causing the toy frog to appear to jump into the air. Children of all ages performed the modeled actions in the modeled order more frequently for novel–enabling sequences than novel–arbitrary sequences, clearly indicating that the presence of an unambiguous, observable goal leads to more accurate imitative behavior in young children. Further, Travis (1997) demonstrated that the presence of an end-state goal in a modeled action sequence led to more frequent imitation of those actions compared with action sequences omitting the goal action. Twenty-four-month-old children were shown interleaved pairs of three-step action sequences similar to the novel–enabling sequences described above. They saw either all six actions (three actions for each pair), or only 1ve actions, with the goal action for one sequence omitted. Interestingly, when shown two-goal pairs, children imitated both action sequences equally. In contrast, when shown one-goal pairs, children imitated more actions from the goal-present sequence than from the goal-absent sequence. In addition, children in both conditions performed actions leading to a particular goal as a temporally contiguous sequence—despite the fact that goal-related actions were not temporally contiguous in the modeled sequence, since they were interleaved with actions from another sequence. Thus, a variety of experimental data indicates that observers interpret the actions they observe above the level of elementary perceptual–motor maps. For instance, the presence of an end-goal in a sequence of actions increases the likelihood that those actions will be imitated and, presumably, it organizes subsequent behavior. It is worth noting that the end-goals used by Bauer, Travis, and colleagues were physical acts involving movement, noise, or both. However, Travis points out that a goal, strictly de1ned, is ‘a mental state representing a desired state of affairs in the world’ (1997, p. 115), and can therefore only be observed in the outcome of intentional actions. Identifying the goals of an observable action requires an inference beyond any mapping or parsing method as described previously. 15.3.1.2 Inferences about action goals Another line of evidence in favor of goal-directed imitation derives from a study by Want and Harris (1998). In their experiment, subjects were shown how to poke an object out of a horizontally mounted transparent tube, in which there was a ‘trap’. Only if the poking was done from one end could the object be obtained. Half of the children saw the action performed perfectly, while the other half saw the model put the stick into the wrong end 1rst, then remove it, and poke from the other end (the same successful performance as shown to the other group). Interestingly, the children



who saw the incorrect sequence did not copy this; however, they did learn signi1cantly more quickly than those who saw only error-free demonstrations. Other examples stressing that goals play an important role in imitative behavior come from developmental psychologists interested in children’s understanding of the intentions of others (Carpenter, Akhtar, and Tomasello 1998; Meltzoff 1995). These experiments demonstrate that even very young children are capable of inferring goals from observed actions, and that inferred goals in2uence imitative behavior. Meltzoff (1995) compared 18-month-old children’s re-enactments of an attempted but failed action, with an attempted and achieved action, using 1ve unique test objects. For example, an adult experimenter moved a rectangular wooden stick toward a rectangular recessed button on a box, and either inserted the stick in the hole, activating a buzzer, or touched an adjacent area on the box, missing the hole and not activating the buzzer. When given the opportunity to manipulate the objects immediately after the adult’s demonstration, children shown an attempted but failed act were just as likely to perform the target act (e.g. inserting the stick in the hole and activating the buzzer) as children shown an attempted and achieved act. This result is especially surprising because children who had seen a failed attempt never actually saw the target act performed. Children in both groups performed the target act approximately four times as often as did children in control conditions. The fact that 18-month-olds imitated intended acts just as often as achieved acts suggests that even very young children infer the goals of others’ behaviors, and imitate those inferred goals. In a similar paradigm, Carpenter, Akhtar, and Tomasello (1998) compared 14- to 18-month-old children’s re-enactments of verbally marked intentional and non-intentional acts. An experimenter performed two unrelated actions on a unique test object, for instance, lifting the top of a bird feeder, and pulling a ring on a string attached to the feeder. These actions were accompanied by vocal exclamations marking each action as either an intended act (‘There!’) or an accidental act (‘Whoops!’), with some children seeing 1rst an intentional and then an accidental act, and others seeing them in the reversed order. After both actions had been performed, a salient event occurred (e.g. a party favor attached to the bird feeder moved and made a noise). Irrespective of the order of the modeled actions, children reproduced the intentional acts approximately twice as often as the non-intentional acts. Together these experiments suggest that imitation in children relies on the presence of unambiguous, observable goals, and, importantly, on inferences about the actor’s intentions of the observed act as well. Furthermore, strong support was found for the notion that these (inferences about) goals or intentions in2uence subsequent imitative behavior.

15.3.1.3 Goal-directed interference effects Traditionally, the importance of goals in action perception and/or imitation has been addressed in developmental psychology. However, as argued elsewhere (e.g. Wohlschläger et al., submitted), we have stressed the view that there is no fundamental reason to assume that children imitate fundamentally differently from adults. Rather, we have argued that children are ideal subjects to investigate the issue of goals because of their limited working-memory capacity. Thus, the fact that adults are able to imitate a contralateral hand movement to the left ear correctly does not mean that the goal of a left ear does not primarily activate the motor program belonging to an ipsilateral hand movement, as seen in children. To investigate this issue in more detail, we recently measured adults’ response latencies in the hand-to-ear task described above (Wohlschläger and Bekkering, submitted). The latency data in

307


308


adults showed the same pattern as the error data in children. That is, although adults made almost no errors, when comparing latencies, contralateral hand movements were clearly initiated later than ipsilateral ones. This result is in parallel with the contra–ipsi error in children. However, in order to make the point that the increased response latency was due to the presence of goals, we also replicated the dot experiment (Exp. 2 of Bekkering et al. 2000; see above) in adults. We asked adults to imitate ipsi- and contralateral 1nger movements presented on a screen. In one block of trials, the 1nger movements presented on the screen were directed towards two red dots. In the other block of trials, the same movements were shown, but now without the dots. The data showed that contralateral 1nger movements were more delayed than ipsilateral 1nger movements, but only if dots were present. Our interpretation of this 1nding is that the presence of dots activates the more ideomotor-like ipsilateral 1nger movement 1rst. The ideomotor theory of action (e.g. Greenwald 1970) states that the anticipatory representation of an action’s sensory feedback (a response image) is used for action control. In other words, the anticipation of the sensory consequences of an action (e.g. the tactile sensation of touching an ear or the visual sensation of covering a dot) is responsible for the action selection processes and favors the response most strongly associated with these sensory consequences. However, adults can inhibit these strong connections, in order to imitate more precisely. This inhibition is time-consuming, as re2ected by the increased response latencies (Wohlschläger and Bekkering, submitted). Stimuli depicting goals cause interference not only when subjects are asked to imitate but also if subjects are instructed to respond in a pre-speci1ed way (Stürmer, Aschersleben, and Prinz, 2000). In their set-up, participants had to either spread their 1ngers apart or make a grasping hand movement from a neutral middle starting position. The stimulus on the screen consisted of a sequence of pictures that showed a similar hand in the same neutral starting position. After a random time period, the hand either spread or it closed to a 1st, before returning to the neutral position again. The subject’s instruction for response selection was provided by a cue, such as the color of the hand. That is, subjects were instructed to make their response as soon as the color was added to the pictures (e.g. ‘Make a grasping movement if the stimulus turns red’). The imperative cue (that is, the color), was presented during different times of the hand movement sequence. The main 1nding was a typical Simon-like correspondence effect. That is, subjects responded faster if the observed hand movement was in correspondence with the instruction given by the color, than in the case of non-correspondence between observed and instructed hand movement. The results of a second experiment were particularly interesting with respect to our goal-directed theory of imitation. In this experiment, instead of dynamic hand movements the static images of the end positions (1st vs. spread 1ngers) were presented to the subjects. Now the compatibility effects were even stronger than those observed in the 1rst experiment. Thus, the 1ndings of Stürmer et al. (2000) suggest that the observation of a static hand posture interferes more with selection processes than seeing the whole hand movement, a 1nding that nicely agrees with the ideomotor theory (responses are selected by their intended effects or goals) mentioned above. In another series of experiments (Brass, Bekkering, Wohlschläger, and Prinz 2000), it was tested whether observed 1nger movements have a stronger in2uence on 1nger movement execution than a symbolic or spatial cue. In the 1rst experiment, we compared symbolic cues with observed 1nger movements using an interference paradigm. Observing 1nger movements strongly in2uenced movement execution, irrespective of whether the 1nger movement was the relevant or the irrelevant stimulus dimension. In the second experiment, effects of observed 1nger movements and spatial 1nger cues were compared. The observed 1nger movement dominated the spatial 1nger cue. A reduction in the



similarity of observed and executed action in the third experiment led to a decrease of the in2uence of observed 1nger movement, which demonstrates the crucial role of the imitative relation of observed and executed action for the described effects.

15.3.1.4 How do we perceive and arrange goals? Having said all this about the relevance of goals in imitation, we would like to 1nish this section with some critical notes about the notion of goal-directed imitation, the concept of goals in general, and some future work that has to be done to clarify some essential issues still to be answered. A major criticism of the theory could be that whatever aspects the imitator imitates from the model’s action, this will be viewed as being the goal of the action. Therefore, the theory is hard to falsify since one cannot individuate the goal hierarchy independently from what the imitator does in an imitation situation. Although this in fact is the basis of our theory—you do imitate what you perceive to be the goal of the model’s movement—we think that we have been able to 1nd some ways to deal with this criticism. First of all, in the dot experiment (Experiment 3 of Bekkering et al. 2000) with the kindergarten children, we were able to predict a change in the imitative behavior of the children by adding or removing goals in the modeled act. That is, children did or did not copy similar hand movements observed, depending on the presence of higher goals like objects, as proposed by the theory. Second, maybe even more convincingly, the response-time differences in the 1nger-imitation experiments of the Wohlschläger and Bekkering (2000) study, show that the observation of goals can in2uence motor programming although the 1nger movements to be programmed are identical. That is, although the adult subjects were able to imitate the observed contralateral 1nger movements, it took them more time if the observed movements on the screen were directed to the dots. Here, a nice dissociation between perceived action goals (a movement to a left dot activates a left ipsilateral 1nger movement) and the executed imitative act (the right contralateral 1nger movement) was arranged, while the presence of a higher goal was still re2ected in the response-time latencies. Third, the studies mentioned above of Brass et al. (2000) and those of Stürmer et al. (2000) have shown that the presence of action goals even outside the scope of imitation can interfere with action initiation. That is, the observation of 1nger movements or hand postures on a computer screen seems to directly in2uence the premotor system in activating another task. Neurophysiological support for a close coupling between the action observation system and the action execution system comes from a 1nding by Rizzolatti and colleagues. They observed single-cell activity in the rostral part of the inferior premotor cortex, area F5, of the monkey during goal-directed hand movements such as grasping, holding, and tearing, but also when the monkey only observed these actions performed by the experimenter (e.g. di Pellegrino, Fadiga, Fogassi, and Rizzolatti 1992). In most of these so-called mirror neurons, there needed to be a clear link between the effective observed movement and that executed by the monkey in order to 1nd discharges in the same neuron (e.g. Gallese, Fadiga, Fogassi, and Rizzolatti 1996), which led the authors to propose that these mirror neurons form a system for matching observation and execution of motor actions (for an overview, see Gallese, this volume). A recent study by Fadiga, Fogassi, Pavesi, and Rizzolatti (1995) took this notion one step further. The results of their transcranial magnetic stimulation experiment showed that the excitability of the motor system increased when an observer watched grasping movements performed by a model. Furthermore, the pattern of muscle activation evoked by the transcranial magnetic stimulation during action observation was very similar to the pattern of muscle contraction present during the execution of a similar action.

309


310


Importantly, the mirror neuron experiments showed that the activity of the F5 neurons is correlated with speci1c hand and mouth motor acts and not with the contractions of individual muscle groups. That is, typically, the neurons are only active when a goal, an object, is present and they stay silent when the same movement is performed without this goal. Also, these neurons are very sensitive to the treatment of the object by the effector. That is, one neuron might 1re when the object is picked up with a precision grip, while it stays silent during a full grip and vice versa. Recently, neurophysiological evidence for the link between ‘imitative’ effects as observed in the Brass et al. (2000) paper and the mirror neuron system were provided by an fMRI study by Iacoboni et al. (1999). Using the stimuli of Brass et al. they found that two distinct areas were more highly activated in an imitative 1nger movement task than in the symbolic 1nger movement task. These were, first, and most interestingly, Broca’s area 44, and also the areas PE/PC in the parietal lobe. The authors proposed that Broca’s area 44, which has been suggested to be the human homologue of monkey area F5, might describe the observed action in terms of its motor goal (e.g. lift the 1nger) without de1ning the precise details of the movement. In contrast, the parietal lobe might code the precise kinesthetic aspects of the movement, as suggested by Lacquaniti, Guigon, Bianchi, Ferriana, and Caminiti (1995).

15.4 Concluding remarks 15.4.1 The concept of ‘goals’ As has been reflected upon in this chapter, a widely used concept for explaining imitative behavior is to say that imitation is ‘goal-directed’. Interestingly, the different theories that use the concept of goals refer to totally different mechanisms. We would like to 1nish this tutorial with an overview of the goal concepts in the major theories, and a taxonomical suggestion is made to capture the different meanings of the word ‘goal’ in the domain of imitation. Per de1nition, the goal-directed theory of imitation is concentrated around the concept of goals. Here, goals typically refer to physical things such as dots and ears. That is, in the experiments mentioned before, imitators always move to the correct goal, such as an object or a particular ear to reach for. However, this theory also uses the concept of goals at another more functional level, as re2ected in the ideomotor principle. The ideomotor principle states that the selected physical goals elicit the motor program with which they are most strongly associated. In other words, the physical goals are represented in certain neural codes and these representations affect the selection and initiation processes of imitative actions. Thus, goals now refer to a functional mechanism necessary to initiate an imitative action. To make things even more complicated, and as mentioned before, Travis already has pointed out that a goal, strictly de1ned, is ‘a mental state representing a desired state of affairs in the world’ (1997, p. 115), and can therefore only be observed in the outcome of intentional actions. Identifying the goals of an observable action requires an inference beyond any mapping or parsing method as described previously. A good example of the role of mental goals can be found in this book in the chapter of Gallese and colleagues (Chapter 17). In agreement with earlier observations of Perrett’s group (see Jellema and Perrett, this volume, Chapter 18) in higher visual areas, mirror neuron cells were also found to be active when they either observe, or execute an action on an object that is hidden at the moment that the action is performed. In this case, the physical goal of the action is only mentally present. The importance and automatic in2uences of such intentional strategies on action



performance has recently received a great deal of attention in the 1eld of social cognition (see, for a review, Gollwitzer and Schaal 1998), but falls behind the scope of this tutorial. The newer version of the active intermodal mapping theory (Meltzoff and Moore 1997) also speaks about goal-directed acts. Here, the goal of an infant is to match their own body with the observed model’s body organ relations, which clearly refers to the functional action level of the goal concept. The emulation theory (Tomasello and colleagues) argues that one can only speak about imitation if not only the same behavioral repertoire but also the same goals are shared between model and observer. In this view, goals typically refer to physical things. The program-level and action-level imitation theory of Byrne and Russon argues that, to imitate, the individual must have a mental apparatus that allows assembling hierarchical frameworks, to organize the goal structure, and to maintain the goal structure while its detailed enactment is built. Here, again, the functional level of the goal concept is meant. To overcome misunderstandings about the different meanings of the word goal we propose the following taxonomy for the domain of imitation: • Physical goals refer to existing objects in the immediate surroundings; • Action goals refer to the functional mechanisms/neural processes that are induced by the physical goals, which are necessary to initiate an imitative action; • Mental goals refer to a desired state of affairs in the world.

15.4.2 A functional view on imitation Instead of arguing about what imitation is, and who can imitate, it might be more interesting to study the functional organization of action recognition and action execution processes per se. In our opinion, the key concept for understanding imitative behavior is that of action goals. As suggested in this tutorial, two sensorimotor mechanisms are involved in the execution and the perception of action goals in imitation, respectively. First, the ideomotor principle—selected physical goals elicit the motor program with which they are most strongly associated, elucidates how both the agent and the imitator translate action goals into the execution of movements. Second, the mirror neuron system—matching the observation of movements with the individual motor actions—gives details of how seeing goal-directed movements can give rise to the recognition of action goals. Thus, the recognition of action goals from the observation of movements and the execution of movements from having action goals can be conceptually clari1ed by proposing two inverted functional mechanisms. The mirror neuron system transforms movements perceived from others into the actor’s own possible action goals, while the ideomotor principle transforms the intended action goals of an actor into movement execution. To conclude, the functional model of imitation, as described here, stresses the importance of the personal action repertoire in both the perception and the execution of goaldirected imitative actions.

References Anisfeld, M. (1996). Only tongue protrusion modeling is matched by neonates. Developmental Review, 16, 149–161.

311


312


Bandura, A. and Mischel, W. (1965). Modi1cation of self-imposed delay of reward through exposure to live and symbolic models. Journal of Personality and Social Psychology, 2, 698–705. Bandura, A. and Walters, R.H. (1963). Social learning and personality development. New York: Holt, Rinehart and Wilson. Bauer, P.J. and Mandler, J.M. (1989). One thing follows another: Effects of temporal structure on 1- to 2-yearolds’ recall of events. Developmental Psychology, 25, 197–206. Bauer, P.J. and Travis, L.L. (1993). The fabric of an event: Different sources of temporal invariance differentially affect 24-month-olds’ recall. Cognitive Development, 8, 319–341. Bekkering, H. and Prinz, W. (in press). Goal representations in imitative actions. In K. Dautenhahn and C.L. Nehaniv (Eds.), Imitation in animals and artifacts. Cambridge, MA: MIT Press. Bekkering, H., Wohlschläger, A., and Gattis, M. (2000). Imitation of gestures in children is goal-directed. Quarterly Journal of Experimental Psychology. Section A: Human Experimental Psychology, 53A, 153–164. Brass, M., Bekkering, H., Wohlschläger, A., and Prinz, W. (2000). Compatibility between observed and executed 1nger movements: Comparing symbolic, spatial and imitative cues. Brain and Cognition, 44, 124–143. Byrne, R.W. (in press a). Imitation without intentionality: Using string parsing to copy the organization of behaviour. Animal Cognition. Byrne, R.W. (in press b). Seeing actions as hierarchically organized structures. Great-ape manual skills. In A. Meltzoff and W. Prinz (Eds.), The imitative mind: Development, evolution, and brain bases. Cambridge, UK: Cambridge University Press. Byrne, R.W. and Byrne, J.M.E. (1993). Complex leaf-gathering skills of mountain gorillas (Gorilla g. beringei): Variability and standardization. American Journal of Primatology, 31(4), 241–261. Byrne, R.W. and Russon, A.E. (1998). Learning by imitation: A hierarchical approach. Behavioral and Brain Sciences, 21, 667–684. Call, J. and Carpenter, M. (in press). Three sources of information in social learning. In K. Dautenhahn and C.L. Nehaniv (Eds.), Imitation in animals and artifacts. Cambridge, MA: MIT Press. Carpenter, M., Akhtar, N., and Tomasello, M. (1998). Fourteen- through 18-month-old infants differentially imitate intentional and accidental actions. Infant Behavior and Development, 21, 315–330. di Pellegrino, G., Fadiga, L., Fogassi, L., Gallese, V., and Rizzolatti, G. (1992). Understanding motor events: A neurophysiological study. Experimental Brain Research, 91, 176–180. Doupe, A.J. and Kuhl, P.K. (1999). Birdsong and human speech: Common themes and mechanisms. Annual Review of Neuroscience, 22, 567–631. Fadiga, L., Fogassi, L., Pavesi, G., and Rizzolatti, G. (1995). Motor facilitation during action observation: A magnetic study. Journal of Neurophysiology, 73, 2608–2611. Gallese, V., Fadiga, L., Fogassi, L., and Rizzolatti, G. (1996). Action recognition in the premotor cortex. Brain, 119, 593–609. Gattis, M., Bekkering, H., and Wohlschläger, A. (in press). Goal-directed imitation. In A. Meltzoff and W. Prinz (Eds.), The imitative mind: Development, evolution, and brain bases. Cambridge: Cambridge University Press. Gleißner, B., Meltzoff, A.N., and Bekkering, H. (2000). Children’s coding of human action: Cognitive factors in2uencing imitation in 3-year-olds. Developmental Science, 3, 405–414. Gleitman, H. (1981). Psychology. W.W. Norton. Gollwitzer, P.M. and Schaal, B. (1998). Metacognition in action: The importance of implementation intentions. Personality-and-Social-Psychology Review, 2, 124–136. Greenwald, A.G. (1970). Sensory feedback mechanism in performance control: With special reference to the ideomotor mechanism. Psychological Review, 77, 73–99. Huber, L. (1998). Movement imitation as faithful copying in the absence of insight. Behavioral and Brain Science, 21, 694. Iacoboni, M., Woods, R.P, Brass, M., Bekkering, H., Mazziotta, J.C., and Rizzolatti, G. (1999). Cortical mechanisms of human imitation, Science, 286(5449), 2526–2528. Köhler, W. (1976). The mentality of apes. (transl. by E. Winter). New York: Liveright. Kuhl, P.K. and Meltzoff, A.N. (1996). Infant vocalizations in response to speech: Vocal imitation and developmental change. Journal of the Acoustic Society of America, 100, 2425–2438. Lacquaniti, F., Guigon, E., Bianchi L., Ferriana, S., and Caminiti, R. (1995). Representing spatial information for limb movement: Role of area 5 in the monkey. Cerebellar Cortex, 5, 391–409. Lefkowitz, M.M., Blake R.R., and Mouton, J.S. (1955). Status factors in pedestrian violation of traf1c signals. Journal of Abnormal and Social Psychology, 51, 704–706.



Leiguarda, R.C. and Marsden, C.D. (2000). Limb apraxis: Higher-order disorders of sensorimotor integration. Brain, 123, 860–879. Lombardino, A.J. and Nottebohm, F. (2000). Age at deafening affects the stability of learned song in adult male zebra 1nches. Journal of Neurosciences, 20, 5054–5064. McKeon, R. (1941). The basic work of Aristotle. New York: Random House. Meltzoff, A.N. (1988). Infant imitation after a 1-week delay: Long-term memory for novel acts and multiple stimuli. Developmental Psychology, 24, 470–476. Meltzoff, A.N. (1995). Understanding the intentions of others: Re-enactment of intended acts by 18-month-oldchildren. Developmental Psychology, 31, 838–850. Meltzoff, A.N. and Moore, M.K. (1977). Imitation of facial and manual gestures by human neonates. Science, 198, 75–78. Meltzoff, A.N. and Moore, M.K. (1983). Newborn infants imitate adult facial gestures. Child Development, 54, 702–709. Meltzoff, A.N. and Moore, M.K. (1994). Imitation, memory, and the representation of persons. Infant Behavior and Development, 17, 83–99. Meltzoff, A.N. and Moore, M.K. (1997). Explaining facial imitation: A theoretical model. Early Development and Parenting, 6, 179–192. Meltzoff, A.N. and Moore, M.K. (1998). Infant intersubjectivity: Broadening the dialogue to include imitation, identity and intention. In S. Braten et al. (Eds.), Intersubjective communication and emotion in early ontogeny. Studies in emotion and social interaction (2nd series), pp. 47–62. New York: Cambridge University Press. Mineka, M.D., Cook, M., and Keir, R. (1984). Observational conditioning of snake fear in rhesus monkey. Journal of Abnormal Psychology, 93, 355–372. Nagell, K., Olguin, R.S., and Tomasello, M. (1993). Processes of social learning in the tool use of chimpanzees (pan troglodytes) and human children (Homo sapiens). Journal of Comparative Psychology, 107, 174–186. Nottebohm, F. (1991). Reassessing the mechanisms and origins of vocal learning in birds. Trends in Neurosciences, 14, 206–211. Piaget, J. (1975). Nachahmung, Spiel und Traum: Die Entwicklung der Symbolfunktion beim Kinde. Stuttgart: Ernst Klett. Poulson, C.L., Kymissis, E., Reeve, K.F., Andreators, M., and Reeve, L. (1991). Generalized vocal imitation in infants. Journal of Experimental Child Psychology, 51, 267–279. Rizzolatti, G., Fadiga, L., Gallese, V., and Fogassi, L. (1996). Premotor cortex and the recognition of motor actions. Cognitive Brain Research, 3, 131–141. Russon, A.E. (1996). Imitation in everyday use: Matching and rehearsal in the spontaneous imitation of rehabilitant orangutans (Pongo pygmaeus). In A.E. Russon, K.A. Bard, et al. (Eds.), Reaching into thought: The minds of the great apes, pp. 152–176. Cambridge, UK: Cambridge University Press. Russon, A.E. and Galdikas, B.M. (1993). Imitation in free-ranging rehabilitant orangutans (Pongo pygmaeus). Journal of Comparative Psychology, 107(2), 147–161. Savage-Rumbaugh, E.S. (1990). Language acquisition in a nonhuman species: Implications for the innateness debate. Developmental Psychobiology, 23(7), 599–620. Siegel, G.M., Cooper, M., Morgan, J.L., and Brenneise-Sarshad, R. (1990). Imitation of intonation by infants. Journal of Speech and Hearing Research, 33(1), 9–15. Spence, K.W. (1937). Experimental studies of learning and higher mental processes in infrahuman primates. Psychological Bulletin, 34, 806–850. Stürmer, B., Aschersleben, G., and Prinz, W. (2000). Correspondence effects with manual gestures and postures: A study of imitation. Journal of Experimental Psychology: Human Perception and Performance, 26(6), 1746–1759. Tchernichovski, O., Lints, T., Mitra, P.P., and Nottebohm, F. (1999). Vocal imitation in zebra 1nches is inversely related to model abundance. Proceedings of the National Academy of Sciences of the United States of America, 96, 12901–12904. Thorndike, E.L. (1898). Animal intelligence: An experimental study of the associative process in animals. Psychological Review Monograph, 2(8), 551–553. Tomasello, M. (1990). Cultural transmission in the tool use and communicatory signaling of chimpanzees? In S. Parker and K. Gibson (Eds.), Language and intelligence in monkeys and apes: Comparative developmental perspectives, pp. 274–311. Cambridge UK: Cambridge University Press.

313


314


Tomasello, M. (1992). The social bases of language acquisition. Social Development, 1, 67–87. Tomasello, M. (1996). Do apes ape? In C. Heyes and B. Galev (Eds.), Social learning in animals: The roots of culture, pp. 319–345. New York: Academic Press. Tomasello, M. and Call, J. (1997). Primate cognition. Oxford: Oxford University Press. Tomasello, M., Kruger, A.C., and Ratner, H.H. (1993a). Cultural learning. Behavioral and Brain Sciences, 16, 495–552. Tomasello, M., Savage-Rumbaugh, E.S., and Kruger, A.C. (1993b). Imitative learning of actions on objects by children, chimpanzees, and enculturated chimpanzees. Child Development, 64, 1688–1705. Travis, L.L. (1997). Goal-based organization of event memory in toddlers. In P.W. van den Broek, P.J. Bauer, and T. Bourg (Eds.), Developmental spans in event comprehension and representation: Bridging 1ctional and actual events, pp. 111–138. Mahwah, NJ: Erlbaum. Vygotsky, L.S. (1978). Prehistory of written speech. Soc. Sci. Inform., 17, 1–17. Want, S.C. and Harris, P.L. (1998). Indices of program-level comprehension. Behavioral and Brain Sciences, 21, 706. Whiten, A. (1998). Imitation of the sequential structures of actions by chimpanzees (Pan troglodytes). Journal of Comparative Psychology, 112, 270–281. Whiten, A. and Ham, R. (1992). On the nature and evolution of imitation in the animal kingdom: Reappraisal of a century of research. In P.J.B. Slater, J.S. Rosenblatt, C. Beer, and M. Milinski (Eds.), Advances in the study of behavior, pp. 239–283. San Diego, CA: Academic Press. Wohlschläger, A. and Bekkering, H. (submitted). Is human imitation based on a mirror-neuron system? Some behavioral evidence. Wohlschläger, A., Gattis, M., and Bekkering, H. (submitted). Mapping means or mapping ends? Towards a goal-directed theory of imitation. Manuscript submitted for publication.


16 Observing a human or a robotic hand grasping an object: differential motor priming effects Umberto Castiello, Dean Lusher, Morena Mari, Martin Edwards, and Glyn W. Humphreys

Abstract. The present paper aims to investigate how functional connections between action and perception may contribute to our imitation of motor acts of other beings. Four experiments examined motor priming effects on imitation from biological and nonbiological effector systems. In Experiment 1 we asked subjects 1rst to observe grasping movements performed by a human actor and by a robotic hand, and subsequently to perform the same movement. For 80% of the cases the movement performed by the primer and that performed by the subjects were directed to the same object (valid trials). For the remaining 20% of cases the subjects were required to perform an action towards an object that differed in size from the object grasped by the primer (invalid trials). Experiment 2 was similar to Experiment 1 except that the ratio between valid and invalid trials was 50%. We found priming effects con1ned to when a human actor served as the primer. In Experiment 3 we showed that the selective effects found for the human primer in Experiment 1 were unrelated to the fact that, in the robot condition, only an arm/hand system was visible while for the human actor both the face and the upper body were visible. Experiment 4 demonstrated some differences between the robot and a human primer even when the kinematics of the human primer did not change as a function of object size. The results demonstrate that motor priming can occur from the observation of biological action. The implications for understanding imitative behaviour are discussed in terms of differential levels of priming: some degree of unspeci1c priming (1rst level) seems to occur whenever the observer is exposed to a human versus a robot arm. There appears to be some conspeci1c advantage, which is completely unrelated to things like object size, trial type, or kinematics. More speci1c forms of priming (levels 2 and 3) appear to be fully dependent on model kinematics. Priming is only seen for the human hand when it operates naturally. It is not seen for the robot and it is not seen for the human hand when its kinematics do not differentiate between conditions.

16.1 Introduction The ability to imitate movements is of fundamental survival value for intelligent organisms, providing an important means of learning as well as a mechanism by which an individual may be accommodated within a group (for a review, see Bekkering and Wohlschläger, this volume, Chapter 15). The role of imitation in the development of human and other animals has long been documented, Darwin (Romanes and Darwin 1884), Thorndike (1898), and Piaget (1951) being three notable contributors to the literature. In recent years interest in the topic has been rekindled by new evidence emerging on the behavioural (for a review, see Prinz, in press) and physiological underpinnings of imitative behaviour (di Pellegrino, Fadiga, Fogassi, Gallese, and Rizzolatti 1992; Gallese, Fadiga, Fogassi, and Rizzolatti 1996; Rizzolatti and Arbib 1998).


316


In the following sections of this introduction we will brie2y review experimental paradigms applied to humans and primates to investigate the various aspects of imitative behaviour, which have led to the present series of experiments.

16.1.1 Behavioural experimental studies A number of paradigms have been implemented and a number of studies have been performed to investigate the mechanisms underlying imitation task performance (for a review, see Prinz, in press; Vogt, in press). For example, Kerzel, Bekkering, Wohlschläger, and Prinz (2000) developed the ‘movement reproduction’ paradigm to investigate the perception and reproduction of intentional actions. In particular, they asked participants to observe two consecutive object movements and then reproduce the 1rst of them as precisely as possible while ignoring the second. The basic paradigms consisted of two disks moving on a display. Participants were required to observe them and reproduce the velocity of the 1rst disk with a stylus movement on a graphic tablet. The results indicated that participants were able to reproduce velocity on the basis of the velocity of the 1rst disk, but, interestingly, the velocity of the second disk also in2uenced the reproduction pattern. Velocity reproduction for the 1rst disk tended to be faster if the velocity of the second disk was higher. While these results suggest a sharing of representational resources between movement perception and movement production, it is with the ‘movement selection’ paradigm that the issues about proper imitation may be better investigated. Stürmer, Aschersleben, and Prinz (2000) developed a paradigm based on gesture selection. This paradigm considered two gestures: hand spreading and hand grasping. In the 1rst case 1ngers were extended, and in the second case 1ngers were 2exed. Participants were required to perform one of these two hand movements as performed by an actor. An important feature for this task was that the identity of the stimulus gesture was irrelevant for the selection of the response gesture (to be performed by the subject). Instead, the relevant cue for the type of movement to be performed was the color of the stimulus hand i.e. red signi1ed 1nger extension; blue signi1ed 1nger 2exion). Similar to evidence on Stroop- and Simon-type compatibility effects, the authors found that the speed of responding was faster when there was correspondence between the stimulus and response gestures than when there was no correspondence. From this the authors argued that similar representational structures are involved in the perception and execution of actions (Prinz 1990, in press). Along the same lines, Brass, Bekkering, Wohlschläger, and Prinz (2000) developed the ‘effector selection paradigm’, where the gesture to be performed was 1xed (lifting of the 1nger) but there was a choice between two effectors to perform the gesture (index or middle 1nger). Two kinds of instructions were utilized, an imitative instruction where participants were required to lift the same 1nger as that lifted by a hand on a display and a symbolic instruction where participants were required to lift the same 1nger as indicated by a cross on the display. It was found that when the 1nger to imitate was the same as the 1nger to be lifted, as in the imitative cueing condition, response times were shorter than when actions were cued symbolically. The aforementioned studies suggest that there is a supramodal representational system, which matches the perceptual information of a seen act with proprioceptive information concerning an executable act. Bekkering, Gattis, and Wohlschläger (2000), however, have recently challenged this idea. In their experiments, preschool children were asked to imitate a set of hand gestures made by an actor positioned in front of them. The gestures varied in complexity. For example, a model touched the left and/or right ear(s) with one or both of the ipsilateral and/or contralateral hand(s). There were three ipsilateral hand movements (left hand to left ear, right hand to right ear, both hands


Observing a human or a robotic hand grasping an object: differential motor priming effects

to ipsilateral ears) and three contralateral hand movements (left hand to right ear, right hand to left ear, and both hands to contralateral ears). The results suggested that the children preferred to use the ipsilateral hand. However, when hand movements were made to only one ear this ipsilateral preference was not observed. Similarly, the ipsilateral preference was not evident when movements were directed at a space rather than a physical object. Their results supported a goal-directed imitation hypothesis where it is advanced that the desired goal of an imitative act is what is extracted from a model’s movement (Bekkering et al. 2000), not a speci1c priming of the effector corresponding to that used by the actor being observed.

16.1.2 Neurophysiological studies A number of physiological studies have also supported the notion that motor structures are involved in action perception as well as production—particularly those concerned with the so-called ‘mirror neurons’(e.g. Rizzolatti and Arbib 1998) in area F5 of the pre-frontal cortex of the macaque monkey (see Gallese et al., this volume, Chapter 17). These neurons are active not only when a monkey grasps and manipulates objects, but also when a monkey observes an experimenter performing a similar gesture. Moreover the cells do not discharge merely in response to object presentation, but rather they require a speci1c observed action in order to be triggered. The tuning of the neurons can also be quite speci1c, coding not only the action but also how it is executed. For example, a neuron might 1re during observation of grasp movements but only when the grasping action is performed with the index 1nger and thumb. Also, if the same grasp is performed using a tool, the neuron may no longer 1re. Neurons showing quite similar properties to those in area F5 have also been reported within the superior temporal sulcus (STS) by Perrett and colleagues (Oram and Perrett 1996; Perrett, Rolls, and Caan 1982; Perrett, Harris, Bevan, and Thomas 1989). For instance, in the lower bank of the STS, cells sensitive to actions of the hand were found. One apparent difference between the neurons in F5 and the STS is that neurons in the STS do not respond to executed motor acts but rather only to perceived ones. Evidence that a similar mirror system exists in humans comes from studies using transcranial magnetic stimulation (TMS) and functional brain imaging. Using TMS, Fadiga, Fogassi, Pavesi, and Rizzolatti (1995) demonstrated a selective increase in motor-evoked potentials when subjects observed various actions. This increase occurred in the muscles that the subjects would usually use for producing the actions they observed. In addition, two PET studies (Grafton, Arbib, Fadiga, and Rizzolatti 1996; Rizzolatti et al. 1996) have shown selective activation in the superior temporal sulcus, the inferior parietal lobule, and the inferior frontal gyrus of the left hemisphere when subjects observe a grasping action performed by an experimenter. Along these lines a recent study conducted using functional magnetic resonance techniques (fMRI; Iacoboni et al. 1999) con1rmed the activation of frontal and parietal areas during an imitation task. These results suggest that the brain may employ specialized circuitry to code stimuli for imitative actions. Furthermore, this circuitry seems responsive to immediate observation of biologically appropriate actions for the organism (e.g. a grasping movement performed by a member of the same species (a conspec)), but not to similar actions that are not biologically appropriate for the organism (e.g. grasping by a tool; Gallese et al. 1996). Whether the same mechanisms are used for longerterm recall and reproduction of action outside of the immediately observed context is less clear, though there is some suggestive evidence from functional brain imaging that similar brain areas

317


318


may be activated in action imitation after immediate observation and in longer-term recall (Decety and Grèzes 1999).

16.1.3 The present study The present study provides a novel and integral contribution to the already existing body of evidence regarding imitation. The issues we tackle in the present paper have never been considered in the previous literature on imitation. For example: does the observed action have to be part of the already existing behavioural repertoire of the observer in order to trigger mechanisms for imitation? We know from the neurophysiology of imitation (Gallese et al. 1996) and from developmental studies (Woodward 1998, 1999) that monkeys and children code and attend to grasping actions performed by a person but not necessarily to those performed by a mechanical device. In the 1rst case, a monkey’s mirror neurons are silent when the object is grasped with forceps or pliers (Gallese et al. 1996). In the second case, 1ndings indicate that six-month-old children selectively attend to an object grasped when a person, but not a mechanical claw, grasps the object (Woodward 1998, 1999). On the basis of these studies, we may expect that observation of an action performed by a conspec should have consequences on subsequent motor behaviour by a human subject. For example, detailed information about the kinematics of the observed action may be used to prime an action made at a later time by the observer, so that, for example, the action is parameterized on the basis of the previously observed action. Moreover, priming an action should occur over and above effects due to recall of the behaviour as previously performed by the observer (at least to the extent that recall might only partially engage the same specialized imitative circuitry). The detailed behavioural consequences of action observation on the kinematics of subsequent actions have not been examined hitherto. This was the aim of the present study. We examined whether there are priming effects produced by observing grasping by a human actor on the execution of a similar action by an observer. If this were the case, then the kinematics of actions to a target would vary according to whether the human model had grasped an object of the same or a different size. In addition, we contrasted the observation of an action performed by a conspec (another human) with the observation of an action performed by a non-conspec, a robot arm. Observation of a robot arm performing the reaching action allows us to investigate three relevant issues: (1) it provides a baseline for assessing whether either mere sight of a prime object of a particular size, or observation of a nonbiological grasping action, was suf1cient to generate action priming; (2) the use of a robotic arm allows also for a comparison between imitation for an action performed within the supposed behavioural repertoire of a normal person (performed by a conspec), and an action outside the normal behavioural repertoire (an action performed by the robot); (3) the study of priming from the robotic arm should match and extend the work with monkeys on neuronal activation associated with actions performed with hands and tools, and also the human developmental work on the same topic.

16.2 The experiments 16.2.1 Experiment 1: visuomotor priming: robotic versus human primer In this initial experiment we used an experimental paradigm based on visuomotor priming (Craighero, Fadiga, Rizzolatti, and Umiltà 1998). Subjects were asked to observe grasping movements performed by a human model or by a robotic hand to a target of one of two sizes. Immediately after this, the



subjects had to grasp a target object that could be the same size as the prime object, or a different (unprimed) size. In Experiment 1, the prime and target were likely to be the same size (on 80% of the trials), so that both the size of the prime and the movement made by the human model were informative of the probable action to the subsequent target. Kinematics of the reach-to-grasp action were recorded. The question addressed was whether effects of the movement performed by the human model or the robotic hand could be observed on the actions performed by the participants. Further, we asked whether such effects, if any, occurred over and above effects due to recall and/or prediction of the action from either the size of the primed object or the type of primer (human or robot).

16.2.1.1 Methods Participants. Eight subjects (4 women and 4 men, aged 20–25 years) volunteered to participate. All were right-handed, all reported normal or corrected-to-normal vision, and all were naive as to the purpose of the experiment. They attended two experimental sessions of 4 hours duration in total. Robot. The arm looked like an average human forearm with a gloved hand. It was mounted on a metal frame with a single motor that moves the arm from a vertical position to a horizontal position. The 4 1ngers and 1 thumb had a common movement so as to mimic the closing of a human hand. The construction was electro-mechanical and was controlled by an 87c751 micro-controller. The hand was constructed from nylon cords for tendons, silicon rubber for joints, and wooden dowels for bones. The movement was provided by a DC electric motor that tensed the tendons to close the hand. Springs were used to store energy and thus reduce the required power and size of the DC motors. Limit sensors on the arm and hand were used by the micro-controller to control movement. The arm length was approximately 0.5 m. The maximum pickup weight was approximately 0.1 kg. A feature of the robot is that it does not differentiate kinematics between large and small objects as humans do. This problem will be tackled in Experiment 4 where it is demonstrated that this difference is not relevant for interpretation of the data from the previous three experiments. Type of stimuli. The stimuli consisted of 2 spherical white foam objects (diameter: ~8 cm and ~2 cm) positioned at 30 cm distance along the mid-sagittal plane. Type of trials. There were two trial types. (1) A ‘valid’ trial, where a robotic arm or a human experimenter performed a reach-to-grasp action towards either the small or the large object and subsequently the subject grasped the same object. This occurred on 80% of the trials. (2) An ‘invalid’ trial, where the robotic arm or the human experimenter performed an action towards the small object and the subject grasped the large object or vice versa. This occurred on 20% of the trials. Apparatus. Re2ective passive markers (0.25 cm diameter) were attached to (a) the wrist, (b) the index 1nger, and (c) the thumb. Movements were recorded with the ELITE motion-analysis system. This consisted of two infrared cameras (sampling rate 100 Hz) inclined at an angle of 30° to the vertical and placed 2 m on the side of the table and 2 m apart. The spatial error measured from stationary and moving stimuli was 0.4 mm. Coordinates of the markers were reconstructed with an accuracy of 1/3000 of the 1eld of view and sent to a host computer. Visual availability of the stimuli was controlled with Plato spectacles (Plato Technologies Inc.). These were lightweight and were 1tted with liquid crystal lenses. The robotic arm or the experimenter was positioned at 90°, in front of the subject. The starting position of the robotic arm and the starting position of the experimenter’s arm were the same (see Fig. 16.1). The distance from the hand of the subject, the hand of the robot, and

319


320


Fig. 16.1 In panel (a) and panel (b) the position of the subject (wearing LCD glasses) and the position of the experimenter or the robot are represented. Panel (c) shows the position of the markers. Panel (d) represents the position of the infrared cameras.

the hand of the experimenter to the target was kept constant (~30 cm). The type of movement performed by the robotic arm and the experimenter differed in that, for the robot, the programmed accelerative and decelerative phases of the movement were similar for small and large objects alike. For humans, these phases differ (as demonstrated by our baseline data where kinematics for the human prime were measured). Both the opacity of the lenses and the initiation of movement by the robotic arm were controlled by the computer. At the beginning of each trial involving the use of the robotic arm, the experimenter at the computer console pressed the data acquisition button, the spectacles cleared, and the robotic arm started to move at a delay of 0.5 s from the opening of the glasses. When the experimenter performed the reach-to-grasp action, he/she started the movement as soon as he/she detected the clearance of the spectacles. Subjects wore earplugs to avoid noise produced by the experimenter during the re-positioning of the object after each trial. It was felt that such noise could provide information regarding the type of trial to be performed (valid or invalid). Procedure. Subjects were asked to perform the task as indicated by a tape-recorded set of instructions. The sequence of events was as follows: after the 1rst opening of the spectacles the subject had to observe the robotic arm or the human grasping the small or the large object. Then the spectacles were shut and when they re-opened the subject had to perform the grasping action towards the same object in the case of ‘valid’ trials or towards a different object in the case of ‘invalid’ trials. The target stayed on throughout the duration of the trial. The participants performed 400 randomized trials (100 per block) over which all possible types of trial/target-size/human–robot combinations occurred. Twenty trials for each combination were subsequently analysed.



Data processing. The ELIGRASP (B|T|S|, 1997) software package was used to analyse the data. This provided a three-dimensional reconstruction of the marker positions. The data were then 1ltered using a 1nite impulse response (FIR) linear 1lter-transition band of 1 Hz (sharpening variable = 2; cut-off frequency = 10 Hz). The reaching component was assessed by analysing the acceleration and the velocity pro1le of the wrist marker. The grasp component was assessed by analysing the distance between the two hand markers. Movement duration was taken as the time between movement onset and the end of the action (when the target was touched). The period following this, in which the target was lifted, was not assessed. Analysis of the reaching component of the action was based on the time of peak acceleration, velocity, and acceleration and the time from peak velocity to the end of the movement (the deceleration time). For the grasp component, the time to reach maximum peak grip aperture, the amplitude of the peak grip aperture, the peak velocity for the opening phase of the 1nger movements, and the time from maximum grip aperture to the end of the movement (the closing time) were analysed. Measurements were also taken for the opening phase of the hand movement in relation to the maximum velocity of the movement and the time at which each occurred. The measurement of the maximum grip aperture was based on the greatest distance reached between the thumb and the index 1nger, and the time of its occurrence. A prolonged deceleration time, and a lower amplitude of peak velocity for the reaching component of a grasp action, for smaller relative to larger stimuli are consistently reported in the reach-to-grasp literature (Castiello 1996; Gentilucci et al. 1991; Jakobson and Goodale 1992; Marteniuk, Leavitt, MacKenzie, and Athenes 1990). Differences in deceleration time and peak velocity should therefore be expected here as a function of the size of the target objects, and these differences are a necessary precondition for tests of priming. For the grasp component, we expect there to be a reduced maximum grip aperture for the smaller of the two stimuli, and the maximum grip aperture to be formed earlier in time (Castiello 1996; Gentilucci et al. 1991; Jakobson and Goodale 1992; Marteniuk et al. 1990). In addition we analysed the peak velocity of the 1ngers as they opened for the grip and the time taken to close the grip on the object, because (a) previous results have demonstrated differences in the rate of 1nger opening as a function of target size (Bon1glioli and Castiello 1998), and (b) closing time provides an index that is sensitive to reach-to-grasp strategies (Hoff and Arbib 1993). For each dependent variable an analysis of variance (ANOVA) was performed with type of primer (human, robot), type of trial (valid, invalid), and object size (small, large) as the within-subjects factors. Post-hoc comparisons were conducted on the means of interest using the Newman–Keuls procedure (alpha level: .05).

16.2.1.2 Results In this and the following experiments priming can be observed at three levels of speci1city:

• First level: by effector. In the Results sections this refers to the main effect of robotic versus human primer.

• Second level: by effector and object size (small vs. large object). In the Results sections this refers to the two-way interaction between target size and type of primer (robotic vs. human primer).

• Third level: by effector and trial type. This refers to the two-way interaction between type of primer and trial type and the three-way interaction between type of prime, type of trial, and size.

321


322


16.2.1.3 Effects of size Consistent with previous results within the reach-to-grasp literature, we found a longer movement duration [873 vs. 845 ms; F(1, 7) = 14.67, p = 0.001], a prolonged deceleration time [457 vs. 433 ms; F(1, 7) = 4.94, p = 0.05], and a lower peak velocity amplitude [F(1, 7) = 8.10, p = 0.01] for smaller than for larger stimuli (Castiello 1996; Gentilucci et al. 1991; Jakobson and Goodale 1992; Marteniuk et al. 1990). For the grasp component the maximum grip aperture occurred earlier in time [525 vs. 548 ms; F(1, 7) = 9.36, p = 0.01] and it was reduced in size for smaller relative to larger stimuli [70 vs. 84 mm; F(1, 7) = 17.32, p = 0.001] (Castiello, 1996; Gentilucci et al. 1991; Jakobson and Goodale 1992; Marteniuk et al. 1990). 16.2.1.4 First-level priming effects This section refers to the main effect of the type of primer (robot vs. human). Several parameters of the movements differed if the primer was the robot arm rather than the human model. In essence this was because subjects tended to adopt the parameters for responding to the large object when the primer was the robot, irrespective of the actual size of the prime. Thus, for the reaching component, the time to reach peak acceleration [236 vs. 257 ms; F(1, 7) = 28.16, p = 0.001], peak velocity [403 vs. 430 ms; F(1, 7) = 11.11, p = 0.01], and peak deceleration [570 vs. 587 ms; F(1, 7) = 8.44, p = 0.01] were decreased for trials where the robot arm was the primer rather than the human. For the grasp component there were differences in the accelerative phase as the 1ngers moved to their maximum aperture. The time to maximum acceleration of the 1ngers occurred earlier [83 vs. 97 ms; F(1, 7) = 9.76, p = 0.01] and the maximum acceleration was greater [2159 vs. 2063 mm/s 2; F(1, 7) = 17.21, p = 0.001] following robotic rather than human primers. 16.2.1.5 Second-level priming effects This section considers the interaction between target size and type of primer. We found differences in the movement parameters between human and robot primers as a function of the size of the target object. In each case, the parameters were close to those found for the large target object, on trials where the primer was a robot (for an example, see Fig. 16.2(a)). In contrast, on human primer trials, there was an effect of the size of the target object. For small relative to large stimuli there was a longer movement duration [F(1, 7) = 20.13, p = 0.001], a prolonged deceleration time [F(1, 7) = 27.02, p = 0.0001], a shorter time to achieve the maximum grip aperture [F(1, 7) = 7.47, p = 0.05], and a lowered size of maximum grip aperture [F(1, 7) = 11.06, p = 0.001]. The results for this section are rather important because they demonstrate that primed movement kinematics can in2uence the execution of grasping movements. This is clearly shown by the contrasting data for the robot condition, where no effects of size were found (though remember that the movement of the robot was similar for different target sizes). 16.2.1.6 Third-level priming effects Priming effects were apparent in two-way interactions involving type of primer (robot, human) and type of trial (valid, invalid), and one 3-way interaction for deceleration time, involving type of primer (robot, human), type of trial (valid, invalid), and target size (small, large). Let us consider the two-way interactions 1rst. Reaching component. There was a (type of prime) × (type of trial) interaction for: the time to reach peak acceleration [F(1, 7) = 7.34, p = 0.01], the time to obtain peak velocity [F(1, 7) = 8.53, p < 0.01], the time to reach peak deceleration [F(1, 7) = 9.12, p = 0.01], and the deceleration time



Fig. 16.2 The two-way interaction (Type of Primer by Size) obtained for Expts. 1, 2, and 4 for the measure amplitude of maximum grip aperture.

itself [F(1, 7) = 6.53, p = 0.01]. For each of these parameters, the values for valid and invalid did not differ if the robot appeared on the priming trials. However, when the primer was human, then in all cases differences emerged between valid trials on the one hand, and invalid trials on the other (all p < 0.05, Newman–Keuls tests). Figure 16.3(a) represents an example of this patterning for the parameter deceleration time. With the human primer, deceleration time was reduced for valid but not for invalid trials. Grasp component. For the grasp component, the interaction between the type of primer and type of trial was signi1cant for the following parameters: time to maximum grip aperture [F(1, 7) = 10.21, p = 0.001], maximum grip aperture [F(1, 7) = 20.11, p = 0.001], time to peak acceleration of 1nger opening for grip [F(1, 7) = 8.82, p = 0.01], and closing time [F(1, 7) = 10.44, p = 0.001]. As for reaching, there were no differences between valid and invalid trials when the primer was the robot arm. With the human primer, the time to peak acceleration of the opening grip and the time to obtain

323



(a) 500

Deceleration time (ms)

324

(b) 500

Experiment 1

450

450

400

400

350

350

300

300

(c) 500

(d) 500

Experiment 3

450

450

400

400

350

350

Experiment 2 Robot Human

Experiment 4

300

300 Valid

Invalid

Valid

Invalid

Fig. 16.3 The two-way interaction (Type of Primer by Type of Trial) obtained for deceleration time for the four experiments.

Fig. 16.4 The three-way interaction (Type of Primer by Type of Trial by Size) obtained for Expts. 1 and 3 for deceleration time.



maximum grip aperture were longer on valid trials; in addition, the maximum grip aperture was smaller and the closing time shorter for valid trials. The three-way interaction between type of primer, type of trial, and object size was signi1cant for deceleration time [F(1, 7) = 13.12, p = 0.001]. As before, deceleration times did not vary as a function of trial type following robot primers. However, following human primers, deceleration times were slower for invalid relative to valid trials, but this only occurred when the target was large. The threeway interaction is illustrated in Fig. 16.4.

16.2.1.7 Discussion These results demonstrate that third-level priming effects occurred on trials with human primers but not on trials with robot primers. When the primer was a robot, we failed to 1nd any changes in movement kinematics as a function of whether trials were valid or invalid. There were also few differences between the kinematics of the responses to large and small targets, with the movement kinematics being set towards large targets (see Fig. 16.2(a)). These responses following robot primers can be understood if subjects adopted a relatively conservative response strategy on these trials, initiating their movement with parameters set for large targets. The data from trials with human primers indicate that, on invalid trials, it was easier to adapt an action parameterized for a large object (from a large prime to a small target) than an action parameterized for a small object (from a small prime to a large target). Thus, following human primers there were few costs on invalid trials for small targets, whilst there were reliable costs for large targets. When subjects make a grasp response to a target, any adjustment from a large to a smaller grasp response will match the natural pattern of action in which the 1nger and thumb reach a maximum grasp aperture that is wider than the target to be grasped and then close around the target under guidance from visual feedback. In contrast, adaptation of a grasp from small to large will operate against the usual patterns of adjustment during reaching for grasping, generating asymmetric costs on performance. With the robot primer, subjects seem to adopt a strategy of minimal adjustment, and so move in all cases (irrespective of the size of the prime object) from an initial parameterization favouring a large grasp. As we have noted, quite different results occurred on trials following observation of the robot and of a human movement. This is interesting because the effects of the prime were mostly to disrupt action. For example, on invalid trials deceleration times were slowed when the prime was small and the target large (Fig. 16.4(a)). It would appear that subjects adapted their behaviour to match the observed primer action even though this was not necessarily bene1cial to their performance. The degree to which this imitation effect is under strategic control was tested in Experiment 2 when we reduced the informativeness of the priming event by only having prime and target actions valid on half the trials. Strategic use of the priming event should be lessened under these conditions. The fact that we found third-level priming effects only with human and not robot primers indicates that priming was not due to subjects’ preprogramming actions based on (a) the size of the priming object (note that this was predictive of the size of the target), and (b) a memory of the action parameters used for the predicted target. If priming were due to either of these factors, then differences between valid and invalid trials should have occurred for robot primers as well as for human primers. 16.2.2 Experiment 2: testing the automaticity of the priming effects Consider a trial where the subject observes a primer action made to the small object. This event may lead subjects to set parameters for small grasp actions, even though a large target may subsequently

325


326


be presented. As a consequence of this, subjects show a cost effect when reaching to a subsequent target. This kind of adjustment could itself operate in one of two ways. One could be strategic, with subjects taking account of the transitionary changes in validity rather than the overall information carried by prime events. The other could be more automatic, based on some form of reinforcement learning operating on a trial-by-trial basis. Whichever is the case, the important point to stress is that similar effects were not found unless subjects observed another human performing the priming action, though they could have used the size of the prime object in a similar way. This raises the question of how strategic preprogramming is. To investigate this preprogramming issue we ran a further study with 50–50 valid to invalid contingencies. Under these conditions it should not be strategically bene1cial for subjects to preprogram the movement.

16.2.2.1 Methods Participants. Eight subjects (4 women and 4 men, aged 20–25 years) with the same characteristics as those in the previous experiment volunteered to participate. None of them had participated in the previous experiment. They attended one experimental session of 1 hour duration in total. Apparatus, materials, procedure and data processing. These were the same as for Experiment 1, except that the ratio between valid and invalid trials was 50%, and the number of performed trials was reduced from an overall value of 400 to 160.

16.2.2.2 Results Data were analysed as for Experiment 1. Effects of size. In contrast to the 1ndings for Experiment 1 we found that deceleration time and the amplitude of peak velocity were similar for smaller and larger stimuli. For the grasp component we found no differences for the time of maximum grip aperture and the amplitude of grip aperture between smaller and larger stimuli, and, for both stimuli, movement parameters seemed to be set for the larger object (cf. Table 16.1 and Fig. 16.2(b)). This suggests that a conservative response strategy was adopted when primes were not valid on a majority of trials.

Table 16.1 Movement duration and kinematic parameters of the subjects’ movements for Small and Large conditions for Exp. 2. SEM in parenthesis Small

Large

F

Sig.

Reaching component Deceleration time (ms) Amplitude peak velocity (mm/s)

339 (16) 696 (31)

353 (13) 707 (36)

2.15

ns

2.24

ns

Grasp component Time to maximum grip aperture (ms) Maximum grip aperture (mm)

495 (32) 91 (3)

515 (29) 88 (3)

3.53

ns

3.12

ns



First-level priming effects. As for Experiment 1, several parameters of the movements were accelerated after seeing the robotic rather than the human arm as the primer. Reaching component. For the reaching component, anticipation was evident for the time to peak acceleration [247 vs. 263 ms; F(1, 7) = 28.14, p = 0.0001], the time to peak velocity [398 vs. 416 ms; F(1, 7) = 20.16, p = 0.001], and the time to peak deceleration [536 vs. 559 ms; F(1, 7) = 10.08, p = 0.02]. Grasp component. For the grasp component it was the accelerative opening phase of the 1nger to a maximum that differed between the robotic and the human conditions. The time to peak gripopening acceleration was earlier [139 vs. 150 ms; F(1, 7) = 6.16, p = 0.05] and greater (2576 vs. 2342 mm/s2; F(1, 7) = 11.65, p = 0.001) for the robotic than for the human condition. Second-level priming effects. The interaction between type of primer (human, robot) and target size was not signi1cant for any of the dependent measures analysed. For example, as represented in Fig. 16.2(b), for both the human and robot primers no differences for the small and the large objects were found for the amplitude of maximum grip aperture. For both human and robot primers, subjects adopted movement parameters for a large object at the start of each trial, leading to a reduction in the overall effects of object size on reaching and grasping. Third-level priming effects. Despite the overall effects of object size being the same on target actions following both human and robot primers, there remained a differential priming effect. The two-way interaction between type of primer and type of trial was signi1cant for both reaching and grasping components measures. An example of this effect for deceleration time is shown in Fig. 16.3(b). Reaching component. For the reaching component the time to attain peak velocity occurred at the same time for valid and invalid trials when the robot was the primer. However, with a human primer, the peak velocity was attained earlier in time on invalid trials relative to valid trials [F(1, 7) = 7.05, p < 0.01]. The same held for the time to peak deceleration [F(1, 7) = 6.20, p = 0.05]. Grasp component. For the grasp component, the interaction between the type of primer and type of trial was signi1cant for the following parameters: time to maximum grip aperture [F(1, 7) = 10.48, p = 0.01] and the amplitude of maximum grip aperture [F(1, 7) = 6.66, p = 0.01]. When the robot was the primer, the time to maximum grip aperture, and the maximum aperture obtained, did not differ for valid and invalid trials. However, with a human primer the maximum grip aperture took place earlier in time on invalid trials than on valid trials. Post-hoc comparisons revealed that the differences between valid trials and invalid trials were signi1cant ps < 0.05.

16.2.2.3 Discussion When primes were not generally valid, subjects tended to adopt a conservative response strategy with human and robot primers alike, with responses tending to be set for the larger of the two target objects. Nevertheless, some third-level priming effects were apparent when the primer was human, whilst there was little differential effect of the robot primer. In particular, maximum grip aperture tended to be reduced, the time taken to reach this point was delayed, and the time to attain peak velocity for the reach component of the movement was more prolonged on valid relative to invalid trials. These results mimic the data from Experiment 1 and suggest some adjustment of the parameters of the movement on valid trials, particularly when the prime was small (with the conservative, ‘wide grip’ parameters being set less often).

327


328


16.2.3 Experiment 3: testing differences between robot and human primes: is how much of a body you can see important? The results from the previous two experiments suggest that there are differences in the different levels of movement priming between a robotic and a human hand. In the third experiment we investigated whether the priming effects found with the human primer in Experiments 1 and 2 were due to the fact that, in the robot condition, only a forearm/hand was visible while, for the human actor, many other cues were available (i.e. the face, the upper body). The motivation for these experiments comes from possible functional differences between, on the one hand, neurons within STS showing selective neuronal responses to the sight of actions of the hand (Perrett et al. 1989) and, on the other hand, neurons found in area F5 (Rizzolatti and Arbib 1998). This difference is that neurons in STS do not respond (as neurons in area F5 do) to executed motor acts, but rather only to perceived ones. Also, in studies of STS and mirror neurons by Perrett and colleagues and Rizzolatti and colleagues, similar to those of Experiments 1 and 2 of the present study, the entire body and face of the experimenter performing the action was visible. Thus it may be possible that our forearm/hand robot did not activate the neural system concerned with movement execution in order to generate motor priming effects in Experiments 1 and 2. To clarify this question we performed an experiment where only the forearm and the hand of the human actor were visible.

16.2.3.1 Methods Participants. Eight subjects (4 women, 4 men, aged 20–25 years), with the same characteristics as those who took part in the previous experiments, volunteered to participate. None of them had participated in the previous experiments. They attended two experimental sessions of 4 hours duration in total. Apparatus, materials, procedure, and data processing. These were the same as for Experiment 1, except that the experimenter was hidden behind a thick black net curtain and only the arm was visible to the subjects. Prime actions were valid on 80% of the trials.

16.2.3.2 Results Data were analysed as a means of comparison between Experiment 1 and Experiment 3. Experiment (1 vs. 3) was the between-subjects factor. Type of primer (human, robot), type of trial (valid, invalid), and object size (small, large) were the within-subjects factors. The four-way interaction between experiment, type of primer, type of trial, and object size was signi1cant for deceleration time [F(1, 7) = 8.54, p = 0.01] and the amplitude of peak velocity [F(1, 7) = 11,23, p = 0.001]. Deceleration time and the amplitude of peak velocity did not vary as a function of trial type and size following robot primes. However, following human primes, deceleration time was shorter and the amplitude of peak velocity was lower for invalid trials relative to valid trials. Further, deceleration time was shorter and the amplitude of peak velocity was lower for small invalid trials (see Fig. 16.4). 16.2.3.3 Discussion The present experiment was performed mainly because the STS region is activated by movements of various body parts (Perrett et al. 1989). In the earlier experiments here, more body information was visible for the human primer than for the robot primer, and this may have in2uenced the subjects’ performance. However, the data from Experiment 3 con1rm the reliability of the present



effects, and con1rm that the effects are not dependent on subjects seeing more than the arm of the human primer. Priming effects at all levels were obtained from the sight of a human arm reaching and grasping an object. With a robot primer, subjects again tended to adopt a conservative response strategy suitable for the larger object, irrespective of whether the target was large or small, and irrespective of the size of the prime object. Sight of the human model body is not necessary for priming to occur.

16.2.4 Experiment 4: testing the differences in kinematics between robot and human primers The contrasting results from the human and robot primers in the earlier experiments could be due to the differences between a conspec and a robot as a primer, or to the fact that the kinematics for the robot did not differentiate between large and small objects. In Experiment 4 we clarify this point, normalizing the kinematics for a human prime with respect to object size. We asked a naïve human primer to perform the movement blindfolded within an allotted time. These constraints were felt suf1cient to normalize the movement of the human primer with respect to object size. We evaluated whether this normalized movement, made by a human primer, was now equivalent to the robot primer.

16.2.4.1 Methods Participants. Six subjects (4 women, 2 men, aged 25–30 years) with the same characteristics as those who took part in the previous experiments volunteered to participate. None of them had participated in the previous experiments. They attended one experimental session of 2 hours duration in total. Apparatus, materials, procedure, and data processing. These were the same as for Experiment 1, except that for this experiment the naïve subject was asked to be the human primer. The human primer was trained to reach for the small and the large object within an interval corresponding to the time employed by the robotic hand to complete the movement (800 ms; ± 25 ms). The interval was de1ned by two sounds (200 ms sound duration; 880 Hz). This interval was the same for the small and the large object. Further, the human primer was blindfolded so he could not see his arm while reaching for the object. In other words, he did not know the size of the object he was grasping. This led to the primer using a movement patterning that was very similar to that for the robot. That is, the hand opened widely and closed on the object only after having touched it with the palm. During the experimental session the subject wore earphones so as not to hear the two sounds when the ‘instructed’ primer was demonstrating the movement. Prime actions were valid on 80% of the trials.

16.2.4.2 Results Of relevance for the present study is that the normalization procedure was successful. Kinematics analyses showed that there were no differences for movements directed to the small and the large object for the ‘instructed’ primer (see Table 16.2, where the values are presented as a percentage of movement duration). Thus, in this study, the human primer acted similarly to the robot primer. As found for Experiment 2 there was a tendency for the subject to program movement in terms of the larger object (see Fig. 16.2(c)). Data were analysed as for Experiment 1 and are summarized as follows. Several parameters of the movements differed if the primer was the robot arm relative to the human model (1rst-level priming effects). Thus, for the reaching component, the times to reach peak acceleration [242 vs. 265 ms;

329


330


Table 16.2 Kinematic parameters of the subjects’ and the human primer’s movements for Small and Large conditions for Exp. 4. Temporal measures are expressed as a percentage of movement duration. SEM in parenthesis Subjects

Human primer

Statistics

Small

Large

Small

Large

F

Sig.

Reaching component Deceleration time (%) Amplitude peak velocity (mm/s)

57 (7) 549 (6)

56 (7) 544 (9)

56 (6) 552 (7)

56 (5) 550 (6)

0.21

ns

1.04

ns

Grasp component Time to maximum grip aperture (%) Maximum grip aperture (mm)

48 (3) 96 (3)

48 (3) 97 (4)

48 (4) 97 (4)

49 (5) 97 (4)

2.12

ns

1.43

ns

F(1, 5) = 17.16, p = 0.001] and peak velocity [421 vs. 446 ms; F(1, 5) = 5.01, p = 0.05] were decreased for trials where the robot arm was the primer rather than the human. For the grasp component there were differences in the accelerative phase as the 1ngers moved to their maximum aperture. The time to maximum acceleration of the 1ngers occurred earlier [100 vs. 115 ms; F(1, 5) = 4.86, p = 0.05] following robotic rather than human primers. In contrast to Experiment 1, second-level priming effects were not found (the interaction between type of primer and size was not signi1cant for any of the dependent measures). These results further suggest that the actions of the ‘normalized’ human primer were interpreted independently of the size of the object. Priming effects were not apparent in the two-way interactions involving type of primer and type of trial (second-level priming effects). Further, no third-level priming effects were found. The interaction type of primer × type of trial was not signi1cant for any of the dependent measures (e.g. Fig. 16.3(d)). The values for valid and invalid trials did not differ for both the robot and the human primes. The data from Experiment 4 con1rm the reliability of the differences between the robot and the human primer for the reach component of performance. We also failed to 1nd evidence of the size of the prime object on movements to the target. This is not surprising, however, given that the kinematics of the human primer did not differ to large and small objects. Despite this, kinematics were delayed following the human primer relative to the robotic primer, as we found in the earlier studies.

16.3 General discussion We have reported four experiments showing that priming effects at all levels seem to occur whenever the observer is exposed to a human versus a robot arm. First, there appears to be some conspec advantage, which is completely unrelated to things like object size, trial type, or kinematics. Second, the more speci1c forms of priming (level 2 and 3) appear to be fully dependent on model kinematics. In Experiments 1 and 4, speci1c priming is only seen for the human hand when it operates naturally.



It is not seen for the robot and it is not seen for the human hand when its kinematics do not differentiate between the two conditions. At 1rst glance it may be argued that the different results obtained for the human model and the robot hand are present because the robot’s hand kinematics do not differentiate between large and small objects. In other words, the whole pattern of results can be interpreted in such a way that the kinematics of the movement prime the action irrespective of whether the kinematics are shown by a human or a robot. However, we clari1ed this issue in Experiment 4, where a human model was constrained to perform the same kinematics for both the large and the small objects. The results for this experiment con1rm those of the other experiments, suggesting that a robot arm is perceived in a way that is different from a human arm (since the reach components of action remained selectively accelerated for robot primers). Consistent with this, studies of functional imaging in humans have found no evidence for either premotor or frontal activation when movements of a hand have been observed in a virtual reality system (Decety et al. 1994). The robot hand here, and the virtual hand in Decety et al. (1994) seem not to engage cells which mediate immediate visually guided action (see also Gallese et al., this volume, Chapter 17). This lack of engagement with a robot arm is particularly evident when looking at the relationship between the type of primer and the type of trials. In the human condition, subjects appeared to preprogram a response based on the prime, and then use this to guide action. As a consequence, on invalid trials they had to amend their motor program to respond to the properties of the stimulus. This leads to acceleration of the action, that is, the accelerative part of both the reaching and the grasp components is anticipated. The effect of the human primer action on deceleration time was longer for ‘invalid’ than for ‘valid’ trials when a large stimulus was presented (third-level priming effects). In contrast, for the robot trials subjects appeared not to preprogram the movement for the ‘valid’ condition, so that no ‘invalid’ effect is noticed. Again, a possible explanation is that the robot’s kinematics are similar for both the small and the large objects, thus the priming effect is not evident because subjects are coding for similar types of actions rather than for different types of objects. Another possible explanation for the present results is to consider the robot as a control condition with kinematics held constant. If viewed in this way, the results from Experiment 1 could be taken to show that kinematics matter. This point is also con1rmed on the basis of the results of Experiment 4 where the kinematics of the human primer did not differ for large and small objects. The conclusion that kinematics is relevant (at least as regards level-2 and level-3 priming) is not only supported by comparing the natural human arm (Experiment 1) with the robot arm (Experiments 1 and 4), but also by comparing it with a nonnaturally moving human arm (Experiment 4). This indicates, in line with the results obtained by Kerzel et al. (2000) and Stürmer et al. (2000), not only that participants were able to reproduce the actor’s pattern, showing correspondence between the stimulus (actor’s movement) and response (observer’s movement) gestures, but also that this similar representational system not only matches the perceptual information of a seen act with proprioceptive information concerning an executable act but also takes movement kinematics into account. Recently, Bekkering and colleagues (2000) postulated a new view on the representations that mediate perception and action in imitation. They suggest a motor-pattern process that is guided by an interpretation of the motor pattern as a goal-directed behaviour. The present results do indicate that the desired goal of the action—for instance, grasping an object—can be preprogrammed from observation of a prior action. Further, it is shown that this computation is chie2y driven by primed movement kinematics rather than by object size. We believe that this is one of the most interesting

331


332


of the present results. It shows that not only target size, as has been shown many times in the literature, but also primed movement kinematics can in2uence the execution of grasping movements. Also, since both the robot and the constrained human primer did not elicit priming effects, it appears that the source of priming was not the perceived size of the prime (from which the predicted size, and associated movement patterns for the target, could be generated). In conclusion, we have demonstrated motor priming effects when human subjects see an action to an object by a human primer. There are also general differences in reach kinematics after observing a human relative to a robotic primer, even when the speci1c grasp components are not primed. We speculate that the neural basis for these priming effects may reside in the specialized circuitry revealed by physiological and functional imaging studies in the superior temporal, inferior parietal, and inferior frontal lobes.

Acknowledgments This work was supported by an NHMRC and by a Wellcome Trust grant to UC. Morena Mari was supported by an NHMRC grant to UC. Professor Glyn Humphreys was supported by a visiting scholar grant awarded to UC by the University of Melbourne, and by grants from the MRC and the Wellcome Trust. Bruce Ferabend is thanked for assembling the robot used in the present study. We would like to thank Gisa Aschersleben, Harold Bekkering, and Wolfgang Prinz for their comments on previous versions of this manuscript.

References Bekkering, H., Gattis, M., and Wohlschläger, A. (2000). Imitation of gestures in children is goal-directed. Quarterly Journal of Experimental Psychology, 53A, 153–164. Bon1glioli, C. and Castiello, U. (1998). Dissociation of covert and overt spatial attention during prehension movements: Selective interference effects. Perception and Psychophysics, 60, 1426–1440. Brass, M., Bekkering, H., Wohlschläger, A., and Prinz, W. (2000). Compatibility between observed and executed 1nger movements: Comparing symbolic, spatial, and imitative cues. Brain and Cognition, 44, 124–143. Castiello, U. (1996). Grasping a fruit: Selection for action. Journal of Experimental Psychology: Human Perception and Performance, 22(3), 582–603. Craighero, L., Fadiga, L., Rizzolatti, G., and Umiltà, C. (1998). Visuomotor priming. Visual Cognition, 5, 109–125. Decety, J. and Grèzes, J. (1999). Neural mechanisms subserving the perception of human actions. Trends in Cognitive Sciences, 3, 172–178. Decety, J., Perani, D., Jeannerod, M., Bettinardi, V., Tadary, B., Woods, R., Mazziotta, J.C., and Fazio, F. (1994). Mapping motor representations with positron emission tomography. Nature, 371, 600–602. di Pellegrino, G., Fadiga, L., Fogassi, L., Gallese, V., and Rizzolatti, G. (1992). Understanding motor events: A neurophysiological study. Experimental Brain Research, 91, 176–180. Fadiga, L., Fogassi, L., Pavesi, G., and Rizzolatti, G. (1995). Motor facilitation during action observation: A magnetic stimulation study. Journal of Neurophysiology, 73, 2608–2611. Gallese, V., Fadiga, L., Fogassi, L., and Rizzolatti, G. (1996). Action recognition in the premotor cortex. Brain, 119, 593–609. Gentilucci, M., Castiello, U., Corradini, M.L., Scarpa, M., Umiltà, C., and Rizzolatti, G. (1991). In2uence of different types of grasping on the transport component of prehension movements. Neuropsychologia, 29, 361–378. Grafton, S.T., Arbib, M.A., Fadiga, L., and Rizzolatti, G. (1996). Localisation of grasp representations in humans by PET: 2. Observation versus imagination. Experimental Brain Research, 111, 103–111. Hoff, B. and Arbib, M.A. (1993). Models of trajectory formation and temporal interaction of reach to grasp. Journal of Motor Behavior, 25, 175–192.



Iacoboni, M., Woods, R.P., Brass, M., Bekkering, H., Mazziotta, J., and Rizzolatti, G. (1999). Cortical mechanisms of human imitation. Science, 286, 2526–2528. Jakobson, L.S. and Goodale, M.A. (1992). Factors affecting higher-order movement planning: A kinematic analysis of human prehension. Experimental Brain Research, 86, 199–208. Kerzel, D., Bekkering, H., Wohlschläger, A., and Prinz, W. (2000). Launching the effect: Representations of causal movements are in2uenced by what they lead to. Quarterly Journal of Experimental Psychology, Section A: Human Psychology, 53, 1163–1185. Marteniuk, R.G., Leavitt, J.L., MacKenzie, C.L., and Athenes, S. (1990). Functional relationships between the grasp and transport components in a prehension task. Human Movement Science, 9, 149–176. Oram, M.W. and Perrett, D.I. (1996). Integration of form and motion in the anterior superior polysensory area (STPa) of the macaque monkey. Journal of Neurophysiology, 76, 109–129. Perrett, D.I., Rolls, E.T., and Caan, W. (1982). Visual neurones responsive to faces in the monkey temporal cortex. Experimental Brain Research, 47, 329–342. Perrett, D.I., Harris, M.H., Bevan, R., and Thomas, S. (1989). Framework of analysis for the neural representation of animate objects and actions. Journal of Experimental Biology, 146, 87–113. Piaget, J. (1951). Play, dreams, and imitation in childhood. W.W. Norton. Prinz, W. (1990). A common-coding approach to perception and action. In O. Neumann and W. Prinz (Eds.), Relationships between perception and action: Current approaches, pp. 167–201. Berlin, New York: Springer-Verlag. Prinz, W. (in press). Experimental approaches to imitation. In A. Meltzoff and W. Prinz (Eds.), The imitative mind: Development, evolution, and brain bases. Cambridge: Cambridge University Press. Rizzolatti, G. and Arbib, M. (1998). Language within our grasp. Trends in Neurosciences, 21, 188–194. Rizzolatti, G., Fadiga, L., Matelli, M., Bettinardi, V., Perani, D., and Fazio, F. (1996). Localization of cortical areas responsive to the observation of hand grasping movements in humans: a PET study. Experimental Brain Research, 111, 246–256. Romanes, G.J. and Darwin, C. (1884). Mental evolution in animals. Appleton and Co. Stürmer, B., Aschersleben, G., and Prinz, W. (2000). Correspondence effects with manual gestures and postures: A study of imitation. Journal of Experimental Psychology: Human Perception and Performance, 26, 1746–1759. Thorndike, E.L. (1898). Animal intelligence: An experimental study of the associative process in animals. Psychological Review Monograph, 2, 551–553. Vogt, S. (in press). Visuomotor couplings in object-oriented and imitative actions. In A. Meltzoff and W. Prinz (Eds.), The imitative mind: Development, evolution and brain bases. Cambridge: Cambridge University Press. Woodward, A.L. (1998). Infants selectively encode the goal object of an actor’s reach. Cognition, 69, 1–34. Woodward, A.L. (1999). Infants’ ability to distinguish between purposeful and nonpurposeful behaviors. Infant Behavior and Development, 22(2), 145–160.

333


17 Action representation and the inferior parietal lobule Vittorio Gallese, Luciano Fadiga, Leonardo Fogassi, and Giacomo Rizzolatti

Abstract. From birth onwards, primates’ cognitive development depends heavily on being able to observe and interact with other individuals. How can individuals assign a meaning to the actions performed by other conspeci1cs? A possible neural correlate of the mechanism allowing action understanding could be represented by a class of neurons (mirror neurons) that we have discovered in area F5 of the ventral premotor cortex of the macaque monkey. We proposed that mirror neurons could be part of a cortical system that, by matching action observation with action execution, enables individuals to ‘understand’ the behavior of others. The present study is aimed at better clarification of the nature and the properties of such a cortical matching system. Neurons responding to the observation of complex actions have been described by Perrett and co-workers in the cortex buried within the superior temporal sulcus (STS). These neurons could be a particularly well-suited source of visual input for F5 mirror neurons. However, area F5 does not receive direct projections from the STS region. One of its major inputs comes from the inferior parietal lobule and in particular from area PF (7b). The inferior parietal lobule, in turn, is reciprocally connected with the STS region. We therefore decided to study the functional properties of area PF by means of single neuron recording experiments. About 20% of the recorded neurons responded both during action execution and action observation, and therefore, in analogy with the neurons described in area F5, we de1ned them as ‘PF mirror neurons’. Furthermore, a subset of PF mirror neurons matched hand action observation to mouth action execution. A possible hypothesis is that this latter class of PF neurons may represent a ‘primitive’ matching system. Taken together, these data indicate that an action observation/execution matching system does also exist in the parietal cortex, possibly constituting a building block of a cortical network for action understanding.

17.1 Introduction Primates are social animals. Their societies are characterized by complex and sophisticated rules disciplining the various types of interactions entertained by the single individuals within their group. This requires the capacity to recognize individuals within a social group, to assign a social rank to oneself as well as to others, and to be able to comply with the rules that such a complex hierarchical social environment entails. From birth onwards, primates’ cognitive development depends heavily on being able to observe and interact with other individuals. Action observation appears therefore to be very important in order to build a meaningful account of conspeci1cs’ behavior. How can individuals assign a meaning to the actions performed by other conspeci1cs? A possible neural correlate of the mechanism allowing action understanding could be represented by a class of neurons (mirror neurons) that we have discovered in area F5 of the ventral premotor cortex of the macaque monkey (di Pellegrino, Fadiga, Fogassi, Gallese, and Rizzolatti 1992; Gallese, Fadiga, Fogassi, and Rizzolatti 1996; Rizzolatti, Fadiga, Gallese, and Fogassi 1996a). Mirror neurons are activated during the execution of purposeful, goal-related hand movements, such as grasping, holding, or manipulating objects, and they also discharge when the monkey observes similar


Action representation and the inferior parietal lobule

hand actions performed by another individual. Mirror neurons require, in order to be activated by visual stimuli, an interaction between the action’s agent (human being or a monkey) and the object. Control experiments showed that neither the sight of the agent alone nor of the object alone were effective in evoking the neuron’s response. Similarly, mimicking the action without a target object or performing the action by using tools were poorly effective (Gallese et al. 1996). Frequently, a strict congruence was observed between the observed action effective in triggering mirror neurons and the effective executed action. In one-third of the recorded neurons the effective observed and executed actions corresponded in terms of both the general action (e.g. grasping) and in the way in which that action was executed (e.g. precision grip). In the other two-thirds only a general congruence was found (e.g. any kind of observed and executed grasping elicited the neuron’s response). We proposed that mirror neurons could be part of a cortical system that, by matching action observation with action execution, enables individuals to ‘understand’ the behavior of others (Gallese et al. 1996; Rizzolatti et al. 1996a). It must be stressed that several studies that used different methodologies have demonstrated the existence of a similar matching system in humans (see Cochin, Barthelemy, Lejeune, Roux, and Martineau 1998; Fadiga, Fogassi, Pavesi, and Rizzolatti 1995; Grafton, Arbib, Fadiga, and Rizzolatti 1996; Grèzes, Costes, and Decety 1998; Hari et al. 1999; Iacoboni et al. 1999; Rizzolatti et al. 1996b). All these studies suggested that humans have a ‘mirror matching system’ similar to that originally discovered in monkeys. Whenever we are looking at someone performing an action, beside the activation of various visual areas, there is a concurrent activation of the motor circuits that are recruited when we ourselves perform that action. Although we do not overtly reproduce the observed action, nevertheless our motor system becomes active as if we were executing the very same action that we are observing. The present study was aimed at better clarification of the nature and the properties of such a cortical matching system in the monkey brain. Neurons responding to the observation of complex actions, such as grasping or manipulating objects, have been described by Perrett and co-workers in the cortex buried within the Superior Temporal Sulcus (STSa, see also Jellema and Perrett, this volume). These neurons, whose visual properties are for many aspects similar to those of mirror neurons, could constitute mirror neurons’ source of visual information. The STS region, however, has no direct connection with area F5, but has links with the anterior part of the inferior parietal lobule (area PF or 7b), which in turn is reciprocally connected with area F5 (Matelli, Camarda, Glickstein, and Rizzolatti 1986; see also Rizzolatti, Luppino, and Matelli 1998). Area PF, or 7b, is located on the convexity of the inferior parietal lobule. It receives inputs from primary sensory areas (mostly area 2) and the second somatosensory area (area SII). It projects caudally to the adjacent areas located on the convexity of the inferior parietal lobule (PFG, PG) and on the lateral bank of the intraparietal sulcus (Pandya and Seltzer 1982). Outside the parietal cortex its main projections are to the ventral premotor cortex (Matelli et al. 1986; see also Rizzolatti et al. 1998), to SII, and to the prefrontal cortex (area 46). Single neuron studies showed that the majority of PF neurons respond to passive somatosensory stimuli (touch, joint displacement, muscle palpation) (Fogassi, Gallese, Fadiga, and Rizzolatti 1998; Graziano and Gross 1995; Hyvärinen 1981; Leinonen and Nyman 1979; Leinonen et al. 1979). The tactile receptive 1elds are large, frequently covering an entire arm, leg, or the chest. A considerable number of neurons can be activated by visual stimuli. About half of them are bimodal neurons responding to both visual and somatosensory stimuli (Graziano and Gross 1995; Leinonen et al. 1979). About one-third of PF neurons 1re during the animal’s active movements (Leinonen et al.

335


336


1979). Reaching with the arm, hand manipulation, and reaching with the mouth are the most frequently represented movements. Area PF, through its connection with STSa, on one hand, and F5, on the other, could play the role of an ‘intermediate step’ within a putative cortical network for action understanding, by feeding to the ventral premotor cortex visual information about actions as received from STSa. We decided therefore to study the functional properties of area PF by means of single neuron recording experiments. Neuron properties were examined during active movements of the monkey and in response to somatosensory and visual stimuli. Visual stimuli also included goal-related hand movements. About one-third of the recorded neurons responded both during action execution and action observation. These data indicate that an action observation/execution matching system does also exist in the parietal cortex, possibly constituting a building block of a cortical network for action understanding. The results of the present study will be discussed within a theoretical framework stressing the role played by the motor system in the representation of intentional actions. A preliminary report of these data appeared elsewhere (Fogassi et al. 1998).

17.2 Methods Electrical activity from single neurons was recorded from the rostral part of the inferior parietal lobule (area PF) in one monkey (Macaca nemestrina). All experimental protocols were approved by the Veterinarian Animal Care and Use Committee of the University of Parma and complied with the European law on the humane care and use of laboratory animals.

17.2.1 Neuron testing and behavioral paradigm During the recording session, the monkey was awake and seated on a primate chair, with the head 1xed. Once a neuron was isolated, its sensory and motor properties were 1rst tested (for a full description, see Rizzolatti et al. 1988). The somatosensory properties of the recorded neurons were tested using touch of the skin, hair bending, light pressure of the tissue, and slow and fast rotation of the joints. All testings were done with eyes open and closed. Visual properties were studied by presenting 3D objects by hand at different space locations and different distances from the monkey. After presentation, they were also moved, starting from different angles, toward and away from the monkey, or along a tangential plane at different distances from the monkey. The borders of the visual responding region (3D visual RF) were considered to be the external limits of that part of space whose crossing gave constant responses. In addition, all recorded neurons were studied by examining their response to the observation of actions performed by the experimenter in front of the monkey (for a full description, see Gallese et al. 1996). In brief, these actions were related to grasping, manipulating, holding, and placing objects. All these actions were performed at different distances from the monkey with the right, the left, or both hands of the experimenter. Furthermore, gestures with or without emotional content, such as threatening gestures, lifting the arms, waving the hand, etc., were executed in front of the monkey. To verify whether the recorded neurons were speci1cally activated by the observation of hand– object interactions, the following actions were also performed: prehension movements of objects performed with tools (e.g. pliers), mimicking object-related hand actions in absence of the target objects.



Motor properties of recorded neurons were studied in both light and dark conditions. Different objects of various sizes and shapes were presented in the different quadrants of the visual space of the monkey that reached for and grasped them. By examining the large variety of proximal–distal movement combinations it was usually possible to assess which proximal or distal movement was effective in triggering a given neuron (for details, see Rizzolatti et al. 1988).

17.2.2 Physiological procedures and data recording The surgical procedures for the construction of the head implant were the same as described in previous studies (for details, see Gentilucci et al. 1988). Single neurons were recorded using tungsten microelectrodes (impedance 0.5–1.0 MΩ, measured at 1 kHz) inserted through the dura. Neuronal activity was ampli1ed and monitored with an oscilloscope. Individual action potentials were isolated with a time–amplitude voltage discriminator (Bak Electronics, Germantown, MD). The output signal from the voltage discriminator was monitored and fed to a PC for analysis. By using a contact-detecting circuit, a signal was sent to a PC whenever the monkey or the experimenter touched a metal surface with their hand or mouth. This signal allowed the alignment of the histograms with the moment in which the motor action, performed either by the monkey or by the experimenter, was completed. The same contact-detecting circuit was also used to record visual and somatosensory responses: a signal was sent to the PC when the stimulus was introduced to the monkey’s visual 1eld, or touched the monkey’s skin, respectively. Response histograms were constructed by summing ten individual trials.

17.2.3 Identi1cation of the recorded region The monkey from which all the neurons presented in this study have been recorded is still alive. Therefore, the identi1cation of the recorded region has been achieved on the basis of its functional properties and of the neighboring regions. We 1rst recorded from the hand and face region of SI and then moved backwards at regular steps.

17.3 Results We recorded from 236 PF neurons in one hemisphere of one monkey. Figure 17.1 shows the region of area PF that was studied. As previously shown (Hyvärinen 1981; Leinonen and Nyman 1979), the vast majority (n = 220, 93%) of PF neurons responded to sensory stimuli. The responsive neurons were subdivided into three categories: ‘somatosensory’ neurons, ‘visual’ neurons, and bimodal ‘somatosensory and visual’ neurons (see Table 17.1).

17.3.1 Somatosensory response properties Of the 220 PF neurons responding to passive stimulation, 196 (89%) were activated by somatosensory stimuli. The somatosensory properties of ‘somatosensory’ and bimodal PF neurons were similar and therefore will be described together. Out of 196 neurons that responded to somatosensory stimulation, 138 (70.5%) were activated by light touch, 52 (26.5%) by pressure applied to the skin or passive movement of the joints, and 6

337


338


Table 17.1

Passive properties of the recorded neurons

Property

No. of neurons

% of neurons

Somatosensory Visual Somatosensory and visual

72 24 124

33 11 56

Total

220

100

Fig. 17.1 Lateral view of the left hemisphere of a standard macaque monkey brain. The posterior parietal cortex is parcellated according to Pandya and Seltzer (1982). The agranular frontal cortex is parcellated according to Matelli et al. (1985). The shaded area indicates the part of area PF explored in this study. The two asterisks indicate the presumed location of two penetrations carried out in SI, where the face and the 1ngers were represented, respectively. (cs = central sulcus; ias = inferior arcuate sulcus; ls = lateral sulcus; sas = superior arcuate sulcus; sts = superior temporal sulcus.)

(3%) by both touch and joint rotation or deep pressure. The tactile receptive 1elds (RFs) of PF neurons were typically large. They were located most frequently on the face, or on the neck, chest, and arm. Table 17.2 summarizes the number and percentage of different body part locations of the RFs of all somatosensory neurons. Of the 126 neurons whose RFs were located on the face, 109 responded to the stimulation of the lower face, 14 to the stimulation of the upper face, and 3 to the stimulation of both parts of the face. Most RFs were contralateral to the recorded side (73%), some extended bilaterally (22%), and a few were strictly ipsilateral (5%).



Table 17.2 Subdivision of neurons with somatosensory responses according to their RF locations No. of neurons

% of neurons

Contralateral

Ipsilateral

Bilateral

Face Hand Arm Neck and trunk Face and hand Large

126 35 10 4 8 13

64 18 5 2 4 7

82 35 10 0 5 11

8 0 0 0 1 0

36 0 0 4 2 2

Total

196

100

143

9

44

Table 17.3 Subdivision of bimodal (somatosensory and visual) neurons according to their RF locations No. of neurons

% of neurons

Contralateral

Ipsilateral

Bilateral

Face Hand Arm Neck and trunk Face and hand Large

68 7 6 1 2 9

73 8 6 1 2 10

46 7 6 0 2 7

1 0 0 0 0 0

21 0 0 1 0 2

Total

93

100

68

1

24

Table 17.3 summarizes the body part location of RFs of bimodal ‘somatosensory and visual’ neurons. If compared with ‘somatosensory’ neurons, the RFs of bimodal neurons tended to be located even more predominantly on the face. Of 68 neurons whose RFs were on the face, 59 had RFs on the lower face, 8 on the upper face, and 1 on both parts of the face. Considering that all ‘large’ RFs included the face, 85% of bimodal neurons had RFs on the monkey’s face. In contrast with tactile responses, proprioceptive and deep pressure responses were mostly evoked by arm and hand stimulation. Proprioceptive and deep pressure responses were most frequently (n = 49, 84%) evoked by stimuli applied contralateral to the recorded hemisphere, while only 9 (16%) of the neurons were activated by bilateral stimuli, and none by ipsilateral ones.

17.3.2 Visual properties Out of 236 recorded neurons, 148 (63%) responded to visual stimuli. According to the type of stimulation effective in activating them, visually responsive neurons were subdivided into three main classes, shown in Table 17.4. The 1rst and most represented class was formed by neurons with RFs located in the space around the monkey (peripersonal space) and responding best to rotational stimuli or stimuli moved along a horizontal, vertical, or sagittal direction. Typically, RFs were large. All neurons of the 1rst class were bimodal ‘somatosensory and visual’ neurons, with their visual RFs located around the tactile

339


340


Table 17.4 Subdivision of neurons with visual responses according to their preferred visual stimuli No. of neurons Peripersonal Far Biological actions Total

% of neurons

71 16 61

48 11 41

148

100

ones. Most RFs were located around the face. Out of the 71 neurons of the 1rst class, 58 (82%) had RFs contralateral to the recorded hemisphere, only one neuron had an ipsilateral RF, and 12 neurons (17%) had 1elds extending bilaterally. Visual responses were elicited any time a three-dimensional object was moved inside the RF. The quality of the object was generally irrelevant to the activation of the neurons, although occasionally neurons were recorded that seemed to respond best when the stimulus was the experimenter’s hand. The second class was formed by neurons responding to stimuli presented outside the peripersonal space. These neurons typically responded any time the stimuli were presented or moved in the monkey’s visual 1eld at a distance greater than 40 cm from the monkey’s body. The stimuli could be pieces of food or different objects at hand in the lab. The third class was composed by neurons responding to the observation of actions. Some of these neurons, in addition, had tactile RFs on the face and visual RFs around the tactile ones. In the next sections we will describe in more detail the neurons of this third class.

17.3.3 Neurons responding to the observation of actions Sixty-one neurons responded to the observation of actions executed by the experimenter in front of the monkey. Out of these 61 neurons, 43 had motor properties, and therefore, in analogy with the neurons described in area F5 (see Gallese et al. 1996; Rizzolatti et al. 1996a), will be referred to as ‘PF mirror’ neurons. Eighteen neurons, which were devoid of motor properties, will be described as ‘PF action-observation’ neurons.

17.3.4 Visual properties of PF mirror neurons Virtually all neurons of this class (n = 39, 91%) responded to the observation of actions in which the experimenter’s hand(s) interacted with objects. Of the four remaining neurons, three responded to the observation of the experimenter’s arm reaching for an object, and one to the observation of the experimenter’s elbow flexion. The responses triggered by these stimuli were consistent and did not habituate. The visual presentation of objects, such as food items or objects at hand in the lab, did not evoke any response. Similarly ineffective or very little effective in driving the neuron response were actions that, although achieving the same goal and looking similar to those performed by the experimenter’s hand, were made with tools such as pliers or pincers. Actions having emotional content, such as threatening gestures, were also ineffective. The distance and the location in space



Table 17.5 Mirror neurons subdivided according to the actions effective in activating them Observed actions

No. of neurons

Bimanual interaction Grasping Manipulating Holding Releasing Placing Reaching

8 4 3 3 2 1 3

Grasping and placing Grasping and holding Grasping and releasing Grasping and bringing to the mouth Placing and holding Placing and manipulating

5 4 2 1 3 1

Elbow 2exion

1

Total

43

with respect to the monkey at which the experimenter’s actions were performed did not appear to modulate the intensity of the response. Out of 43 PF mirror neurons, 25 were driven by the observation of a single action. Eighteen neurons were activated by the observation of two of them. The properties of neurons responding to two actions were the same as those responding to one action, apart from their lower speci1city. Table 17.5 shows the observed actions effective in activating the neurons and the number of mirror neurons activated by each of them. Only the actions listed in Table 17.5 were effective, among the many tested, in driving the neurons. An example of a PF mirror neuron responding to the observation of a single action is shown in Fig. 17.2. This neuron (bimanual interaction neuron) responded to the observation of both hands of the experimenter interacting with an object. Figure 17.3 shows another example of a PF mirror neuron responding to the observation of a single action. This neuron responded to the observation of grasping. The discharge started immediately before contact with the object and diminished immediately after the experimenter’s hand took possession of it. The neuron responded also during the monkey’s active execution of grasping with the mouth. This neuron, in addition to responses during grasping observation, also showed bimodal properties. The association between mirror and bimodal properties was found in about one quarter of PF mirror neurons. This interesting association will be dealt with below. About 40% of mirror neurons responded to the observation of two actions. An example of a PF mirror neuron discharging during the observation of two actions—grasping and releasing—is shown in Fig. 17.4.

341


342


Fig. 17.2 Visual responses of a ‘bimanual interaction’ mirror neuron. Each panel shows rasters and histograms recorded during 10 consecutive trials of the corresponding behavioral condition. This neuron discharged when both hands of the experimenter held an object. The discharge was tonically present during the whole holding period. If holding was performed by a single hand the discharge was either much weaker (left hand) or completely absent (right hand). Mimicking a bimanual holding action without the object did not evoke any response. The neuron discharged also when the monkey grasped an object with its hand (not shown in the 1gure). Rasters and histograms are aligned (small gray bars and black vertical bar, respectively) with the moment in which the experimenter’s hand started moving toward his other hand holding the object (1rst panel from top), toward his other hand without object (last panel from top), or to show the object to the monkey (second and third panel from top). Abscissae: time; ordinates: spikes per second.



Fig. 17.3 Visual and motor responses of a ‘grasping’ mirror neuron. Rasters and histograms are aligned with the moment in which the experimenter’s hand touched the object ( 1rst panel from top), started moving to present the object to the monkey (second panel from top), started moving the object with a top-down trajectory (third panel from top). Rasters and histograms of the last panel from top are aligned with the moment in which the monkey’s mouth touched the object. Other conventions as in Fig. 17.2.

343


344


Fig. 17.4 Visual and motor responses of a ‘grasping and releasing’ mirror neuron. This neuron started 1ring about 300 ms before the experimenter’s hand touched the object. The discharge continued until the experimenter’s hand took possession of the object, ceased during the holding phase, and started again during the releasing action. This neuron displayed a speci1city for the observed grip: the observation of grasping achieved by opposing the index 1nger to the thumb (pre-cision grip, PG) was much more effective than the observation of grasping achieved by 2exing all 1ngers around the object (whole hand prehension, WH). This selectivity was reciprocated by the neuron’s motor selectivity: the neuron’s discharge was higher when the monkey grasped the object using a precision grip than when using a whole hand prehension. Rasters and histograms of all panels, except for the last from top on the left row, are aligned with the moment in which either the experimenter’s or the monkey’s hand touched the object. Rasters and histograms of the last panel from top on the left row are aligned with the moment in which the experimenter’s hand started moving to present the object to the monkey. Other conventions as in Fig. 17.2.

This neuron displayed another property observed in several (see below) PF mirror neurons; namely, the hand performing the observed action markedly in2uenced the discharge intensity. The discharge was higher when the observed actions were performed by the experimenter’s left hand as opposed to the right hand. Out of 43 mirror neurons, 15 (35%) responded best to the observation of actions performed by one hand: 10 preferred the left hand and 5 the right hand. It is interesting to note that the experimenter’s left hand in a face-to-face stance corresponds to the observer’s right hand, that is, as in the present experiment, the hand contralateral to the recorded hemisphere.



Fig. 17.5 Tactile and visual properties of a ‘grasping’ mirror neuron. The mirror properties of this neuron are shown in Fig. 17.3. Rasters and histograms are aligned with the moment in which the tactile stimulus, a three-dimensional object, touched the monkey’s skin ( 1rst and third panel from top), or with the moment in which the visual stimulus, a three-dimensional object, entered the monkey’s visual 1eld (second and last panels from top). In A, tactile and visual stimuli were moved along a top-down trajectory. In B, tactile and visual stimuli were moved along a bottom-up trajectory. The two drawings illustrate the location of the tactile RF (shaded area), which encompassed the entire hemiface contralateral to the recorded hemisphere, and of the peripersonal visual RF (solid), which extended approximately 15 cm from the monkey’s skin. The two arrows indicate the direction along which the tactile and visual stimuli were moved. Other conventions as in Fig. 17.3.

345


346


A set of PF mirror neurons (n = 10, 23%), in addition to action observation, responded to tactile stimuli applied to the monkey’s face and to visual stimuli (three-dimensional objects) moved in the peripersonal space around the tactile RF. Unlike the other mirror neurons, all ‘bimodal mirror’ neurons were excited during the monkey’s active movements of the mouth. Figure 17.5 shows an example of a bimodal mirror neuron. Its mirror properties were illustrated in Fig. 17.3. The tactile RF was located around the hemiface contralateral to the recorded hemisphere. The visual RF was located around the tactile one, extending in depth for about 15 cm. Both tactile and visual RFs were directionally selective: stimuli moved top-down were far more effective than stimuli moved bottomup in evoking the neuron discharge. It is interesting to note that this directional selectivity matched with the selectivity for the direction along which the observed hand approached and grasped the object, as shown in Fig. 17.3. The neuron discharged only when the experimenter’s hand approached the object from above, with a top-down trajectory. However, moving the object held by the experimenter’s hand with the same trajectory was much less effective in driving the neuron’s response. Object presentation was also ineffective.

17.3.5 Visual properties of PF action-observation neurons Among the 61 neurons responding to the observation of actions, 18 were devoid of motor properties (‘action-observation’ neurons). Out of 18 of these neurons, 8 were driven by the observation of a single action. Ten neurons were activated by the observation of two or three actions. Table 17.6 shows the observed actions effective in activating the neurons and the number of action-observation neurons activated by each of them. Only the actions listed in Table 17.6 were effective, among the many tested, in driving the neurons. Figure 17.6 shows an example of an action-observation neuron. This neuron discharged when the monkey observed the experimenter’s hand grasping and holding the object. The discharge onset preceded the moment in which the experimenter’s hand touched the object by about 300 ms, and

Table 17.6 Action-observation neurons subdivided according to the actions effective in activating them Observed actions Bimanual interaction Grasping Holding Reaching Grasping and holding Bimanual interaction and holding Placing and holding Grasping and releasing Grasping, placing, and bimanual interaction Total

No. of neurons 3 2 2 1 5 2 1 1 1 18



Fig. 17.6 Visual properties of a ‘grasping and holding’ action-observation neuron. Rasters and histograms are aligned with the moment in which the experimenter’s hand touched the object (1rst and second panel from top), started moving to present the object to the monkey with a stick (third panel from top), or started holding the object. Other conventions as in Fig. 17.2.

then continued during the whole holding period. The hand used by the experimenter markedly influenced the discharge intensity of this neuron. Hand preference was present in three out of 18 actionobservation neurons, of which two preferred the left hand and one, shown in Fig. 17.6, the right one. When the experimenter performed the observed action with his left hand, the discharge was almost absent. Similarly ineffective were object presentation, or the observation of a holding action not preceded by a grasping action, even when performed by the preferred hand.

347


348


The vast majority of action-observation neurons (n = 12, 67%) had also bimodal properties. These neurons responded to tactile stimuli applied to the monkey’s face and to visual stimuli (three-dimensional objects) moved in the peripersonal space around the tactile RF.

17.3.6 Motor properties Out of 236 recorded neurons, 130 (55%) discharged in association with active movements of the monkey. Of those, 16 (12%) were purely motor, 38 (30%) also responded to somatosensory stimuli, 17 (13%) to visual stimuli, and 59 (45%) to both somatosensory and visual stimuli. Table 17.7 shows the effectors whose movement determined the responses of PF neurons. Almost all neurons (90%) were activated by movements of the mouth, of the hand, or of both. As previously described (Hyvärinen 1981; Leinonen et al. 1979), typically, PF neurons were selectively activated during speci1c actions. These were: grasping with the hand (n = 27), manipulation (n = 15), grasping and manipulation (n = 6), grasping and holding (n = 2), grasping with the mouth (n = 40), grasping with the hand and the mouth (n = 24), arm reaching (n = 3), bringing to the mouth (n = 8), arm reaching and bringing to the mouth (n = 1), and associated actions of the hand, arm, and mouth (n = 3). Only one neuron responded during a movement of the monkey—elbow 2exion. The motor properties of PF mirror neurons were indistinguishable from those of the other PF neurons. Seventeen of them (40%) discharged during hand actions: 13 were activated by hand grasping, and four by manipulation. Eleven neurons (25.5%) discharged during mouth grasping, and another 11 neurons (25.5%) discharged during hand and mouth grasping. Three neurons (7%) responded during arm and forearm movements: two of them responded to reaching, and one to elbow 2exion. Finally, one more neuron (2%) discharged during mouth grasping and arm reaching.

17.3.7 Relationship between visual and motor properties of mirror neurons In most mirror neurons there was a clear relationship between the observed action they responded to and the executed action that drove their discharge. Using as a classi1cation criterion the relationship between the effectors whose action observation or execution triggered the neurons’ discharge, we distinguished three broad classes. The 1rst class (n = 23, 53%) comprised neurons that responded during observation and execution of hand actions. In eight of these neurons the effective observed and executed action corresponded both in terms of action goal (e.g. grasping) and in terms of the way in which the goal was achieved

Table 17.7

Properties of motor neurons

Effector Mouth Hand Arm and forearm Mouth and hand Mouth, arm, and hand Total

No. of neurons

% of neurons

40 50 13 24 3

31 39 10 18 2

130

100



(e.g. precision grip). In six neurons the effective observed and executed actions were similar but not identical. In some of them the motor response was more speci1c than the visual one, in others the opposite was true. In six neurons the visual response could be interpreted as logically related to the motor response. For example, the effective observed action could be placing a piece of food on a tray, while the effective executed action could be grasping the piece of food. Finally, in three neurons there was no clear-cut relationship between the effective observed and executed actions. The second class (n = 12, 28%) comprised neurons that responded during observation of hand actions and during the execution of mouth actions. All bimodal mirror neurons fell in this class. Finally, the third class comprised neurons (n = 8, 19%) that responded during observation of hand actions and during execution of hand and mouth actions. If one considers only the hand actions, in two neurons the effective observed and executed actions corresponded in terms of both action goal and of the way in which the goal was achieved, while in six neurons the effective observed and executed actions were similar but not identical.

17.4 Discussion 17.4.1 General properties of area PF In agreement with previous reports (Graziano and Gross 1995; Hyvärinen 1981; Leinonen and Nyman 1979; Leinonen et al. 1979), the present study showed that the majority of PF neurons respond to passive somatosensory stimuli, to visual stimuli, or to both. A considerable percentage of neurons in area PF were also endowed with motor properties. All these motor neurons (except one) discharged during actions such as grasping, manipulating, and reaching objects. This 1nding stresses the role of the posterior parietal cortex as a region of the brain where not only different sensory modalities are integrated but also sensory information is used for motor actions. Neurons responding to visual stimuli were subdivided into three classes. Neurons of the 1rst class had RFs located in the space around the monkey (peripersonal space). Neurons of the second class responded to objects presented outside the peripersonal space. Finally, neurons of the third class responded to the observation of actions. In the remaining part of the Discussion we will focus on neurons of the 1rst and third class. Virtually all neurons of the 1rst class were also driven by cutaneous stimulation. These bimodal neurons responded independently to visual and tactile stimulation. Two-thirds of bimodal somatosensory and visual neurons had visual peripersonal RFs around the face, most frequently around the mouth. Tactile RFs were also located predominantly on the lower part of the face. Tactile and visual RFs were therefore usually ‘in register’. Directional selectivity, when present, as in the neuron shown in Fig. 17.5, was the same in both modalities. It is noteworthy that most bimodal neurons with motor properties discharged during mouth grasping movements. The association of motor properties related to a given effector (e.g. mouth, head, or arm) with bimodal receptive 1elds anchored to the same effector, is displayed also by neurons of the premotor area F4 (Fogassi et al. 1992, 1996; Gentilucci, Scandolara, Pigarev, and Rizzolatti 1983; Graziano, Yap, and Gross 1994; Graziano, Hu, and Gross 1997). The ‘visual’ responses of F4 neurons, rather than being truly visual, are probably potential actions. In other words, they consist of the automatic activation of the motor programs required by the effector whose visual receptive field is crossed by a stimulus to interact with the same stimulus (see Fogassi et al. 1996; Rizzolatti, Fadiga, Fogassi,

349


350


and Gallese 1997a). Although one cannot exclude that the responses observed in the parietal cortex are true visual responses, coding the position of stimuli in the peripersonal space, it is more likely that in area PF too the peripersonal space located around the mouth is a ‘grasping space’. This space would enable a fast grasping action of the mouth whenever an appropriate stimulus approaches its surrounding space (see also below). Previous experiments in which area PF was ablated produced de1cits that are consistent with the neuron sensory and motor properties just described (Faugier-Grimaud, Frenois, and Stein 1978; Matelli, Gallese, and Rizzolatti 1984; Rizzolatti, Gentilucci, and Matelli 1985). Following damage to PF, monkeys tend to use the hand ipsilateral to the lesion. Distal movements of the contralateral hand become clumsy. Tactile stimuli applied to the face contralateral to the lesion are frequently neglected. The coordinated head–mouth movements necessary to reach food when the lips are touched with it are slower on the affected side. Visual de1cits are present in the peribuccal space, but not in the extrapersonal space. When the monkey 1xates a central stimulus, the movement of a piece of food near the mouth on the side of the lesion produces an immediate mouth-grasping response, whereas the same stimulus shown contralaterally is ignored. With two stimuli moved simultaneously, the one ipsilateral to the lesion is always preferred.

17.4.2 Neurons responding to the observation of actions The most important 1nding of this study was the discovery that a considerable percentage of PF neurons (mirror and action-observation neurons) responded to the observation of actions performed by other individuals such as grasping, placing, holding, reaching, and the like. The majority of these neurons had also motor properties matching the visual ones. As in the premotor area F5 (di Pellegrino et al. 1992; Gallese et al. 1996; Rizzolatti et al. 1996a), the PF mirror and action-observation neurons required an interaction between the agent of the action and the object target of the action in order to be visually driven. The sight of the agent alone or of the object alone was not effective. Similarly, the same actions when mimicked without the target object were much less effective. Hand actions were the most effective stimuli. The use of tools to imitate the effective observed actions usually did not signi1cantly affect the neuron response. Another similarity between PF and F5 mirror and action-observation neurons was the broad degree of generalization across different instances of the observed actions evoking the neuron’s discharge. In many neurons the distance and location in space with respect to the monkey of the observed actions was not crucial. About two-thirds of PF neurons responding to action observation have properties almost indistinguishable from those characterizing F5 mirror and action-observation neurons. This result indicates that the ‘mirror’ system, matching action observation to action execution, is not a prerogative of the premotor cortex, but extends to the posterior parietal lobe as well. How could one relate these PF neurons with the properties of F5 mirror neurons? A problematic issue since the discovery of F5 mirror neurons has been the source of their visual input. Neurons activated by complex biological, meaningful visual stimuli had been previously described in the macaque brain. Early studies showed that in the inferotemporal lobe there are neurons that discharge selectively to the presentation of a face or a hand (Gross et al. 1972; see also Perrett, Rolls, and Caan 1982). More recently, Perrett and co-workers demonstrated that in a region of the upper bank of the superior temporal sulcus (STSa) there are neurons, apparently devoid of motor properties (it must be noted, however, that such properties were never tested), selectively activated by the sight of hand actions (Perrett et al. 1989; Perrett, Mistlin, Harries, and Chitty 1990; see



also Jellema and Perrett, this volume). The results of the present study suggest that area PF could represent an intermediate step leading from a ‘visual’ description of actions, carried out in STSa, to a motor representation of the same actions coded in the premotor cortex. Area PF, together with the STS region and the premotor area F5 could compose a cortical network supporting action recognition. Evidence from brain-imaging studies in humans suggests that this is more than a speculative hypothesis. Several PET and fMRI studies (Grafton et al. 1996; Grèzes et al. 1998; Iacoboni et al. 1999; Rizzolatti et al. 1996b; for a review, see Allison, Puce, and McCarthy 2000; Decety and Grèzes 1999) have shown that whenever subjects observed meaningful goal-related hand actions, three cortical regions were consistently activated: the STS region, the anterior part of the inferior parietal lobule (BA 40), and a sector of the premotor cortex (BA 44–45). It is noteworthy that in a recent fMRI study (Buccino et al. 2001) it has been shown that a mirrormatching system is present for mouth and foot actions as well, thus suggesting that this system is not con1ned to hand actions, but is likely to underpin the understanding of a huge variety of actions.

17.4.3 Matching hand actions on mouth actions A consistent number of PF mirror neurons matched observed hand actions on mouth actions. This apparent discrepancy between the different effectors, the observation and active movement of which drive these neurons, thus needs to be addressed. A possible hypothesis is that these PF neurons represent a ‘primitive’ matching system based on mouth movements. Ontogenetically speaking, the mouth is the effector by means of which all primates, humans included, not only start to feed themselves, but also start to explore the surrounding world. Through the medium of mouth actions the world can be readily classi1ed in categories (good/bad, edible/non-edible) that are very likely to form the building blocks of the future, more comprehensive account of the environment. Consistent with this ontogenetic hypothesis is the high degree of mouth–hand synergies observed in infants. After extensive practice, paralleled by the development of corticospinal projections, the two effectors can easily be used independently in a successful way. When the infant primate acts with its mouth, nevertheless, it most frequently can see its hand. An association between mouth and hand actions can therefore be established. According to this hypothesis, this association may lead to the origin of mirror neurons that have mouth-related responses on the output side, and hand-related responses on the input side. This relation, which initially is established between the infant’s mouth and its own grasping hand, could be generalized later to the hands of other individuals. One could object that these neurons were recorded in an adult monkey. Such a ‘primitive’ matching system, however, could persist in adulthood, even when a more sophisticated hand–hand matching system is developed, in order to provide an ‘abstract’ categorization of the observed actions: what is recognized is a particular action goal, regardless of the effector enabling its achievement. Some additional words should be spent to discuss the properties of those PF mirror neurons that showed bimodal properties. The visual RFs of these mirror neurons were located around the face, mostly around the mouth, and their tactile RFs were located on the lips and the peribuccal region. All these neurons discharged during mouth grasping actions and all responded to the observation of hand actions. What can the function of these RFs be, when combined with mirror properties? As stated above, a visual peripersonal RF located around a mouth tactile RF can be interpreted as a motor space, by means of which the visual stimuli that cross it are ‘translated’ into suitable motor

351


352


plans (e.g. a mouth grasping action), enabling the organism endowed with such RFs to successfully interact with the same stimuli (Fogassi et al. 1996; Rizzolatti, Fadiga, Fogassi, and Gallese 1997a). The visual stimulus that most frequently crosses the peripersonal visual RFs of these PF mirror neurons is likely to be the monkey’s own hand, while bringing food to the mouth. A hand approaching the mouth can therefore pre-set the motor programs controlling grasping with the mouth. Through a process of generalization between the monkey’s own moving hand, interpreted as a signal to grasp with the mouth, and the object-directed moving hands of others, any time the monkey observes another individual’s hand interacting with food, the same mouth action representation will be evoked. According to this ontogenetic hypothesis, the peripersonal visual RF around the mouth would enable a primitive matching between the vision of a hand and the motor program controlling the mouth. Once this equivalence is put in place, a mirror system matching hand actions observation to mouth actions execution can be established.

17.4.4 Action, perception, and the parietal cortex What is the link between acting and observing someone else acting? This question raises the broader issue of the relationship between action and perception. Since the early eighties, the dominant view on the cortical processing of visual information has been the ‘what’ and ‘where’ theory, as proposed by Ungerleider and Mishkin (1982). According to these authors, the ventral stream has its main role in object recognition, while the dorsal stream analyzes an object’s spatial location. This point of view was in accordance with the classical notion of the parietal cortex as the site for unitary space perception. Since the early nineties, Milner and Goodale (Goodale and Milner 1992; Milner and Goodale 1995) have argued against this theory, emphasizing the role of the dorsal stream in the ‘on-line’ control of action. This point of view, primarily triggered by clinical data, has been subsequently substantiated by neurophysiological evidence. The posterior parietal cortex is now thought to consist of a mosaic of areas, each receiving speci1c sensory information (Colby and Duhamel 1996; Rizzolatti, Fogassi, and Gallese 1997b; Rizzolatti et al. 1998). Within the dorsal stream, there are parallel cortico-cortical circuits, each of which elaborates a speci1c type of visual information in order to guide different types of action. The peculiarity of these circuits resides in the fact that different effectors are provided with the most suitable type of visual information required by their motor repertoire. This 1rm connection between vision and action seems to be the organizing principle within the circuitry connecting the parietal cortex with the agranular frontal cortex of the monkey (see also Gallese, Craighero, Fadiga, and Fogassi 1999). The present data point to an important involvement of the posterior parietal lobe, and in particular of areas such as area PF, strictly linked with the premotor cortex, in mediating processes traditionally considered to be ‘high level’ or cognitive, such as action recognition. Mirror neurons, such as those presented in this study, represent a perfect instantiation of this view. This matching mechanism can be framed within theories postulating a shared representational domain for action and perception (Gallese 2000a,b; Jeannerod 1994, 1997; Prinz 1997; Rizzolatti, Fogassi, and Gallese 2000). Such a mechanism offers the great advantage of using a repertoire of coded actions in two ways at the same time: at the output side to act, and at the input side, to analyze the visual percept. The link is constituted by the presence in both instances of a goal. Our proposal is that the goal of the observed action is recognized and ‘understood’ by the observer by mapping it on to a shared motor representation.



Acknowledgements This work was supported by MURST and by HFSP.

References Allison, T., Puce, A., and McCarthy, G. (2000). Social perception from visual cues: Role of the SRS region. Trends in Cognitive Sciences, 4, 267–278. Andersen, R.A., Asanuma, C., Essick, G., and Siegel, R.M. (1990). Corticocortical connections of anatomically and physiologically de1ned subdivisions within the inferior parietal lobule. Journal of Comparative Neurology, 296, 65–113. Buccino, G., Binkofski, F., Fink, G.R., Fadiga, L., Fogassi, L., Gallese, V., Seitz, R.J., Zilles, K., Rizzolatti, G., and Freund, H.-J. (2001). Action observation activates premotor and parietal areas in a somatotopic manner: An fMRI study. European Journal of Neuroscience, 13, 400–404. Cavada, C. and Goldman-Rakic, P.S. (1989a). Posterior parietal cortex in rhesus monkey: II. Evidence for segregated corticocortical networks linking sensory and limbic areas with the frontal lobe. Journal of Comparative Neurology, 287, 422–445. Cavada, C. and Goldman-Rakic, P.S. (1989b). Posterior parietal cortex in rhesus monkey: I. Parcellation of areas based on distinctive limbic and corticocortical connections. Journal of Comparative Neurology, 287, 393–421. Cochin, S., Barthelemy, C., Lejeune, B., Roux, S., and Martineau, J. (1998). Perception of motion and qEEG activity in human adults. Electroencephalography and Clinical Neurophysiology, 107, 287–295. Colby, C.L. and Duhamel, J.-R. (1996). Spatial representations for action in parietal cortex. Cognitive Brain Research, 5, 105–115. Decety, J. and Grèzes, J. (1999). Neural mechanisms subserving the perception of human actions. Trends in Cognitive Sciences, 3, 172–178. di Pellegrino, G., Fadiga, L., Fogassi, L., Gallese, V., and Rizzolatti, G. (1992). Understanding motor events: A neurophysiological study. Experimental Brain Research, 91, 176–180. Fadiga, L., Fogassi, L., Pavesi, G., and Rizzolatti, F. (1995). Motor facilitation during action observation: A magnetic study. Journal of Neurophysiology, 73, 2608–2611. Faugier-Grimaud, S., Frenois, C., and Stein, D.G. (1978). Effects of posterior parietal lesions on visually guided behavior in monkeys. Neuropsychologia, 16, 151–168. Fogassi, L., Gallese, V., di Pellegrino, G., Fadiga, L., Gentilucci, M., Luppino, G., Matelli, M., Pedotti, A., and Rizzolatti, G. (1992). Space coding by premotor cortex. Experimental Brain Research, 89, 686–690. Fogassi, L., Gallese, V., Fadiga, L., Luppino, G., Matelli, M., and Rizzolatti, G. (1996). Coding of peripersonal space in inferior premotor cortex (area F4). Journal of Neurophysiology, 76, 141–157. Fogassi, L., Gallese, V., Fadiga, L., and Rizzolatti, G. (1998). Neurons responding to the sight of goal-directed hand/ arm actions in the parietal area PF (7b) of the macaque monkey. Society of Neuroscience Abstracts, 24, 257.5. Gallese, V. (2000a). The acting subject: Towards the neural basis of social cognition. In T. Metzinger (Ed.), Neural correlates of consciousness, pp. 325–333. Cambridge, MA: MIT Press. Gallese, V. (2000b). The inner sense of action: agency and motor representations. Journal of Consciousness Studies, 7, 23–40. Gallese, V., Fadiga, L., Fogassi, L., and Rizzolatti, G. (1996). Action recognition in the premotor cortex. Brain, 119, 593–609. Gallese, V., Craighero, L., Fadiga, L., and Fogassi, L. (1999). Perception through action. Psyche: http://psyche. cs.monash.edu.au/v5/psyche-5-21-gallese.html. Gentilucci, M., Scandolara, C., Pigarev, I.N., and Rizzolatti, G. (1983).Visual responses in the postarcuate cortex (area 6) of the monkey that are independent of eye position. Experimental Brain Research, 50, 464–468. Gentilucci, M., Fogassi, L., Luppino, G., Matelli, M., Camarda, R., and Rizzolatti, G. (1988). Functional organization of inferior area 6 in the macaque monkey: I. Somatotopy and the control of proximal movements. Experimental Brain Research, 71, 475–490. Goodale, M.A. and Milner, D. (1992). Separate visual pathways for perception and action. Trends in Neuroscience, 15, 20–25.

353


354


Grafton, S.T., Arbib, M.A., Fadiga, L., and Rizzolatti, G. (1996). Localization of grasp representations in humans by PET: 2. Observation compared with imagination. Experimental Brain Research, 112, 103–111. Graziano, M.S.A and Gross, C.G. (1995). The representation of extrapersonal space: A possible role for bimodal visual-tactile neurons. In M.S. Gazzaniga (Ed.), The cognitive neurosciences, pp. 1021–1034. Cambridge, MA: MIT Press. Graziano, M.S.A., Yap, G.S., and Gross, C.G. (1994). Coding of visual space by premotor neurons. Science, 266, 1054–1057. Graziano, M.S.A., Hu, X., and Gross, C.G. (1997). Visuo-spatial properties of ventral premotor cortex. Journal of Neurophysiology, 77, 2268–2292. Grèzes, J., Costes, N., and Decety, J. (1998). Top-down effect of strategy on the perception of human biological motion: A PET investigation. Cognitive Neuropsychology, 15, 553–582. Gross, C.G. et al. (1972). Visual properties of neurons in inferotemporal cortex of the monkey. Journal of Neurophysiology, 35, 96–111. Hari, R., Forss, N., Avikainen, S., Kirveskari, S., Salenius, S., and Rizzolatti, G. (1999). Activation of human primary motor cortex during action observation: A neuromagnetic study. Proceedings of the National Academy of Science, 95, 15061–15065. Hyvärinen, J. (1981). Regional distribution of functions in parietal association area 7 of the monkey. Brain Research, 206, 287–303. Iacoboni, M., Woods, R., Brass, M., Bekkering, H., Mazziotta, J.C., and Rizzolatti, G. (1999). Cortical mechanisms of human imitation. Science, 286, 2526–2528. Jeannerod, M. (1994). The representing brain: Neural correlates of motor intention and imagery. Behavioral Brain Sciences, 17, 187–245. Jeannerod, M. (1997). The cognitive neuroscience of action. Oxford: Blackwell. Leinonen, L. and Nyman, G. (1979). II. Functional properties of cells in anterolateral part of area 7 associative face area of awake monkeys. Experimental Brain Research, 34, 321–333. Leinonen, L., Hyvärinen, J., Nyman, G., and Linnankoski, I. (1979). I. Function properties of neurons in lateral part of associative area 7 in awake monkeys. Experimental Brain Research, 34, 299–320. Matelli, M., Gallese, V., and Rizzolatti, G. (1984). De1cit neurologici conseguenti a lesione dell’area parietale 7b nella scimmia. Bollettino della Società Italiana di Biologia Sperimentale, 60, 839–844. Matelli, M., Luppino, G., and Rizzolatti, G. (1985). Patterns of cytochrome oxidase activity in the frontal agranular cortex of the macaque monkey. Behavioral Brain Research, 18, 125–137. Matelli, M., Camarda, R., Glickstein, M., and Rizzolatti, G. (1986). Afferent and efferent projections of the inferior area 6 in the Macaque Monkey. Journal of Comparative Neurology, 251, 281–298. Milner, A.D. and Goodale, M.A. (1995). The visual brain in action. Oxford: OUP. Pandya, D.N. and Seltzer, B. (1982). Intrinsic connections and architectonics of posterior parietal cortex in the rhesus monkey. Journal of Comparative Neurology, 204, 196–210. Perrett, D.I., Rolls, E.T., and Caan, W. (1982).Visual neurons responsive to faces in the monkey temporal cortex. Experimental Brain Research, 47, 329–342. Perrett, D.I., Harries, M.H., Bevan, R., Thomas, S., Benson, P.J., Mistlin, A.J., Chitty, A.K., Hietänen, J.K., and Ortega, J.E. (1989). Frameworks of analysis for the neural representation of animate objects and actions. Journal of Experimental Biology, 146, 87–113. Perrett, D.I., Mistlin, A.J., Harries, M.H., and Chitty, A.K. (1990). Understanding the visual appearance and consequence of hand actions. In M.A. Goodale (Ed.), Vision and action: The control of grasping, pp. 163–180. Norwood, NJ: Ablex. Petrides, M. and Pandya, D.N. (1997). Projections to the frontal cortex from the posterior parietal region in the rhesus monkey. Journal of Comparative Neurology, 228, 105–116. Prinz, W. (1997). Perception and action planning. European Journal of Cognitive Psychology, 9, 129–154. Rizzolatti, G., Gentilucci, M., and Matelli, M. (1985). Selective spatial attention: One center, one circuit or many circuits? In M.I. Posner and O. Marin (Eds.), Attention and Performance, XI: Conscious and nonconscious information processing, pp. 251–265. Hillsdale, NJ: Erlbaum. Rizzolatti, G., Camarda, R., Fogassi, L., Gentilucci, M., Luppino, G., and Matelli, M. (1988). Functional organization of inferior area 6 in the macaque monkey: II. Area F5 and the control of distal movements. Experimental Brain Research, 71, 491–507. Rizzolatti, G., Fadiga, L., Gallese, V., and Fogassi, L. (1996a). Premotor cortex and the recognition of motor actions. Cognitive Brain Research, 3, 131–141.



Rizzolatti, G., Fadiga, L., Matelli, M., Bettinardi, V., Paulesu, E., Perani, D., and Fazio, G. (1996b). Localization of grasp representations in humans by PET: 1. Observation versus execution. Experimental Brain Research, 111, 246–252. Rizzolatti, G., Fadiga, L., Fogassi, L., and Gallese, V. (1997a). The space around us. Science, 277, 190–191. Rizzolatti, G., Fogassi, L., and Gallese, V. (1997b). Parietal cortex: From sight to action. Current Opinion in Neurobiology, 7, 562–567. Rizzolatti, G., Luppino, G., and Matelli, M. (1998). The organization of the cortical motor system: New concepts. Electroencephalography and Clinical Neurophysiology, 106, 283–296. Rizzolatti, G., Fogassi, L., and Gallese, V. (2000). Cortical mechanisms subserving object grasping and action recognition: A new view on the cortical motor functions. In M.S. Gazzaniga (Ed.), The cognitive neurosciences, (2nd edn), pp. 539–552. Cambridge, MA: MIT Press. Seltzer, B. and Pandya, D.N. (1984). Further observations on parietotemporal connections in the rhesus monkey. Experimental Brain Research, 55, 301–312. Ungerleider, L.G. and Mishkin, M. (1982). Two visual systems. In D.J. Ingle, M.A. Goodale, and R.J. Mans1eld (Eds.), Analysis of visual behavior, pp. 549–586. Cambridge, MA: MIT Press.

355


18 Coding of visible and hidden actions Tjeerd Jellema and David I. Perrett

Abstract. We review the properties of cells in the temporal cortex of the macaque monkey, which are sensitive to visual cues arising from the face and body and their movements. We speculate that the responses of populations of cells in the cortex of the anterior superior temporal sulcus (STSa) support an understanding of the behaviour of others. Actions of an agent, including whole body movements (e.g. walking) and articulations (of the limbs and torso), made during the redirecting of attention and reaching are coded by STSa cells in a way which: (1) allows generalization over different views and orientations of the agent with respect to the observer; (2) utilizes information about the agent’s current and (3) imagined position while occluded from sight; and (4) is sensitive to sequences of the agent’s movements. The selectivity of cells is described from the perspective of hierarchical processing, which presumes that early processing establishes sensitivity to simple body cues and later coding combines these cues to specify progressively more subtle and abstract aspects of behaviour. The action coding of STSa cells is discussed in terms of dorsal and ventral cortical systems, the binding problem, and the functional architecture, which allows hierarchical information processing.

18.1 Introduction 18.1.1 Starting simple: perspectives on hierarchical processing One of the most fundamental tasks for scientists attempting to understand vision is to realize its purpose. This is necessary even before de1ning a computational theory of how that purpose can be achieved (Marr 1982). Of course, there need not be one single purpose, a ‘holy grail’, though authors have championed various singular causes—for example, Marr (1982) suggested that the purpose of vision is to build representations and, more recently, Goodale and Milner (1992) have stressed the function of vision in guiding actions. This paper assumes that one goal of vision is to enable the viewer to understand the behaviour of others (which may in turn afford social interactions, avoid predation, etc.). Given this goal, vision needs to achieve the ability to detect and discriminate meaningful actions. These are abstract, complex visual events, which are obviously not de1ned by the presence of a single edge or particular colour in the image. Early vision, which provides the basis for discriminating colour or orientations, is not suf1cient for understanding actions. An act such as ‘knocking over an object’ involves analysis of a huge number of elementary visual features and their movements. Detecting this act may even necessitate realization of the motives of another (i.e. that the knocking over was intentional rather than accidental). The thesis followed in our work is that understanding of such complex acts is achieved by the initial detection of simple events and the subsequent detection of combinations of these simple events. This reiterative process supports a hierarchy of complexity of the visual con1gurations


Coding of visible and hidden actions

detected and allows progressively more subtle meanings to be realized. There is no reason that readers should be alarmed by the properties of any one stage of this hierarchy. The detection of light, of edges, shapes, gaze, and attention are all manifestations of the same wonderful biological processes, whereby cells take on inputs representing sensory data, perform a statistical assessment of these (Riesenhuber and Poggio 2000), and provide output to other cells. No stage of processing needs to be seen as exceptional, since the machinery and operations at each level are equivalent. The problem with our notion of hierarchies is our anthropomorphism. We assume that being higher in the hierarchy is somehow more important, perhaps because people higher in the chain of command are seen as more responsible and thus deserving of higher salaries. Cells, on the other hand, are paid equally and all do a similar job of detecting patterns of input and providing output action potentials wherever they are in the brain. Anthropomorphic thinking about hierarchies in the nervous system leads to a hunt for the ghost in the machine. Once processing a few synapses away from the sensory receptors is understood, it is allocated the lowly status of sensory processing; awareness and consciousness, even of those simple qualities made apparent by the sensory processing, is relocated to some higher level. An alternative, more egalitarian view is that at each stage neural processing can contribute to awareness. Under this view, cells in the primary visual cortex can contribute to awareness of orientation and spatial frequency, but are not able to contribute directly to awareness of facial patterns. Cells sensitive to facial patterns, on the other hand, may contribute to the awareness of faces but not to the orientation of edges. The purpose of this paper is to provide an overview of how the brain builds progressively more abstract descriptions of the actions of others. Investigations into visual processing can be made by recording the activity of individual brain cells at different stages in the visual system of experimental animals. Since the work of Gross and colleagues (Gross et al. 1972), it has become clear that some cells in the temporal cortex of the macaque monkey respond selectively to the sight of biologically important objects (such as hands and faces). The properties of such cells, therefore, offer a unique opportunity to study directly the brain mechanisms involved in processing complex visual patterns and their meaning. In this paper, we focus on the visual cues used by cells to specify the posture and actions of others. We review previously published 1ndings, but extend these accounts with new examples and observations on the sensitivity of single cells within the temporal cortex that exhibit tuning for speci1c body postures, movements, and components of behaviour. The paper traces the historical progression from cell properties that have been described for over two decades (i.e. selectivity to view of the face) to more recently discovered and unusual properties. Hopefully, it will be apparent that even the most complex neural descriptions of behaviour can in principle be derived from cellular sensitivity to relatively standard visual attributes (such as the form of individual body components, view, direction of motion, and position).

18.2 Integration of form 18.2.1 Getting attention The responses of cells to faces, particularly the cells within the superior temporal sulcus (STSa), are consistent with these cells playing a role in the perception of social signals (Emery and Perrett 1994). One type of social signal that appears to be analysed extensively within the STSa is where another animal is directing its attention (Perrett et al. 1990c, 1991). We refer to these signals as social attention (see Langton, Watt, and Bruce 2000).

357


358


Fig. 18.1 Discrimination of view for head and body. Upper: Schematic illustration of the 3-D test stimuli. Lower: The mean and standard error (SE) of response of one cell to different views of the head and body. With the body occluded from sight, the cell gave an excitatory response to the left pro1le view of the head but gave zero response to the right pro1le view. With the head occluded from sight, the cell responses showed a preference for the left pro1le view of the body (torso and limbs). Statistical analysis of responses supported these observations. A 2-way ANOVA (with view and body components as main factors) indicated a signi1cant effect of view on responses [F(1,14) = 7.7, p < 0.02], no effect of body part [F(1,14) = 0.4, p = 0.52], and no interaction between factors [F(1,14) = 0.30, p = 0.93]. Thus, the cell responses signal the sight of the head or the body facing in the same direction. A role in the visual analysis of direction of another’s attention may account for the selectivity of different cells in STSa to not only the face but also many other views of the head (Hasselmo, Rolls, Baylis, and Nalwa 1989; Perrett et al. 1991). Different cells in the STSa are selective for different views of the head; some respond only to the front or face view, others respond selectively to the left pro1le view of the head (e.g. Fig. 18.1), or to the right pro1le, while yet others respond to the back view of the head. Further cells respond to the head raised or the head lowered (e.g. Fig. 18.2). We speculate that a cell maximally responsive to the face seen in left pro1le may signal that the attention of another individual is directed to the observer’s left. Likewise, a cell responsive to the head lowered might signal attention down.

18.2.2 What you looking at? The hypothesis that STSa cells responsive to faces may signal attention direction suggests the existence of a variety of response properties, if the cells are to be useful in social situations. In many



cases the direction in which another individual’s head is pointing is not a reliable index of where that individual’s attention lies. Gaze direction can be a better guide to the focus of another individual’s attention and should therefore affect STSa cell responses. These predictions for a role in signalling attention direction are borne out since most cells selective to a particular head view are also sensitive to gaze direction (Perrett et al. 1985a, 1990c, 1991, 1992). Moreover, the direction of gaze and the head view to which cells are maximally sensitive tend to be aligned. For example, cells responsive to the head directed towards the observer (full face) are more responsive to eye contact than to laterally averted gaze. By contrast, cells responding to the head turned away from the observer towards pro1le also respond more to laterally averted gaze. Similarly, in the vertical plane, many of the cells that are sensitive to the head pointing downwards are also sensitive to gaze directed down (Fig. 18.2), and many of those selective for the face directed upwards are also selective for the eyes directed upwards (Perrett et al. 1990c). This is illustrated for one cell in Fig. 18.2, which was responsive to the sight of the head oriented towards the ground, and whether the head was seen from the front or seen from pro1le (not shown). For the front view of the head facing the ground, the eyes are not visible, so that responses must be based on visual information derived from other parts of the face. The cell is unresponsive to the face directed towards the camera/observer, but does respond to the face when the eyes are directed downward. An ineffective view of the head becomes effective provided that the eyes point in the correct direction: downwards, not straight ahead or upwards. There are circumstances in which the direction of gaze of another individual is not clear, for example when the eyes lie in shadow. In these cases, the direction of attention can still be analysed from the direction in which the head is pointing. Head angle thus provides a parallel cue to attention direction. Cells showing combined sensitivity to head view and gaze direction (e.g. Fig. 18.2) are therefore capable of signalling the direction of another individual’s attention under a variety of viewing conditions. Since gaze direction can be a more accurate cue to direction of attention than head view, a prediction can be made that if head and gaze cues were put in con2ict, cell sensitivity to gaze direction should override sensitivity to head view. Experimental results support this prediction. Changing the gaze direction can decrease cell response to an effective head view or elevate response to an ineffective head view (Perrett et al. 1985a, 1990c, 1992). Langton et al. (2000) note that, for humans, head and gaze cues may contribute independently (rather than hierarchically) to the analysis of attention direction. Indeed, at the cellular level the interaction of head and gaze cues can be additive rather than prioritized.

18.2.3 Bodies count too! It is also possible to derive visual indications of the direction of attention from an individual’s body posture. It turns out that 60% of cells within the STSa that are responsive to the face have been found to process visual information about the body in addition to the head (Wachsmuth et al. 1994). Visual information arising from body cues appears to contribute to cell sensitivity in a way that is also consistent with the cells’ role in analysing the direction of attention (Perrett et al. 1992). For example, Fig. 18.1 illustrates the responses of one cell to the left pro1le view of the head or the left pro1le view of the body. The cell responses could contribute to signalling that an individual is attending in a direction to the observer’s left. This signalling function could occur in situations where either the head or the body are partially occluded from sight.

359


360


Fig. 18.2 Sensitivity to gaze direction, head, and body postures indicative of attention directed down. Upper parts of (A and B): Schematic illustrations of real 3-D stimuli used for testing. Lower parts of (A and B): Mean (+ /− 1SE) of response of one cell to the stimuli. (a) Sensitivity to head and eye gaze directed down. The cell responded more to a view of the head in which the face was rotated towards the ground than to full-face views ( p < 0.002, each comparison, Newman–Keuls). With the full-face view the cell responded more when the gaze was directed down than to gaze directed at the camera (viewer) or to gaze averted upwards or spontaneous activity (SA, p < 0.005 each comparison). [Overall effect of conditions F(4, 20) = 29.6, p < 0.0005.] (b) Sensitivity of the same cell to body posture. With the head covered, the cell responded more to the quadrupedal posture than to the bipedal posture ( p < 0.0005). With the head visible the cell responded more when the head was pointing at the ground than when it was level ( p < 0.0005). [Overall effect of conditions F(4, 36) = 20.5, p < 0.0005.] (Adapted from Perrett et al. 1992.)



Figure 18.2 illustrates a further example of the responses of a cell sensitive to head view and body posture. The sensitivity of this cell to head down and gaze down has already been discussed. Some of the cells with this type of sensitivity were also found to be responsive to the sight of the body in quadrupedal but not bipedal posture (Perrett et al. 1990c, 1992). For the cell illustrated in Fig. 18.2, sensitivity to the quadrupedal posture was found with the head occluded from sight. These results indicate that three independent types of visual cue, arising from the eyes, head, and body, all impact on the cell. Moreover, the visual information from the head appears to take some priority over the visual cues from the body, since the ineffective bipedal body posture becomes effective when the head is visible and oriented downward, and the effective quadrupedal body posture becomes ineffective when the head is visible but oriented level rather than downwards. The visual sensitivity of this type of cell to the eyes, head, and body posture are each consistent with a role in coding the sight of an individual ‘attending down’ (Perrett et al. 1992). Generally, the conjoint sensitivity to gaze direction, head view, and body posture indicates the cells’ ability to utilize very different types of visual information, yet all of the information to which the cells are sensitive is compatible with the same conceptual interpretation of another individual’s direction of attention. These cells appear to signal where in the world someone else is looking. One can imagine that separate visual descriptions would be built initially for the appearance of the eyes, head, and body and, at later stages in the analysis, the outputs of the appropriate versions of these lower level descriptions would be combined hierarchically to establish selectivity for multiple components of the body. This hierarchical processing scheme 1ts the range of neural sensitivities observed, with some cells sensitive for component facial cues and other cells sensitive to combinations of gaze, facial, and bodily cues (Perrett et al. 1992). Such a hierarchical scheme is analogous to that proposed for the 2ow of information from cell populations with view-speci1c response sensitivity to cell populations with view-general sensitivity (Logothetis, Pauls, and Poggio 1995; Perrett et al. 1984, 1985a, 1989, 1991, 1992; Riesenhuber and Poggio 2000; Seibert and Waxman 1992). Some evidence for such a hierarchical organization with speci1c tuning being combined to establish more conceptually abstract tuning is apparent from analysis of cell response latencies: cells with more abstract properties show onset latencies that are longer than cells with more simple properties (Perrett et al. 1992).

18.3 Integration of form and motion 18.3.1 Time for action The neuronal mechanisms described so far have involved the processing of static visual stimuli. Area STSa receives visual information about the motion of ‘animate’ objects from posterior motion processing areas (Boussaoud, Ungerleider, and Desimone 1990). Comprehension of what other individuals are ‘doing’ may depend on the combined analysis of the individuals’ appearance (their form) and the way they move. Within the STSa there are many cell populations which appear specialized for the coding of speci1c limb or whole body movements (Perrett et al. 1985b). These cells re2ect combined sensitivity to both the form of the body and the nature of its movements (Oram and Perrett 1996). Collectively the cells can be thought of as providing a visual encyclopaedia of recognizable actions of the face and body. Again, the cell types can be arranged in a conceptual hierarchy, starting with cells sensitive to the simple movements of single limbs and ending with cells sensitive to the complex patterns of

361


362


articulation characteristic of whole body movements and cells sensitive to actions in which the body movements are related to objects or goals within the environment (Jellema et al. 2000a; Perrett et al. 1989). Cells sensitive to individual limb movements code articulations in speci1c directions towards or away from the observer, or code changes in limb elevation up or down. Some cells are responsive solely to particular head movements, others to leg or arm movements and others to movements of smaller body or facial components (1ngers, eyes, or mouth). All of these cell types display form sensitivity in that they respond speci1cally to one body part moving but not to equivalent movements involving a different body part (e.g. arm movements but not leg movements). Most cells are unresponsive to control stimuli constructed to resemble the effective body part in size and shape. For some cells, however, the form sensitivity is sometimes reduced such that stick 1gures, which articulate in the same way as the whole body or a speci1c limb, can evoke cell responses (Oram and Perrett 1994; Perrett et al. 1990a). Such sensitivity to patterns of articulation allows the cells to respond to a variety of body types with different colouration and patterning of fur or of clothes (in the case of humans).

18.3.2 Good intentions One key to the coding of intention in actions is sensitivity to information about the direction of attention of the individual performing the action. Usually an agent will attend to the goal of an action that is intended; by contrast the agent’s attention may be elsewhere when an action is unintentional or accidental. We recently studied a population of STSa cells which seems to combine information about the direction of attention of an agent with the action performed by that agent (Jellema et al. 2000a,b); this enables the cells to be sensitive to the intentionality of the action. These cells typically responded when an agent performing a reaching action focused attention onto the intended target-site of the reach. When the agent performed an identical reaching action but directed attention 90 degrees away from the position in which the reaching was aimed, the cells did not respond. These cells can be thought of as combining the outputs from cells speci1cally responsive to arm reaching with the outputs of cells speci1cally responsive to the direction of attention (as conveyed by the face, gaze, and body orientation, e.g. Figs. 18.1, 18.2). The presence of a speci1c object at the target position of the reaching, such as a banana located on a tray, did not affect the responses of these cells. The goal of the reaching in these cases appeared to be a position rather than an object.

18.3.3 Stepping out There is a whole menagerie of cell types located within the STSa. One type that occurs in large numbers is responsive to the sight of whole body movements that are witnessed during walking (Perrett et al. 1985b). Most cells coding whole body movements within the STSa appear to use a ‘viewer-centred’ frame of analysis. That is, changing the observer’s view of the moving body changes the cell’s response. For example, one cell might respond to the right pro1le view of the body moving to the observer’s right, but not to the left pro1le view moving in the same direction. Different cells are sensitive to whole body movements in different directions (left, right, towards, away, up, and down). This is illustrated in Fig. 18.3 for one cell that responds to the front view of the body approaching the observer. Note that changing the view of the approaching body, to left or right pro1le or to the back of the body, eliminates the cell’s response. Likewise, movements maintaining the front view of the body but directed to the left, right, or away from the observer fail to



provoke the cell’s response. Here, then, both the body view and the movement direction must be speci1ed correctly with respect to the observer before the cell responds. Such behaviour is typical of 95% of STSa cells responsive to whole body movements made during walking (Oram and Perrett 1996; Perrett et al. 1985b). The cells differ in the choice of direction and view of the body coded; some respond to walking to the observer’s left, some to the observer’s right. A small number (5%) of STSa cells have been studied that are capable of responding in more abstract terms to whole body movements and behave as if the vantage point of the observer was irrelevant (Figs. 18.4–6). Some of these cells respond to ‘walking forwards’ (e.g. Fig. 18.4), others to ‘walking backwards’ (e.g. Figs. 18.5 and 18.6). The former cells respond best to movements away from the observer when the back view of the body is seen moving; yet for movements towards the observer the same cells respond best to the front view of the body; for movements to the observer’s left the left pro1le view is the optimal one and for movements to the right it is the right pro1le view (e.g. Fig. 18.4). For each direction of movement the body view is critical or for each view of the body the direction of motion is critical (e.g. Fig. 18.5). Such descriptions have ‘object-centred’

Fig. 18.3 Viewer-centred coding for walking towards the observer. Upper: Schematic representation of the view and type of movement. Lower: Mean (+/−1SE) of responses of one cell. The cell responds to the experimenter walking towards the observer so long as the body faces the direction of movement (i.e. walks forwards). The front view of the body seen moving left, right, or away from the observer fails to provoke a response. The experimenter approaching with different body views (left or right pro1le, or back of the body) also produces less response as does a body-sized control object approaching and the static front view of the body. The cell requires a particular combination of body view and direction of motion. [1-way ANOVA main effect of test condition F(8, 57) = 4.95, p < 0.0002; front view approaching greater than all other conditions, p < 0.001.]

363


364


properties because the description walking forwards does not depend on the observer’s vantage point (Hasselmo et al. 1989; Marr and Nishihara 1978; Oram and Perrett 1996; Perrett et al. 1985a,b, 1989). It is easy to speculate that such abstract descriptions of walking forwards and backwards are built hierarchically by combining the outputs of several view-speci1c descriptions of motion. For

Fig. 18.4 Object-centred coding of forward walking or forward translation. Mean (+ /−1SE) of responses of one cell to video clips of walking towards, left, right, or away from the observer so long as the body faces the direction of movement (i.e. walks forward). A schematic representation of the view and type of movement is shown adjacent to each of the responses. Responses to compatible walking are higher than to incompatible walking (where the body is facing away from the direction of motion). [2-way ANOVA, overall effect of compatibly of walking (forwards, backwards), F(1, 15) = 75.8, p < 0.000005; direction of motion (towards, away, to the right, to the left), F(3, 15) = 0.8, p = 0.5; interaction, F(3, 15) = 2.1, p = 0.15.] The cell responses continue to discriminate compatible and incompatible body movements for video 1lm of an actor moved on a trolley without articulation. [2-way ANOVA main effect of compatibility F(1, 14) = 50.5 p < 0.000005, direction of motion F(3, 14) = 1.3 p = 0.3, interaction F(3,14) = 0.3 p = 0.82.]



Response (spikes/sec)

30

Incompatible Compatible Control

24 18 12 6 0

Move towards

Move left

Move away

Move right

30

Response (spikes/sec)

Move left Move right

15

0 0

90

180 Angle of view

270

360

Fig. 18.5 Coding of walking backwards to the right and left. Upper: Schematic representation of the view and type of movement. Lower: Mean (+/−1SE) of responses of one cell to 8 different views of the body moving to the left or right. Curves display the best 1t second order cardioid function (see Perrett et al. 1991). The cell tuning responses show different view tuning for two directions of motion. For each direction the optimal view is one in which the body is oriented the opposite way to the motion.

example, walking forwards independent of view can be manufactured by combining the outputs of viewer-centred cells sensitive to compatible motion directed towards, away, to the left, and right of the observer (Perrett et al. 1985b, 1989, 1990a,b). There are two observations consistent with this speculation. First, latencies of cells responding to whole body movements in a viewer-centred manner tend to be shorter than latencies of cells

365


366


Fig. 18.6 Cell coding upright and inverted backward walking. Mean (+/−1SE) of responses of one cell to walking towards, left, right, and away from the observer, so long as the body faces the opposite direction of movement (i.e. walks backwards). [2-way ANOVA, main effect of compatibility F(1, 38) = 38.4, p < 0.000005, effect of direction of motion F(3, 38) = 3.7, p = 0.02, interaction F(3, 38) = 3.66, p = 0.02.] The cell responses continue to discriminate compatible and incompatible body movements for inverted 1lm of a person walking away from the camera/observer [F(1, 11) = 29.9 p < 0.0002].

responding in the object-centred manner. A similar difference in latencies is apparent for cells responsive to view-speci1c and view-general static information about the head and body (Perrett et al. 1992). Second, one can 1nd cells that show sensitivity to more than one but not all directions of movement. When this occurs we 1nd that the cells show selectivity for one type of view compatible with the movement directions: we do not 1nd cells showing selectivity for compatible forward motion in one direction and incompatible backward motion in another direction. Thus, the cells’ view and direction selectivity seem to re2ect logical combinations rather than random combinations. This is illustrated in Fig. 18.5 for a cell responsive to motion to the left and right of the observer. This cell failed to respond to directions towards or away from the observer. For movements directed left and right, tests were made comparing responses to video 1lms showing eight views of the body moving. It can be seen that for both directions the cell is tuned for the view of the body consistent with it walking backwards.

18.3.4 Articulation that doubles you up Rather than responding to the net displacement of bodies through space, a different type of cell codes for the articulation of the body where one limb or multiple components of the body move with respect to other components. Cells sensitive to articulation come in different ‘2avours’: one apparent division of labour between the cells is whether they code for vertical or horizontal rotation. The articulatory movements of the body can again be described relative to the observer (viewer-centred) or they can be described relative to some other component of the body itself (object-centred: Hasselmo et al. 1989; Perrett et al. 1985b, 1990a,b). When referenced to the observer, horizontal rotations can be speci1ed by the view they bring to confront the observer: rotation towards the observer



bringing the head and body to a front view; rotation away taking the face away and presenting a pro1le or rear view (Perrett et al. 1985b, 1990b). Vertical rotations may move the observed body or some component of it with respect to gravity, for example, lowering the face or chest towards the ground or raising it towards the sky. Figure 18.7 illustrates one cell typical of those sensitive to vertical articulation. For the sight of a human body seated (normally) in a chair, the cell responds to motion increasing the angle of 2exion at the hips and resulting in the chest and face taking on a more skyward orientation. Note that the angle of 2exion through which the cell is responsive is speci1c: it is only when the angle between upper torso and legs exceeds 90° that the cell responds. The body can 2ex in four ways, symbolized by different arrows in Fig. 18.7. We have studied 38 cells of this type responsive to vertical body 2exion. The majority (94%) of these cells were sensitive to bending motion through one of the four quadrants illustrated. Examples are illustrated for bending backwards from upright (Fig. 18.7), and for bending forwards from upright (Fig. 18.8). Thus, the cells code a speci1c type of 2exion or extension (cf. Hasselmo et al. 1989). For this type of cell, we have studied the importance of different component body movements, 2exing just the head, or the head and upper torso. Most of these cells did not require the torso to move and responded to the head movement alone. For a small number of cells (e.g. Fig. 18.7C), the response to the head movement alone was present but reduced compared with the combined head and torso motion, implying the importance of the torso movement. Thus, like the cells coding static information about body posture, some of the cells coding body movement also appear to integrate information from multiple components of the body. Such integration of visual cues from head and torso motion is indicated in Fig. 18.7. Here the head and torso either move in the same direction or twist in different directions. When the movements of the head and upper torso occur in different directions, the movement of the face downwards can suppress the response to the chest moving upwards.

18.4 Generalization across similar actions 18.4.1 Turning the world upside down Further dramatic indications that cell coding can generalize to conceptually similar movements come from tests in which the video of the walking person was vertically inverted. This is illustrated in Fig. 18.6 for one cell that is responsive to walking backwards, when witnessed in the normal upright orientation. The cell remains capable of discriminating compatible and incompatible movement for inverted videos of walking. Similar generalization across viewing orientation is found in cells tuned to articulation of the body and head (see Fig. 18.8). One immediately thinks that no subject has seen inverted walking—so why are their cells capable of generalizing to such unusual movements, and how do their inputs allow the cells to respond? Monkeys and apes (including humans, particularly in their youth) see their companions moving in all sorts of ways: climbing up and down, scrambling forwards and backwards while suspended upside down. Observers, too, will occasionally hang upside down while watching the movements of others in their gravitationally normal lives. So retinal stimulation by movements will occur in a variety of orientations. Temporal cortex appears sensitive to viewing conditions: cells in this area develop selectivity for the appearance of faces and bodies in those conditions in which they are experienced (Ashbridge, Perrett, Oram, and Jellema 2000). Tuning also appears to re2ect the duration of experience (Perrett et al. 1998).

367


368


Fig. 18.7 Coding of the body bending backwards independent of view. Upper left: Schematics depicting the test views of the experimenter relative to the subject. The direction of body articulation is indicated by arrows (numbered 1–4). A: Mean (+ /−1SE) of response to different articulations. Articulations at the hips that bent the body backwards, increasing the distance between the chest and face and the knees, produced larger responses than other articulations (movement number 2; dark arrow and 1lled histogram bars). The cell responded vigorously to this action when seen from the front, back, right, and left side. [2-way ANOVA, showed a main effect of articulation type (movements 1–4), F(3, 128) = 154.1, p < 0.000005; and of view, F(3, 128) = 3.67, p = 0.014; but no interaction, F(9, 128) = 1.42, p = 0.18.] B: Independent sensitivity to head movement. Rotation of the head backwards, starting from the upright position, produced more activity from the cell than rotating the head forwards along the same trajectory and ending in the upright position [F(1, 15) = 20.6, p < 0.0004]. Head movements 1 and 4 did not excite the cell (not shown). C: Interaction of head and body articulation. The responses of the cell to upper body movements were modulated by rotation of the head. Head movements are indicated by small arrows. The data shown represent the averaged responses across front, back, right, and left views. The response to the backward bending of the torso was reduced when the head simultaneously turned downwards (p < 0.0002), whereas the response to the forward bending of the torso was increased when the head simultaneously turned upwards (p < 0.004). [ANOVA, type of bending, F(3, 93) = 88.9, p < 0.00005.]



Fig. 18.8 Coding of the body bending forwards independent of view and orientation. Upper: Schematics depicting the test views and orientations of the experimenter relative to the subject. The directions of body articulations are indicated by arrows (numbered 1–4). Lower: Mean (+ /−1SE) of response to different articulations. With the body in an upright orientation, articulations at the hips that bent the body forwards and brought the chest and face closer to the knees (movement number 4; dark arrow and 1lled bars) produced larger responses than other articulations. [2-way ANOVA, showed a main effect of articulation type (movements 1–4), F(3, 106) = 147.8, p < 0.00005; view, F(3, 106) = 3.51, p = 0.018; and interaction, F(9, 106) = 2.1, p = 0.034.] With the body oriented horizontally the cell continued to respond selectively for the same type of articulation. [2-way ANOVA main effect of articulation type (movement 1 vs. 4) F(1, 35) = 70.2, p < 0.00005; view, F(1, 35) = 0.018, p = 0.89; interaction, F(1, 35) = 1,30, p = 0.26.]

18.4.2 Gliding along Perhaps the real clue to the cells’ capacity to generalize across orientation comes from consideration of the mechanisms by which cells respond selectively to body movements. There are two

369


370


main ways; some of the cells code speci1c patterns of articulation, others code the combination of the form available at each instant plus the direction of displacement. The former cells respond to ‘biological motion’ displays where only a few points of light attached to the body need be visible (Johansson 1973; Oram and Perrett 1994; Perrett et al. 1990a). The latter cell type, which is much more numerous, requires only the form of the face and/or body to be seen translating or changing in scale. This is shown in Fig. 18.4 for one cell, which responded in an object-centred manner to walking forwards in all directions. The cell continued to discriminate forward from backward motion for videotape stimuli in which the body was moved without articulation (or effort) on a mobile trolley, although responsiveness was less than that observed for a real human walking. Indeed, for many cells the movement required can be further simpli1ed. For example, for cells selective to a body walking towards the observer, simply zooming a slide of a face to increase its magni1cation can be suf1cient to evoke responses. Likewise, cells responsive to walking right can respond to a slide projection of the right pro1le of a human body made to drift to the right across a projection screen. Some cells are a little more sophisticated and require translation of the face/body relative to background elements (Hietanen and Perrett 1996a,b; Oram and Perrett 1996). Cells within the STSa that are sensitive to body movements appear to combine two different signals, one available from an STSa cell population that speci1es direction of motion but lacks form selectivity (Oram, Perrett, and Hietanen 1993) and a second STSa population that speci1es the form of the face and body independent of its movement (Perrett et al. 1984). These two sources of information can be seen to arrive separately (i.e. at different times) on individual cells conjointly sensitive to form and motion (Oram and Perrett 1996). Given that the cells selective for whole body movements can combine these two types of input, then it follows that, since about 20% of the cells specifying face and body form generalize over orientation (Ashbridge et al. 2000), some of the cells sensitive to walking will inherit this orientation tolerance and display the trick of coping with upside-down walkers. For cells generalizing to articulations across different orientations, generalization may depend on different mechanisms. For these cells, generalization could depend on coding the relative separation between two parts of the body (Perrett et al. 1990a; e.g. the top of head moving closer to the knees), or the relative speed of motion of different parts of the body towards a further part of the body (e.g. forehead moving towards the hips faster than chin).

18.5 Cortical visual pathways and the location of actions 18.5.1 What, where, how, and why? Our understanding of visual processing has been dominated by the ‘what’ versus ‘where’ dichotomy, proposed by Ungerleider and Mishkin (1982). Their model envisages a separation of visual processing into two distinct cortical streams: a dorsal stream, from V1 into the inferior parietal cortex, which deals with the spatial relationships of objects (the ‘where’ stream), and a ventral stream, extending from V1 into the inferior temporal cortex, dealing with the shape and identity of objects (the ‘what’ stream) (e.g. Desimone and Ungerleider 1989; Haxby et al. 1991, 1993; Köhler, Kapur, Moscovitch, Winocur, and Houle 1995).



Where in this dichotomy do the cells described here 1t? The abundance of cells within STSa that code the visual appearance of the face and body, and their apparent lack of sensitivity to retinal position, orientation, luminance, and colour (e.g. Ashbridge et al. 2000; Ito, Tamura, Fujita, and Tanaka 1995; Lueschow, Miller, and Desimone 1994; Perrett et al. 1984, 1989; Rolls and Baylis 1986) suggest a role in object recognition, and allocate STSa to the ventral stream. In fact the STS runs between dorsal and ventral streams. The assignation of the STS to the dorsal–ventral anatomical dichotomy is therefore ambiguous. The posterior sections of the STS include the motion processing areas (V5/MT and MST) and are traditionally thought of as belonging to the dorsal stream, since they project heavily to the parietal cortex. The anterior sections of the STS lie in the temporal lobe and therefore belong anatomically to the ventral division. The functions of STSa cells in coding the direction of motion of animate objects and the direction of attention of other animals have a decidedly spatial 2avour, which could be allied with dorsal operations. Indeed, such visual information could be sent to the dorsal cortical systems to facilitate control of the observer’s own attention (Hoffman and Haxby 2000; Lorincz, Baker, and Perrett 1999) via the dense anatomical projections to the parietal cortex from the STSa (Harries and Perrett 1991). It is more important to consider the possible functions of STSa cells than to try to 1t their properties into circumscribed functions already assigned to dorsal and ventral processing streams. Attempts to shoe-horn cell types into one or other system blind us to functions that the STSa cells may serve, that are independent of those currently associated with dorsal and ventral streams. If, as we have speculated, STSa cells play a role in social cognition, then they may utilize all sorts of information (including spatial position) that has previously been allocated to the dorsal or to the ventral cortical streams. A number of 1ndings at the neuropsychological level challenge the strict ‘what–where’ dorsal– ventral dichotomy and indicate that object properties may also be coded dorsally (e.g. Goodale, Milner, Jakobson, and Carey 1991; Murphy, Carey, and Goodale 1998). These are substantiated by reports at the cellular level—for example, cells in parietal cortex code for the size and orientation of objects that have to be grasped (Murata, Gallese, Kaseda, and Sakata 1996; Sakata et al. 1998). Sereno and Maunsell (1998) found cell selectivity for passively viewed two-dimensional shapes in the lateral intraparietal cortex. Conversely, there have been recent reports of spatial coding within the ventral stream. Dobbins, Jeo, Fiser, and Allman (1998) reported a high proportion (> 60%) of cells in area V4 displaying changes in response with viewing distance, independent of retinal image size. Even cells in V1 code for certain volumes of space (Trotter and Celebrini 1999). This evidence suggests that both form and spatial cues may be processed in each of the two cortical visual streams. Milner and Goodale (Goodale and Milner 1992; Milner and Goodale 1993, 1995) have reformulated the function of dorsal and ventral visual streams and emphasize the visuomotor nature of processing within parietal areas (‘how’ to deal with objects). One major implication of this revised type of model is that form and space are processed in both pathways but for different purposes. The ventral stream is thought to serve visual ‘perception’, that is, object and scene recognition (cf. Marr and Nishihara 1978) and recognition of ‘why’ an action is occurring (Walsh and Perrett 1994). In the ventral stream, object representations are thought to bene1t from allocentric spatial coding to represent the enduring characteristics of objects and relationships between object components. By contrast the dorsal stream is thought to serve the visual control of ‘action’, and to utilize egocentric spatial coding for short-lived representation of views of an object that are essential for guiding visuomotor interactions with objects. The functions of ventral and dorsal streams emphasize visionfor-perception versus vision-for-action.

371


372


18.5.2 Using your position Lately, we have become aware that spatial position is integrated with form and movement cues to support the comprehension of animals and their actions within the temporal cortex (e.g. Baker et al. 2000, 2001; Jellema and Perrett 1999). Our working hypothesis with respect to the functional signi1cance of spatial coding in STSa is that it plays a role in the visual analysis of the intentions and goals of others’ actions, that is, in social cognition (cf. Abell et al. 1999; BaronCohen 1994, 1996; Brothers 1995; Emery and Perrett 1994, 1999; Jellema et al. 2000a). The spatial locations that individuals occupy are especially relevant in hierarchically organized primate societies. Our preliminary results suggest that spatial coding may indeed be widespread in STSa. The reason why previous studies did not observe it is probably that, given the predominant view of the functions of the dorsal and ventral visual streams, most studies on the ventral stream have been biased towards investigating object processing, neglecting possible effects of position. A convergence of information about the motion and form of objects within STSa has been con1rmed at the cellular level (Oram and Perrett 1994, 1996; Tanaka et al. 1999), but a possible spatial in2uence on STSa cells has not been studied systematically. Anatomical studies reveal that STSa receives an abundance of projections, which could provide the position information: from the parahippocampal cortex (Seltzer and Pandya 1994), the entorhinal cortex (Good and Morrison 1995), the posterior parietal cortex (Baizer, Ungerleider, and Desimone 1991; Seltzer and Pandya 1984), the anterior part of IT (Baizer et al. 1991; Boussaoud et al. 1990; Morel and Bullier 1990), and posterior regions of STS (Boussaoud et al. 1990). For instance, the distance sensitivity observed in cells in area V4 (Dobbins et al. 1998) may well extend into STSa, since V4 forms the main visual input onto IT, and IT projects heavily onto STSa. The hippocampus and/or parahippocampal gyrus may provide the spatial input to STSa via its projection onto the perirhinal cortex (Seltzer and Pandya 1994). Thus, information about space has a profound in2uence in the temporal lobe but its utilization in the visual processing of complex objects and actions is only just becoming apparent. Baker et al. (2001) described substantial numbers of cells with responses that were selective for static views of the body and which were additionally sensitive to the distance of the body from the observer. It now appears that the cells in the STSa tuned to body movement can also possess spatial selectivity. Based on preliminary data, we 1nd sensitivity in STSa to walking depends on the combination of each of the three factors (form, motion direction, and spatial position). Changing just one factor can abolish the response, while no single factor is suf1cient to evoke the response: spatial cues are necessary but not suf1cient to produce a response. Preliminary data show that, for cells sensitive to approaching or retreating movements of the experimenter, there is a tendency for maximal responses at the ‘near’ location for cells responsive to compatible walking towards the subject, and at the ‘far’ location for cells responsive to compatible walking away from the subject (Jellema and Perrett 1999). This suggests that the cells favour certain combinations of location, form, and direction of motion above others. Such spatial sensitivity appears to be present in at least some cells selective for walking directed left or right. This is illustrated for one cell in the upper half of Fig. 18.9. The cell responded to the sight of walking to the right, more than to the sight of walking to the left. The cell shows spatial sensitivity, with greater responses to the experimenter when in positions on the right-hand side of the room compared with the left-hand side. The spatial selectivity of this cell is relatively weak compared with that exhibited by other STSa cells sensitive to movement in depth towards and away from the observer.



Fig. 18.9 Coding visible and hidden movement. Responses (mean + /− 1 SE) of one cell to the sight of the experimenter moving (A) in and (B) out of sight. The monkey’s view of the lab is illustrated on the right. Grey rectangles denote the position of occluding screens in the middle and on the right of the experimental room 4 m from the subject. Solid arrows indicate the position and direction of walking while the experimenter was visible (a–c). Dotted arrows indicate position of the experimenter when hidden from sight and the direction of movement prior to occlusion (d–f). [2-way ANOVA, visibility of movement (in sight vs. out of sight), F(1, 24) = 15.7, p = 0.0006; direction and position of movement (3 levels; left position move right, right position move left, right position move right), F(2, 24) = 57.8, p < 0.00005; interaction, F(2, 24) = 2.4, p = 0.12; n = 5 for each condition.] The response to the experimenter out of sight on the right (f) was greater than all other conditions (a–e), p < 0.004, each comparison. Walking towards the right screen (c) evoked a greater response than walking towards the middle screen (a and b) [c vs. b, p < 0.0005; c vs. a, p < 0.05]. Walking towards the middle screen produced a larger response when starting from the left (a) than when starting from the right (b) [p < 0.002]. (From Jellema et al. 2002.)

18.6 Actions as events extending over time 18.6.1 Now you see me, now you don’t Actions often become partially or completely hidden from view. Since the predictability of impending sensory stimuli has a pervasive in2uence on STSa responses to tactile and motion stimuli (Hietanen and Perrett 1993a,b, 1996a,b; Mistlin and Perrett 1990), we have investigated how cells respond when actions such as walking become hidden from sight. These studies revealed a population of visual cells in STSa that respond maximally when individuals are seen to ‘hide’ behind an occluding screen. Of particular relevance to the discussion of spatial coding was the 1nding that all of the cells studied in this population (n = 30) were sensitive to the location of the hidden person (Baker et al.

373


374


2001). Thus these cells responded maximally after the individual had moved out of sight at a particular location in the lab. For example, in Fig. 18.9 the cell illustrated responded more in the 3 s following disappearance from sight behind screens than in the prior 3 s when the experimenter was visible and moving towards the screens. Many cells have no detectable response to visible movements but start responding 1–4 s after the person moving has become completely hidden. The cell illustrated in Fig. 18.9 responded maximally when the experimenter was hidden behind a screen located at the far right side of the experimental room (response to condition f is greater than to all other conditions a–e). Hiding behind a screen located in the middle of the room at the same distance from the subject (d and e) produced less response. The cell’s responses to the experimenter walking in-sight were consistent with the out-of-sight responses in that a larger response was evoked when the experimenter walked towards the right screen (c) than towards the middle screen (a and b). Additionally, walking towards the middle screen produced a larger response when starting from the left (a) than when starting from the right (b). Thus, the maximal out-of-sight response was obtained when the experimenter hid behind the right-hand screen, and the maximal in-sight response was obtained when the direction of walking was towards the right-hand screen and when the site of walking was closest to the right-hand screen. These in- and out-of-sight responses are consistent with the idea that this cell codes not only for the presence of the experimenter behind the right screen, but also for the intention of the experimenter to go behind that screen. For this interpretation, we need only assume that walking towards the right screen re2ects the intention to move behind that screen. The use of spatial information in temporal cortex contrasts sharply with (1) the spatial sensitivity of the hippocampal ‘view’ cells (Rolls et al. 1997, 1998), which do not require an object at the optimal position or view, and (2) spatial sensitivity in premotor cortex (Graziano, Yap, and Gross 1994; Graziano, Hu, and Gross 1997), which applies to any object which happens to be in the near space. The speculation here is that position sensitivity of STSa cells may help the interpretation of current impending actions, providing indications as to what is likely to happen next.

18.6.2 What happens next? A 1nal property exhibited by some cells within the STSa is intriguing in this context. We have just found that the perceptual history or sequence of events is critical in determining the response to the current scene. To some extent this is apparent for the cells that respond when animate objects become hidden from sight. For example, in Fig. 18.9, when the person has walked out of sight, the scene is identical, yet the cell’s response depends critically on what was last seen—where the experimenter was and in which direction he was moving before disappearing. More dramatically, we 1nd that cell responses to static views of the head and body in a particular posture can depend entirely on the preceding movement (Jellema, Baker, Wicker, and Perrett 2000b). We have studied 31 cells for which responses occur when one particular movement leads to the posture, but responses are absent if the same posture is presented from behind a shutter with no perceptual history, or is presented after a different preceding movement. Actions and behaviour are complex: much of their complexity derives because actions are composed of complex sequences of movements extending over time and developing in different ways. What we are beginning to see within the STSa is that cells are sensitive to elementary sequences of two events. It is quite likely that comprehension of complex and lengthy action sequences can be stitched together, hierarchically, from sensitivity to these elementary sequences.



18.7 Discussion 18.7.1 Bondage without pain The cells reviewed here require two or more visual attributes to 1re: for example, motion to the left and the left pro1le view of the body (Oram and Perrett 1996). In essence, the cells require the two features to be ‘bound together’, that is, to arise from one object rather than from two independent objects (one object moving left, and a second object with the left pro1le form; Oram and Perrett 1996). Philosophers, psychologists, and neuroscientists continue to be fascinated by the ability of the nervous system to detect such conjunctions and solve the ‘binding problem’. It is therefore appropriate to discuss how the process may be achieved. Indeed, it need be no mystery; the essence of a mechanism to solve the binding problem without the use of spooky codes has been available for 30 years (Wicklegren 1969; for review see Perrett and Oram 1998). Binding happens whenever simple features are combined to allow a more complex pattern to be detected. To understand binding we therefore need to understand how hierarchical feature processing works, for example, how a diamond shape is detected and differentiated from an X, which has many of the same features but in a different order. The same problem exists for cells tuned to face patterns, which may require the visibility of several eye-like and mouth-like features to 1re but can remain unresponsive when the same features are presented in a jumbled con1guration. Details of the way temporal cortex binds information together to detect facial patterns are given elsewhere (Perrett and Oram 1998). Here we can restrict ourselves to simpler examples, but it should be clear how such examples extend to more complex properties. Consider the detection of a diamond shape (◊). The activity in feature detectors tuned for individual orientations does not reveal the diamond pattern explicitly, nor does activity in a collection of detectors sensitive to pairs of oriented elements as angular corners, e.g. >, 1/3 the biological trend was exaggerated. Finally, the movement was truly biological only for 1 – β = 1/3. There were 1ve different trajectories, each of which was presented 12 times with a different initial distribution of velocity determined by selecting β at random among 12 values. Figure 21.2 shows


Motor competence in the perception of dynamic events: a tutorial

the results for a typical subject. Panels (a) to (e) (one for each trajectory) show the progressive convergence of the control parameter toward the value for which velocity appeared constant. Panel (f) shows mean and standard deviation of the selected β for each trajectory. The outcome of the experiment was quite clear. All subjects consistently and accurately selected the biological movement (i.e. β = 2/3) as the best approximation to a constant velocity stimulus. Moreover, when the initial β was positive, nobody, while searching for the appropriate setting of the parameter, ever approached the value β = 1 corresponding to true constant velocity. Similar results were obtained both when the random trajectory was replaced by ellipses of various eccentricities, and in several control conditions. The size of the illusion revealed by these experiments is surprisingly large: velocity variations as large as 250% that occurred in the biological case were not detected. After the experiment, we showed to the subjects what a true constant-velocity movement looked like; we also showed a point moving on a straight path with the velocity pro1le that had been perceived as constant. Both demonstrations were always received with the utmost skepticism. The results described above lead to two distinct (albeit related) conclusions. On the one hand, in formulating velocity judgments, we have access to some implicit knowledge of the motor rule expressed by the Two-thirds Power Law. On the other hand, movements that comply with this rule are perceived as uniform. It is worth stressing that the 1rst conclusion does not entail the second. We could ‘resonate’—as Gibson (1966) would put it—to biological movements while acknowledging that they are not uniform. Thus, the reason why we disregard the large velocity variations present in our stimuli remains to be explained. Runeson (1974), noting that certain motions arising in nature are similar to that of a mass that is acted upon by a continuing force in the presence of viscous friction, argued that the mass–spring–dashpot model is implicitly assumed as a model for all (one-dimensional) displays and concluded that ‘only natural motions look constant’ (p. 11). Along similar lines, one may surmize that biological movements are important enough from the ecological point of view to justify their role of default model for many dynamic displays. If so, one can make the further assumption that perceiving such displays involves a comparison with the best 1tting model prediction, and that accelerations and decelerations are perceived only when the actual velocity deviates from the predicted one. Are motor–perceptual interactions con1ned to vision? The fact that perceptual phenomena like apparent motion have their counterpart in other sensory modalities (Sherrick and Rogers 1966; Perrot 1974) suggests that these interactions may be more pervasive. A demonstration that this is in fact the case was provided by a recent experiment on kinesthetic perception (Viviani, Baud-Bovy, and Redol1 1997). The logic of the experiment and the stimuli were very similar to those of the Viviani and Stucchi (1989) study described above. The key difference was that the dynamic elliptic stimuli were not presented visually, but fed into a computerized robotic arm which drove the passive right arm of the blindfolded subject (Fig. 21.3). The movement continued until the subject identi1ed the orientation of the major axis of the ellipse, which he knew to be either vertical or horizontal. As in the visual experiment, the eccentricity in the 1rst trials was so large that the orientation was easily detected, but it decreased after each correct response, making the task increasingly dif1cult. Also, we tested the same three cinematic conditions described above. In the 1rst condition velocity was constant. In the other two conditions the velocity pro1les would be biological if the trajectory were an ellipse (eccentricity: 0.7) with a horizontal or a vertical major axis, respectively. The results were even clearer than in the case of vision (Fig. 21.4). When the biological model 1t well the kinesthetic in2ow near the point of objective equality (constant velocity for quasi-circular

419


420


Fig. 21.3 Experimental set-up used for studying kinesthetic perception. A six-degrees-of-freedom robotic arm drives the subject’s right arm through elliptic trajectories with a controllable velocity pro1le. In one experiment the subject has to indicate the orientation of the major axis of the ellipse. In a second experiment (see Fig. 21.7) the subject has to reproduce with the left arm the movement imposed on the right one (from Viviani et al. 1997). trajectories), the aspect ratio of the stimulus was perceived with a small constant error (CE) and a small differential limen (JND). Large, systematic CEs and large JNDs were measured in the other two conditions where the modulation of velocity was inconsistent with the quasi-constant curvature of the trajectory. When the movement decelerated at the right and left extremes of the trajectory, the subjective circle was in fact a vertical ellipse. The opposite bias was present in the other condition. The fact that kinesthetic estimates of vertical and horizontal extent are biased just like visual estimates invites an obvious inference: because two sensory channels that have little in common display the same sensitivity to the relation between form and velocity, the site of the interaction between perception and implicit motor competence must be upstream of the primary sensory mechanisms, perhaps at a level where stimuli are represented in some amodal format.

21.2.4 Only lawful perceptual stimuli guide action effectively Many actions are accomplished successfully even when the main source of information that is meant to provide guidance is incomplete or corrupted. One reason for such robustness is the ability to exploit secondary sources of information that would otherwise be redundant. The second, even more important reason is the brain’s ability to 1ll in for the missing information by drawing from its stock of preconceptions about the way the world goes. Of course, this very same ability can actually make things worse if the course of events in the world actually fails to comply with our preconceptions. Conversely, any evidence that an action performed under the guidance of a sensory input is not performed as accurately as expected points to the possibility that the input deviates from some tacit expectation. The four experiments summarized below were designed for the purpose of identifying one peculiar form of deviance, and, by the same token, one peculiar type of expectation.



0.8 0.6 0.4 0.2 0.0 – 0.2 – 0.4 – 0.6 – 0.8

Condition A

0.8 0.6 0.4 0.2 0.0 – 0.2 – 0.4 – 0.6 – 0.8

Condition B

0.8 0.6 0.4 0.2 0.0 – 0.2 – 0.4 – 0.6 – 0.8

Condition C

Eccentricity

v

0

Flat ellipses

Tall ellipses Flat ellipses

Tall ellipses Flat ellipses

Tall ellipses 5 10 15 20 25 30 35 40 45 50 55 60 65 70 Trial number

Fig. 21.4 Perception of the aspect ratio (horizontal axis/vertical axis) of an elliptic motion is affected by the relation between velocity and curvature. Left: velocity (V) distribution of the right hand driven by the robotic arm. Velocity (heavy lines) is plotted in polar coordinates using the hand trajectory as zero reference. The three distributions are exempli1ed in the special case of a circular trajectory; however, the same distributions were used for all trajectories within a condition. Condition A: constant velocity, i.e. the biological distribution for circular motions. Condition B: velocity was maximum at the top and bottom portions of the trajectory; this is the biological distribution for a horizontal ellipse with eccentricity 0.7. Condition C: velocity was maximum at the leftmost and rightmost portions of the trajectory; this is the biological distribution for a vertical ellipse with eccentricity 0.7. Right: Results for one typical subject. Staircase traces describe the variations of the eccentricity (ordinate) of the trajectory for all trials (abscissa) of a complete experiment. By convention, positive and negative values of the eccentricity refer to horizontal and vertical ellipses, respectively. Ascending and descending series alternated randomly within the experiment. After each correct answer, the eccentricity was reduced (in absolute value); after each mistake the eccentricity was increased. The experiment ended when there had been at least 15 inversions in both series indicating that responses were at chance level. The trajectory perceived as circular was identi1ed by the average eccentricity (dotted lines) calculated from the last 10 inversions. There is very little bias in Condition A where the imposed movement is biological near the point of perceived circularity. The trajectories perceived as circles in Conditions B and C are vertically and horizontally elongated ellipses, respectively (reproduced from Viviani et al. 1997).

421


422


All the experiments involved variants of the pursuit-tracking task in which the hand or the eye is asked to follow dynamic visual targets. In the 1rst experiment (Viviani and Mounoud 1990), the task was to follow, with a stylus, a two-dimensional point light target which was tracing an elliptic trajectory. The experimental factors included the rhythm of the target, the orientation of the major axis of the ellipse, and the way the instantaneous velocity varied along the trajectory. For each combination of rhythm and orientation there were two cinematic conditions (Fig. 21.5). In the 1rst one, (N), the velocity varied according to the biological model. In the second one, (T), the velocity varied in a way that would have been biological for an ellipse oriented at 90 deg with respect to the actual trajectory. Therefore, although both average and maximum velocity were identical in the two cases, the second condition forced the subject to perform a movement that he would never have produced spontaneously. In the biological condition tracking was quite accurate, even at the highest rhythm. Deviations from the target trajectory were very modest, and the delay of the hand with respect to the target was small and constant. By contrast, in the non-biological condition tracking was poor in both space and time domains. The pursuit trace deviated systematically from the target; in addition, it was rotated by as much as 30 deg with respect to the orientation of the ellipse. Things were even worse in the time domain where the interval between target and pursuit oscillated between leads and lags. Aside from the rotation, all the deviations could be accounted for in a simple way: although the task required a strict coupling with the target, the hand went on performing a distorted but highly identi1able biological movement, as if the target was unable to provide appropriate guidance. Similar results were also obtained in another experiment (Viviani, Campadelli, and Mounoud 1987) in which the target followed unpredictable trajectories generated by recording actual scribbling movements. Once again, the key experimental factor was the velocity pro1le, which could be either the original (biological) one, or a constant velocity pro1le which violated the Two-thirds Power Law. One may ask whether the performance errors demonstrated by these two tracking studies have to do with the hand control system per se. A recent experiment involving eye movements (de’Sperati and Viviani 1997) showed that the problem is as general as the perceptual biases described in the previous section. We asked subjects to follow with the gaze a point light tracing ellipses on the computer screen, and recorded the eye tracking movements with high-accuracy scleral-coil lenses. By keeping the cycling rhythm constant, and varying the eccentricity of the ellipses, we explored the upper range of the velocities that, in the case of one-dimensional, predictable targets, are still compatible with accurate smooth pursuit. As in the hand pursuit experiments, the main controlled variable was the velocity pro1le of the target along the trajectory. Each eccentricity was tested with several pro1les computed by inserting a different value of the exponent in the general expression of the power law (eqn 1). Only the pro1le for β = 2/3 corresponded to that of a biological movement. In ocular tracking tasks, smooth pursuit phases are generally interspersed with catch-up saccades, the number and size of which depend on the dif1culty of the task. For each velocity pro1le, tracking accuracy was estimated by two indexes based on retinal position error (RPE) before and after catch-up saccades. The 1rst index measured the increase in the distance between gaze and target during the pursuit phase between successive saccades. The second index measured the difference between the RPE immediately before and after each saccade. Both indexes had a clear minimum around the value β = 2/3, demonstrating that even pursuit eye movements are most effective when the target follows a biological movement. The similarity between hand and eye tracking behavior is further emphasized by comparing the instantaneous values of the RPE around the trajectory (Fig. 21.6) with the analogous plots of the instantaneous hand-target delay (cf. Fig. 21.5). Despite the large difference between hand and eye control systems, both are constrained to generate movements that comply


Fig. 21.5 Non-biological movements cannot be tracked by the hand. Results of a visuo-manual pursuit-tracking experiment (one typical subject). Subjects had to track with a stylus a light point target tracing elliptic trajectories (eccentricity: 0.9). N: Biological condition; the instantaneous velocity of the target complies with the Two-thirds Power Law (i.e. the target mimics a voluntary hand movement tracing the same horizontal ellipse). T: Non-biological condition; the target velocity is that of voluntary hand movement following a vertical ellipse with eccentricity 0.9. Average, maximum, and minimum velocities are identical in the two conditions, but the orientation of the extrema is rotated by 90° (inset polar diagrams). Lower panels: Average pursuit trace (heavy lines) superimposed on target trajectory (thin lines). Upper panels: polar plots of the instantaneous delay between target and pursuit using the target trajectory as zero reference. By convention, values inside (outside) the trajectory indicate that the pursuit leads (lags) the target. In the non-biological condition, tracking is inaccurate in the space domain, and totally in the time domain deviant (Reproduced from Viviani and Mounoud 1990.)


424


Fig. 21.6 Non-biological movements cannot be tracked by the eye. Results of an experiment in which subjects had to pursue-track a light point which was tracing elliptic trajectories. The main experimental variable was the velocity pro1le of the target, which was controlled by setting the value of the exponent (β) in the power law (eqn (1) in the text). Only one pro1le corresponded to a biological movement (β = 2/3), all others representing graded departures from this model. The angular distance between target and gaze (Retinal Position Error, RPE) affords a measure of the effectiveness of the tracking. The four polar plots describe the RPE (average over all trails and all subjects; data points pooled over the indicated 16 sectors) for four selected values of β and one trajectory (eccentricity: 0.968). The RPE is small and approximately constant in the biological case. It becomes larger and more variable as the stimulus departs more markedly from the biological model. with the power law, and both resist in a similar manner all attempts to make them move in any other way. To conclude this section on tracking, let us consider again the case of kinesthetic stimuli. The computer-controlled robotic arm used to test the perception of the aspect ratio, was also used to test the ability to reproduce with the left hand the passive displacements imposed on the right hand. In this experiment the trajectories imposed by the robot were always ellipses elongated either horizontally or vertically. The task was to rotate voluntarily the left hand in synchrony with right one, following the same perceived trajectory (cf. Fig. 21.3). The rotation had to be performed both symmetrically (i.e. by engaging homologous muscles of the arms) and anti-symmetrically (i.e. by pairing the agonist muscles of one arm with the antagonist muscles of the other). Because neither arm was visible during the task, the imitation was based only on the kinesthetic inputs coming from the displaced limb. There were two conditions that corresponded to those in the Viviani and Mounoud (1990) visuo-manual tracking experiment, described earlier. In the first condition the velocity distribution was congruous with the eccentricity and the direction of the elongation of the trajectory according to eqn (1). In the second condition, in contrast, there was maximum discrepancy because the movement decelerated at points of low curvature and accelerated at points of high curvature. Finally, there was a third condition in which velocity was constant. Only in the congruous case did the left hand faithfully reproduce the movement of the right hand. In the second condition the trajectory was distorted and much more variable from cycle to cycle



(Fig. 21.7). The results for the constant velocity condition were somewhat intermediate between the other two. The inability of the left hand to reproduce the motion of the right one was particularly obvious in the time domain. Instead of being small and constant, the delay between hands varied systematically within a cycle in a manner that resembled strikingly both the results of the Viviani and Mounod study (cf. Fig. 21.5), and the RPE plots from the ocular pursuit experiments (cf. Fig. 21.6). In all three cases, the variations in the delay simply re2ected the fact that the tracking movement remained close to the biological model instead of reproducing the imposed template.

Fig. 21.7 Non-biological movements cannot be mimicked. The left hand is trying to reproduce the vertical elliptic movement imposed by the robotic arm (cf. Fig. 21.3) on the right arm (eccentricity: 0.9). Column A: Biological condition; the velocity of the imposed movement reproduces that of a voluntary active movement following the same vertical elliptic trajectory. Column B: First nonbiological condition; the imposed movement has constant velocity. Column C: Second non-biological condition; the velocity of the imposed movement is that of a voluntary movement tracing a horizontal ellipse. Row D: Polar plots of the velocity (heavy lines) using the trajectory (G) as zero reference. Also shown the trajectory (K) that would make the velocity distribution compatible with the Two-thirds Power Law. Row E: Continuous polar plots of the instantaneous delay between left and right hands (heavy lines) for one representative trial. The right-hand trajectory is used as a reference for measuring delays: data points outside (inside) this reference indicate that the left hand lags (leads) the right one. Inside the reference line are also shown the ten movement cycles of the left hand. Row F: Average and standard deviation of the delay computed over all trials in all participants, pooling the data points for the indicated eight sectors. Note the similarity with the analogous data from the visuo-manual pursuit tracking experiment (Fig. 21.5) and with the distribution of the RPE in the eye tracking experiment (Fig. 21.6). (Reproduced from Viviani et al. 1997.)

425


426


In summary, all four studies demonstrated that neither visual nor kinesthetic inputs provide appropriate guidance for the movement unless they comply with the prescriptions of the power law. Although the required responses were well within the acceptable dynamic range of the hand and of the eye, the pursuit of non-biological targets was somewhat disorganized. Moreover, whatever regularity remained in the motor responses was the expression of a tendency to remain close to the biological model normally followed by unguided, spontaneous movements. It must be stressed that, in all but one study, stimuli were periodic and highly predictable. Yet there was no evidence, even after many cycles, that the sensorimotor loop would learn how to take advantage of such regularity.

21.2.5 A helping hand So far, we have seen that all sorts of strange things happen when innate motor schemata are applied to dynamic stimuli that violate certain biological constraints. To conclude this section, I will emphasize instead the good things that may happen when the biological constraints are satis1ed. The popularity of mental imagery, already a subject of interest by the end of the nineteenth century, has been increasing ever since the techniques of brain imaging opened a window into the underlying mechanisms. Motor images, both when they refer to the imager being engaged in an action (grasping, pointing, running, etc.), and when they refer to the representation of an external agent, presuppose a voluntary (albeit imaginary) effort. It is debated whether this fact alone sets motor images apart from other types of imagery (e.g. mental rotations of geometrical 1gures), or whether, as argued by Annet (1995), the distinction has no real ground. Be that as it may, it is now 1rmly established that motor images elicit activity in many cortical areas normally engaged in the planning of actual movements (cf. Crammond 1997; Jeannerod and Decety 1995). Although the activation of the primary motor area is still controversial (Kawamichi, Kikuki, Endo, Takeda, and Yoshizawa 1998), motor images follow the general rule that applies also to visual, auditory, and tactile images, namely that many cortical areas that are responsive to actual sensimotor events are equally responsive to events evoked from within. Moreover, as is the case for visual images (Shepard and Cooper 1982), temporal and cinematic properties of mental images mimic those of the real, represented events (Parsons 1994; Sirigu et al. 1995, 1996). In monkeys, neurons in the superior temporal polysensory area respond selectively to visual stimuli representing biological motion (Oram and Perrett 1994). More importantly, certain cells (the so-called ‘mirror’ neurons) of section F5 of area 6 that are normally activated during the performance of grasping movements are also activated when the animal observes someone else performing the same gesture (di Pellegrino, Fadiga, Fogassi, Gallese, and Rizzolatti 1992; Gallese, Fadiga, Fogassi, and Rizzolatti 1996). This important discovery has been generalized to humans by showing that the inferior frontal gyrus is active when one simply observes a grasping gesture. Moreover, a PET study (Decety et al. 1994) has shown that the same cortical area is also active when grasping is only imagined. Interestingly, the meaningfulness of the observed gesture appears to be crucial for the involvement of the frontal area (Decety et al. 1997). Also, the fact that section F5 in monkeys partially overlaps with Broca’s area in humans suggests the fascinating hypothesis that mirror neurons may have played a role in the genesis of language (Rizzolatti, Fadiga, Gallese, and Fogassi 1996). The observations summarized above point to a close connection among the mechanisms responsible for interpreting a perceived action, imaging performing that action, and preparing for its performance. They also invite inferences about the functional signi1cance of such a connection. As for the relation



between perception and action, Rizzolatti et al. (1996)—echoing Liberman—have suggested that ‘when an external stimulus evokes a neural activity similar to that which, when internally generated, represents a certain action, the meaning of the observed action is recognized because of the similarity between the two representations, the one internally generated and that evoked by the stimulus’ (p.137). In the same vein, Parsons and Fox (1998) noted that ‘The implicit knowledge the brain possesses about movements it can actually generate may also in2uence our interpretation of observed actions’ (p. 599). As regards motor imagery, instead, the hypothesis that motor templates (covertly) activated by the imager (presumably similar to those that would be engaged in a real action) are also instrumental for perceiving the action that they are supposed to realize was formulated by Annet (1969), even before supporting physiological evidence became available. The fact that the con1gural aspects of handwritten letters can be described more accurately while imagining the letter being drawn than while looking at the actual patterns (Zimmer 1982) supports the even stronger hypothesis that motor imagery represents the medium in which recognition or identi1cation of dynamic events take place. In a more subdued vein, Decety et al. (1994) suggested that mentally evoked acts ‘involve rehearsal of neural pathways related to cognitive stages of motor control’ (p. 600). Recent experiments have lent support to these general views. People are generally good at identifying objects seen from different viewpoints, the subjective report being that identi1cation often occurs after some mental manipulation of the image. Discriminating images of the left and right parts of the body (hands, feet) is a somewhat special case because, rather than manipulating the image, observers report that they compare the stimulus to a mental representation of their own body, moving the hand in imagination until it matches the stimulus (Parsons 1987). These subjective reports were con1rmed by an experiment (Parsons 1987) in which subjects were shown pictures of either the left or the right hand in different orientations. In one condition, the task was simply to reproduce with a real movement of the appropriate hand the posture being shown. In a second condition no overt movement was required, the task being simply to tell whether the stimulus was a right or a left hand. In both conditions response times varied as a function of the orientation of the hand. More importantly, the two response times were highly correlated, suggesting that left/right judgements were indeed reached by comparing an initial perceptual cue with a simulated movement of the corresponding hand. More recently (Parsons and Fox 1998), a PET study in which hand images were presented in either hemi-1eld has con1rmed that left/right judgments elicit limb-speci1c activation of the hand contralateral to the side of the presentation. The perceptual in2uence of covert action and the role of handedness were investigated by de’Sperati and Stucchi (1997). Right- and left-handed subjects were shown computer animations of a screwdriver rotating along its main axis, and had to decide whether the tool was screwing or unscrewing (Fig. 21.8). The orientation of the screwdriver and the sense of rotation varied from trial to trial. For some orientations the posture for gripping the tool was quite natural for the right hand, and awkward for the left. The converse was true for other orientations. In one condition subjects were told to rely only on visual cues. In a second condition they were encouraged to imagine their dominant or non-dominant hand grasping the screwdriver. As expected on the basis of previous studies on mental rotation (cf. Shepard and Cooper 1982), response times depended on stimulus orientation (Fig. 21.9). The new 1nding was that response times were longer when subjects were instructed to imagine using the non-dominant hand for a grip that would have been more natural for the dominant one. Again, the results suggest that eliciting a mental image of the dominant hand was the natural strategy for responding, whether or not this was suggested. When instructions explicitly con2icted with this tendency, the response took as much as 0.7 s longer than when no instruction was given.

427


428


Fig. 21.8 Static representation of the stimuli used for demonstrating the role of motor imagery. Actual stimuli were presented in a random order at the center of the screen and were rotating (180 deg/sec) around their main axis. Subjects had to indicate as soon as possible whether the rotation was clockwise (screwing) or counterclockwise (unscrewing). In the observation condition, subjects were not encouraged to use motor imagery. In the imagery conditions, subjects were encouraged to imagine grasping the screwdriver with either their dominant or non-dominant hand before answering. Both right- and left-handers were tested (reproduced from de’Sperati and Stucchi 1997).

21.2.6 Perceptual anticipation Reading a handwritten text is a most impressive perceptual feat. On the one hand, letter templates are recovered quite reliably in spite of the extravagant idiosyncratic variations to which they are subjected. On the other hand, the recovery process does not discount the variations, for otherwise we would not be able, for instance, to recognize a familiar handwriting. Because theories of handwriting recognition working their way up from abstracted geometrical features to templates (cf. Gibson and Levin 1975) proved inadequate, attempts have been made to incorporate the contribution of motor knowledge. By taking inspiration from what graphologists have always claimed, Zimmer (1982) conjectured that recognition uses information about the way the letter was written, and that such information is derived from tacit knowledge of the writing method. A recognition experiment (Freyd 1983) involving an arti1cial character set supported the conjecture. Subjects who had learned one of two methods of tracing the characters in the template form performed well when asked to identify characters that had been distorted in a way that was congruous with the method they had learned. Performance dropped when the distortion was instead consonant with the other method. A subsequent experiment (Babcock and Freyd 1988) tested the sensitivity to variations in the handwritten trace. It was found that, when asked to reproduce arti1cial letters from memory, subjects unconsciously adopted the stroke direction used to trace the template set that they had memorized. Therefore, motor competence makes it possible to extract from a memory trace information



Right-handers (n=15) 5.5

5.0

4.5 4.0 3.5

Left-handers (n=15) 5.5

OBS RHI LHI

5.0 Response time (s)

(b)

Response time (s)

(a)

v

3.0 2.5

4.5 4.0 3.5 3.0 2.5

2.0

2.0

1.5

1.5 –165 –115 –65 –15 15

65 115 165

Screwdriver orientation (deg)

OBS RHI LHI

–165 –115 –65 –15 15

65 115 165

Screwdriver orientation (deg)

Fig. 21.9 Response times (averaged over all trials and all subjects) as a function of the experimental factors (OBS: Observation condition; RHI: Right-hand imagery condition; LHI: Left-hand imagery condition). Screwdriver orientation has a major effect of all response times. Response times in the observation condition were not statistically different from those measured when the subjects imagined using the dominant hand. Instead, latencies were signi1cantly longer when subjects had to imagine using the non-dominant hand (reproduced from de’Sperati and Stucchi 1997).

relating to production. A recent study to be described below demonstrated that motor competence is also instrumental in exploiting anticipatory affects. Planning and execution of complex sequences of movements involve a signi1cant amount of look-ahead revealed by the fact that units of motor action being executed often carry the imprint of yet-to-be-executed units. Anticipatory adjustments are present in many language-related movements such as speech (Benguérel and Cowan 1974), Morse code (Bryan and Harter 1897), typing (Viviani and Laissard 1996), and handwriting (Thomassen and Schomaker 1986). In handwriting, anticipatory adjustments can be used to predict the letter that is about to be traced (Orliaguet, Kandel, and Boë 1997). Kandel, Orliaguet, and Viviani (2000) investigated the basis of this predictive ability. Speci1cally, we tested the hypothesis that reliable predictions can be made only when the stimuli complied with the Two-thirds Power Law. We recorded a set of 100 instances each of the trigrams LLL and LLN handwritten by one individual (Fig. 21.10, A, B). Two template traces of the middle L were generated from each set by selecting and averaging the 10 traces with the least within-set temporal variance, and the least between-set geometric variance (Fig. 21.10, C, D). The shapes of the templates were almost indistinguishable, and their total duration was normalized to 1 sec. However, the velocity pro1les of the middle L embedded in the two trigrams were different (Fig. 21.10, E, F). In one variant of the experiment, subjects were shown the templates traced on the computer screen by a light point. They were informed that the trace was excerpted from a continuous writing movement, and that the following letter (not presented) could be either another L or a N. The task was to guess this letter, but subjects had the option of not answering if they had no clue. Following the technique introduced by Viviani and Stucchi (1992),

429


430


(a)

Trigram : LLL

(c)

Trigram : LLN

1 cm

(d)

1 cm

(b) V (cm/sec) 30 20

R (cm) V (cm/sec) 40 30 (e) 30 20 20

10

30 20

10 10

0 0.0

R (cm) 40 (f)

0 0.5 1.0 Time (sec)

10 0 0.0

0.5 Time (sec)

0 1.0

Fig. 21.10 Stimuli used to demonstrate that only in biological movements do coarticulatory cues provide the basis for perceptual anticipation. (a), (b): Typical instances of the complete traces of the two trigrams used in the experiment. The traces are recordings of actual writing movements. Only the portion between the two dots was actually shown. The stimulus was a light point tracing the middle letter on the computer screen. Immediately after the disappearance of the trace, subjects had to guess the third letter. The experimental variable was the velocity pro1le of the light point. Using the exponent of the power law (eqn (1) in the text) as a parameter, we tested seven velocity pro1les, only one of which mimicked the biological writing movement. (c), (d): Average trajectories (templates) of the middle letter computed over 10 recordings of the trigrams in (a) and (b), respectively. Bands around the averages indicate the geometrical variability. (e), (f): average and standard deviation of the tangential velocity for the two templates in (c) and (d), respectively. Durations have been normalized to 1 s and velocities have been scaled accordingly. Also shown (lower traces) is the instantaneous value of the radius of curvature of the templates (reproduced from Kandel et al. 2000). the invariable trajectory of each template was traced on the screen with 7 different velocities computed from eqn (1) by setting the exponent β to the values 1/6, 2/6, 3/6, 4/6, 5/6, 6/6, 7/6. At the middle value (β = 4/6) the velocity covaried with the curvature of the trace as it does in biological movements. For all other values of β, stimuli departed in a controlled fashion from the biological model. The results were unequivocal (Fig. 21.11). When the velocity mimicked that of the original handwriting movement, the following letter was guessed with a much higher probability than chance. Moreover, the rate of ‘No answer’ remained very low even though subjects were a priori quite skeptical about their guessing ability. The performance degraded with increasing distance from the biological



1.0 Trigram : LLL Trigram : LLN

0.9

Response probabilit y

0.8

Correct answer

0.7 0.6 0.5 0.4 0.3 Wrong answer 0.2

No answer

0.1 0.0

1/6

2/6

3/6

4/6 5/6 Exponent

6/6

7/6

Fig. 21.11 Response probabilities (average over all subjects) as a function of the exponent of the power law. The middle value (4/6) corresponds to the Two-thirds Power Law approximation to a biological movement. Note that the rate of wrong answers increases well beyond chance level when the stimulus departs drastically from the biological model (reproduced from Kandel et al. 2000).

model. At the four extreme values of β there was even a paradoxical inversion of the predictions. These 1ndings are in keeping with the view that discriminal information is evoked by the stimuli through the interaction with implicit motor knowledge. The only objective basis for predicting the following letter was the internal timing of the traces, which remained invariant across β values. One can then make the hypothesis that discriminal information gained from this only cue was used to trigger an internal simulation of the complete gesture (i.e. including the next letter), and that the response was selected on the basis of the outcome of the simulation. If so, the high error rate at extreme β values may not be such a paradox. Perhaps the invariant timing of the stimulus suggests an initial (correct) guess. However, when the velocity–curvature covariation is grossly at variance with the simulation (which, by de1nition, follows the Two-thirds Power Law), the observer is induced to reject the initial guess and opt for the (wrong) alternative.

21.3 Speculations I have attempted to marshal empirical evidence for the claim that perception of dynamical events involves, inter alia, framing the sensory data within a set of relational constraints derived from our motor competence. I do not expect the case to be watertight, if only because the sample of studies reviewed here is neither exhaustive nor random. Moreover, certain issues still need to be addressed in order for the claim to make contact with current theorizing on sensorimotor interactions. In this closing section, I will try to state these issues as concisely as possible by asking two questions. First, where do constraints come from? Second (and, relatedly), what is their format?

431


432


21.3.1 Where do constraints come from? In phrasing the point of view defended here in terms of constraints acting on internal representations, I have adopted a clear cognitive stance departing both from the Helmholtzian doctrine that percepts are constructed by simply combining sensory cues, and from the Gibsonian doctrine that the only constraints that we need to take into account are the invariances that nature imposes from without on the sensory 2ow. Actually, I have used the term ‘model’ somewhat interchangeably with the term ‘constraints’, to emphasize the basic tenet of cognitive psychology that the transition between sense data and percepts is realized when the former are framed within an internally available script. Internal constraints (or model, or scripts) may be innate or emerge from the internalization of signi1cant, recurrent regularities present in the world. Linguistic constraints (Universal Grammar) that map multidimensional conceptual representations onto one-dimensional sequences of expressive gestures are a prime example of the former possibility; whatever their remote origin, they are now well entrenched in the genetic code. Although the nature of linguistic constraints is still controversial, it remains that if a child does not talk by age 4, something is awry in his brain—whereas executing a novel musical score in real time is a skill based on the harmonic and melodic rules that some individuals have interiorized through practice: we are all born to talk, but no one is a born pianist. Where does the perception of dynamic events stand between these two extremes? One in2uential opinion was expressed by Shepard (1984). Shepard argued that the process of internalization is not an all-or-none affair, and that the difference between temporary attitudes and dispositions, at one end of the continuum, and genetic determinants, at the opposite end, is a matter of degree, not of substance (actually, a similar view has been defended even in the case of language; see Lieberman 2000). As regards the perception of dynamic events, Shepard acknowledges the reality of short-term phenomena, such as priming and cueing (e.g. the path-guided apparent motion; Shepard and Zare 1983). The emphasis, however, is on entrenched constraints. In his words: ‘There are good reasons why the automatic operations of the perceptual system should be guided more by general principles [ . . . ] than by speci1c principles governing the different probable behaviors of particular objects’(p. 426), and ‘What is perceived is determined [ . . . ] by much more general abstract invariants that have instead been picked up genetically over an enormous history of evolutionary internalization’ (p. 431). Favoring innate factors over learning sounds very reasonable and uncontroversial. Because it is so important for survival, coding in the genes the principles that make dynamic perception effective is a far better solution than acquiring these principles from scratch during each individual life. Two other components of Shepard’s credo are more substantive and more open to debate. First, he states that, no matter how far back in the past, the ultimate source of inspiration for the principles he is advocating has to be traced to the regularities present in the external world. Even ‘[syntactic rules] may have been built upon already highly evolved rules of spatial representation and transformation, [and may be] to some extent traceable to abstract properties of the external world’ (note 6, p. 431). Then, he further quali1es his view by stating that, as far as perception of motion is concerned, the single most important source of inspiration is cinematic geometry which ‘governs motions of rigid objects, or of local parts of rigid objects, during brief moments of time’ (p. 422). Both statements deserve comment. As for the 1rst one, it is indeed likely that some of the most general principles (e.g. the impossibility of two solid objects occupying the same position at the same time, or the fact that the duration of a displacement cannot be zero) re2ect mandatory world-constraints. However, because at any stage of its development the brain has also been part of the world, there seems to be no reason



to rule out the hypothesis that some speci1c principles re2ect constraints pertaining to the functioning of the brain itself. Actually, some amount of constraining from within may turn out to be particularly useful whenever perception requires a preliminary parsing of the sensory messages. Language perception provides a prime example of such a situation, and it is no accident that one of the most articulated motor theories of perception is the one proposed by Liberman to account for the effectiveness with which phonemes are extracted from the acoustic 2ow (Liberman, Cooper, Shankweiler, and Studdert-Kennedy 1967; Liberman and Mattingly 1985). Liberman argued that the objects of speech perception are the intended phonetic gestures of the speaker represented in the listener’s brain as the invariant motor commands that he himself would have issued to produce the same sounds. If this premise is valid (recall the problematic presence of categorical sound perception in animals), the principles at work must have become engrained in the perceptual system pari passu, and in strict association with the development of the speci1c mechanisms for speech production, quite independently of any regularity present in nature. By analogy, the same internal coupling between production and perception might have evolved in other perceptual domains, such as the facial expression of emotions (cf. Eckman and Davidson 1994), expressive hand gestures, and, more generally, in all those cases where the best possible normative model for interpreting incoming sensory information is provided by the very same production rules that we would apply to broadcast that information. After all, non-living objects do not laugh, conduct an orchestra, or make threatening gestures. How could they compete with us as a model for all these activities? Moreover, given that our motor competence provides a direct, reliable basis for such a model, is it really necessary to invoke the genetic interiorization of unspeci1c principles as Shepard did? The hypothesis that cinematic geometry is a source of inspiration of our perceptual models, is clearly supported by robust phenomena (cf. Shepard and Cooper 1982). However, inasmuch as it countenances only cinematic quantities, this hypothesis makes no provision for distinguishing among types of movement (see above). Speci1cally, it does not acknowledge the distinction between biological and non-biological movements that I have stressed throughout this chapter. In fact, such distinction is mentioned only cursorily in Shepard’s 1984 paper. For sure, some simple biological movements (e.g. the swing of the legs during walking) can be described within the framework of kinematics. This is not the case, however, for the vast majority of expressive limb movements (the very same movements for which I have advocated a direct production–perception coupling) inasmuch as they carry the imprint of the logic of the underlying control system. As a simple illustration of this point, consider the well-known positive correlation between the distance to be covered and the average velocity of pointing movements (Fitts’ law). This correlation is the overt manifestation of some intrinsic rule of the motor system and is certainly a perceptually salient cue in the identi1cation of biological gestures. Yet moving objects do not display such a correlation unless the driving force was planned accordingly—an implausible circumstance in the early days of evolution. In short, while I agree that cinematic geometry might have contributed to shaping perceptual models, I would rather assign a preeminent role to biological dynamics. By reversing the balance suggested by Shepard, I am here suggesting that, even when this may not be appropriate, humans tend to adopt biological motion as the default model for dynamic perception.

21.3.2 What is the format in which constraints are represented? A discussion of the neuronal bases of the production–perception coupling is beyond the scope of this chapter. Instead, I will adopt a functionalist point of view by asking just what we might mean by

433


434


saying that motor competence provides a framework for dynamic perception. Before attempting an answer, however, I have to hark back to a more general question. Traditionally, the fact that perception and action are subserved by distinct anatomical structures has motivated the inference that perceptual contents and action goals are represented separately in different formats (indeed, the very fact of countenancing a production–perception coupling presupposes a degree of functional autonomy between the underlying representations). This seemingly uncontroversial assumption has been challenged by the Theory of Event Coding (TEC) which holds that perceived and to-be-executed events are coded within a common representational medium (cf. Hommel, Müsseler, Aschersleben, and Prinz 2001). The theory’s key concept is the event code, which is construed – both for percepts and motor gestures – as a collection of features describing identi1able properties of the event. By necessity, the reference frame adopted by the theory is uncompromisingly allocentric: the features coded internally refer to the distal events, not to the proximal stimulation. Moreover, features are amodal: in their perceptual role, they integrate the contribution of all available sensory channels; in their motor role, they modulate the activities of various components of the motor system. The theory provides a natural framework for interpreting both priming and interference effects. Suppose that two events share a feature. Perceiving or planning one event will 1rst activate all its features, including the feature shared with the other event. Thus, in this preliminary phase, priming prevails. In contrast, interference sets in at a subsequent stage, when the features of the selected event are bound together into a coherent unit, and the shared feature is no longer available to the non-selected event. Inasmuch as TEC posits that the distinction between percepts and actions is not the re2ection of a type distinction between their internal representations, but rather the re2ection of the different roles that event codes are called to play in any given circumstance, it would appear to be ideally suited to explaining why motor competence biases perception. Upon re2ection, however, things may turn out to be less straightforward. To exemplify, let us consider again Shiffrar and Freyd’s experiment (cf. Fig. 21.1) from the point of view of TEC. Presumably the sequential presentation of the body images activates two distinct events, corresponding to a small and a large rotation of the forearm, respectively. Which one wins over the other and reaches consciousness is determined by the relative strength of competing biasing factors, i.e. the timing of the presentation, and the degree of biomechanical compatibility. The fact that both have a chance, however, poses a problem to TEC because no event corresponding to a small rotation should exist: after all, why should one want to generate a code that, qua movement, would break the elbow? The more general point that I want to emphasize by this example is that positing a type identity between percepts and gestures does not do justice to the fact that perception has more degrees of freedom than movement. It is indeed possible that the internal representation of percepts relative to gestures is intimately related to, or even coincides with the representation of the gestures. It is also possible, as we have seen, that gestures provide a privileged model for interpreting sensory data. It is questionable, however, that TEC’s basic assumption holds true in the case of percepts (and there are many) for which the model is hopeless because there is no action that may be sensibly related to that percept. In the spirit of TEC, the notion of coupling adopted here becomes almost redundant. Instead, if my criticism is correct, it still makes sense to ask which form the in2uence of motor competence on perceptual representations takes. Ironically, a source of inspiration comes from the notion of resonance introduced by the very same Gibson who did not think much of internal representations anyway. The 1rst attempt to spell out a theory of perception based on resonance is again due to Shepard (1984) who used actual resonating systems as a metaphor for identifying the features that recom-



mend this concept as a basis for perception. The major features of linear resonators (the only type considered by Shepard) are: (1) Productivity. Resonators are designed to respond maximally to just one category of stimuli; however, they also exhibit the spontaneous tendency to respond in a sustained, characteristic manner to unstructured impulses from within. (2) Tunability. Many resonating systems can be tuned to different frequencies by acting on just one parameter (e.g. the length of the diapason’s rod, or the value of the capacity in an oscillating electrical circuit). Tunability is important insofar as it entails controllability. (3) Robustness. A resonator tuned to one frequency also responds (although to a lesser degree) to stimuli of different frequency, as well as to incomplete or corrupted stimuli. This is a desirable feature for a system that has to deal reliably with the vagaries of the environment. (4) Sympathy. A collection of resonators, only one of which is activated (from without or from within, see point 1 above), tend to join in, responding indirectly to the activity of the precursor. This spontaneous spreading of activation provides a hypothetical but congenial basis for such well-known phenomena as perceptual completion. Ever since Shepard’s initial proposal, the scope of the resonator metaphor has broadened considerably, leading to the so-called dynamical systems theory. Mostly under the impulsion of S. Kelso, M. Turvey, E. Thelen, and their collaborators, a full-2edged theory of motor coordination has been developed (cf. Kelso 1995; Kugler and Turvey 1987; Thelen and Smith 1994) by cross-breeding and generalizing certain seminal ideas that had been around for quite a while. One idea, 1rst advertised by Bernstein (1967), is that the variables involved in controlling movements are organized in groups (synergies) by a network of mutual constraints. Synergies within a dynamic system can be characterized by collective variables in a control space with much lower dimensionality than that of the system itself. Another idea is that, to a 1rst approximation, certain movements resemble the oscillations of resonators. In order for these intuitions to become fruitful, the original resonator concept had to be generalized along two directions. Firstly, it was expedient to consider non-linear resonators (or oscillators as they are now more commonly called), that is, systems in which displacement and velocity-dependent energy terms are no longer related by linear equations. Secondly, the notion of sympathetic resonance (point 4 above) had to be extended to include explicit coupling among a set of oscillators, thus proving a basis for the establishment of synergies. A characteristic property of this generalized class of dynamical systems is that they posses limit cycles, that is, trajectories in the phase space (basin of attraction) towards which the behavior of the system converges for any initial state within a set. A system may possess more than one limit cycle, each with its own basin of attraction, allowing the (asymptotic) stable state to depend on the initial conditions. Moreover, dynamical systems have the property that a small change in their parameters may result in a sudden (catastrophic) change in the qualitative type of its attractors (bifurcations), thus providing a way to make continuity and discreteness coexist within a uni1ed framework. Finally, the single most signi1cant feature of the dynamical approach is that the way the process being described unfolds in real time is taken into account at the same time as the sequence of states through which it achieves the desired goal. Unlike most computational models, which specify only the nature and sequence of the intervening processing stages, dynamical models also emphasize the time course of the process, which becomes one of their testable predictions. Although the dynamical systems approach was developed initially to account for motor behavior, a growing number of cognitive scientists are now advertising the possibility—indeed the desirability —of applying the approach to many other domains of cognitive science, including perception and development, for which, until recently, the computational paradigm has been the reference guide (cf. Port and van Gelder 1995). In particular, multiple limit cycles seem ideally suited to modeling such

435


436


Sensory inputs

Motor resonators

Perceptual resonators

Coupling

Mirror neurons??

Motor commands

Fig. 21.12 A functional scheme for describing motor–perceptual interactions; full description in the text. well-known phenomena as bistable and categorical perception. To conclude, I wish to suggest a further extension, by arguing that the dynamic systems approach may also be conducive to addressing the problem of representing the production–perception coupling. The scheme I am entertaining here (Fig. 21.12) features two sets of hierarchically organized oscillators, subserving the generation of percepts and the organization of actions, respectively. Each set comes with a full complement of within-set couplings (some genetically speci1ed, some acquired through learning) responsible for integrating and orchestrating the activation of the individual components in the respective domains. The scheme also contemplates between-set couplings which, again, may be inborn or acquired. Consonant with what I said before, the perceptual set is granted a richer repertoire of resonant modes (or limit cycles) than the motor set. Which perceptual mode prevails at any one time depends largely on the sensory in2ow. However, the selection of the winner is also in2uenced by the couplings with the motor set. Through these couplings, resonant modes are induced in the motor oscillators even in the absence of any direct activation originating from the will to move. In fact (and here I am taking direct inspiration from TEC) there is no reason to posit a type difference between motor and perceptual oscillators. Oscillations (as well as between- and within-set couplings) may well originate from the same physiological mechanisms, as long as the respective roles are assigned by their speci1c source of activation. The important point is that, whether activation spreads from the perceptual to the motor set or the other way around, the result is itself an integrated, superordinate limit cycle. Thus, some familiar perceptual experiences would be precluded to an individual in whom the motor set of oscillators were not functional. Conversely (but, perhaps, to a lesser extent) suppressing perceptual resonances should alter the normal motor repertoire, not only because reafferences would be missing, but also because some global resonant modes would no longer be available. To give a feeling of how this scheme might work, let us see how it accounts for some of the observations summarized in the preceding section. Consider 1rst the peculiar motor–perceptual interaction described by Stricker (see Section 21.1). Imagining the moving clouds involves triggering from within (i.e. from visual memory) a complex pattern of resonant modes in the perceptual set. By itself, the trigger would not be suf1cient for sustaining the pattern. A stable percept emerges because the initial activation spreads to the motor set through a coupling that, presumably, has been established by the fact that pursuit eye movements are normally associated with that perceptual pattern.



If the motor set is silenced (damped) by a act of will, its necessary contribution to the establishment of the global pattern is no longer there, and the clouds stop. As a second example, let us again take up Shiffrar and Freyd’s 1990 experiment. Here the trigger comes from without in the form of a sequence of images and, as argued before, excites two distinct perceptual modes. Again, activation spreads to the motor set. In this case however, the contribution of the motor set is highly selective, because no resonance exists there which would correspond to an elbow-breaking torsion. Reinforcement is given to only one of the two competing perceptual modes. The experiment has shown that this contribution is decisive provided that it overcomes the biasing action of the timing of the sequence. The 1nal example is the misperception of the aspect ratio demonstrated by Viviani and Stucchi (1989). In the control condition of that experiment (condition A), when the light point is tracing a circle at a constant velocity, the perceptual resonant mode is reinforced by the coupling with the motor set, which is itself capable of sustaining a similar mode (recall that we spontaneously draw circles at constant velocity). In condition B, the same circular trajectory is traced with a velocity that would be natural for a horizontally elongated ellipse. The two determinants of the motion (shape and velocity) no longer correspond to a limit cycle in the phase space of the motor set. Therefore, the sympathetic resonance of the motor set is not stable. Striving for equilibrium, the complete dynamic system ends up distorting the perceived shape (possibly, the velocity as well) in a direction that reduces the discrepancy. Surely, the conceptual scheme presented above is highly speculative, as the title of the section promised. However, it is not beyond the reach of empirical tests. In particular, I believe that experiments in which subjects are asked to make dynamic perceptual judgments while performing independent motor tasks should be able to address the basic hypothesis behind the scheme. I also believe that the scheme provides one of the simplest solutions to the problem of representing time, a most important ingredient for perceiving dynamic events. It has been argued (Freyd 1987) that represented and external time must be related isomorphically, so that the 1rst shares with the second the properties of continuity and directionality (cf. Palmer 1978). In other words, the representation itself should be construed as a process unfolding with its own time-scale. If one accepts these premises, oscillators stand out as ideal candidates for the role of timekeepers. Moreover, the implicit knowledge that is supposed to affect perception via the coupling between motor and perceptual sets may well concern the internal timing of the perceptual process.

Acknowledgement The preparation of this chapter was partly supported by FNRS Research Grant, 31–55620.98.

References Annet, J. (1969). Feedback and human behaviour. Harmondsworth: Penguin Books. Annet, J. (1995). Motor imagery: Perception or action? Neuropsychologia, 33, 1395–1417. Anstis, S. and Ramachandran, V. (1985). Kinetic occlusion by apparent motion. Perception, 14, 145–149. Babcock, M.K. and Freyd, J.J. (1988). The perception of dynamic information in static handwritten forms. American Journal of Psychology, 101, 111–130.

437


438


Bassili, J.N. (1978). Facial motion in the perception of faces and of emotional expressions. Journal of Experimental Psychology: Human Perception and Performance, 4, 373–379. Beardworth, T. and Bukner, T. (1981). The ability to recognize oneself from a video recording of one’s movement without one’s body. Bulletin of the Psychonomic Society, 18, 19–22. Bell, C. (1811). On the motions of the eye, and illustration of the uses of the muscles and nerves of the orbit. Philosophical Transactions of the Royal Society, London, 113, 166–168. Benguérel, A.P. and Cowan, H.A. (1974). Coarticulation of upper lip protrusion in French. Phonetica, 30, 41–55. Berkeley, G. (1709). An essay towards a new theory of vision. Dublin. [reprinted: London: J.M. Dent (1969)]. Bernstein, N. (1967). The coordination and regulation of movements Oxford, UK: Pergamon Press. Bertenthal, B.I. and Pinto, J. (1994). Global processing of biological motions. Psychological Science, 5, 221–225. Bertenthal, B.I., Prof1tt, D.R., and Cutting, J. (1984). Infant sensitivity to 1gural coherence in biomechanical motion. Journal of Experimental Child Psychology, 37, 213–230. Bertenthal, B.I., Prof1tt, D.R., Spetner, N.B., and Thomas, A. (1985). Development of the perception of biomechanical motions. Child Development, 56, 531–543. Bertenthal, B.I., Prof1tt, D.R., and Kramer, S.J. (1987). Perception of biomechanical motion in infants: Implementation of various processing constraints. Journal of Experimental Psychology: Human Perception and Performance, 13, 577–585. Börjesson, E. and von Hofsten, C. (1972). Spatial determinants of depth perception in two-dot motion patterns. Perception and Psychophysics, 11, 263–268. Börjesson, E. and von Hofsten, C. (1973). Visual perception of motion in depth: Application of a vector model to three-dot motion patterns. Perception and Psychophysics, 13, 169–179. Bornstein, M.H. (1987). Perceptual categories in vision and audition. In S. Harnad (Ed.), Categorical perception, pp. 287–300. Cambridge: Cambridge University Press. Braunstein, M.L. and Andersen, G.J. (1984). Shape and depth perception from parallel projections of threedimensional motion. Journal of Experimental Psychology: Human Perception and Performance, 10, 749–760. Bryan, W.L. and Harter, N. (1897). Studies in the physiology and psychology of the telegraphic language. Psychological Review, 4, 27–53. Caramazza, A. and Shelton, J.R. (1998). Domain-speci1c knowledge systems in the brain: The animate–inanimate distinction. Journal of Cognitive Neuroscience, 10, 1–34. Claparède, E. (1902). Expériences sur la vitesse du soulèvement des poids de volumes différents [Experiments on the speed of lifting of weights with different volumes]. Archives de Psychologie, 1, 69–94. Coren, S. and Girgus, J.S. (1978a). Seeing is deceiving: The psychology of visual illusion. Hillsdale, NJ: Erlbaum. Coren, S. and Girgus, J.S. (1978b). Visual illusions. In R. Held, H.W. Leibowitz, and H.L. Teuber (Eds.), Handbook of sensory physiology, Vol.8: Perception, pp. 549–568. New York: Springer-Verlag. Crammond, D.J. (1997). Motor imagery: Never in your wildest dream. Trends in Neuroscience, 20, 54–57. Cutting, J.E. (1981). Coding theory adapted to gait perception. Journal of Experimental Psychology: Human Perception and Performance, 7, 71–87. Cutting, J.E. and Kozlowski, L.T. (1977). Recognizing friends by their walk: Gait perception without familiarity cues. Bulletin of the Psychonomic Society, 9, 353–356. Cutting, J.E., Prof1tt, D.R., and Kozlowski, L.T. (1978). A biomechanical invariant for gait perception. Journal of Experimental Psychology: Human Perception and Performance, 4, 357–372. Cutting, J.E., Moore, C., and Morrison, R. (1988). Masking the motion of human gait. Perception and Psychophysics, 44, 339–347. Decety, J., Perani, D., Jeannerod, M., Bettinardi, V., Tadary, B., Woods, R., Mazziotta, J.C., and Fazio, F. (1994). Mapping motor representations with positron emission tomography. Nature, 371, 600–602. Decety, J., Grèzes, J., Costes, N., Perani, D., Jeannerod, M., Procyk, E., Grassi, F., and Fazio, F. (1997). Brain activity during observation of actions: In2uence of action content and subject’s strategy. Brain, 120, 1763– 1777. de’Sperati, C. and Stucchi, N. (1997). Recognizing the motion of a graspable object is guided by handedness. NeuroReport, 8, 2761–2765. de’Sperati, C. and Viviani, P. (1997). The relationship between curvature and velocity in two-dimensional smooth pursuit eye movements. Journal of Neuroscience, 15, 3932–3945. di Pellegrino, G., Fadiga, L., Fogassi, L., Gallese, V., and Rizzolatti, G. (1992). Understanding motor events: A neurophysiological study. Experimental Brain Research, 91, 176–180.



Dittrich, W.H. (1993). Action categories and the perception of biological motion. Perception, 22, 15–22. Duhamel, J.-R., Colby, C.L., and Goldberg, M.E. (1992). The updating of the representation of visual space in parietal cortex by intended eye movements, Science, 255, 90–92. Eckman, P. and Davidson, R.J. (Eds.) (1994). The nature of emotion: Fundamental questions. Oxford: Oxford University Press. Eimas, J.L., Siqueland, E.R., Jusczyk, P., and Vigorito, J. (1971). Speech perception in infants. Science, 171, 303–306. Farah, M.J., McMullen, P.A., and Meyer, M.M. (1991). Can recognition of living things be selectively impaired? Neuropsychologia, 29, 185–193. Finke, R.A. and Freyd, J.J. (1985). Transformation of visual memory induced by implied motions of pattern elements. Journal of Experimental Psychology: Learning, Memory, and Cognition, 11, 780–794. Flash, T. (1990). The organization of human arm trajectory control. In J. Winters and S. Woo (Eds.), Multiple muscle systems: Biomechanics and movement organization, pp. 282–301. Berlin: Springer-Verlag. Fodor, J.A. (1980). Fixation of belief and concept acquisition. In M. Piattelli Palmarini (Ed.), Language and learning: The debate between Jean Piaget and Noam Chomsky. London: Routledge and Kegan. Freyd, J.J. (1983). Representing the dynamics of a static form. Memory and Cognition, 11, 342–346. Freyd, J.J. (1987). Dynamic mental representations. Psychological Review, 94, 427–438. Freyd, J.J. and Finke, R.A. (1984). Representational momentum. Journal of Experimental Psychology: Learning, Memory, and Cognition, 10, 126–132. Freyd, J.J. and Finke, R.A. (1985). A velocity effect for representational momentum. Bulletin of the Psychonomic Society, 23, 443–446. Gallese, V., Fadiga, L., Fogassi, L., and Rizzolatti, G. (1996). Action recognition in the premotor cortex. Brain, 119, 593–609. Gibson, J.J. (1966). The senses considered as perceptual systems. London: George Allen and Unwin. Gibson, J.J. (1979). The ecological approach to visual perception. Hillsdale, NJ: Erlbaum. Gibson, E.J. and Levin, H. (1975). The psychology of reading. Cambridge, MA: MIT Press. Gillam, B. (1971). A depth processing theory of the Poggendorff illusion. Perception and Psychophysics, 10, 211–216. Grieser, D. and Kuhl, P. (1989). Categorization of speech by infants: Support for speech-sound prototypes. Developmental Psychology, 25, 577–588. Grüsser, O.J. (1986). Interaction of efferent and afferent signals in visual perception. A history of ideas and experimental paradigms. Acta Psychologica, 63, 3–21. Helmholtz, H. von (1867). Handbuch der Physiologischen Optik. Leipzig: Voss. [English translation: J.P.C. Southall (Ed. and transl.) A treatise on physiological optics. New York: Dover, 1962.] Heptulla Chatterjee, S., Freyd, J., and Shiffrar, M. (1996). Con1gural processing in the perception of apparent biological motion. Journal of Experimental Psychology: Human Perception and Performance, 22, 916–929. Hertz, H. (1894/1956). The principles of mechanics. (D.E. Jones and J.T. Walley, transl.) New York: Dover. Hoffman, D.D. and Flinchbaugh, B.E. (1982). The interpretation of biological motion. Biological Cybernetics, 42, 195–204. Holst, E. von and Mittelstaedt, H. (1950). Das Reafferenzprinzip. Naturwissenschaften, 37, 464–476. [English translation in: R. Martin (Ed. and transl.) Selected papers of Erich von Holst: The behavioral physiology of animals and man, Vol. 1. London: Methuen, 1973]. Hommel, B., Müsseler, J., Aschersleben, G., and Prinz, W. (2001). The Theory of Event Coding (TEC): A framework for perception and action planning. Behavioral and Brain Sciences (in press). James, W. (1906). Psychology. London: Macmillan. Jeannerod, M. and Decety, J. (1995). Mental motor imagery: A window into the representational stages of action. Current Opinions in Neurobiology, 5, 727–732. Johansson, G. (1950). Con1gurations in events perception. Uppsala: Almqvist and Wiksell. Johansson, G. (1973). Visual perception of biological motion and a model for its analysis. Perception and Psychophysics, 14, 201–211. Johansson, G. (1975). Visual motion perception. Scienti1c American, 232, 76–88. Johansson, G. (1976). Spatio-temporal differentiation and integration in visual motion perception. Psychological Research, 38, 379–393. Johansson, G., von Hofsten, C., and Jansson, G. (1980). Event perception. Annual Review of Psychology, 31, 27–63. Jusczyk, P. (1986). Speech perception. In K.R. Boff, L. Kaufmann, and J.P. Thomas (Eds.), Handbook of perception and human performance, Vol. II: Cognitive processes and performance (pp. 1–57). New York: Wiley.

439


440


Kandel, S., Orliaguet, J.-P., and Viviani, P. (2000). Perceptual anticipation in handwriting: The role of implicit motor competence. Perception and Psychophysics, 62, 706–716. Kawamichi, H., Kikuki, Y., Endo, H., Takeda, T., and Yoshizawa, S. (1998). Temporal structure of implicit motor imagery in visual hand-shape discrimination as revealed by MEG. Cognitive Neuroscience, 9, 1127– 1132. Kelso, J.A.S. (1995). Dynamic patterns: The self-organization of brain and behavior. Cambridge, MA: MIT Press. Kolers, P. and Pomerantz, P. (1971). Figural change in apparent motion. Journal of Experimental Psychology, 87, 99–108. Korte, A. (1915). Kinematoskopische Untersuchungen [Cinematoscopic investigations]. Zeitschrift fuer Psychologie, 72, 194–296. Kozlowski, L.T. and Cutting, J.E. (1977). Recognizing the sex of a walker from a dynamic point-light display. Perception and Psychophysics, 21, 575–580. Kugler, P.N. and Turvey, M.T. (1987). Information, natural law, and the self-assembly of rhythmic movement. Hillsdale, NJ: Erlbaum. Kuhl, P.K. (1987). Perception of speech and sound in early infancy. In P. Salapatek and L. Cohen (Eds.), Handbook of infant perception, Vol. II: From perception to cognition, pp. 274–382. New York: Academic Press. Kuhl, P.K. (1991). Human adults and human infants show a ‘perceptual magnet effect’ for the prototypes of speech categories, monkeys do not. Perception and Psychophysics, 50, 93–107. Kuhl, P.K. and Meltzoff, A. (1982). The bimodal perception of speech in infancy. Science, 218, 1138–1141. Kuhl, P.K. and Miller, J.D. (1978). Speech perception by the chincilla: Identi1cation functions for synthetic VOT stimuli. Journal of the Acoustical Society of America, 63, 905–917. Kuhl, P.K. and Padden, D.M. (1983). Enhanced discriminability at the phonetic boundaries for the place feature in macaques. Journal of the Acoustic Society of America, 73, 1003–1010. Lacquaniti, F., Terzuolo, C.A., and Viviani, P. (1983). The law relating kinematic and 1gural aspects of drawing movements. Acta Psychologica, 54, 115–130. Liberman, A.M. and Mattingly, I.G. (1985). The motor theory of speech perception revisited. Perception, 21, 1–36. Liberman, A.M., Cooper, F.S., Shankweiler, D.P., and Studdert-Kennedy, M. (1967). Perception of speech code. Psychological Review, 74, 431–461. Lieberman, P. (2000). Human language and our reptilian brain: The subcortical bases of speech, syntax, and thought. Cambridge, MA: MIT Press Lotze, R.H. (1852). Medizinische Psychologie oder Physiologie der Seele [Medical psychology or the physiology of the soul]. Leipzig: Weidmann. Mach, E. (1885). Beiträge zur Analyse der Emp1ndungen [English translation: Contributions to the analysis of sensations. La Salle, IL: Open Court (1897)]. Marean, G.C., Werner, L.A., and Kuhl, P. (1992). Vowel categorization by very young infants. Developmental Psychology, 28, 396–405. Massey, J.T., Lurito, J.T., Pellizzer, G., and Georgopoulos, A.P. (1992). Three-dimensional drawing in isometric conditions: Relation between geometry and kinematics. Experimental Brain Research, 88, 685–690. Mateef, S. (1978). Saccadic eye movements and localization of visual stimuli. Perception and Psychophysics, 24, 215–224. Mather, G., Radford, K., and West, S. (1992). Low-level visual processing of biological motion. Proceedings of the Royal Society of London, 249, 149–155. Matin, L. (1986). Visual localization and eye movements. In K.S. Boff, L. Kaufmann, and J.P. Thomas (Eds.), Handbook of perception and human performance, Vol. 1: Sensory processes and perception, pp. 1–45. New York: Wiley. Mays, L.E. and Sparks, D.L. (1981). Saccades are spatially, not retinocentrally, coded. Science, 208, 1163–1165. Michotte, A. (1946). La perception de la causalité [The perception of causality]. Louvain: Publications Universitaires de Louvain. Oram, M. and Perrett, D. (1994). Responses of anterior superior temporal polysensory (STPa) neurons to ‘biological motion’ stimuli. Journal of Cognitive Neuroscience, 6, 99–116. Orliaguet, J.-P., Kandel, S., and Boë, L.J. (1997). Visual perception of cursive handwriting: In2uence of spatial and kinematic information on the anticipation of forthcoming letters. Perception, 26, 905–912. Palmer, S.E. (1978). Fundamental aspects of cognitive representation. In E. Rosch and B.B. Lloyd (Eds.), Cognition and categorization, pp. 259–303. Hillsdale, NJ: Erlbaum. Parsons, L.M. (1987). Imagined spatial transformations of one’s hands and feet. Cognitive Psychology, 19, 178–241.



Parsons, L.M. (1994). Temporal and kinematic properties of motor behavior re2ected in mentally simulated action. Journal of Experimental Psychology: Human Perception and Performance, 20, 709–730. Parsons, L.M. and Fox, P.T. (1998). The neural basis of implicit movement used in recognising hand shape. Cognitive Neuropsychology, 15, 583–615. Pastore, R.E., Ahroon, W.A., Buffuto, K.A., Friedman, C.J., Puleo, J.S., and Fink, E.A. (1977). Common factor model of categorical perception. Journal of Experimental Psychology: Human Perception and Performance. 4, 686–696. Pellizzer, G. (1997). Transformation of the intended direction of movement during motor trajectories. Cognitive Neuroscience and Neuropsychology, 8, 3447–3452. Perrot, D.R. (1974). Auditory apparent motion. Journal of Auditory Research, 14, 163–169. Poincaré, H. (1905). La Science et l’Hypothèse. Paris: Flammarion. [English translation: Science and hypothesis. New York: The Science Press.] Port, R.P. and van Gelder, T. (1995). Mind as motion: Explorations in the dynamics of cognition. Cambridge, MA: Bradford Book/MIT Press. Restle, F. (1979). Coding theory of the perception of motion con1gurations. Psychological Review, 86, 1–24. Rizzolatti, G., Fadiga, L., Gallese, V., and Fogassi, L. (1996). Premotor cortex and the recognition of motor actions. Cognitive Brain Research, 3, 131–141. Runeson, S. (1974). Constant velocity—not perceived as such. Psychological Research, 37, 3–23. Runeson, S. and Frykholm, G. (1981). Visual perception of lifted weights. Journal of Experimental Psychology: Human Perception and Performance, 7, 733–740. Scheerer, E. (1984). Motor theories of cognitive structure: A historical review. In W. Prinz and A. F. Sanders (Eds.), Cognition and motor processes, pp. 77–97. Berlin: Springer-Verlag. Scheerer, E. (1987). Muscle sense and innervation feelings: A chapter in the history of perception and action. In H. Heuer and A.F. Sanders (Eds.), Perspective in perception and action, pp. 171–194. Hillsdale, NJ: Erlbaum. Schwartz, A.B. (1994). Direct cortical representation of drawing. Science, 265, 540–542. Schwartz, S.P. (1977). Naming, necessity and natural kinds. Ithaca, NY: Cornell University Press. Sechenov, I. (1878). The elements of thought. [English translation in: Selected physiological and psychological works. Moscow: Foreign Languages Publishing House, 1956, pp. 265–401.] Shepard, R.N. (1984). Ecological constraints on internal representation: Resonant kinematics of perceiving, imagining, thinking, and dreaming. Psychological Review, 91, 417–447. Shepard, R.N. and Cooper, L.A. (1982). Mental images and their transformations. Cambridge, MA: MIT Press/Bradford Books. Shepard, R.N. and Zare, S. (1983). Path-guided apparent motion. Science, 220, 632–634. Sherrick, C.E. and Rogers, R. (1966). Apparent haptic movement. Perception and Psychophysics, 1, 175–180. Shiffrar, M. and Freyd, J.J. (1990). Apparent motion of the human body. Psychological Science, 1, 257–264. Shiffrar, M. and Freyd, J.J. (1993). Timing and apparent motion path choice with human body photographs. Psychological Science, 4, 379–384. Shiffrar, M., Lichtey, L., and Heptulla Chatterjee, S. (1997). The perception of biological motion across apertures. Perception and Psychophysics, 59, 51–59. Sirigu, A., Cohen, L., Duhamel, J.-R., Pillon, B., Dubois, B., and Agid, Y. (1995). Congruent unilateral impairments for real and imagined movements. Neuroreport, 6, 997–1001. Sirigu, A., Duhamel, J.-R., Cohen, L., Pillon, B., Dubois, B., and Agid, Y. (1996). The mental representation of hand movements after parietal cortex damage. Science, 273, 1564–1568. Soury, J. (1892). Les fonctions du cerveau [The functions of the brain]. Paris: V.ve Babé, Libraire Editeur. Stricker, W. (1882). Studien über die Bewegungsvorstellungen [Studies on the representation of movement]. Vienna: Hankel. Sumi, S. (1984). Upside-down presentation of the Johansson moving light-spot pattern. Perception, 13, 283–286. Thelen, E. and Smith, L.B. (1994). A dynamic systems approach to the development of cognition and action. Cambridge, MA: Bradford Books/MIT Press. Thomassen, A.J.W.M. and Schomaker, L.R. (1986). Between-letter context effects in handwriting trajectories. In H.S. Kao, G.P. Van Galen, and R. Hoosain (Eds.), Graphonomics: Contemporary research in handwriting, pp. 253–272. Amsterdam: Elsevier. Thornton, I., Pinto, J., and Shiffrar, M. (1998). The visual perception of human locomotion. Cognitive Neuropsychology, 15, 535–552. Viviani, P. and Flash, T. (1995). Minimum Jerk, Two-thirds Power Law, and isochrony: Converging approaches to movement planning. Journal of Experimental Psychology: Human Perception and Performance, 21, 32–53.

441


442


Viviani, P. and Laissard, G. (1996). Motor templates in typing. Journal of Experimental Psychology: Human Perception and Performance, 22, 417–445. Viviani, P. and Mounoud, P. (1990). Perceptuo-motor compatibility in pursuit tracking of two-dimensional movements. Journal of Motor Behavior, 22, 407–443. Viviani, P. and Schneider, R. (1991). A developmental study of the relationship between geometry and kinematics in drawing movements. Journal of Experimental Psychology: Human Perception and Performance, 17, 198–218. Viviani, P. and Stucchi, N. (1989). The effect of movement velocity on form perception: Geometric illusions in dynamic displays. Perception and Psychophysics, 46, 266–274. Viviani, P. and Stucchi, N. (1992). Biological movements look constant: Evidence of motor-perceptual interactions. Journal of Experimental Psychology: Human Perception and Performance, 18, 603–623. Viviani, P. and Terzuolo, C.A. (1982). Trajectory determines movement dynamics. Neuroscience, 7, 431–437. Viviani, P., Campadelli, P., and Mounoud, P. (1987). Visuo-manual pursuit tracking of human two-dimensional movements. Journal of Experimental Psychology: Human Perception and Performance, 13, 62–78. Viviani, P., Baud-Bovy, G., and Redol1, M. (1997). Perceiving and tracking kinesthetic stimuli: Further evidence of motor–perceptual interactions. Journal of Experimental Psychology: Human Perception and Performance, 23, 1232–1252. Warrington, E.K. and Shallice, T. (1984). Category speci1c semantic impairments. Brain, 107, 829–854. Wertheimer, M. (1912). Experimentelle Studien über das Sehen von Bewegung [Experimental studies on the perception of movement]. Zeitschrift für Psychologie, 61, 161–265. Wundt, W. (1893). Grundzüge der physiologischen Psychologie [Foundations of physiological psychology]. Leipzig: Engelmann. Zimmer, A. (1982). Do we see what makes our script so characteristic or do we only feel it? Modes of sensory control in handwriting. Psychological Research, 44, 165–174. Zinchenko, V.P. and Vergiles, N.Y. (1972). Formation of visual images: Studies of stabilized images. New York: Plenum Press Consultants Bureau.


22 Eliminating, magnifying, and reversing spatial compatibility effects with mixed location-relevant and irrelevant trials Robert W. Proctor and Kim-Phuong L. Vu

Abstract. Regardless of whether stimulus location is relevant or irrelevant to a task, responses are faster and more accurate when stimulus location and response location correspond than when they do not. Stimulus– response compatibility (SRC) effects of this nature are robust and typically considered to be automatic consequences of stimulus-response associations that are either hard-wired or acquired through years of experience. An exception to the robustness of SRC effects has been shown to occur when compatible and incompatible mappings are mixed. In the present paper, we review the literature on mixing compatible and incompatible mappings and show that mixing does not always reduce the SRC effect. We then present results from studies in which location-irrelevant (LI) trials are mixed with location-relevant (LR) trials. The SRC effect for LR trials of physical locations to keypresses is eliminated when stimulus color, rather than location, is relevant on half of the trials. However, the SRC effect for LR trials is unaffected by mixing when the location information is conveyed by arrows and ampli1ed when it is conveyed by words. With vocal location responses, the SRC effects for all three stimulus types are enhanced by mixing. In addition, regardless of stimulus type and response modality, mixing intensi1es the correspondence effect for trials on which location is irrelevant when the mapping for LR trials is compatible and eliminates or reverses the effect when the mapping is incompatible. These results show that SRC effects are not as hard-wired as previously depicted and are affected by the demands on subjects imposed by the task environment.

22.1 Introduction Most researchers in cognitive psychology are aware that Attention and Performance II, in 1968, was the Donders Centenary Symposium on Reaction Time, held in honor of F.C. Donders’ pioneering contributions to the use of reaction time (RT) to measure mental processes. The proceedings of this symposium included an English translation of Donders’ (1868/1969) seminal paper, ‘On the speed of mental processes’, in which he outlined his view that the processes intervening between stimulus onset and response can be decomposed into discrete stages, whose durations can be measured. What is not widely known is that Donders reported results demonstrating that some stimulus–response (S–R) relations yield faster selection of responses than do others, a phenomenon that is known today as S–R compatibility (SRC). Donders estimated that the time to choose between responses with the left and right hands was 66 ms when the stimulus was an electrical impulse presented to the left or right foot, and 122 ms when the stimulus was a light of red or white color. He also noted that the estimated choice time for repeating one of two vowel sounds was 56 ms. From these effects, Donders concluded that choice among two response alternatives is faster when the stimuli are paired with their natural responses.


444


The only author of whom we are aware who acknowledges Donders’ (1868/1969) demonstrations of mode SRC effects is Prinz (1997), who described them as ‘a beautiful set of compatibility experiments—certainly the very 1rst ones in the history of psychology’ (p. 250). Ironically, this eloquent depiction of Donders’ prescience, though included in an edited book devoted to SRC (Hommel and Prinz 1997), is presented in a chapter titled, ‘Why Donders has led us astray.’ The sentiment expressed in Prinz’s title re2ects his belief that too little emphasis has been placed on the role of intention, or task set, in the choice RT literature, a de1ciency he attributes in some degree to Donders not explicitly addressing this topic. Due in part to Prinz’s efforts, this trend has been reversed in recent years, and the experiments we describe in this paper are among an increasing number that emphasize the role of task set. Although Prinz (1997) pointed out Donders’ (1868/1969) work on mode SRC effects, he did not mention that Donders also brie2y described the 1rst instance of spatial mapping effects. Donders noted that, in comparison to the condition in which the assigned response was on the same side as the stimulus, ‘when movement of the right hand was required with stimulation of the left side or the other way round, then the time lapse was longer and errors common’ (p. 421). It was not until 85 years later that the concept of SRC was formalized by Paul Fitts, to whom the proceedings of the 1rst Attention and Performance meeting were dedicated. Thus, at least with regard to SRC, it seems that rather than Donders leading us astray, we should have followed his lead more closely. Fitts and Seeger (1953) used an eight-choice task with three different S–R arrangements to demonstrate that responses were faster when the con1guration of stimulus locations matched that of the response locations. The study of Fitts and Deininger (1954), which also used eight-choice tasks, showed that the mapping of stimuli to responses affected performance: responses were fastest when they corresponded to the stimuli, next fastest when they were systematically related, and slowest when they were unrelated. According to Fitts and Deininger, ‘Compatibility effects are conceived as resulting from hypothetical information transformation processes (encoding and/or decoding) that intervene between receptor and effector activity. The rate of processing information is assumed to be maximum when these recoding processes are at a minimum’ (p. 483). Although Fitts and colleagues used eight-choice tasks, many subsequent studies have used twochoice tasks (e.g. Broadbent and Gregory 1962), as Donders (1868/1969) did in his early work, typically with visual stimuli. A standard 1nding from two-choice tasks is a spatial SRC effect such that when left–right stimulus locations are mapped to left–right keypresses, the mapping of left to left and right to right yields faster responding than the opposite mapping. Consistent with Donders’ 1ndings of mode SRC effects, for two-choice tasks in which the stimuli and responses are left and right, the more natural pairings of spatial–manual and verbal–vocal S–R sets are more compatible than the pairings of spatial–vocal and verbal–manual S–R sets (Proctor and Wang 1997; Wang and Proctor 1996). A third SRC effect, called the Simon effect, is found in two-choice tasks for which stimulus location is irrelevant and the relevant stimulus dimension is non-spatial (e.g., color). Performance is better when the irrelevant stimulus location corresponds with the response location than when it does not (e.g. Lu and Proctor 1995; Simon 1990; Umiltà and Nicoletti 1990). Beginning with Donders (1868/1969), most accounts of SRC effects have emphasized learned associations between stimuli and responses. For example, in answering the question why a choice required less time when repeating vowel sounds versus making manual responses to color, Donders said, ‘The answer is the response given to the sound is the simple imitation which has become natural by training, more so than the conventional response with the right or the left hand in the case of differences in colour’ (p. 421). Donders accounted for performance with the compatible mapping


Eliminating, magnifying, and reversing spatial compatibility effects

Fig. 22.1 Illustration of the dimensional overlap model by Kornblum et al. (1990). The top route depicts automatic activation of the corresponding response, and the bottom route depicts identi1cation of the assigned response by intentional S–R translation. From ‘Dimensional overlap: Cognitive basis for stimulus–response compatibility—A model and taxonomy’, by S. Kornblum, T. Hasbroucq, and A. Osman 1990. Psychological Review, 97, p. 257. Copyright 1990 by the American Psychological Association.

for S–R locations in a similar manner, stating, ‘The tendency to respond in this way is already present as a consequence of habit or training’ (p. 421). Fitts (1964) also attributed SRC, at least in part, to habit strength, stating, ‘As we progress from tasks low in compatibility to ones of relatively high compatibility, Ss [subjects] are presumably making more and more use of very well-established habits (i.e. using responses which show strong population stereotypes)’ (p. 271). The penchant for attributing SRC effects in large part to learned associations has continued to the present. The most popular accounts of SRC effects currently are dual-route models, in which response activation can occur either automatically or through intentional translation. The most well known exemplar is that of Kornblum, Hasbroucq, and Osman (1990). In their model (see Fig. 22.1), activation of the corresponding response occurs via a direct route whenever a stimulus dimension overlaps with (i.e. is similar to) a response dimension, and this activation is independent of the S–R mapping de1ned for the task. Automatic activation is therefore presumed to facilitate responding when the mapping is compatible and to interfere when it is incompatible. Response identi1cation occurs by way of an indirect route, through retrieval or generation by rule of the assigned response. A related way of characterizing automatic activation is to distinguish activation produced by longterm S–R associations, typically described as learned or hard-wired, from that produced by short-term S–R associations de1ned for the speci1c task (Barber and O’Leary 1997; Umiltà and Zorzi 1997). The long-term associations correspond to the direct response-selection route and the short-term associations to the indirect translation route. Zorzi and Umiltà (1995) implemented the distinction between short- and long-term associations in a model for the Simon effect. Their model is a connectionist network consisting of three groups of interconnected processing nodes (see Fig. 22.2). The position of the imperative stimulus is encoded by position nodes, the value of the relevant stimulus

445


446


Fig. 22.2 Illustration of a connectionist model for the Simon effect by Zorzi and Umiltà (1995). Position nodes are connected to responses by the long-term memory (LTM) links and feature nodes are connected by short-term memory (STM) links. From ‘The role of LTM links and STM links in the Simon effect,’ by M. Tagliabue, M. Zorzi, C. Umiltà, and F. Bassignani, 2000. Journal of Experimental Psychology: Human Perception and Performance, 26, p. 660. Copyright 2000 by the American Psychological Association. attribute is encoded by feature nodes, and the two response alternatives are represented by response nodes. The position nodes and response nodes are connected by long-term links, and the feature nodes are connected to short-term memory nodes, which in turn are connected to the response nodes by short-term links. In Zorzi and Umiltà’s model, the Simon effect is attributed to the long-term links, consistent with the prevailing view that SRC effects are to a large extent due to pre-experimental associations and, consequently, are relatively immutable. Despite the robustness of SRC effects, occasional exceptions have been reported. Shaffer (1965) mixed compatible and incompatible mappings of location-relevant trials. On each trial, a mapping signal was presented simultaneously with the imperative stimulus (a light in a left or right position), to indicate whether the S–R mapping on the trial was compatible or incompatible. With pure blocks of compatible or incompatible mappings, the SRC effect was 54 ms. However, when compatible and incompatible trials were mixed, the SRC effect was a nonsigni1cant −8 ms. With regard to the Simon effect, Hedge and Marsh (1975) found a performance advantage for trials on which stimulus location did not correspond with response location. Speci1cally, subjects responded to the color (red or green) of a stimulus, which could occur in a left or right location, by moving a hand from a home key to a red or green key located to the left or right. For some conditions, the mapping of stimulus color to key color was incompatible, that is, the green key was mapped to the red stimulus, and vice versa. With this mapping, a reverse Simon effect was evident for which responses were faster when the stimulus location was on the side opposite the response key rather than on the same side. Thus, this ‘Hedge and Marsh’ reversal is a violation of the principle that responses are faster when S–R locations correspond than when they do not. Although these violations of SRC have been known for some time, and both the effects of mixing and the Hedge and Marsh reversal have attracted considerable interest, accounts of SRC effects have continued to emphasize pre-existing associations. The primary concern of the present paper is to examine the role of task-de1ned S–R associations on performance of, for the most part, two-choice reaction tasks by mixing task sets. We restrict consideration mainly to spatial compatibility effects,



broadly de1ned to include those obtained when the location information is conveyed by location words, arrow directions, or physical locations. We begin by summarizing prior research for which compatible and incompatible mappings of spatial locations were mixed and the models that have been proposed to explain the results. Then we describe recent research we have conducted using mixed and blocked presentations of location-relevant (LR) and location-irrelevant (LI) trials. We next examine effects obtained when the location information is conveyed by arrow direction or location word, rather than by spatial location. The 1nal set of experiments examines conditions in which the LR stimuli are presented in one mode and the LI stimuli in another mode. Overall, the results of the mixing studies imply that SRC effects are not an automatic consequence of long-term associations, as often assumed, and that the short-term S–R associations de1ned by the task set contribute substantially to the pattern of results obtained.

22.2 Mixing compatible and incompatible mappings When subjects perform a two-choice SRC task, the instructions de1ne the S–R associations for the task. The instructions also specify an identity rule for the compatible mapping and opposite rule for the incompatible mapping, which allows the possibility of generating the correct response to a stimulus by applying the appropriate rule, rather than retrieving the associated response (Duncan 1977b). According to most dual-route models, the long-term associations between each stimulus location and its corresponding response also affect performance. These associations may be in accord with the task-de1ned associations (the compatible mapping) or counter to them (the incompatible mapping). The advantage for compatible mappings is attributed at least in part to the direct activation of the corresponding response produced by the long-term associations. Since Shaffer’s (1965) study, researchers have been concerned with how to characterize performance for situations in which compatible and incompatible mappings are mixed. Duncan (1977b) introduced a method in which four stimulus locations arranged in a row are mapped to four keypress responses (made with the index and middle 1ngers of each hand). In his original study, the two inner stimulus locations had one mapping and the two outer locations the alternative mapping, and mixing slowed RT equally for both mappings. Subsequent studies have also used this condition in which the mappings differ between inner and outer locations, as well as ones in which each mapping is assigned to the two left or two right locations or to alternate stimulus locations. The typical 1nding, however, has been that the mapping effect is reduced but not eliminated when the mappings are mixed (Duncan 1977a, 1978; Ehrenstein and Proctor 1998; Stoffels 1996b). At least three types of explanations have been proposed for the mixing effects: pure translation (translation ef1ciency), two-step response selection, and alternative-routes accounts.

22.2.1 Pure translation (translation ef1ciency) Shaffer (1965) explained the greater effect of mixing on compatible than incompatible spatial mappings as follows: ‘It is as though when I [the mapping] was variable S [the subject] selected at each trial from a set of equally dif1cult transformations: when I was 1xed or known in advance the null transformation could be considered as a special class and was easier to compute’ (p. 287). This pure translation account implies that intentional S–R translation occurs more ef1ciently when all trials are compatible (possibly because an identity rule can be applied) than when the trials are of mixed mappings.

447


448


22.2.2 Two-step response selection The second explanation, proposed by Duncan (1977a,b, 1978), is explicitly a rule-based translation account. Response selection with mixed mappings occurs in two steps: a decision is made as to whether the mapping is compatible or incompatible, and then the appropriate mapping rule is applied. Duncan’s (1977b) model was based initially on his atypical 1nding of additive effects for mixed-blocked presentation and SRC in the four-choice task version. However, the model can explain the more common 1nding of a reduced SRC effect when mappings are mixed, by assuming that a reduction from four to two choices is relatively more bene1cial for the incompatible mapping than for the compatible mapping. Evidence consistent with the model is that errors tend to be the response that would be correct if the alternative mapping rule were applicable for the trial (Duncan, 1977a, 1978; Ehrenstein and Proctor 1998; Stoffels 1996b).

22.2.3 Alternative-routes model The third account is a version of the dual-route model according to which the direct route contributes to response selection only when the mapping is compatible on all trials (De Jong 1995; Van Duren and Sanders 1988). Response selection occurs by way of the indirect translation route when all trials are incompatible or when compatible and incompatible trials are mixed. An intuitive way to characterize this alternative routes model is that a response can be selected on the basis of initial response tendencies if these are going to be correct, but not if they lead to the wrong response on a signi1cant proportion of trials. The fact that the dominant 1nding has been a signi1cant reduction of the mapping effect under mixed conditions, and that this reduction is evident in Shaffer’s (1965) task version in which the number of stimuli and responses is always two, has led to the alternative routes view being favored. However, Stoffels (1996b) and Ehrenstein and Proctor (1998) have suggested that it is necessary to include rule-based translation along with the alternative-routes model to explain the full range of results. Precuing studies provide evidence that the compatible mapping bene1ts from direct activation when it is known in advance that the trial will be compatible. In Shaffer’s (1965) study, one group of subjects received the mapping stimulus 333 ms before the imperative stimulus. The precued group showed an SRC effect similar to that obtained in the pure mapping conditions, rather than the absence of effect found for the group that was not precued. Stoffels (1996b) and Ehrenstein and Proctor (1998) obtained similar results using variations of Duncan’s (1977b) four-choice procedure. A precue designated two of the four stimuli as possible, with the two stimuli being the compatible subset, the incompatible subset, or a mixed subset (one from the compatible and one from the incompatible subset). In most cases, the bene1t at the longest precuing interval was larger for the compatible subset than for the incompatible subset, indicating that the SRC effect increased in magnitude. If the precue only allowed advance selection of the appropriate mapping, as Duncan’s two-stage model suggests, then the SRC effect should be additive with that of the compatibility manipulation. The larger precuing bene1t for compatible trials than for incompatible trials is consistent with the alternative routes model, assuming that subjects adjust their strategies to use the direct processing route when informed that the mapping will be compatible. Most studies on mixed compatible and incompatible mappings have reported repetition analyses, with the typical 1nding being that repetition of the task category from the preceding trial is bene1cial. Shaffer (1965, 1966) obtained additive bene1ts of repetition for both the mapping signal



and the stimulus location in the two-choice task version, with the bene1t being larger for repeating the mapping signal than repeating the stimulus location. He noted, ‘The relative magnitude of these transition effects corresponded to the relative reductions in RT obtained by presenting I [the mapping signal] or M [the stimulus position] in advance’ (p. 287). Stoffels (1996b, Exp. 3) also noted that the pattern of repetition effects in the four-choice task version was similar to the pattern of precuing bene1ts. For pure blocks of one mapping, repetition of the stimulus location was more bene1cial for the incompatible than compatible mapping, as is typically found. However, for mixed blocks, repetition of the task, but without repetition of the speci1c stimulus, was more bene1cial for the compatible mapping than for the incompatible mapping, resulting in an increased SRC effect. An additional bene1t of repeating the same stimulus was also obtained that was of similar magnitude for the compatible and incompatible mappings. Stoffels suggested that the bene1t of task repetition for the compatible mapping is due to the direct, or automatic, response-selection route not being inhibited on repetition trials. In summary, regardless of which task version is used, the SRC effect is reduced, if not eliminated, when compatible and incompatible mappings are mixed. The SRC effect is larger for trials on which the task is repeated than for trials on which it is not. The effect is also reinstated when the mapping is precued suf1ciently far in advance of the imperative stimulus to allow preparation for the cued mapping. On the whole, the 1ndings are most consistent with the view that a direct response-selection route is used (a) for pure blocks of compatible S–R mappings, (b) on task repetition trials in mixed blocks, and (c) when the mapping is precued in advance in mixed blocks.

22.3 Mixing location-relevant and location-irrelevant trials We have conducted experiments that focus on mixing compatibly or incompatibly mapped LR trials with LI trials. Stimulus color is used to designate whether location is relevant or irrelevant on each trial. Mixing LR and LI trials allows issues to be examined that cannot be through mixing compatible and incompatible LR trials. One is whether the SRC effect is eliminated by inclusion of LI trials, or whether elimination only occurs when both tasks require responding to spatial location but with different mappings. If the SRC effect is eliminated, other manipulations can be used to determine why. Another issue is the extent to which the LR mapping intrudes on performance when location is irrelevant. The contributions to performance made by the task-de1ned and long-term associations of S–R locations on the LI trials can be evaluated. Results from our experiments, described below, show that by mixing LR and LI trials, it is possible to eliminate and magnify the SRC effects for both trial types, as well as to reverse the effect for the LI trials. A characterization of the task set in terms of associations for the conditions in which LR and LI trials are mixed is as follows. When stimulus location is relevant, the instructions specify short-term associations between stimulus locations and responses that are either consistent (compatible mapping) or inconsistent (incompatible mapping) with the long-term associations. Hence, according to most dual-route models, the advantage for compatible mappings is due at least in part to direct activation of the corresponding response produced by the long-term associations. For a Simon task, the taskde1ned associations relate the non-spatial stimulus dimension to the response locations. Thus, when stimulus location is relevant on some trials and irrelevant on others, task-de1ned associations of both stimulus dimensions to responses must be maintained in an active state. The questions of interest are whether the associations de1ned for one task intrude on performance of the other task and whether the contribution of the long-term associations is altered.

449


450


22.3.1 Mapping effects for location-relevant trials We have conducted experiments that used mixed presentation of physical location stimuli in which the LR mapping was compatible or incompatible (Marble and Proctor 2000, Expts. 1 and 4 Proctor and Vu 2001, Exp. 1; Proctor, Vu, and Marble in press, Exp. 1). Stimuli were 1lled circles presented in left or right locations of the display screen, and responses were left and right keypresses (see Fig. 22.3). Half of the trials were LR (white circles) and half LI (red or green circles). Proctor et al.’s and Proctor and Vu’s studies also included pure LR conditions, with the mapping being compatible or incompatible. For the pure conditions, performance was better with the compatible than incompatible mapping; however, for the mixed conditions, there was no SRC effect for the LR trials (see Table 22.1). Thus, the two-choice SRC effect is eliminated when LI and LR trials are mixed, as it is for mixed compatible and incompatible mappings (Shaffer 1965). Proctor and Vu’s (2001, Exp.1: physical location stimuli) data were submitted to a task repetition analysis on LR trials that partitioned the trials into three types: task repetition of the same or different S–R pair, and task nonrepetition. Responses were slowest for nonrepetition trials, in which the previous task was LI. For task repetition trials, responses were faster when the same S–R pair was repeated than when it was not. When the mixed conditions were compared to conditions of pure LR trials, there was a signi1cant interaction of repetition type (same or different), condition, and compatibility. For the pure condition, the incompatible mapping showed a 25 ms repetition bene1t for the same S–R pair, but the compatible mapping showed a repetition cost of 25 ms. Although this cost of repetition for the compatible mapping seems surprising, Shaffer (1965) reported a similar repetition cost when the compatible and incompatible mappings were mixed. For the mixed condition (Proctor and Vu 2001), the task repetition bene1t was similar for the compatible and incompatible mappings. However, the compatible mapping showed less bene1t of a task repetition when the alternative stimulus was presented than did the incompatible mapping (Mean Differences (MDs) = 85 and 124 ms, respectively). This difference was offset by the compatible mapping showing a larger additional bene1t than the incompatible mapping for

Table 22.1 Mean SRC effect for reaction time (in milliseconds) and percentage of error (in parentheses) for LR trials as a function of experiment and condition Experiment

Condition Mixed

Left–right codes on both location-relevant and irrelevant trials Marble and Proctor (2000, Exp. 1) −8 (−1.95%) Marble and Proctor (2000, Exp. 4) −6 (−0.55%) Proctor, Vu, and Marble (in press, Exp. 1) −16 (−1.17%) Proctor and Vu (2001, Exp. 1) −16 (−1.67%) Proctor, Vu, Marble (in press, Exp. 3, mixed) −1 (0.10%) Left–right codes on only location-relevant trials Proctor, Vu, Marble (in press, Exp.2) Proctor, Vu, Marble (in press, Exp. 4)

54 (0.73%) 71 (1.20%)

Pure – – 68 (1.41%) 77 (2.11%) 117 (0.20%)

56 (1.61%) –



Marble & Proctor (2001; Exps. 1&4) Proctor et al. (in press; Exp.1) Proctor & Vu (2001; Exp.1)

Proctor et al. (in press; Exp.2)



Fig. 22.3 Components of the overall task partitioned by location-relevant (white stimuli) and location-irrelevant (colored stimuli) trials as a function of experiment. * Participants are responding to color of stimuli with spatial location irrelevant. As shown in the figure, participants press the left key if the circle is red and the right key if it is green. Although not illustrated, half the subjects responded to the red stimuli by pressing the right key and green stimuli by pressing the left key. ** For half the participants, white circles appeared in the top row and colored ones in the bottom row, as illustrated. The other half received the colored circles in the top row and white ones in the bottom row.

451


452


repeating the stimulus as well as the task (MDs = 124 ms and 68 ms, respectively). The greater task repetition bene1t for the incompatible mapping when the stimulus location changed could be due to subjects being primed for ‘opposite’ with that mapping but ‘same’ with the compatible mapping. The elimination of the SRC effect when LI trials are mixed with the LR trials is consistent with the view of the alternative routes hypothesis that compatible mappings bene1t from activation by way of the direct route only when all trials are compatible. From this view, there are several possible reasons why mixing LI trials with LR trials could eliminate the contribution of the direct route. First, the requirement of having to respond to color on some trials, and thus to maintain a task set of color-to-response associations as well as location-to-response associations, may be the crucial factor. Second, the contribution of the direct route may be reduced because the two types of stimuli occur in the same physical locations, thus precluding a location distinction being used as a basis for immediate responding. Third, left and right codes are generated for both trial types, meaning that the presence of a left or right location code is not suf1cient for signaling the response. Experiments 2–4 of Proctor et al. (in press) evaluated these possibilities (see Fig. 22.3). In Proctor et al.’s (in press) Experiment 2, the LI stimuli were displayed in the center of the screen instead of left–right locations. In this case, the mixed condition showed an SRC effect similar to that obtained in the pure condition. Thus, the requirement to respond to color is not the critical factor. In Experiment 3, stimuli were presented in left–right locations above or below 1xation. For the pure LR conditions, subjects responded to the left or right stimulus location with a compatible or incompatible mapping, ignoring the top–bottom distinction. This condition yielded an SRC effect of 117 ms. The mixed–random condition was similar to the mixed conditions of the previous experiments in that the LR and LI stimuli could occur in any location. This condition yielded a nonsigni1cant 32 ms SRC effect. Of most importance was another mixed condition, in which the LR stimuli occurred in one row and the LI stimuli in the other, that showed no SRC effect (MD = − 1 ms). Thus, even when the stimuli for the two tasks occur in distinct locations, the SRC effect is eliminated if both trial types produce left–right location codes. Experiment 4 of Proctor et al.’s (in press) study presented LR stimuli in left–right positions and LI stimuli in top–bottom positions. With this method, a location code is generated for both trial types, but the LR code is left–right and the LI code is top–bottom. In this case, the presence of a left–right code provides a suf1cient basis for responding because it is only present on LR trials. Consequently, we predicted that a typical SRC effect would be evident. The results supported this prediction, showing a 71-ms advantage for the compatible mapping over the incompatible mapping. Together, Proctor et al.’s Experiments 3 and 4 indicate that a distinct location code (top or bottom) for the LR and LI tasks is insuf1cient to allow relatively rapid responding with the compatible mapping if left–right location codes are present for both tasks. Only when the LI task does not activate left–right codes is the SRC effect evident for the LR task. Overall, the results obtained with physical location stimuli indicate that the SRC effect is eliminated when LR and LI trials are mixed, as when compatible and incompatible mappings are mixed (Shaffer 1965). Consequently, they can be interpreted in terms of the alternative routes model, according to which the direct route is not a factor for situations in which it would lead to an incorrect response. This interpretation is that the direct route is used for compatible mappings when a location code provides a suf1cient basis for responding, but not when the same location codes occur for both of the mixed tasks. However, this picture is complicated by experiments, described later, that used left and right pointing arrows and left and right location words.



22.3.2 The Simon effect for location-irrelevant trials When LR and LI trials are mixed, the primary question regarding the LI trials is how the magnitude and direction of the Simon effect are in2uenced by the LR mapping. Our experiments (Marble and Proctor 2000; Proctor and Vu 2001; Proctor et al. in press) also allowed this question to be answered (see Table 22.2). With the compatible LR mapping, the Simon effect averaged 39 ms across the experiments. With the incompatible LR mapping, a reverse Simon effect averaging −53 ms was obtained. Thus, relative to the pure blocks of LI trials, the Simon effect is enhanced when the LR trials are compatibly mapped and reversed when they are incompatibly mapped. Since the reverse Simon effect was as large as the positive Simon effect, there apparently was no direct activation of the corresponding response, independent of the task mapping, from long-term associations. This is because any activation of the corresponding response would add to the positive effect in the mixed compatible condition and subtract from the reversed effect in the mixed incompatible condition. Marble and Proctor (2000) reported a task repetition analysis on the RT data for LI trials that showed responses to be 104 ms faster when the trial was a task repetition than when it was not. The task repetition effect had negligible impact on the positive and reverse Simon effects. When the LR mapping was compatible, the positive Simon effect was 46 ms on repetition trials and 47 ms on nonrepetition trials. When the LR mapping was incompatible, the reverse Simon effect was 60 ms on repetition trials and 78 ms on nonrepetition trials. Mordkoff (1998), Leuthold et al. (1999), and Valle-Inclán, Hackley, and de Labra (2002; this volume, Chapter 23) reported that the Simon effect occurs when the previous trial is corresponding, but not when it is noncorresponding. Mordkoff and Leuthold et al. attributed this pattern of results to the direct route being suppressed following a noncorresponding trial. A repetition analysis of the pure LI condition of Marble and Proctor’s Experiment 1 showed similar results: the Simon effect was positive when the S–R locations corresponded on the preceding trial (M = 73 ms) and negative when they did not (M = −31 ms). For the mixed condition, the task repetition trials showed a similar, but stronger, pattern. When the LR mapping was compatible, the Simon effect was 103 ms following a corresponding trial and –27 ms following a noncorresponding trial. When the LR mapping was incompatible, the Simon effect was 17 ms following a corresponding trial and −144 ms following a noncorresponding trial. The magnitude of the difference in effects following corresponding and noncorresponding trials did not depend signi1cantly on whether the LR mapping was compatible or incompatible. The 1nding of a Table 22.2 Mean Simon effect for reaction time (in milliseconds) and percentage of error (in parentheses) for LI trials as a function of experiment and condition Experiment

Condition Pure

Mixed compatible

Mixed incompatible

Marble and Proctor (2000, Exp. 1) Marble and Proctor (2000, Exp. 4) Proctor, Vu, and Marble (in press, Exp. 1) Proctor and Vu (2001, Exp. 1)

21 (2.8%) 15 (2.5%) – –

44 (8.0%) 26 (2.0%) 42 (4.4%) 42 (5.3%)

−64 (−7.5%) −42 (−7.0%) −61 (−5.5%) −44 (−5.4%)

Average

18 (2.7%)

39 (4.9%)

−53 (−6.4%)

453


454


reverse Simon effect in the pure and mixed compatible conditions does not conform with the suppression account because it predicts that any effect in these conditions should be positive. One interpretation of these experiments is that the S–R associations de1ned by the LR mapping are applied on the LI trials. This process could be under the participant’s control or occur relatively involuntarily as a function of the task requirements. This issue can be evaluated by precuing the task prior to presentation of the imperative stimulus (e.g. De Jong 1995; Shaffer 1965, 1966). In Marble and Proctor’s (2000) Experiment 2, an incompatible mapping was used for the LR task. The task was precued with 100% validity by the word ‘COLOR’ or ‘SPACE’ presented at the center of the screen at a stimulus onset asynchrony (SOA) of 150, 300, 600, 1200, or 2400 ms. The precue was effective, with RT being faster at the longest SOA than at the shortest for both cued LI and LR trials. However, the reverse Simon effect for LI trials was evident at all SOAs. Thus, when cued by as much as 2400 ms before the presentation of the imperative stimulus, subjects were not able to prevent application of the LR mapping. When the LR and LI tasks are mixed in equal number, the corresponding response is correct on 75% of the trials and the noncorresponding response on 25% of the trials if the LR mapping is compatible, and vice versa if it is incompatible. This relation suggests that the in2uences of the LR mapping on the Simon effect could be due to a bias to respond consistent with the mapping. This possibility seems plausible because the Simon effect and related effects of irrelevant information reverse when incongruent trials are more frequent than congruent trials (Greenwald and Rosenberg 1978; Hommel 1994; Logan 1980; Logan and Zbrodoff 1979; Toth et al. 1995). Marble and Proctor’s (2000) Experiment 4 evaluated the biasing possibility by measuring performance in pure blocks of LI trials for which the response was corresponding for 75% of the trials and non-corresponding for 25%, or vice versa. Performance on the pure blocks was compared with the mixed conditions in which the LR mapping was compatible or incompatible. Relative to the Simon effect for a baseline LI condition in which the S–R locations corresponded on 50% of the trials (MD = 15 ms), the Simon effect was enhanced when the corresponding relation predominated and reversed when the noncorresponding relation did. But there were differences in the result patterns for the pure LI trials and the mixed trials. Speci1cally, for the frequency manipulation in the pure blocks the reverse Simon effect when noncorresponding trials predominated (MD = −36 ms) was smaller than the Simon effect when corresponding trials predominated (MD = 58 ms), as in Hommel’s (1994) and Toth et al.’s (1995) studies. However, in the mixed conditions, the reverse Simon effect obtained with an incompatible LR mapping (MD = −42 ms) was at least as large as the positive Simon effect obtained with a compatible mapping (MD = 26 ms). Thus, although biasing may contribute to the results obtained with mixed presentation, it is not the sole factor. Another difference between the pure and mixed blocks in Marble and Proctor’s (2000) Experiment 4 was apparent when the LI trials were divided into two groups based on whether the S–R locations corresponded on the previous trials (for the mixed condition, previous trial type was not a factor). For the pure Simon trials, responses were 40 ms faster when the correspondence relation on the present trial (corresponding or noncorresponding) was a repetition of that for the previous trial than when it was not. This difference was independent of the relative frequency manipulation. In contrast, for the mixed conditions, the bene1t of correspondence repetition depended on the LR mapping. With the compatible mapping, a large bene1t of repeating the correspondence relation was evident for the noncorresponding trials, but not for the corresponding trials. However, when the LR mapping was incompatible, a large bene1t of repeating the correspondence relation was evident for the corresponding trials, but not for the noncorresponding trials. Thus, in contrast to the pure



Simon blocks, the bene1t for repeating the correspondence relation for mixed blocks occurred only on those trials for which the spatial S–R relation was opposite to that in effect on LR trials. More detailed analyses that partitioned the repetition trials for the mixed condition according to whether the previous trial was LR or LI showed that the asymmetric pattern described above was due primarily to a substantial cost of task switching. This cost is re2ected in the RTs for a change from corresponding to noncorresponding trials with the compatible mapping and from noncorresponding to corresponding trials with the incompatible mapping. With the incompatible mapping, the reverse Simon effect was approximately twice as large on the task switch trials (−68 ms) as on the task repetition trials (−36 ms), but it was evident even when the task was repeated. Thus, the reversal is not simply due to re-applying the opposite rule from the preceding trial. Additional evidence that bias is not the sole factor and that subjects do not have complete control over application of the task-de1ned associations comes from studies in which subjects practiced with an incompatible location mapping and then were transferred to a Simon task. Proctor and Lu (1999, Exp. 2) had subjects practice three sessions responding to the location in which the letter H or S appeared with an incompatible mapping. In a fourth session, subjects were transferred to a Simon task in which one of the letters was assigned to the left response and one to the right response. A signi1cant reverse Simon effect was evident in both the RT (−14 ms) and error data (−2.0%). Because all of the transfer trials were LI, the reversal could not be due to a response bias induced by differential frequencies of the compatible and incompatible location relations. The reverse Simon effect in the transfer session implies that the task-de1ned associations between incompatible S–R locations continue to be activated even though they are no longer relevant to the task. In Proctor and Lu’s (1999) Experiment 2, the same stimuli were used for both the LR practice trials and the LI transfer trials. Therefore, it is impossible to tell whether the location associations are linked to the speci1c stimulus set that was used. In their Experiment 3, one group of subjects performed in practice and transfer sessions with the letter stimuli, as in Experiment 2, and another group with color stimuli during the practice session and letters during the transfer session. The reverse Simon effect was of similar magnitude for the two conditions (−25 ms in the former and −30 ms in the latter). Thus, the location associations are independent of the symbolic characteristics of the stimuli for which the incompatible mapping is practiced. Tagliabue et al. (2000) conducted similar experiments, but with subjects receiving the initial LR task for only a single short session. When the transfer session was conducted without delay after practice with the incompatible mapping, no Simon effect was apparent. In other experiments, a 24-hour or 7-day delay was introduced, and the Simon effect still was absent, regardless of whether the stimulus properties were the same as those in the practice condition. The fact that the Simon effect did not reverse after practice with the incompatible mapping in Tagliabue et al.’s study is likely due to the use of fewer practice trials than in Proctor and Lu’s (1999) study. Tagliabue et al. 1t a modi1ed version of Zorzi and Umiltà’s (1995) connectionist model to their data by allowing practice to in2uence either the short-term links or the long-term links. Based on goodness of 1t, they concluded that the short-term links, rather than the long-term links, were affected by practice with the incompatible mapping and continued to exert their in2uence in the transfer session. In summary, the effect of correspondence between S–R locations on LI trials varies as a function of the LR mapping. When that mapping is compatible, the Simon effect is enhanced; when it is incompatible, the Simon effect is reversed. Repetition analyses show that the in2uence of the LR mapping is not restricted solely to trials on which the preceding trial was LR. A bias toward the LR S–R relations may play a role in the mixing effects, but it is not the major factor. The in2uence of the

455


456


LR mapping on performance occurs when the LI task is precued well in advance, as well as when the LI task is performed alone after practicing the LR task. Thus, the task-de1ned associations between stimulus and response locations exert a substantial effect on performance even when the subject is aware that they are irrelevant to the current task.

22.4 Spatial information in symbolic and verbal modes Spatial information can be signaled not only by physical locations, but also by left–right pointing arrows and ‘left’–‘right’ words. An SRC effect is obtained for both; the mapping of left stimulus to left keypress and right stimulus to right keypress yields better performance than the alternative mapping (Wang and Proctor 1996). When vocal left–right responses are compared with keypresses, the pairings of physical locations or arrow directions with keypress responses and words with vocal responses are more compatible than the opposite pairings (Wang and Proctor 1996). When irrelevant location information is conveyed by arrow directions or location words, Simon effects are obtained, although the effect is small for words (Baldo, Shimamura, and Prinzmetal, 1998; Barber and O’Leary 1997; Lu and Proctor 2001). Other evidence also suggests that arrows tend to automatically activate their corresponding responses (Eimer 1995). Thus, with left–right keypresses, spatial locations are the most compatible stimulus mode and location words the least compatible mode (Wang and Proctor 1996).

22.4.1 Mixing compatible and incompatible mappings We recently conducted an experiment similar to Shaffer’s (1965) study, in which all trials were LR and the responses were left–right keypresses (Vu and Proctor 2001, Exp.1). As in his study, compatible and incompatible mappings were mixed, but conditions were examined in which the location information was conveyed by arrow direction and location word, as well as physical location. Trial type was signaled by stimulus color, with red signaling the compatible mapping and white the incompatible mapping, or vice versa. For physical locations, the SRC effect was 71 ms for the pure condition, but a nonsigni1cant 5 ms when the two mappings were mixed, replicating Shaffer’s 1ndings. Arrow directions showed similar results, with the SRC effect being 80 ms in the pure condition and 3 ms in the mixed condition. However, for location words the SRC effect increased from 35 ms in the pure condition to 117 ms in the mixed condition. Vu and Proctor (2001) performed repetition analyses for the pure and mixed conditions (see Table 22.3). For the pure condition, physical locations showed a slight cost for repeating the same S–R pair for both compatible and incompatible mappings. In contrast, arrows and words showed bene1ts for repeating the same S–R pair for both mappings. For the mixed conditions, all stimulus types showed similar patterns of results: RT was faster for task-repetition than task-nonrepetition trials. Physical locations and arrows showed little SRC effect when the stimulus position changed, regardless of whether the trial was a task-repetition or nonrepetition. However, when the stimulus position repeated, the compatible mapping bene1ted relative to the incompatible mapping when the task repeated but showed a cost when the task changed. Words showed a similar pattern, but superimposed on the large overall bene1t for the compatible mapping. De Jong (1995, Exp. 1) also used a method similar to Shaffer’s (1965), with the location information conveyed by an upright arrow tilted to the left or right. At the shortest SOA of 100 ms between the mapping and imperative stimuli, the SRC effect was reduced to 18 ms from a value of 45 ms obtained in pure blocks of only one mapping. As Shaffer found for physical locations, presenting



Table 22.3 Mean reaction time (in milliseconds) for pure and mixed compatible and incompatible mappings of Vu and Proctor’s (2001) study as a function of experiment, stimulus mode, and repetition type Condition Stimulus mode:

Repetition type Nonrepetition Same

Nonrepetition Different

Task repetition Different

Task repetition Same

Comp

Comp

Incomp

Comp

Incomp

Comp

Incomp

– – –

– – –

312 374 465

388 462 504

325 367 445

394 440 476

665 773 781

664 776 909

572 633 685

578 646 829

474 498 568

520 549 708

– – –

– – –

– – –

434 486 505

464 535 627

436 483 499

455 518 635

753 768 754

841 820 738

901 874 811

716 711 690

736 740 773

593 613 619

632 670 721

Incomp

Experiment 1: Keypress responses Pure Physical locations – – Arrow directions – – Location words – – Mixed Physical locations 647 606 Arrow directions 733 664 Location words 728 796 Experiment 2: Vocal responses Pure Physical locations – Arrow directions – Location words – Mixed Physical locations 793 Arrow directions 806 Location words 719 Note:

Comp = Compatible; Incomp = Incompatible

the mapping stimulus 600 ms prior to the imperative stimulus restored the SRC effect. Repetition analyses showed that the reduction of the SRC effect at the 100-ms SOA was much stronger when the trial mapping changed than when it did not. De Jong (1995) included a manipulation of the relative proportion of compatible and incompatible trials. At the 10-ms SOA, the SRC effect was numerically larger when 67% of the trials were compatible (29 ms) than when 50% of the trials were compatible (23 ms). When only 33% of the trials were compatible, the SRC effect was reduced to 6 ms. For none of these conditions was the effect at the 10-ms SOA as large as the 41 ms effect obtained for all frequency conditions at the 800-ms SOA. Thus, mixing reduced the SRC effect, and the bias induced by the relative frequency manipulation acted to increase and decrease the base effect. This is similar to the results obtained when relative frequency of corresponding and noncorresponding trials is manipulated for the Simon task (Hommel 1994; Toth et al. 1995). De Jong interpreted this pattern of results in terms of the alternative routes model, proposing that the degree of suppression of the automatic route is an increasing function of the percentage of incompatible trials in the sequence.

457


458


Vu and Proctor’s (2001) Experiment 2 was similar to their Experiment 1, but with vocal ‘left’– ‘right’ responses instead of keypresses. The SRC effect for location words was reduced from 131 ms in the pure condition to 75 ms in the mixed condition, instead of being enhanced. Physical locations and arrows showed similar trends: for pure conditions, the SRC effect was 24 ms for physical locations and 42 ms for arrows, and for mixed conditions, the effect was 18 and 26 ms, respectively. Repetition analyses for the pure condition showed that, for all stimulus types (physical locations, arrows, and words) there was little difference between repeating the same or different S–R pair for both mappings (see Table 22.3). For the mixed condition, for both task repetitions and nonrepetitions, all stimulus types showed a bene1t for the compatible mapping when the stimulus position changed. With physical locations and arrows, the SRC effect was larger when the task changed than when it repeated; also, when the stimulus position repeated, the compatible mapping bene1ted when the task repeated but showed a cost when the task changed. Words showed a similar pattern, but superimposed on a large overall bene1t for the compatible mapping. The reduction of SRC effects for mixed mappings of words to vocal responses is not restricted to location words. Van Duren and Sanders (1988) conducted a study similar to our word condition, but using vocal numeral responses to digits: subjects responded to the digits 2 and 3 by naming them and the digits 4 and 5 by naming the opposite member of the pair. When presented in pure blocks of compatible or incompatible mappings, a 90 ms SRC effect was obtained. Mixing the two mappings slowed only the responses for the compatible mapping, reducing the SRC effect to 25 ms. Morin and Forrin (1962) conducted a similar study in which digit names were spoken in response to digit stimuli or shapes. The correct response to a digit stimulus was always its name, making this mapping compatible. Because digit names were arbitrarily assigned to the shapes, this task can be considered an ‘incompatible’ mapping. The difference between the incompatible conditions used by Morin and Forrin and by Van Duren and Sanders (1988) is that the shape stimuli are related to the digit names only by task-de1ned associations, whereas the digit stimuli have long-term associations to their corresponding responses. Mixing the tasks had virtually no effect on the incompatible shape naming task but slowed the compatible digit-naming task considerably. Forrin and Morin (1967) proposed a model to explain their mixing effects that combines the two-step model and the alternative routes model (see Ehrenstein and Proctor 1998, for detailed discussion of the model). To test the model, they conducted an experiment in which they varied the number of numerals and shapes. Their primary prediction from the model was that the set size for the alternative trial type would not have an in2uence on RT for a particular trial because after the appropriate category was selected, the response-selection route appropriate to the task would be used. However, there was a signi1cant 10-ms effect of set size for the shapes on numeral naming RT, which Forrin and Morin interpreted as counter to the model. Forrin and Morin also included blocks of trials in which the stimulus subset was precued 1 s prior to the imperative stimulus. Consistent with the studies of SRC, the precue reduced the effect of mixing by 14 ms, with this effect being independent of set size for digit stimuli but not for the shape stimuli. Forrin (1975) reported an experiment in which the stimuli were letters and digits, all of which were to be named. For pure lists, RT was 17 ms shorter for digit-naming than letter-naming. However, for mixed presentation, RT to digits increased 29 ms, whereas that to letters increased only 6 ms, making the RTs for the two categories similar in magnitude. This 1nding indicates that the condition that yields the shortest RT in pure blocks can be slowed by mixing, even when all responses are highly compatible and consistent with the long-term associations. This seems to be problematic for the alternative routes interpretation of the mixing effects because there is no obvious reason why the



direct route could not continue to be used under mixed conditions when the response triggered by each stimulus should be correct. Repetition analyses showed a 7-ms bene1t for category repetition and no additional bene1t for repetition of the speci1c stimulus. However, Marcel and Forrin (1974) obtained both category and item repetition bene1ts in the digit-letter naming task at response–stimulus intervals of 300 and 1600 ms, but only a category repetition effect at the 2900 ms interval. More importantly, they showed that this category repetition effect was eliminated by precuing the appropriate category 1000 ms prior to the imperative stimulus. In summary, when the location information was conveyed by arrow direction, results similar to those for physical locations were obtained: mixing increased RT more for compatible than for incompatible mappings, and this effect was reduced by precuing the mapping. In addition, the SRC effect was reduced less when the compatible mapping was repeated than when the preceding trial was incompatible. For location words, mixing reduced the SRC effect when the responses were vocal, but enhanced it considerably when the responses were keypresses. The studies that used digit/letter or digit/shape stimuli and vocal responses showed mixing, repetition, and precuing effects similar to those obtained in the studies of spatial SRC. These studies provide the additional information that mixed presentation reduces the bene1t of a compatible mapping even when the mixed stimuli have no long-term associations to responses from the same category and when they are also compatible but produce slower responses under blocked presentation.

22.4.2 Mapping effects for location-relevant trials mixed with location-irrelevant trials We conducted experiments with mixed LI and LR trials in which the location information for both trial types was arrow direction, location word, or physical location (Proctor and Vu 2001; Proctor et al. 2000). Proctor and Vu’s Experiment 1 used conditions in which the relevant location mapping was either compatible or incompatible. The physical location stimuli showed the pattern of results described previously, a 77 ms SRC effect with pure presentation of LR trials and a nonsigni1cant −16 ms effect with mixed presentation. When the location information was conveyed by arrows, an SRC effect of 42 ms was obtained with mixed presentation that was similar to the 32 ms effect with pure presentation. De Jong (1995) similarly obtained an SRC effect of 33 ms when the location information was conveyed by a left or right tilting upward pointing arrow and color was the relevant dimension on LI trials. Our study showed that for location words, the SRC effect was larger with mixed presentation (M = 172 ms) than with pure presentation (M = 21 ms), as was found when compatible and incompatible LR mappings were mixed. Thus, mixing location mappings with color mappings did not reduce the SRC effect for arrow directions or words, as it does for physical locations, and even sharply increased the effect for location words. Task repetition analyses for LR trials similar to those described previously for physical location stimuli were performed for the arrows and words (see Table 22.4). The ordering of repetition types for these stimuli was similar to that for physical locations: responses were slowest for nonrepetition trials, intermediate for task repetition–different, and fastest for task repetition–same. For words, but not arrows, this effect was quali1ed by an interaction with compatibility. This interaction for words was mainly due to the difference between the repetition and nonrepetition trials being smaller for the compatible (MD = 121 ms) than incompatible mapping (MD = 265 ms). When the task was repeated, the pure blocks for arrow stimuli showed little advantage of repeating the identical S–R pair from the previous trial (MD = 10 ms), in contrast to the bene1t shown in the mixed blocks (MD = 77 ms). Words showed a similar pattern, with the advantage for repeating the same S–R pair being less in the pure blocks (MD = 54 ms) than in the mixed blocks (MD = 126 ms).

459


460


Proctor and Vu’s (2001) Experiment 2 was similar to their Experiment 1, except that vocal ‘left’– ‘right’ responses were used. For all stimulus modes, the SRC effect was larger with mixed presentation of LR and LI trials than with pure presentation of LR trials, being larger for words (182 ms) than for arrows (67 ms) or physical locations (65 ms). A repetition analysis was also conducted (see Table 22.4). All stimulus types showed the same pattern of results. Responses were slowest on nonrepetition trials, intermediate on task repetition–different and fastest on task repetition–same. In addition, there was little repetition bene1t for the speci1c item in pure conditions (MDs = 6 ms for physical locations, 4 ms for arrows, and 13 ms for words). Thus, all three stimulus types showed similar patterns of repetition effects, including a large task-repetition bene1t. Across Proctor and Vu’s (2001) two experiments, the results indicate that whenever there is a verbal component to the task, either for the stimuli or responses, mixing magni1es the SRC effect. A possible explanation for why mixing increases the SRC effect when the task mode is verbal is as follows. For pure compatible mappings, the name of the stimulus is the correct response, and it is activated easily and executed rapidly. For pure incompatible mappings, the stimulus name is strongly associated with the assigned location response (e.g. ‘right’ is the highest associate of ‘left’; Proctor et al. 2000). Thus, subjects can generate the assigned response relatively quickly by being prepared to respond with the highly associated alternative location. For the mixed conditions, response selection for both mappings is mediated, at least in part, by 1rst naming the stimulus. With the compatible mapping, the response consistent with this name can be emitted quickly once the subject decides that location is relevant. With the incompatible mapping, the subject is not prepared to emit the assigned response but must engage in the additional time-consuming process of generating the correct response by applying an ‘opposite’ rule to the stimulus-location name.

22.4.3 Simon effects for location-irrelevant trials mixed with location-relevant trials With keypress responses, the arrow stimuli showed a response pattern for LI trials similar to that obtained with physical locations. The Simon effect was 75 ms when the LR mapping was compatible and −37 ms when it was incompatible. The major difference is that the reverse Simon effect

Table 22.4 Mean reaction time (in milliseconds) for location-relevant trials for mixed conditions of Proctor and Vu’s (2001) study as a function of experiment, stimulus mode, and repetition type Stimulus mode

Repetition type Nonrepetition

Task-repetition alternative

Complete repetition

Experiment 1: Keypress responses Physical locations 738 Arrow directions 714 Location words 935

633 585 806

538 508 680

Experiment 2: Vocal responses Physical locations 770 Arrow directions 688 Location words 726

686 609 688

602 553 628



obtained with arrows was not as large as the positive Simon effect, whereas for physical locations the negative effect is at least as large as the positive effect. De Jong (1995, Exp. 3), using left or right upward pointing arrow stimuli for mixed LR and LI trials, obtained an even stronger asymmetry. The Simon effect was 100 ms when the LR mapping was compatible and was reduced to a positive effect of about 30 ms when the LR mapping was incompatible. Location words showed positive Simon effects of 49 ms when the LR mapping was compatible and 29 ms when it was incompatible. In other experiments, the incompatible mapping condition for words has shown a small reverse Simon effect (Proctor et al. 2000, Experiments 1, 2, and 4). Proctor et al. pointed out that the Simon effect obtained with incompatibly mapped location words has a bimodal distribution, with subjects showing either a large negative or positive effect. The variability across experiments appears to re2ect probabilistic sampling from the bimodal distribution. This suggests that under mixed presentations with incompatible LR mappings, some subjects use verbal mediation in which they name the word before applying an opposite rule. Consistent with this interpretation, Proctor et al. demonstrated that mean RT was slower for the subjects showing a positive Simon effect than for those showing a negative Simon effect. Task repetition analyses of Proctor and Vu’s (2001) study indicated that, for all stimulus types, responses were faster when the LI task was repeated than when the previous trial was LR (see Table 22.5). None of the stimulus types showed two-way interactions of correspondence and task repetition or a three-way interaction of those variables with LR mapping. This indicates that the overall pattern of Simon effects (positive Simon effects with a compatible LR mapping for all stimulus types, negative Simon effects with an incompatible mapping for physical locations and arrows, and a positive Simon effect with an incompatible mapping for location words) was evident when the preceding trial was LR and when it was LI. Another analysis partitioned the LI repetition trials according to whether the preceding trial was corresponding or not (see Table 22.6). All three stimulus types showed similar results. For both compatible and incompatible mappings, the Simon effect was positive following a corresponding trial and negative following a non-corresponding trial. Table 22.5 Mean reaction time (in milliseconds) for location-irrelevant trials for mixed conditions of Proctor and Vu’s (2001) study as a function of experiment, stimulus mode, and repetition type Stimulus mode

Repetition type Task repetition

Task nonrepetition

Experiment 1: Keypress responses Physical locations 665 Arrow directions 580 Location words 687

774 707 929

Experiment 2: Vocal responses Physical locations 643 Arrow directions 614 Location words 675

766 715 817

461


462


Table 22.6 Mean Simon effects for reaction time (in milliseconds) for locationirrelevant trials for mixed conditions of Proctor and Vu’s (2001) study as a function of experiment, stimulus mode, compatibility, and correspondence on previous trial Stimulus mode

Correspondence on previous trial Corresponding

Experiment 1: Keypress responses Compatible mapping Physical locations 124 Arrow directions 58 Location words 78 Incompatible mapping Physical locations 32 Arrow directions 10 Location words 66 Experiment 2: Vocal responses Compatible mapping Physical locations Arrow directions Location words Incompatible mapping Physical locations Arrow directions Location words

Non-corresponding

−65 −31 −21 −133 −102 −8

108 103 118

−13 −2 20

24 60 74

−65 −60 −32

With vocal responses, all stimulus modes showed similar results for the Simon effect. When the LR mapping was compatible, positive Simon effects of similar magnitude were obtained for physical locations (56 ms), arrow directions (61 ms), and location words (61 ms). In contrast, when the LR mapping was incompatible, the Simon effect showed a small reversal or was not signi1cant. For physical locations and arrows, the reverse effect (−22 and −17 ms, respectively) was smaller than the positive one obtained with the compatible LR mapping. For location words, the mean Simon effect was positive (8 ms) even when the LR mapping was incompatible, but was not signi1cant. These results suggest that when the response mode is verbal, a signi1cant portion of subjects name the stimulus before selecting the response. Thus, when either the stimuli or responses are verbal in nature, there is a tendency for a stimulus to directly activate its corresponding name. A repetition analysis of the vocal-RT data for the LI trials showed that for all stimulus types, responses were faster when the task was repeated than when it was not (see Table 22.5). For physical locations and arrow directions, there was a four-way interaction between repetition, stimulus position, response position, and compatibility. Both the positive Simon effect for compatible mappings and the negative Simon effect for incompatible mappings were smaller when the trial was a repetition than when it was a nonrepetition (compatible mappings: 45 vs. 94 ms for physical locations and 56 vs. 78 ms for arrows; incompatible mappings: −15 vs. −36 ms for physical locations and −2 vs. −30 ms



for arrows). The repetition trials were partitioned, as for the keypress responses. All three stimulus types showed similar results (see Table 22.6). For compatible mappings, the Simon effect was positive when following a corresponding trial and negative or reduced considerably when following a noncorresponding trial. For incompatible mappings, the Simon effect was positive when following a corresponding trial and negative when following a noncorresponding trial. For each stimulus type, the effect of the preceding trial being corresponding or noncorresponding was of similar magnitude for the compatible and incompatible LR mappings. In summary, with keypress responses, when location information is conveyed by arrow direction, the Simon effect is enhanced if the LR mapping is compatible and reversed if it is incompatible, as for physical locations. However, the reversed effect is not as large as the positive effect. The in2uence of an incompatible LR mapping is even less when the location information is conveyed by word, with there being little or no Simon effect. With vocal responses, the effects of mixing are similar for all three stimulus types: a large Simon effect is obtained when the LR mapping is compatible, and the effect is eliminated or reversed slightly when the mapping is incompatible. As with physical locations, for trials on which the LI task is a repetition, the Simon effect is positive if the previous trial was corresponding and typically negative if the previous trial was noncorresponding. Thus, the occurrence of the Simon effect only on those trials that follow a corresponding trial is a general phenomenon and not one that is restricted to physical locations.

22.5 Mixing modes of location information The reverse Simon effect for LI trials when the LR mapping is incompatible indicates that stimuli produce activation via the task-de1ned location–response associations even on the LI trials. These associations could be independent of, or dependent on, stimulus mode. In terms of connectionist models, the associations could involve links from concept nodes to output nodes (e.g. Zhang and Kornblum 1998) or links from mode-speci1c nodes to output nodes (e.g. Cohen, Dunbar, and McClelland 1990). We conducted experiments to evaluate these alternatives by using different modes for the location information on LR and LI trials (Proctor et al. 2000). In Proctor et al.’s (2000) study, all combinations of physical locations, arrows, and words were evaluated in which the location information was conveyed in different stimulus modes on the LR and LI trials (see Table 22.7). The LR mapping was incompatible for all conditions, and Simon effects of −5 to 33 ms were obtained. Clearly, none of the stimulus types showed a strong reversal of the Simon effect similar to that obtained when the LR mapping is incompatible and the two trial types are presented in the same mode. The possibility exists that the form distinction between the two stimulus modes is the factor that eliminated the reversal of the Simon effect. To explore this possibility, Proctor et al.’s Experiment 4 used form distinctions of stimuli within the same mode. That is, for physical positions, the stimuli were circles for one trial type and were squares for the other (large and small arrows were used for arrow stimuli and upper-case and lower-case words were used for location words). Reverse Simon effects of −65 ms, −44 ms, and −17 ms were obtained for physical locations, arrow directions, and location words, respectively. These effects are of similar magnitude to those obtained when the two trial types were conveyed by stimuli of the same shape and size. Thus, a form distinction is not suf1cient to eliminate the reverse Simon effect. Proctor and Vu (2001, Exp. 3) showed that mode differences reduce the effect of mixing on the SRC effect for LR trials. Four groups were tested with compatible or incompatible LR mappings, with physical position relevant and location word irrelevant, or vice versa. The SRC effect for physical

463


464


Table 22.7 Mean Simon effect for reaction time (in milliseconds) for location-irrelevant trials when location-relevant trials are incompatibly mapped in Proctor, Marble, and Vu’s (2000) study as a function of stimulus mode for each trial type Location–relevant mode

Location–irrelevant mode

Spatial compatibility effect

Physical locations Location words Location words Arrow directions Arrow directions Physical locations

Location words Physical locations Arrow directions Location words Physical locations Arrow directions

32 4 33 13 3 −5

locations was 100 ms, in contrast to the non-signi1cant −16 ms effect when LI stimuli were presented in the same mode. The SRC effect for location words was 110 ms, which is smaller than the 172 ms effect obtained when location-irrelevant stimuli were presented in the same mode, but considerably larger than the 21 ms effect obtained in pure blocks of compatibly or incompatibly mapped location words. Thus, whereas presenting the LI information in words eliminates the effect of mixing on the SRC effect for physical locations, presenting the LI information in physical locations only reduces the effect of mixing for words.

22.6 General discussion The studies investigating mixed presentation of compatible and incompatible LR mappings and of mixed LR and LI trials show that task set is a major determinant of the SRC effects that are obtained. The SRC effect for the LR trials varies systematically as a function of the nature of the intermixed tasks, as does the Simon effect for LI trials.

22.6.1 Summary of major 1ndings 22.6.1.1 Reduction, elimination, and enhancement of the SRC effect Shaffer (1965) showed that, for two-choice SRC tasks in which stimuli were left–right locations and responses left–right keypresses, mixing trials with compatible and incompatible mappings eliminated the SRC effect. We replicated Shaffer’s results using stimulus color, rather than a separate mapping signal, to designate the mapping (Vu and Proctor 2001). We also included conditions in which left– right keypresses were made to left–right arrow directions or location words. For arrows, mixing compatible and incompatible mappings eliminated the SRC effect, as for physical locations. However, for words, mixing enhanced the SRC effect. In contrast, when the responses were the spoken words ‘left’ and ‘right’, mixing tended to reduce, but not eliminate, the SRC effect for all three stimulus modes. Thus, mixing compatible and incompatible mappings in two-choice tasks does not always eliminate, or even reduce, the SRC effect. Our experiments in which compatibly or incompatibly mapped LR trials were mixed with LI trials showed that the SRC effect was eliminated for physical locations mapped to keypresses (Marble and Proctor 2000; Proctor and Vu 2001; Proctor et al. 2000; Proctor et al. in press). Proctor et al. (in



press) showed that the elimination of the SRC effect with physical locations and keypresses occurs primarily for situations in which the LR and LI trials share left–right spatial codes. When the LI stimuli are presented in a centered location, a normal SRC effect is obtained. However, when the LI and LR stimuli are presented in two rows, above or below 1xation, no SRC effect is evident. In both cases, the LI stimuli appear in locations that are distinct from those in which the LR stimuli occur. The major difference is that left and right codes are present for both trial types in the latter experiment, but not in the former. Finally, an SRC effect is obtained when the LI stimuli vary along the vertical dimension and the LR stimuli along the horizontal dimension. Thus, the elimination of the SRC effect under mixed presentation is apparently a consequence of uncertainty about whether the compatible response can be made to the location code that is formed upon stimulus presentation. With keypress responses, mixing LI and LR trials had different effects on SRC for arrow directions and location words than for physical locations (Proctor and Vu 2001). For arrows, the SRC effect was of similar magnitude to that obtained with pure presentation of LR trials, whereas for words, mixing increased the magnitude of the SRC effect. With vocal ‘left’–‘right’ responses, mixing LR and LI trials increased the SRC effect for all stimulus types. Thus, mixing LR and LI trials eliminated the SRC effect only for physical locations mapped to keypress responses. When either stimuli or responses were of a verbal nature, mixing the trial types increased the SRC effect. This outcome implies that response selection is mediated in these cases to a considerable extent by activation of the stimulus name.

22.6.1.2 Enhancement and reversal of the Simon effect With physical location stimuli and keypresses, the reverse Simon effect obtained when the LR mapping was incompatible was at least as large as the enhanced positive Simon effect obtained when the mapping was compatible (Marble and Proctor 2000; Proctor and Vu 2001; Proctor et al. 2000; Proctor et al. in press). When the location information was conveyed by arrow direction or location words and the responses were keypresses, mixing had the following effects. For arrows, mixed compatible LR trials enhanced the Simon effect and mixed incompatible LR trials reversed it. However, unlike physical location stimuli, the reversed effect was not as large as the positive Simon effect. For location words, a positive Simon effect was obtained when the LR mapping was compatible, but only a small negative or positive Simon effect when the LR mapping was incompatible. With vocal responses, physical location and arrow stimuli showed a small reverse Simon effect when LI trials were mixed with incompatibly mapped LR trials. For location words, a small positive Simon effect was obtained when the LR mapping was incompatible. When the LR mapping was compatible, all stimulus types showed a large positive Simon effect. Thus, the task-de1ned associations of location information to responses in2uence performance for symbolic and verbal modes, as well as for the physical mode. 22.6.1.3 Mixed location modes Presenting LR and LI trials in distinct modes reduces the in2uence of mixing on LR trials. When physical locations convey the LR information and location words the LI information, the SRC effect is reinstated, and possibly even enhanced (Proctor and Vu 2001). When words convey the LR information and physical locations convey the LI information, the SRC effect is obtained: it is reduced in magnitude compared with mixed presentation of location words for both trial types, but the effect is larger than that obtained with blocked presentation of LR trials. Presenting location information in distinct modes on LR and LI trials also reduces the impact of the LR mapping on the Simon effect (Proctor et al. 2000). With the incompatible mapping, the reverse Simon effect is

465


466


eliminated regardless of which stimulus mode is used for LR trials and which for LI trials, with a positive Simon effect being reinstated fully for some mode combinations. Thus, the results for both the Simon effect and the SRC effect imply that the task-de1ned associations of location information to responses are relatively mode speci1c.

22.6.1.4 Precuing and repetition effects When compatible and incompatible mappings of physical locations or arrows are mixed, precuing the mapping reinstates the SRC effect (De Jong 1995; Shaffer 1965): Subjects can prepare for the cued mapping in a way that allows the normal bene1t for the compatible mapping to occur. When incompatibly mapped LR trials are mixed with LI trials, precuing the trial type improves performance but does not eliminate the reverse Simon effect for the LI trials. Because only one location mapping (incompatible) is in effect for the block of trials, these S–R associations continue to be applied on the LI trials. The reverse Simon effect also is obtained when subjects practice with an incompatible spatial mapping prior to performing a Simon task (Proctor and Lu 1999). That the reverse Simon effect is obtained when location is known to be irrelevant indicates that the effect is not simply a consequence of uncertainty about which stimulus dimension is relevant. Task repetition typically produces a substantial RT bene1t compared to a task switch. When compatible and incompatible LR mappings are mixed, the compatible mapping bene1ts more from repetition than does the incompatible mapping (Vu and Proctor 2001). When LR and LI trials are mixed, bene1ts occur for repeating the same task and repeating the same S–R pair. The task repetition bene1t on LR trials is of similar magnitude for compatible and incompatible mappings (e.g. Proctor et al. 2001). Thus, although compatible trials receive an extra bene1t from repetition when mixed with incompatible trials, they do not receive an extra bene1t relative to the incompatible trials when either type is mixed with LI trials in separate blocks. Another way of describing the aforementioned relation for the mixed LR and LI conditions is that the SRC effect is present when an LR trial follows either an LR or LI trial. The positive and reverse Simon effects for compatible and incompatible LR mappings, respectively, also occurred regardless of whether the LI task followed an LR or LI trial. However, for task repetition trials, the Simon effect was dependent on the correspondence relation of the preceding trial. Regardless of the condition (i.e. pure Simon, mixed-compatible, or mixed-incompatible), the Simon effect was always positive when following a corresponding trial and negative following a noncorresponding trial, although the relative magnitudes of the positive and negative effects did differ across conditions. Thus, the repetition effects of the type reported by Mordkoff (1998), Leuthold et al. (1999), and Valle-Inclán et al. (2002) for pure Simon trials occur equally when LI and LR trials are mixed. However, the method of examining correspondence effects with respect to whether the previous trial was corresponding or not can be problematic because different combinations of trial types are collapsed together (Hommel 2000). We conducted an analysis of Marble and Proctor’s (2000) Experiment 1 that examined the same trials partitioned into conditions in which stimulus color (also response) and stimulus position were repeated or changed on consecutive trials. With pure LI trials, performance was better when both the stimulus color and position repeated or changed compared with conditions in which only one changed. With mixed LI and LR trials, collapsed across location mapping, repetition of the LI task bene1ted only when both the stimulus color and position were repeated and not when they both changed. For the Simon effect, when the mixed LR mapping was compatible, the effect was larger when both stimulus color and position were changed or repeated



than when only one was repeated. When the mixed LR mapping was incompatible, this relation held for the reverse Simon effect: the effect was larger when both stimulus color and position were changed or repeated than when only one was repeated. This pattern of repetition effects seems more consistent with an account in terms of integration of stimulus and response features from the previous trial, as Hommel (1998b) suggests, than with voluntary gating of the direct route.

22.6.2 Evaluation of the pure translation, two-step, and alternative routes accounts When two tasks or trial types, for which the same set of responses are mapped to stimuli in different manners, are mixed, the subject must maintain multiple intentions, or task sets. To respond with high accuracy, a determination must be made when the stimulus is presented as to which task set is appropriate, and this task set must ultimately control response selection. We evaluate the three accounts, described in the Introduction, that have been proposed to explain the effects on performance of mixing different mappings of stimuli and responses.

22.6.2.1 Pure translation (translation ef1ciency) Shaffer (1965) proposed that for the conditions of his study in which all S–R pairs were known in advance to be compatible, ‘the null transformation could be considered a special class and was easier to compute’ (p. 287). In current terms, this amounts to saying that the bene1t for the pure compatible conditions arises from use of an identity rule (i.e. respond with the corresponding response) that cannot be used when some mappings are incompatible. This identity rule is more ef1cient than searching for the assigned response or application of an opposite rule (i.e. respond with the noncorresponding response). Shaffer’s pure translation account has not received much attention, although it seems to account for several results. One 1nding that creates dif1culty for the account is that when incompatible and compatible mappings are mixed, errors tend to be the response that would be correct if the alternative mapping were appropriate for the stimulus that was presented (Duncan 1977a,b, 1978; Ehrenstein and Proctor 1998; Stoffels 1996b). This result implies that response selection occurs at least in part through application of mapping rules even when compatible and incompatible trials are mixed. The pure translation account also does not provide an adequate explanation for the greater repetition bene1t shown for the compatible mapping than for the incompatible mapping under mixed conditions. One would have to propose that the identity rule was applied after a compatible trial but not if the preceding trial was incompatible, which does not seem plausible. With mixed presentation of LR and LI trials, a pure translation model cannot account for the 1nding that repetition effects are equal for compatible and incompatible mappings. 22.6.2.2 Two-step response selection Duncan (1977a,b, 1978) proposed that with mixed lists of compatible and incompatible mappings, response selection proceeds in two steps: determination is made as to which of the mappings is applicable for the trial, and then that mapping rule is applied. A distinguishing feature of this model is that only one rule is applied on any given trial, with the rule being the appropriate one in most cases and the inappropriate one when an error is made in the 1rst step. The strongest support for this model is that errors are typically the correct response for the inappropriate rule for that trial (Duncan, 1977a,b, 1978; Ehrenstein and Proctor 1998; Stoffels 1996b). However, the two-step model alone cannot account readily for several 1ndings. The most straightforward prediction of the model

467


468


is that the effect of mixing should be additive with that of mapping because the major difference between mixed and pure blocks is the additional mapping-selection step required for all trials. The model can accommodate the more customary 1nding of larger mixing costs for compatible than incompatible mapping by assuming that the incompatible trials have a bene1t due to reduction in number of S–R alternatives that the compatible trials do not. However, even with this assumption, the model cannot explain the fact that a precue is more bene1cial for the compatible than incompatible mapping (Ehrenstein and Proctor 1998; Shaffer 1965; Stoffels 1996b). This is because the additional factor that produces the bene1t for the incompatible mapping should still be contributing to performance when the mappings are precued. With respect to mixing LR and LI trials, the two-step model loses more ground. It does not directly predict the elimination of the SRC effect when location is irrelevant on half of the trials. Moreover, the model does not differentiate between the experiments for which mixing eliminates the SRC effect (presenting the LR and LI stimuli in the same locations; presenting both trial types in left and right locations on different rows) and those for which the SRC effect is evident (presenting the LI stimuli in the center of the screen; presenting the LI stimuli along the vertical dimension and LR stimuli along the horizontal dimension). Also, because mapping selection is presumed to precede response selection within the mapping, the extent of the crosstalk effects of the LR mapping on the LI trials seems to greatly exceed the magnitude that would be expected. At a minimum, the two-step model must be combined with additional properties to explain a reasonable range of 1ndings. For example, Ehrenstein and Proctor (1998) suggested that when combined with the alternative routes account, the two-step model can account for most data from tasks in which compatible and incompatible mappings are mixed. Another possibility suggested by the considerable cross-activation for the LR and LI trial types is that activation proceeds initially with regard to the associations de1ned for both tasks, with the decision about which associations are appropriate for the current trial occurring relatively late in processing (e.g. Hommel, 1998a).

22.6.2.3 Alternative routes model Van Duren and Sanders (1988) proposed that compatible mappings bene1t from long-term S–R associations and thus from a direct response-selection route when it is known in advance that all trials will be compatible. However, when all trials are incompatibly mapped or the two trial types are mixed, the direct route cannot be used because the S–R mapping is based on task-de1ned associations that require search from memory. Another way to describe the alternative routes model is that the direct route is used only if all mappings are compatible because it leads to the correct response on all trials, and the direct route is suppressed when all mappings are incompatible or when mappings are mixed because it leads to the wrong response on many trials. The alternative routes model is the most popular explanation of mixing effects. Unlike the two-step model, it can account for the precuing bene1ts obtained with mixed compatible and incompatible mappings. In this case, the compatible mapping bene1ts more than the incompatible mapping because the precue allows the direct route to be used. If it is assumed that the direct route is suppressed following an incompatible trial but not following a compatible trial (e.g. Stoffels 1996a,b), then the fact that mixing primarily reduces the SRC effect for trials on which the mapping is not repeated can be accommodated as well. With regard to mixing LR and LI trials, the alternative routes model can explain the pattern of SRC effects obtained: the direct route is suppressed for situations in which the location codes cannot serve as unambiguous indicators of a compatible response but not for situations in which they



can. The alternative routes model does not make speci1c predictions regarding the Simon effects obtained on LI trials because the direct route should not be a contributing factor. However, the results are generally consistent with the model if it is assumed that the task-de1ned associations of locations in the indirect translation route produce activation on LI trials. The alternative routes model predicts that any in2uence of mixing on the SRC effect should be to reduce its magnitude. Therefore, it cannot account for the fact that, for location words mapped to keypress responses, the SRC effect increased with mixing of compatible and incompatible LR trials and mixing of LR and LI trials. It also cannot account for the increase in the SRC effect that occurred for mixed LI and LR trials for vocal responses, with all stimulus modes. However, the results can be accommodated by the model if it is modi1ed to allow stimuli to activate their corresponding names, regardless of mapping, for a signi1cant portion of trials in conditions for which either the stimulus or response is verbal. A more serious problem for the alternative routes model arises from the results of studies that used mixed mappings for digit naming: mixing reduces the advantage for the compatible mapping under conditions where the direct route should provide an appropriate basis for response selection. Van Duren and Sanders’s (1988) 1nding that the advantage for compatibly mapped digit stimuli and naming responses is eliminated when some digits are incompatibly mapped to digit names is consistent with the model, as they concluded. However, when shapes mapped to digit names are mixed with compatibly mapped digit stimuli, as in Morin and Forrin’s (1962; Forrin and Morin 1967) studies, the correct response to a digit is always its corresponding name; thus, the argument would have to be made that the direct route is suppressed in order to prevent the shapes from being named. Even more problematic, the advantage for digit naming is eliminated when the compatibly mapped digits are mixed with compatibly mapped letters (Forrin 1975), in which the direct route should yield the correct responses for all stimuli. Although the mixing effects in Forrin’s study were small, they suggest that mixing stimulus categories is suf1cient to reduce the bene1t for the easier task even when both tasks require compatible responses.

22.6.3 Alternative theoretical accounts Overall, the alternative routes model, modi1ed to accommodate the results obtained with verbal stimuli or responses, seems to explain more 1ndings than the two-step and pure translation models. However, all of the models, including the alternative routes model, cannot explain certain 1ndings easily. Los (1996) classi1es these models as structural models, for which the mixing costs are primarily strategic in nature. An alternative view is that mixing costs are stimulus-driven in nature, being a function primarily of the greater intertrial variability, which in turn necessitates more trial-to-trial adjustments. Los suggests that the effects of precuing and repetition are particularly diagnostic for deciding which class of models, strategic or stimulus-driven, can best account for mixing effects. Speci1cally, he argues that if precuing eliminates mixing costs and sequential effects are absent, a strategic explanation is implicated; whereas if sequential effects are found and precuing is ineffective, then a stimulus-driven account is favored. Several studies have shown that precuing a compatible or incompatible mapping largely eliminates the cost of mixing the two mappings (De Jong 1995; Ehrenstein and Proctor 1998; Forrin and Morin 1967; Shaffer 1965, 1966; Stoffels 1996a,b), consistent with a strategic account. However, precuing a trial as LI does little to reduce the effect of an incompatible LR mapping (Marble and Proctor 2000). Also, the mapping continues to be applied after the task has been changed to one for which

469


470


location is no longer relevant (Proctor and Lu 1999; Tagliabue et al. 2000). Thus, there clearly is not complete strategic control of response selection. Perhaps the most striking and consistent aspect of the mixing literature is that large, systematic repetition effects are obtained in all task variations. When repetition analyses have been performed, sequential effects were apparent that, in the case of mixed compatible and incompatible mappings, mimic the precuing results relatively closely. The possibility that subjects engage in trial-to-trial strategic alterations of task set cannot be ruled out, but, as Los (1996) stated, ‘Sequential effects should be considered as at least slightly embarrassing for a strategic account of mixing costs’ (p. 172). More speci1cally, Hommel (2000) showed that the pattern of repetition effects that Mordkoff (1998) and Leuthold et al. (1999) interpreted as indicating voluntary gating of the direct route is more consistent with a stimulus-driven account when repetition analyses do not collapse across the different combinations of stimulus color and position. Our analyses described earlier in the ‘General discussion’ support this point. Research conducted on task switching, in which two tasks are presented in a 1xed order (e.g. on alternate trials), has obtained results suggesting that there is a large component of changing task set that is not under voluntary control. Unlike the mixing studies described in this paper, because the sequence is orderly, subjects are aware of what the speci1c task will be on each trial. Even though the task sequence is known, the typical 1nding is that performance is worse when the task alternates between trials than when only single a task is presented for all trials (Jersild 1927). This switching cost is evident even when the interval between trials is suf1cient to allow preparation for the forthcoming task (Allport, Styles, and Hsieh 1994; Rogers and Monsell 1995), suggesting that there is a component that is not under the subject’s control. Allport et al. conducted an experiment using a Stroop color-naming task, which is a close relative of the Simon tasks used in our experiments, with either the color word or physical color named on alternate trials. The easier word-reading task was slowed considerably, compared to a baseline condition, when it followed the color-naming task, whereas the more dif1cult color-naming task was unaffected. This asymmetric effect was evident at response–stimulus intervals up to 1100 ms. Thus, costs for the easier task occur with the switching method much as with randomized presentation of the two task types, even when there is more than one second to prepare for the appropriate task. Allport et al. (1994) called the component of task switching that is not under the subject’s control task set inertia, and Rogers and Monsell (1995) called it exogenous task-set recon1guration. Regardless of the exact nature of this component, the point is that a signi1cant portion of the switching costs is not under strategic control. Rogers and Monsell (Exp. 6) used sequences of four task repetitions and then a switch to the alternate task for four consecutive trials, and so on, and found that the switch costs were eliminated after the 1rst trial of the new task. Their results imply that the distinction between task-repetition and switch trials should be of importance when trials with different mappings or relevance of the location dimension are randomly intermixed, as is in fact the case. Thus, a substantial portion of the mixing effects is stimulus driven.

22.7 Conclusion Donders (1868/1969) was the 1rst to recognize that performance of two-choice reaction tasks with different S–R sets and mappings afford considerable insight into response selection in particular and the interaction between perception and action in general. Although Donders did not speci1cally examine the role of task set in his experiments, two-choice tasks can also provide much valuable



information about this role. The experiments described in this paper demonstrate that SRC effects obtained with location information are quite susceptible to the in2uence of task set. When compatible and incompatible LR mappings are mixed, the bene1t for the compatible mapping typically is reduced or eliminated, but it is enhanced when locations words are mapped to keypresses. When LI and LR trials are mixed, the SRC effect for the LR trials is eliminated for physical locations mapped to keypresses, unaffected for arrows mapped to keypresses, and enhanced when the stimulus or response mode is verbal in nature. The mappings de1ned for the LR task are applied on trials for the LI task and, thus, largely determine the pattern of Simon effects. For physical locations assigned to keypress responses, the presence of codes along the same dimension on the intermixed trials (LI or incompatible spatial mapping) precludes rapid responding on the basis of these codes for trials on which the location mapping is compatible. When either the stimulus or response is verbal, a tendency for activating the stimulus name exists, regardless of LR mapping, which increases the bene1t for the compatible mapping under mixed conditions. The major accounts for the effects of mixing attribute them to controlled processing strategies. However, the systematic repetition effects obtained with mixed presentation of different types of trials imply that a substantial part of the mixing effects may be involuntary and stimulus driven. Regardless of the ultimate explanation, there is little doubt hat SRC effects involve much more than learned associations between stimuli and responses and that task set plays a signi1cant role.

References Allport, A., Styles, E.A., and Hsieh, H. (1994). Shifting intentional set: Exploring the dynamic control of tasks. In C. Umiltà and M. Moscovitch (Eds.), Attention and performance XV, pp. 421–452. Cambridge, MA: MIT Press. Baldo, J.V., Shimamura, A.P., and Prinzmetal, W. (1998). Mapping symbols to response modalities: Interference effects on Stroop-like tasks. Perception and Psychophysics, 60, 427–437. Barber, P. and O’Leary, M. (1997). The relevance of salience: Towards an activation account of irrelevant stimulus– response compatibility effects. In B. Hommel and W. Prinz (Eds.), Theoretical issues in stimulus–response compatibility, pp. 135–172. Amsterdam: North-Holland. Broadbent, D.E. and Gregory, M. (1962). Donders’ B- and C-reactions and S–R compatibility. Journal of Experimental Psychology, 63, 575–578. Cohen, J.D., Dunbar, K., and McClelland, J.L. (1990). On the control of automatic processes: A parallel distributed processing account of the Stroop effect. Psychological Review, 97, 332–361. De Jong, R. (1995). Strategical determinants of compatibility effects with task uncertainty. Acta Psychologica, 88, 187–207. Donders, F.C. (1868/1969). On the speed of mental processes. In W.G. Koster (Ed.), Acta Psychologica, 30, Attention and performance II, pp. 412–431. Amsterdam: North-Holland. Duncan, J. (1977a). Response selection errors in spatial choice reaction tasks. Quarterly Journal of Experimental Psychology, 29, 415–423. Duncan, J. (1977b). Response selection rules in spatial choice reaction tasks. In S. Dornic (Ed.), Attention and performance VI, pp. 49–61. Hillsdale, NJ: Erlbaum. Duncan, J. (1978). Response selection in spatial choice reaction: Further evidence against associative models. Quarterly Journal of Experimental Psychology, 30, 429–440. Ehrenstein, A. and Proctor, R.W. (1998). Selecting mapping rules and responses in mixed compatibility fourchoice tasks. Psychological Research, 61, 231–248. Eimer, M. (1995). S–R compatibility and automatic response activation: Evidence from psychophysiological studies. Journal of Experimental Psychology: Human Perception and Performance, 21, 837–854. Fitts, P.M. (1964). Perceptual–motor skill learning. In A.W. Melton (Ed.), Categories of human learning pp. 243–285. New York: Academic Press.

471


472


Fitts, P.M. and Deininger, R.L. (1954). S–R compatibility: Correspondence among paired elements within stimulus and response codes. Journal of Experimental Psychology, 48, 483–492. Fitts, P.M. and Seeger, C.M. (1953). S–R compatibility: Spatial characteristics of stimulus and response codes. Journal of Experimental Psychology, 46, 199–210. Forrin, B. (1975). Naming latencies to mixed sequences of letters and digits. In P.M.A. Rabbitt and S. Dornic (Eds.), Attention and performance V, pp. 345–356. New York: Academic Press. Forrin, B. and Morin, R.E. (1967). Effects of context on reaction time to optimally coded signals. Acta Psychologica, 27, 188–196. Greenwald, A.G. and Rosenberg, K.E. (1978). Sequential effects of distraction stimuli in a selective attention reaction time task. In J. Requin (Ed.), Attention and performance VII, pp. 487–504. Hillsdale, NJ: Erlbaum. Hedge, A. and Marsh, N.W.A. (1975). The effect of irrelevant spatial correspondences on two-choice response time. Acta Psychologica, 39, 427–439. Hommel, B. (1994). Spontaneous decay of response-code activation. Psychological Research, 56, 261–268. Hommel, B. (1998a). Automatic stimulus–response translation in dual-task performance. Journal of Experimental Psychology: Human Perception and Performance, 24, 1368–1384. Hommel, B. (1998b). Event 1les: Evidence for automatic integration of stimulus–response episodes. Visual Cognition, 5, 183–216. Hommel, B. (2000). Stimulus and response feature integration masquerading as information gating and route suppression. Manuscript submitted for publication. Hommel, B. and Prinz, W. (Eds.) (1997). Theoretical issues in stimulus–response compatibility. Amsterdam: North-Holland. Jersild, A.T. (1927). Mental set and shift. Archives of Psychology, Whole No. 89. Kornblum, S., Hasbroucq, T., and Osman, A. (1990). Dimensional overlap: Cognitive basis for stimulus– response compatibility: A model and taxonomy. Psychological Review, 97, 253–270. Leuthold, H., Stürmer, B., Soetens, E., Schroter, H., and Sommer, W. (1999). Suppression of location-based priming in the Simon task: Behavioral and electrophysiological evidence. Manuscript submitted for publication. Logan, G.D. (1980). Attention and automaticity in Stroop and priming tasks: Theory and data. Cognitive Psychology, 12, 523–553. Logan, G. and Zbrodoff, N.J. (1979). When it helps to be misled: Facilitative effects of increasing the frequency of con2icting stimulus in a Stroop-like task. Memory and Cognition, 7, 166–174. Los, S.A. (1996). On the origin of mixing costs: Exploring information processing in pure and mixed blocks of trials. Acta Psychologica, 94, 145–188. Lu, C.-H. and Proctor, R.W. (1995). The in2uence of irrelevant location information on performance: A review of the Simon effect and spatial Stroop effects. Psychonomic Bulletin and Review, 2, 174–207. Lu, C.-H. and Proctor, R.W. (2001). In2uence of irrelevant information on human performance: Effects of S-R association strength and relative timing. Quarterly Journal of Experimental Psychology, 54A, 95–136. Marble, J.G. and Proctor, R.W. (2000). Mixing location-relevant and location-irrelevant choice-reaction tasks: In2uences of location mapping on the Simon effect. Journal of Experimental Psychology: Human Perception and Performance, 26, 1515–1533. Marcel, T. and Forrin, B. (1974). Naming latency and the repetition of stimulus categories. Journal of Experimental Psychology, 103, 450–460. Mordkoff, T. (1998). The gating of irrelevant information in selective-attention tasks (Abstract). Abstracts of the Psychonomic Society, 3, 193. Morin, R.E. and Forrin, B. (1962). Mixing of two types of S–R associations in a choice reaction time task. Journal of Experimental Psychology, 64, 137–141. Prinz, W. (1997). Why Donders has led us astray. In B. Hommel and W. Prinz (Eds.), Theoretical issues in stimulus–response compatibility, pp. 247–267. Amsterdam: North-Holland. Proctor, R.W. and Lu, C.-H. (1999). Processing irrelevant location information: Practice and transfer effects in choice–reaction tasks. Memory and Cognition, 27, 63–77. Proctor, R.W. and Vu, K.-P.L. (2001). Mixing location irrelevant and relevant trials: In2uence of stimulus mode on spatial compatibility effects. Manuscript submitted for publication. Proctor, R.W. and Wang, H. (1997). Differentiating types of set-level compatibility. In B. Hommel and W. Prinz (Eds.), Theoretical issues in stimulus–response compatibility, pp. 11–37. Amsterdam: North-Holland. Proctor, R.W., Marble, J.G., and Vu, K.-P.L. (2000). Mixing incompatibly mapped location-relevant trials with location-irrelevant trials: Effects of stimulus mode on the reverse Simon effect. Psychological Research/ Psychologische Forschung, 64, 11–24.



Proctor, R.W., Vu, K.-P.L., and Marble, J.G. (in press). Mixing location-relevant and irrelevant tasks: Spatial compatibility effects eliminated by stimuli that share the same spatial codes. Visual Cognition. Rogers, R.D. and Monsell, S. (1995). Cost of a predictable switch between simple cognitive tasks. Journal of Experimental Psychology: General, 124, 207–231. Shaffer, L.H. (1965). Choice reaction with variable S–R mapping. Journal of Experimental Psychology, 70, 284–288. Shaffer, L.H. (1966). Some effects of partial advance information on choice reaction with 1xed or variable S–R mapping. Journal of Experimental Psychology, 72, 541–545. Simon, R.J. (1990). The effects of an irrelevant directional cue on human information processing. In R.W. Proctor and T.G. Reeve (Eds.), Stimulus–response compatibility: An integrated perspective, pp. 31–86. Amsterdam: North-Holland. Stoffels, E.J. (1996a). On stage robustness and response selection routes: Further evidence. Acta Psychologica, 91, 67–88. Stoffels, E.J. (1996b). Uncertainty and processing routes in the selection of a response: An S–R compatibility study. Acta Psychologica, 94, 227–252. Tagliabue, M., Zorzi, M., Umiltà, C., and Bassignani, F. (2000). The role of LTM links and STM links in the Simon effect. Journal of Experimental Psychology: Human Perception and Performance, 26, 648–670. Toth, J.P., Levine, B., Stuss, D.T., Oh, A., Winocur, G., and Meiran, N. (1995). Dissociation of processes underlying spatial S–R compatibility: Evidence for the independent in2uence of what and where. Consciousness and Cognition, 4, 483–501. Umiltà, C. and Nicoletti, R. (1990). Spatial stimulus–response compatibility. In R.W. Proctor and T.G. Reeve (Eds.), Stimulus–response compatibility: An integrated perspective, pp. 89–116. Amsterdam: North-Holland. Umiltà, C. and Zorzi, M. (1997). Commentary on Barber and O’Leary: Learning and attention on S–R compatibility. In B. Hommel and W. Prinz (Eds.), Theoretical issues in stimulus-response compatibility, pp. 173–178. Amsterdam: North-Holland. Valle-Inclán, F., Hackley, S.A., and de Labra, C. (2002). Does stimulus-driven response activation underlie the Simon effect? This volume, Chapter 23. Van Duren, L. and Sanders, A.F. (1988). On the robustness of the additive factors stage structure in blocked and mixed choice reaction designs. Acta Psychologica, 69, 83–94. Vu, K.-P.L. and Proctor, R.W. (2001). Mixing compatible and incompatible mappings: Elimination, reduction, and enhancement of spatial compatibility effects. Manuscript submitted for publication. Wang, H. and Proctor, R.W. (1996). Stimulus–response compatibility as a function of stimulus code and response modality. Journal of Experimental Psychology: Human Perception and Performance, 22, 1201–1217. Zhang, H. and Kornblum, S. (1998). The effects of stimulus-response mapping and irrelevant stimulus– response and stimulus–stimulus overlap in four-choice Stroop tasks with single-carrier stimuli. Journal of Experimental Psychology: Human Perception and Performance, 24, 3–19. Zorzi, M. and Umiltà, C. (1995). A computational model of the Simon effect. Psychological Research, 58, 193–205.

473


23 Does stimulus-driven response activation underlie the Simon effect? Fernando Valle-Inclán, Steven A. Hackley, and Carmen de Labra Abstract. The most in2uential explanations of the Simon effect assume that a spatial stimulus code automatically activates the spatially compatible response. However, the degree to which activation of the compatible response should be considered as an automatic, stimulus-driven process is uncertain. To assess this presumed automaticity we conducted two experiments to analyze the in2uence on the Simon effect of repetitions versus alternations of stimulus, response, and stimulus–response compatibility (SRC). The results indicate that the Simon effect is produced only when the previous trial is compatible. Furthermore, the lateralized readiness potential showed clear signs of incorrect response activation on incompatible trials when previous trial was compatible, but not when the previous trial was incompatible. These results indicate that SRC sequences are critical for the appearance of the Simon effect and that the stimulus spatial code alone does not automatically activate the spatially corresponding response.

23.1 Introduction In their seminal 1967 report, Simon and Rudell showed that spatial stimulus–response compatibility (SRC) effects could be obtained even when stimulus location was task irrelevant. They presented the word ‘left’ or ‘right’ randomly at either ear and required subjects to press a key located on the left or right side depending on the meaning of the word. Although the spatial source of the sound was irrelevant, performance deteriorated when the stimulus and response were on opposite sides. Since then, this special case of SRC has been clearly established in different sensory modalities and in a variety of experimental tasks. The general pattern is that reaction time (RT) is shorter and response accuracy is greater when stimulus and response are on the same side (compatible trials) than when they are on opposite sides (incompatible trials). Research on this phenomenon, known as the Simon effect, has established the following points: (1) the Simon effect cannot be explained by commisural crossing delays (Simon, Hinrichs, and Craft 1970); (2) it is not restricted to a particular sensory modality and can be obtained with crossmodal stimulation (Simon 1982; Simon and Acosta 1982); (3) the size of the effect tends to decrease from fast to slow responses (De Jong, Liang, and Lauber 1994; Hommel 1993a); and (4) at least one locus of the effect is the response selection stage, a proposition that receives support from studies of movement-related brain potentials (de Jong, et al. 1994; Valle-Inclán 1996a, 1996b). The emergent picture is that of a highly automatic bottom-up process: the brain produces a spatial code for the stimulus location that, in turn, activates the spatially corresponding response (see the reviews by Lu and Proctor 1995, and Simon 1990). It is controversial, however, how the spatial code is formed (Valle-Inclán, Hackley, and de Labra 2001) and the degree to which the activation of the compatible response should be considered an obligatory stimulus-driven process. The latter issue is explored in the following experiments.


Does stimulus-driven response activation underlie the Simon effect?

Central to many accounts of the Simon effect is the notion of a transient automatic activation of the spatially compatible response triggered by stimulus onset (Hommel 1993a; Kornblum, Hasbroucq, and Osman 1990; Zorzi and Umiltà 1995) or by stimulus identi1cation (Kornblum et al. 1999). The term automatic is intended to mean that this process is stimulus driven, unintentional, and unavoidable, although it might be modulated by other processes (Kornblum et al. 1990). This automatic route works in parallel with a second route which is governed by the S–R mapping de1ned in the experiment. The interaction of these two routes at the response selection stage causes the Simon effect. These models can account for much of the existing data, but there are several lines of evidence that cast doubt on their generality. The stimulus-driven hypothesis predicts facilitation on compatible trials and interference on incompatible trials with respect to neutral trials (i.e. nonlateralized stimuli). This pattern of results is usually obtained when compatible, neutral, and incompatible trials are mixed in the same block of stimuli (Hommel 1993b; Simon and Craft 1970; Umiltà, Rubichi, and Nicoletti 1999). However, when neutral trials are presented in a separate block there is no facilitation; RT on neutral trials is the same as on compatible trials (Simon and Acosta 1982) or even faster (Craft and Simon 1970; Simon and Small 1969). The appearance and disappearance of facilitation as a function of the blocking scheme is not what would be expected, a priori, of an automatic bottom-up process. The strongest doubts about stimulus-driven explanations are fed by reversals of the Simon effect (i.e. compatible trials slower than incompatible trials), something that should not occur if stimulus presentation automatically activates the spatially corresponding response. Hedge and Marsh (1975) were the 1rst to obtain a reverse Simon effect by instructional manipulation. They used colored keys and instructed subjects to press the key of the same color as the stimulus (identity mapping), or to press the key with the alternative color (alternate mapping). The Simon effect appeared with identity mapping and was reversed when the alternate mapping was used. Hedge and Marsh proposed that the Simon effect and its reversal were due to the unintentional application of mapping rules (identity and alternate) to the irrelevant stimulus dimension (location), a notion that is known as the ‘logical recoding’. More complex models can account for this reversal of the Simon effect and still maintain the automatic route linking stimulus and response codes. For example, De Jong et al. (1994) proposed a dual-process model that incorporates the automatic activation of the compatible response and the idea of logical recoding. Another type of Simon effect reversal is obtained when responses can be coded in different spatial positions. An example of this type of reversal was provided by Hommel (1993b; see Riggio, Gawryszewski, and Umiltà 1986, for related 1ndings). The stimuli were tones presented on left or right side. The response keys were also located on the left and right side and were connected to lights in a parallel fashion (i.e. left–left, right–right) or in a crossed way (left–right, right–left). Hommel instructed subjects to react to the tones either by pressing the keys or by turning on the lights. He found a normal Simon effect when subjects were instructed to press the keys. Under instructions to turn on the lights, however, the Simon effect reversed when keys and lights were connected in a crossed way. Hommel interpreted these data to indicate that the Simon effect re2ected the spatial correspondence of stimulus and goal location. A slightly different interpretation would be that subjects coded the light locations as response locations, as people usually do when playing computer games, for example. This type of coding will produce a reverse Simon effect in the cross connected condition. What these experiments indicate is that the spatial relationship between stimulus and response codes depends on factors other than a direct route.

475


476


The Simon effect also reverses or disappears when subjects practice with spatially incompatible S–R mappings during the previous days (Proctor and Lu 1999; Tagliabue, Zorzi, Umiltà, and Bassignani 2000). Proctor and colleagues have also shown in a recent series of experiments that mixing location-relevant trials in an otherwise typical Simon task enhances the Simon effect when location-relevant trials are spatially compatible, and reverses it when location-relevant trials are incompatible (Proctor and Vu, this volume, Chapter 22). These results, as noted by the authors, are problematic for the direct or automatic route accounts of the Simon effect. From the evidence just reviewed, it is clear that the Simon effect can be reversed by task instructions, presence of alternative spatial codes, practice with incompatible SR mappings, or mixing location-relevant trials with location-irrelevant trials. All these manipulations share in common their strategic nature, and suggest that top-down control can reverse the Simon effect. It is an open question whether the effect itself is also a product of strategy-driven processes. Strategic control has been demonstrated in several con2ict tasks by manipulating the proportion of compatible and incompatible trials. The spatial Stroop effect (Logan 1980, 1985; Logan and Zbrodoff 1979), the 2anker compatibility effect (Gratton, Coles, and Donchin 1992), and the Simon effect (Hommel 1994; Stürmer and Leuthold 1998; Toth et al. 1995) all decrease their size, or even reverse, when compatible trials become infrequent. The interpretation of these results, however, poses a problem, since when the proportion of compatible and incompatible trials is quite different (20/80, for example), the stimulus irrelevant dimension is only formally irrelevant. Consequently, it could be expected that subjects would pay attention to it, in addition to or instead of, the nominally relevant dimension. Whether or not this is the case, these results indicate that variations in global probability (i.e. subjects’ expectancies) do exert a strong in2uence on the Simon effect and other con2ict tasks. Another way to look for contextual in2uences in con2ict tasks has been to study 1rst-order sequential dependencies (i.e. the in2uence of trial N-1 on the performance of trial N). First-order compatibility dependencies have been found in the Stroop (Verleger 1991) and in the 2anker compatibility effects (Botvinick, et al. 1999; Gratton et al. 1992). Recently, the 1rst-order sequential effects in the Simon effect have begun to receive attention. It has been reported that the Simon effect reverses (Valle-Inclán, Hackley, and McClay 1998), vanishes (Stürmer and Leuthold 1998), or decreases (Praamstra, Kleine, and Schnitzler 1999) when the previous trial is incompatible. Accounting for these sequential effects would seem to require a revision of the automatic link between spatially compatible S–R pairs. Stürmer and Leuthold (1998) proposed that the automatic route was inhibited following an incompatible trial (see also Stoffels 1996). Obviously, this interpretation could explain the reduction of the Simon effect after an incompatible trial, but it is dif1cult to see how it could account for the reversal found by Valle-Inclán et al. (1998). Hommel (2000) explained these sequential effects as a result of repetitions and alternations of stimulus and response feature conjunctions (see below). Thus, there are data that challenge the purely stimulus-driven explanations of the Simon effect. If these results are not considered as aberrations, then strong modulations of the automatic route (leading even to abolition and reversal of the Simon effect) would have to be allowed in stimulus-driven models to account for the 1ndings. The consequence is that much of the variance in the Simon effect would be accounted for by the operation of those modulating mechanisms. The next two experiments present behavioral and physiological data on the in2uence of sequential dependencies on the Simon effect. First-order dependencies in choice reaction time tasks are ubiquitous and can take the form of repetition effects (faster RT when consecutive trials are identical in



some respect) or alternation effects (faster reaction when the preceding trial is different). Repetition effects are regarded as a manifestation of automatic facilitation, or priming, whereas alternations are considered to re2ect strategic behavior (an example of the gambler’s fallacy, Kirby 1980). These sequential effects are dependent on the response-to-stimulus interval (RSI), the S–R compatibility, and the categorizability of the stimulus set. In the studies to be described, the RSI was long (> 1 s), the S–R compatibility was low, and the stimuli were not categorizable. Under these conditions, stimulus repetition effects are expected (e.g. Bertelson, 1963). Response repetition effects, however, are not expected since they tend to appear when stimuli and responses are easily categorizable (e.g. letters assigned to one response and digits assigned to the other response, Campbell and Proctor 1993; Pashler and Baylis 1991). By contrast, response alternation effects have been shown to appear with long RSIs and noncategorizable stimuli (as in Experiment 2). Another sequential effect that is relevant to our research is the SRC repetition effect (faster RT for repetitions than for alternations of stimulus–response compatibility or incompatibility). Duncan (1977) proposed that this repetition effect indicated that SR mapping rules, not just the physical correspondence of stimulus and response, are selected during the response selection processes. Hommel (2000) has proposed that S and R features that co-occur in time become temporally associated (bound). As a result, facilitation would appear when the previous binding can be used, and interference would be evident when previous associations trigger the incorrect response. According to this proposal, the fastest RT should be found on complete repetitions and complete alternations (because previous associations are not triggered), and the slowest responses would appear in partial repetitions (previous associations would interfere with current processing). This last prediction is contradicted by the results in two-to-one mappings assignments in which stimuli are not categorizable. In these cases, the typical 1nding (e.g. Campbell and Proctor 1993) is that partial repetitions (i.e. response repetition and stimulus alternation) are faster than complete alternations. The generality of Hommel’s account is undermined, but it remains an adequate model for most data generated in Simon tasks. In such tasks, complete repetitions and complete alternations (the fastest RT according to the theory) are to be found in CC (compatible–compatible) and II (incompatible–incompatible) sequences, while partial repetitions (the slowest RT) occur in the IC and CI sequences.

23.2 Experiment 1 The behavioral data to be reported were collected in an experiment with electrophysiological measures, but these measures will not be considered here (see Valle-Inclán et al. 1998). The experiment aimed to explore the in2uence of sequential dependencies in a Simon task. Given the similarities of the Simon, Stroop, and 2anker compatibility effects, it was predicted that the Simon effect should decrease, or even reverse, if the previous trial was incompatible (see Gratton et al. 1992; Verleger 1991). The experiment comprised a cross-modal Simon task with visual imperative stimuli presented at 1xation and irrelevant noise bursts (i.e. accessory stimuli). Accessory stimuli and responses were aligned along the vertical meridian, as described below. Subjects. Sixteen students (19–25 years) volunteered for the experiment and received academic credits for their participation. Procedure. Visual targets (the letters S and T subtending 1°) were presented in the center of a VGA monitor simultaneously with a noise burst (65 dB, 100 ms) coming from a speaker located about 1 m above or below the monitor. The noise location and letter were selected randomly on

477


478


every trial. Subjects placed their hands on an inclined keyboard slanted perpendicularly towards the screen. Thus, the hand closer to the screen operated the upper keys and the other hand operated the lower keys. Subjects reacted to the letter by pressing a sequence of three keys with the index, ring, and middle 1ngers. The upper keys were ‘8’, ‘i’, and ‘k’ and the lower keys were ‘3’, ‘w’, and ‘a’. We used the vertical meridian, instead of the more common horizontal S–R arrangement, to avoid contamination of motor potentials by lateralized visual ERPs (see Valle-Inclán 1996a). The three-key sequence for responding was used because Hackley and Miller (1995) have shown that the lateralized readiness potential (LRP) amplitudes are larger with complex movements. RT was measured from visual stimulus onset to the 1rst keypress. The experiment consisted of six blocks of 112 trials each. The intertrial interval varied randomly between 500 ms and 4000 ms in 250 ms steps. The assignment of letter to hand and the placement of hands (above/below) were counterbalanced across subjects. Trials with an incorrect response or with RT greater than 2000 ms or less than 100 ms were excluded from the analysis. The accepted trials were classi1ed according to the compatibility of previous and current trial, generating four sequences: compatible–compatible (CC), compatible– incompatible (CI), incompatible–compatible (IC), and incompatible–incompatible (II). Mean RT for each of the 1rst and higher-order sequences was calculated including only the cases with correct responses on all trials. For the accuracy analysis, only cases in which the previous trials were accurate were included. The data were analyzed with 2 × 2 repeated measures MANOVAs with factors of compatibility on previous trial and compatibility on current trial.

23.2.1 Results The overall results showed a small (7 ms) but not signi1cant Simon effect. Since our interest concerned sequential dependencies in the Simon effect, we selected those subjects (n = 8) who did show

Fig. 23.1 Experiment 1. Mean RT for the four possible compatibility sequences within the whole group (left panel) and for the subset of subjects who showed a Simon effect (right panel). The 1gures in parenthesis are the percentage of errors on each condition.



a Simon effect and analyzed them separately. Figure 23.1 plots the mean RT as a function of previous and current trial compatibility for all subjects (left panel) and for those subjects who showed a Simon effect (right panel). In the analysis of the whole group, there was a 21 ms Simon effect when previous trial was compatible (CI–CC trials) and a reverse Simon effect (–14 ms) when previous trial was incompatible (II–IC trials). CC trials were 18 ms faster than IC trials, and II trials were 17 ms faster that CI trials. These results indicate a clear SRC repetition effect, and were con1rmed by a strong interaction between previous and current compatibility, F(1, 15) = 52.55, p < 0.0001. Further comparisons showed that all the differences among the four conditions were signi1cant. The right panel of Fig. 23.1 contains the mean RTs (and proportion of errors) for those subjects who did exhibit a Simon effect. A normal Simon effect of 32 ms appeared between CI and CC trials. An 11 ms reversal of the Simon effect was found between II and IC trials. There was also an SRC repetition effect with similar sizes for compatible trials (IC–CC trials, 19 ms) and for incompatible trials (CI–II trials, 14 ms). These results yielded a signi1cant main effect for current compatibility, F(1, 7) = 10.26, p < 0.01, and an interaction between previous and current compatibility, F(1, 7) = 57.89, p < 0.0001. The percentage of errors (see Fig. 23.1) was low for all conditions, and the results for this measure should be regarded cautiously. The Simon effect (i.e. less errors on compatible than on incompatible trials) was not signi1cant for the whole group, nor for the subgroup of subjects who had a Simon effect. As in the RT results, the previous-by-current compatibility interaction was signi1cant for the whole group, F(1,15)=11.28, p < 0.004, and also for those subjects with a Simon effect, F(1,7) = 9.05, p < 0.02. The SRC sequences were categorized according to stimulus (and response) repetition and alternation sequences. For example, half of CC and II sequences consisted of repetition of the visually presented letter, the location of the auditory accessory stimulus, and the key-press response (a complete repetition), and the other half consisted of complete alternations. Therefore, distinct measurements of the SRC repetition effect (CI + IC – II – CC) were obtained when stimulus and response were repeated and when they alternated. The results showed no signi1cant differences; thus, it seems that this SRC repetition effect is independent of other sequential effects.

23.2.2 Discussion The experiment showed that the sign of the Simon effect depends on previous trial compatibility. There was a normal Simon effect when the previous trial was compatible (the difference between CC and CI trials) and a reverse Simon effect when the previous trial was incompatible (the difference between IC and II trials). Performance was better when compatibility stayed the same (CC and II trials) than when compatibility changed from one trial to the next (IC and CI trials), suggesting the presence of an SRC repetition effect. SRC repetition effects have been described by Duncan (1977) in a four-choice RT task, but in that experiment the SRC sequences were confounded with the stimulus and response sequences. In our experiment, the SRC repetition effect had the same magnitude for stimulus repetitions and alternations, which suggests that these sequential effects were independent. First-order compatibility effects in the Simon effect have been reported by Stürmer and Leuthold (1998) and Praamstra et al. (1999). These authors’ interpretation was that the automatic S–R route was inhibited after an incompatible trial and was active after a compatible trial. This interpretation allows for decrements in size of the Simon effect, but further assumptions would be needed to

479


480


account for an actual reversal, as was found in the present study. One possibility would be to assume that the SRC repetition effect can override the automatic tendency to make a compatible response, thereby yielding a reversal of the Simon effect when previous trial was incompatible. This assumption, however, is inconsistent with the assumption of some authors (e.g. Kornblum et al. 1990) that the automatic tendency to react compatibly cannot be overridden. Note that while the SRC repetition effect is an empirical fact, the notion of automatic activation of the compatible response is an explanatory concept which implies the existence of long-term associative pathways between spatially compatible stimulus and response codes. Such long-term S-R connections should presumably be dif1cult to override; however, Proctor and Lu (1999) and Tagliabue and colleagues (2000) have shown that even a relatively short practice session with incompatible S–R mapping reverses the Simon effect. The implication is that it is not the current spatial S–R relationship that is the primary determinant of behavior, but rather, its interaction with previously learned spatial S–R associations. These effects could re2ect the in2uence of subjects’ expectancies about the proportion of compatible and incompatible trials. Consider now that in everyday life the vast majority of spatial S–R associations are compatible or neutral. Prying with a lever, steering a boat, and avoiding objects on the side of the road while driving are three examples of incompatible mapping in natural environments, but there are not many more. Therefore, it is tempting to think that subjects tend to expect compatible trials. A variant of this idea would be that the mapping subjects have most recently encountered before entering the lab was presumably compatible, and this temporary SR association would be responsible for the overall advantage of compatible trials (see Proctor and Lu 1999; Tagliabue et al. 2000). In functional terms, the effects of the biased expectancy toward compatible trials are very similar to an automatic activation of the compatible response. The critical difference is that the automatic route would be 1xed (even hardwired in some accounts) whereas expectancies are 2exible. In conclusion, we propose that the present results are best explained by a two-factor model comprised of SRC repetition effects and biased expectancies about the proportion of compatible trials. This two-factor account would predict the fastest RT when both factors activate the correct response (CC trials) and slowest when both factors activate the incorrect response (CI trials). Intermediate RT would be expected for IC and II trials, since the correct and incorrect response would both be activated, one by expectancy, the other by SRC repetition.

23.3 Experiment 2 The second experiment was intended to isolate the SRC sequences from S and R sequences (which were partially confounded in the previous experiment). To this end, we adapted a Simon task to a design originally developed by Bertelson (1965) for localizing 1rst-order sequential dependencies. Bertelson (1965) used a two-choice RT task in which two stimuli were assigned to each response. This so-called many-to-one mapping yields three types of transitions on consecutive trials. The two trials can be Identical (I, same stimulus and response), Equivalent (E, different stimuli but same response), or Different (D, different stimuli and responses). Soetens (1998, Experiment 2) has previously used this approach to study the Simon effect. He concluded that subjects build up expectancies across trials concerning the irrelevant stimulus dimension that correspond with the spatial arrangement for the responses. He found that the size of Simon effect was larger on Equivalent and Different than on Identical sequences. This suggests that the Simon effect is due, at least in part, to the in2uence of strategic behavior on the response side of the S–R pathway.



In Experiment 2, we also studied the higher-order SRC sequences. Typically, long RSIs such as those employed in this study can produce both higher-order repetition and higher-order alternation effects (see Soetens 1998). The higher-order alternation effects are considered to re2ect expectancies, while higher-order repetitions represent automatic facilitation and, possibly, expectancies. We analyzed fourth-order SRC sequences following the procedure of Soetens et al. 1985. In addition to behavioral measures, the electroencephalogram (EEG) was registered and eventrelated potentials (ERP) were extracted from these recordings. ERP research on the Simon effect has consistently shown that P300 latency increases on incompatible trials. This 1nding was 1rst described by Ragot, Renault, and Rémond (1980) and has been replicated many times since then. P300 amplitude is typically smaller on incompatible trials (e.g. Valle-Inclán 1996a). In recent years, the Simon effect has been investigated using the lateralized readiness potential (LRP). This is a physiological index of response preparation that was developed independently by Coles and Gratton (1986) and Smid, Mulder, and Mulder (1987). The LRP is a measure of the differential activation of the two hemispheres as recorded from electrodes placed over the motor cortex on each side. Preceding a hand or 1nger movement, a bilateral and initially symmetrical scalp negativity is recorded over the motor cortex (Kornhuber and Deecke 1965). Then, several hundreds of milliseconds before response execution, the voltages at sites contralateral to the side of the intended movement become more negative than those measured at ipsilateral sites (Kutas and Donchin 1974). These potentials are largest for recordings at sites near C3 and C4 of the 10/20 system. When these recordings are subtracted, the sign of the difference corresponds with the side of the intended movement. Speci1cally, the subtraction C3–C4 results in a positive de2ection for left-hand movements and a negative de2ection for right-hand movements. Subtracting trials with right-hand responses from those with left-hand responses yields the LRP. Computed in this manner ([(C3–C4 left response)–(C3–C4 right response)]/2), the correct response activation is manifested as a positive de2ection and the incorrect response activation as a negative de2ection (see Osman, Bashore, Coles, Donchin, and Meyer 1992). In functional terms, the LRP offers a ms-by-ms index of the preferential activation of one response over the other (Coles 1989; Miller and Hackley 1992). Using the LRP, activation of the incorrect response on incompatible trials has been demonstrated in both the 2anker compatibility (Gratton et al. 1988, 1990, 1992; Smid et al. 1987, 1990) and Simon tasks (de Jong et al. 1994; Valle-Inclán 1996a, 1996b). These effects are generally restricted to stimulus-locked LRP, but have recently been reported on response-locked LRP (Masaki, Takasawa, and Yamazaki, 2000; van der Lubbe and Woestenburg 1999). The LRP signature of incorrect response activation on incompatible trials has been interpreted as strong support for stimulus-driven models (De Jong et al. 1994). However, what these LRP results strictly indicate is that the incorrect response is activated on incompatible trials. They do not necessarily imply that the response activation is automatically triggered by stimulus presentation, per se. Incorrect response activation has also been found on slow reaction trials in a choice RT task that involved centrally presented stimuli and no response con2ict (Smulders, Kenemans, and Kok 1996). In accordance with the results and interpretation of the previous experiment, we predicted that LRP signs of incorrect response activation would be large on incompatible trials preceded by a compatible trial and small when preceded by an incompatible trial. In addition, the LRP onset latencies can be used to localize the SRC sequential effects. Effects at loci prior to response selection should produce latency differences on the time interval extending from stimulus onset to LRP onset and no effects on the interval from LRP onset to keypress. By contrast, late motoric effects would be re2ected only within the LRP-to-keypress interval (see Hackley and Valle-Inclán 1998).

481


482


23.3.1 Subjects Sixteen women (aged 19–29 years, all right-handed, with normal or corrected-to-normal vision) volunteered for the experiment and were given academic credits. All subjects were naive as to the purpose of the experiment.

23.3.2 Stimuli and EEG recording and processing The task was a many-to-one mapping with four stimuli and two responses. Subjects viewed the display at a distance of 60 cm, with their chins on a chin rest. Their index 1ngers rested on two keys placed perpendicularly to the screen. These keys were labeled during the instructions and at the beginning of each block as the ‘upper key’ (the key closer to the monitor) and the ‘lower key’ (the key closer to the subject). The task was divided into 20 blocks of 50 stimuli each with short breaks between two blocks. On each trial, a white character on dark background, randomly chosen among X, H, 3, and 6, was presented for 112 ms (above or below a 1xation point in the midline of a VGA monitor). One key was assigned to the stimuli X and 3, and the other to H and 6, counterbalanced across subjects. The interstimulus interval was 2225 ms. The EEG was recorded from an array of 29 tin scalp electrodes referred to left ear lobe. Eye movements and blinks were recorded from two pairs of electrodes above and below the eye and near the external canthi of each eye. Signals were ampli1ed with a 0.01–100 Hz bandpass 1lter and digitized at 250 Hz. Electrode impedance was below 5 kΩ. Eye movements and blinks artifacts were corrected with the procedure of Gratton, Coles, and Donchin (1983). Trials with incorrect responses or that were preceded by an incorrect reaction, trials with RT larger than 2000 ms or smaller than 100 ms, and trials with EEG values larger than 75 uV were excluded from the analysis. The EEG epochs were low-pass 1ltered to 8 Hz before averaging. The LRP was calculated as described above. LRP deviations from zero, indicating the preferential activation of one response over the other, were tested for each condition using a t-test on each digitized point. Differences in LRP onset were tested using the jackknife procedure developed by Miller, Patterson, and Ulrich (1998).

Table 23.1 MANOVA results for RT and accuracy results from Experiment 2. ‘Transition’ refers to identical, equivalent, and different sequences. ‘SRC repetition’ refers to repetition/alternation of compatibility from trial N-1 to trial N. ‘Compatibility’ refers to compatibility on trial N Source

Transition SRC repetition Compatibility Transition × SRC repetition Transition × Compatibility SRC repetition × Compatibility Transition × SRC rep × Comp.

RT

Accuracy

Df

F value

p

Common Mechanisms in Perception and Action (Attention and Performance (Oxford)) (v. 19)

Handbook of Perception and Action, Volume 3: Attention (Handbook of Perception & Action)

Handbook of Perception and Action, Volume 3: Attention (Handbook of Perception & Action)

Object Recognition, Attention, and Action

Space and Time in Perception and Action

Space and Time in Perception and Action

Space and Time in Perception and Action

Visual Space Perception and Action

Visual Perception and Action in Sport

Action in Perception (Representation and Mind)

Perception and Action in Medieval Europe

Studies in perception and action V: Tenth International Conference on Perception and Action : Aug. 8-13, 1999, Edinburgh, Scotland

Handbook of Perception and Action, Volume 1: Perception (HANDBOOK OF PERCEPTION AND ACTION)

Logic, Thought, and Action 19

Architecture and Tourism: Perception, Performance and Place

MiG-19 Farmer in Action

MiG-19 Farmer in action

Advances in Geophysics: v. 19

Decision Making, Affect, and Learning: Attention and Performance XXIII

Suspensions of Perception: Attention, Spectacle, and Modern Culture

Effortless Attention: A New Perspective in the Cognitive Science of Attention and Action (Bradford Books)

Studies in perception and action VI: Eleventh International Conference on Perception and Action : June 24-29, 2001, Storrs, CT, USA

Handbook of Perception and Action, Volume 2: Motor Skills (Handbook of Perception & Action)

Handbook of Perception and Action: Motor Skills (Handbook of Perception & Action)

Handbook of Perception and Action: Motor Skills (Handbook of Perception & Action)

Multicast in Third-Generation Mobile Networks: Services, Mechanisms and Performance

Changing Forests: Collective Action, Common Property, and Coffee in Honduras

Protecting the Self: Defense Mechanisms in Action

Attention and Implicit Learning

Cognition in the Globe: Attention and Memory in Shakespeare's Theatre (Cognitive Studies in Literature and Performance)

Common Mechanisms in Perception and Action (Attention and Performance (Oxford)) (v. 19)

Handbook of Perception and Action, Volume 3: Attention (Handbook of Perception & Action)

Handbook of Perception and Action, Volume 3: Attention (Handbook of Perception & Action)

Object Recognition, Attention, and Action

Space and Time in Perception and Action

Space and Time in Perception and Action

Space and Time in Perception and Action

Visual Space Perception and Action

Visual Perception and Action in Sport

Action in Perception (Representation and Mind)

Perception and Action in Medieval Europe

Studies in perception and action V: Tenth International Conference on Perception and Action : Aug. 8-13, 1999, Edinburgh, Scotland

Handbook of Perception and Action, Volume 1: Perception (HANDBOOK OF PERCEPTION AND ACTION)

Logic, Thought, and Action 19

Architecture and Tourism: Perception, Performance and Place

MiG-19 Farmer in Action

MiG-19 Farmer in action

Advances in Geophysics: v. 19

Decision Making, Affect, and Learning: Attention and Performance XXIII

Suspensions of Perception: Attention, Spectacle, and Modern Culture

Effortless Attention: A New Perspective in the Cognitive Science of Attention and Action (Bradford Books)

Studies in perception and action VI: Eleventh International Conference on Perception and Action : June 24-29, 2001, Storrs, CT, USA

Handbook of Perception and Action, Volume 2: Motor Skills (Handbook of Perception & Action)

Handbook of Perception and Action: Motor Skills (Handbook of Perception & Action)

Handbook of Perception and Action: Motor Skills (Handbook of Perception & Action)

Multicast in Third-Generation Mobile Networks: Services, Mechanisms and Performance

Changing Forests: Collective Action, Common Property, and Coffee in Honduras

Protecting the Self: Defense Mechanisms in Action

Attention and Implicit Learning

Cognition in the Globe: Attention and Memory in Shakespeare's Theatre (Cognitive Studies in Literature and Performance)

Recommend Documents