Seeing Spatial Form

Seeing Spatial Form This page intentionally left blank Seeing Spatial Form Edited by Michael R. M. Jenkin, Ph. D La...

Author: Michael R. M. Jenkin | Laurence R. Harris

132 downloads 3079 Views 26MB Size Report

This content was uploaded by our users and we assume good faith they have the permission to share this book. If you own the copyright to this book and it is wrongfully on our website, we offer a simple DMCA procedure to remove your content from our site. Start by pressing the button below!
Report copyright / DMCA form

DOWNLOAD PDF

Seeing Spatial Form

This page intentionally left blank

Seeing Spatial Form Edited by Michael R. M. Jenkin, Ph. D Laurence R. Harris, Ph. D

OXFORD UNIVERSITY PRESS

2006

OXFORD UNIVERSITY PRESS Oxford University Press, Inc., publishes works that further Oxford University's objective of excellence in research, scholarship, and education. Oxford New York Auckland Cape Town Dar es Salaam Hong Kong Karachi Kuala Lumpur Madrid Melbourne Mexico City Nairobi New Delhi Shanghai Taipei Toronto With offices in Argentina Austria Brazil Chile Czech Republic France Greece Guatemala Hungary Italy Japan Poland Portugal Singapore South Korea Switzerland Thailand Turkey Ukraine Vietnam

Copyright © 2006 by Michael R. M. Jenkin and Laurence R. Harris Published by Oxford University Press, Inc. 198 Madison Avenue, New York, New York 10016 www.oup.com Oxford is a registered trademark of Oxford University Press All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording, or otherwise, without the prior permission of Oxford University Press. Library of Congress Cataloging-in-Publication Data Seeing spatial form / edited by Michael R. M. Jenkin and Laurence R. Harris p. cm. Includes bibliographical references and indexes. ISBN-13 978-0-19-517288-1 ISBN 0-19-517288-4 1. Form perception. 2. Space perception. I. Jenkin, Michael, 1959II. Harris, Laurence, 1953QP492.S44 2005 152.14—dc22 2004056814

9 8 7 6 5 4 3 2 1 Printed in the United States of America on acid-free paper

Preface This book is in appreciation of the contributions of David Martin Regan. He continues to be an inspiration to many. We would like to thank Teresa Manini, who ran the conference; Jim Zacher for his technical assistance; and our wives for their enduring support. The CD-ROM that accompanies this book contains color imagery and video clips associated with various chapters and the York Vision Conference itself. The CDROM is presented in HTML format, and is viewable with any standard browser (e.g., Netscape Navigator or Microsoft Internet Explorer). To view the videos on the CDROM you will need Quicktime, which is available free from Apple. To view the CDROM, point your browser at the file index.htm on the CD-ROM. York University, Ontario, Canada Winter 2004

Michael Jenkin Laurence Harris

v


Contents Contributors

xv

1 Seeing Spatial Form Laurence R. Harris and Michael Jenkin 1.1 1.2

Processing by the Brain The Structure of This Book

3 5

1 Form Vision

9

2 Pictorial Relief Jan J. Koenderink, Andrea J. van Doom, andAstridM. L. Kappers 2.1 2.2 2.3 2.4

2.5

2.6

1

Introduction Some History Psychophysics: Methods Findings 2.4.1 Veridicality 2.4.2 Influence of Viewing Mode 2.4.3 Influence of Pictorial Cues 2.4.4 Global versus Local Representation 2.4.5 Influence of Method Geometry of Pictorial Space 2.5.1 Simple Introduction to the Geometry: The 2D Case . . . . 2.5.2 The 3D Case 2.5.3 The Panoramic Visual World What Next?

3 Geometry and Spatial Vision Gerald Westheimer

11 11 12 13 17 17 18 20 21 21 23 24 26 27 29 35

VII

viii

Contents

4 The Inputs to Global Form Detection David R. Badcock and Colin W. G. Clifford 4.1 4.2 4.3 4.4

Introduction Seeing Glass Patterns A Model of the Functional Architecture of Global Form Detection . . Conclusions

5 Probability Multiplication as a New Principle in Psychophysics Michael Morgan, Charles Chubb, and Joshua Solomon

6

II

43 43 44 52 53 57

5.A1 Methods 5.A2 Models and Theory 5.A2.1 The Late-Noise Reichardt Model 5.A2.2 The Opponent (Contrast Discrimination) Model 5.A2.3 The Probability-Multiplication Model 5.A2.4 The Convoy Model (not considered here and a poor fit to all the data)

63 64 64 65 65

Spatial Form as Inherently Three Dimensional Christopher W. Tyler

67

6.1 6.2 6.3 6.4 6.5

72 75 80 83 85

Surface Representation through the Attentional Shroud Interpolation of Object Shape within the Generic Depth Map . . . . Transparency Object-Oriented Constraints on Surface Reconstruction Conclusion

Motion and Color

7 White's Effect in Lightness, Color, and Motion Stuart Anstis 7.1 7.2 7.3 7.4 7.5 7.6

Introduction Experiment 1. White's Effect Increases with Spatial Frequency ... Experiment 2. A Colored White's Effect Shows Both Contrast and Assimilation Experiments. Colored White's Effect: Spatial Frequency Experiment 4. An Isotropic Brightness Illusion: "Stuart's Rings" . . Experiment 5. White's Effect and Apparent Motion

8 The Processing of Motion-Defined Form Deborah Giaschi 8.1 8.2

The Motion-Defined Letter Test Dissociations Between Motion-Defined Form and Simple Motion Processing

66

89 91 91 92 93 93 95 96 101 102 103

Contents

8.3

8.4

ix 8.2.1 Multiple Sclerosis 103 8.2.2 Cortical Lesions 104 8.2.3 Normal Development 105 8.2.4 Abnormal Development 106 Role of the M/Dorsal Pathways in Motion-Defined Form Processing . 110 8.3.1 Parkinson's Disease 110 8.3.2 Reduced Visual Acuity 113 8.3.3 Functional Neuroimaging 113 Conclusions 115

9 Vision in Flying, Driving, and Sport Rob Gray 9.1 9.2 9.3 9.4 9.5

Introduction 121 9.1.1 Some Basic Limitations of the Visual Processing System . 122 Vision in Flying 122 9.2.1 Visual-Motor Control in Approach and Landing 123 9.2.2 Visual-Motor Control in Low-Level Flight 131 Vision in Driving 136 Vision in Sports 140 Conclusions 146

10 Form-from-Watercolor in Surface Perception, and Old Maps Lothar Spillmann, Baingio Pinna, and John S. Werner 10.1 Introduction 10.2 General Methods 10.3 Experiment 1: How to Create Two Geographical Maps by Using One Boundary 10.4 Experiment 2: Watercolor Effect vs. Proximity and Parallelism . . . 10.5 Experiment 3: Watercolor Effect vs. Good Continuation and Prdgnanz 10.6 Experiment 4: Watercolor Effect Used to Disambiguate Grouping and Figure-Ground Organization 10.7 Experiment 5: Why Did the Old Maps Fail to Elicit Strong LongRange Coloration Effects? 10.8 Conclusion

III

121

Eye Movements

153 153 156 156 157 159 161 162 164

167

11 The Basis of a Saccadic Decision: What We Can Learn from Visual Search and Visual Attention 169 Eileen Kowler 11.1 Prologue 11.2 Saccadic Decisions 11.3 Search and Optimal Search

169 170 172

x

Contents 11.4 Saccades during Natural Visual Tasks 11.5 Saccades and Visual Search: An Investigation of the Costs of Planning a Rational Saccade 11.6 The Role of Attention in the Programming of Saccades 11.7 Saccadic Decisions, Search, and Attention 11.8 Final Comments

173 174 180 183 184

12 Handling Real Forms in Real Life R. M. Steinman, W. Menezes, and A. N. Herst

187

IV Neural Basis of Form Vision

213

13 The Processing of Spatial Form by the Human Brain Studied by Recording the Brain's Electrical and Magnetic Responses to Visual Stimuli 215 David Regan and Marian P. Regan 13.1 Introduction 13.2 Human Brain Electrophysiology: The Early Days 13.3 My Introduction to the Mathematical Analysis of Nonlinear Behavior and to the Joys of Collaborative Research 13.4 Brain Responses to Depth-Defined Form and to "Short-Range" Apparent Motion 13.5 Dissociation of the Brain's Magnetic Responses to Texture-Defined Form and to Texton Change 13.6 Three Subsystems in the Steady-State Evoked Potential to Flicker and a Magno Stream/Parvo Stream Correlate in Human 13.7 The Frequency Tagging Technique: Some Early Applications . . . . 13.8 The Sweep Method: A Fast Hybrid Technique for Gathering Data within a Short Period and for Measuring Adaptation and Other Nonstationary Processes 13.9 Response Spectrum Recorded at Ultra High Resolution: Nondestructive Zoom-FFT 13.10 Measurement of the Orientation, Spatial Frequency, and Temporal Tuning of Spatial Filters by Exploiting the Nonlinearity of Neurons Sensitive to Spatial Form 13.11 A Visual-Auditory Convergence Area in the Human Brain 13.12 A Frequency Domain Technique for Testing Nonlinear Models of the Human Visual System 13.13 Appendix 13.A1 Linear Systems and the Wide and Wild World of Nonlinear Systems . 13.A2 Some Definite Time Elapses between Stimulation of the Eye or Ear and the Occurrence of any Given Evoked Response 13.A3 A Method for Deriving the Response of Asymmetric Nonlinearities to a Sum of Two Sinewaves 13.A3.1 Half-Wave Linear Rectifier: Response to a Single Sinusoid

215 215 223 233 234 236 238 239 243 247 250 250 256 256 258 259 259

xi

Contents 13.A3.2 Half-Wave Linear Rectifier: Response to the Sum of Two Sinusoids 13.A3.3 Half-Wave Square Law Rectifier: Response to the Sum of Two Sinusoids 13.A3.4 Half-Wave Square Root Rectifier: Response to the Sum of Two Sinusoids 13.A3.5 Two Cascaded Half-Wave Rectifiers, AC Coupled 13.A3.6 Cascaded Compressive Rectifiers 13.A3.7 Two Parallel Rectifiers whose Summed Outputs Pass through a Third Linear Rectifier: The Dichoptic Case 13.A3.8 Half-Wave Rectifier Combining Accelerating and Compressive Segments

260 266 267 268 269 270 272

14 Linking Psychophysics and Physiology of Center-Surround Interactions in Visual Motion Processing 279 Duje Tadin and Joseph S. Lappin 14.1 Introduction: Moving Image Information 14.1.1 Linking Macroscopic and Microscopic Analyses of Visual Information Row 14.1.2 Inferring Perception from Physiology 14.2 Center-Surround Interactions in Motion Processing 14.2.1 Center-Surround Mechanisms Found in MT and Elsewhere 14.2.2 Perceptual Correlates of Center-Surround Antagonism . . 14.2.3 Interim Conclusions 14.3 Segregating Surfaces 14.3.1 Psychophysics of Motion-Based Figure-Ground Segregation 14.3.2 Neurophysiology of Motion-Based Figure-Ground Segregation 14.3.3 Interim Conclusion 14.4 Perceiving 3D Surface Shape 14.4.1 Psychophysics of 3D Shape-from-Motion 14.4.2 Contribution of MT to Shape-from-Motion 14.4.3 Interim Conclusions 14.5 General Conclusions

279 280 284 285 285 287 291 291 292 295 299 299 299 300 304 304

15 Transparent Motion: A Powerful Tool to Study Segmentation, Integration, Adaptation, and Attentional Selection 315 Thomas Papathomas, Zoltan Vidnydnszky, and Erik Blaser 15.1 15.2 15.3 15.4 15.5 15.6 15.7

Introduction 315 Stages of Motion Processing 316 Transparent and Non-Transparent Bi-Vectorial Motion 318 Neural Mechanisms of Motion Segmentation and Integration . . . . 320 Integration of Motion Directions during the Motion Aftereffect (MAE) 322 MAE with Transparent Motion - Integration during MAE 323 Nature of Attentional Selection in Motion Processing 324

xii

Contents 15.8 Conclusions

326

16 Neurological Correlates of Damage to the Magnocellular and Parvocellular Visual Pathways: Motion, Form, and Form from Motion after Cerebral Lesions 335 James A. Sharpe, Ji Soo Kim, and Jos ee Rivest 16.1 16.2 16.3 16.4

Introduction Methods Results Discussion

336 337 339 341

17 The Effect of Diverse Dopamine Receptors on Spatial Processing in the Central Retina: A Model 347 Ivan Bodis-Wollner and Areti Tzelepi 17.1 17.2 17.3 17.4

Retinal Circuitry Receptive Fields of Ganglion Cells Retinal Processing and Dopamine's Action Dopaminergic Effects on the PERG in the Monkey 17.4.1 Retinal Spatial Tuning in the MPTP Primate Model . . . . 17.4.2 The Effects of Intravitreal 6-OHDA on Spatial Tuning . . 17.4.3 The Effect of the D2 Receptor Blocker L-Sulpiride on Spatial Frequency Tuning 17.4.4 The Effect of CY 208-243, a Dl Agonist, on Spatial Tuning 17.4.5 Synthesis of Experimental Results 17.5 TheModel 17.5.1 The Normal Retina 17.5.2 The Dopamine-Deficient Retina 17.6 Dopamine's Role in Retinal Mechanisms

347 350 351 352 352 352 354 354 355 356 356 357 359

V Development

369

18 Improving Abnormal Spatial Vision in Adults with Amblyopia Uri Polat

371

18.1 Background 18.2 Amblyopia 18.2.1 Abnormal Spatial Vision in Amblyopia 18.2.2 Contrast Sensitivity and Amblyopia 18.2.3 Models of Amblyopia 18.2.4 Abnormal Spatial Interactions and Crowding 18.3 Perceptual Learning and Neural Plasticity 18.3.1 Plasticity in Amblyopia 18.4 Treatment of Adult Amblyopia

371 372 372 372 374 374 375 375 376

xiii

Contents 18.4.1 18.5

Perceptual-Learning-Based Technique for Treating Amblyopia

Summary

19 Visual Development with One Eye Martin J. Steinbach and Esther G. Gonzalez

376 380 385

19.1 Introduction 385 19.2 Form (Contrast, Texture and Motion Denned), Motion (including OKN), and Depth 386 19.2.1 Form 387 19.2.2 Motion 390 19.2.3 Monocular Practice 391 19.2.4 Optokinetic Nystagmus (OKN) 391 19.2.5 Time to Collision 391 19.3 Depth 391 19.4 Egocenter: Role of Binocular Experience 392 19.4.1 The Visual Direction of Objects 392 19.4.2 Hints from a Birthday Party 392 19.4.3 The "Cyclops Effect" 394 19.4.4 The Egocenter is "Built In" 395 19.4.5 The Egocenter is not so "Plastic" 395 19.4.6 Learning to Perform Monocular Tasks 396 19.4.7 Performance on a Monocular Task sans Feedback 397 19.5 Conclusions 398 A Appendix: Selected Publications of David Regan

405

Author Index

421

Subject Index

435


Contributors Stuart Anstis, Department of Psychology, University of California at San Diego, 9500 Oilman Dr., La Jolla, CA 92093-0109 USA. email: [email protected] David R.Badcock, Human Vision Laboratory, School of Psychology, The University of Western Australia, 35 Stirling Highway, Nedlands, Western Australia 6907. email: [email protected] Erik Blaser, Department of Psychology, University of Massachusetts, 100 Morrissey Blvd., Boston, MA 02125 USA. email: [email protected] Ivan Bodis-Wollner, Department of Neurology, State University of New York, Health Science Center at Brooklyn, 450 Clarkson Ave., Box 1213, Brooklyn, NY 11203 USA. email: [email protected] Charles Chubb, Department of Cognitive Sciences, 3151 Social Sciences Plaza, University of California, Irvine, CA 92697-5100 USA. email: [email protected] Colin W. G. Clifford, Visual Perception Unit, School of Psychology, The University of Sydney, Sydney, NSW Australia 2006. email: [email protected]

xv

xvi Deborah Giaschi, University of British Columbia, Department of Ophthalmology, Room A146 - 4480 Oak Street, Vancouver, BC V6H 3V4 Canada. email: [email protected] Esther G. Gonzalez, Toronto Western Research Institute, Toronto Western Hospital, 399 Bathurst Street Toronto, Ontario M5T 2S8 Canada. email: [email protected] Robert Gray, Department of Applied Psychology, Arizona State University East, 7001 E Williams Field Rd., Mesa, AZ 85212 USA. email: [email protected] Laurence R. Harris, Centre for Vision Research and Department of Psychology, York University, 4700 Keele Street, Toronto, Ontario M3J 1P3 Canada. email: [email protected] A. N. Herst, Department of Psychology, University of Maryland, College Park, MD 20742-4411 USA. email: [email protected] Michael Jenkin, Centre for Vision Research and Department of Computer Science, York University, 4700 Keele Street, Toronto, Ontario M3J 1P3 Canada. email: [email protected] Astrid M. L. Kappers, Industrial Design Delft, Delft University of Technology, Landbergstraat 15,2628CE Delft, The Netherlands. email: [email protected]

Contributors

Contributors Ji Soo Kim, Department of Neurology, College of Medicine, Seoul National University, 300 Gumi-dong, Bundang-ku, Seongnam-si, Gyeonggi-do, 463-707 Korea. email: [email protected]

Jan J. Koenderink, Helmholtz Instituut, Universiteit Utrecht, Buys Ballot Laboratorium, Princetonplein 5, 3584CC Utrecht, The Netherlands. email: [email protected] Eileen Kowler, Department of Psychology, 152 Frelinghuysen Road, Rutgers University, Piscataway, NJ 08854 USA. email: [email protected] Joseph S. Lappin, Vanderbilt Vision Research Center and Department of Psychology, Vanderbilt University, 111 21st Ave. South, Nashville, TN 37203 USA. email: [email protected] Wayne Menezes, Department of Electrical and Computer Engineering, University of Maryland, College Park, MD 20742-4411 USA. email: [email protected] Michael Morgan, Department of Optometry, City University, Northampton Square, London EC1V OHB United Kingdom. email: [email protected] Thomas Papathomas, Department of Biomedical Engineering and Laboratory of Vision, Research, Rutgers University, 152 Frelinghuysen Road, Piscataway, NJ 08854-8020 USA. email: [email protected] Baingio Pinna, Facolta di Lingue e Letterature Straniere, Universita di Sassari, Via Roma 151,07100 Sassari-Italy. email: [email protected]

xvii

xviii

Uri Polat, Goldschleger Eye Research Institute, Tel Aviv University, Sheba Medical Center, Ramat Can, Tel Hashomer, Israel 52621. email: [email protected] David M. Regan, Centre for Vision Research and Department of Psychology, York University, 4700 Keele St., Toronto, Ontario M3J 1P3 Canada. email: [email protected] Marian P. Regan, Centre for Vision Research and Department of Psychology, York University, 4700 Keele St., Toronto, Ontario M3J 1P3 Canada. email: [email protected] Josee Rivest, Department of Psychology, Glendon College York University, 4700 Keele St., Toronto, Ontario M3J 1P3 Canada. email: [email protected] James A. Sharpe, Division of Neurology, University Health Network, University of Toronto, Toronto, Ontario M5T 2S8 Canada. email: [email protected] Joshua Solomon, Department of Optometry, City University, Northampton Square, London EC IV OHB United Kingdom. email: [email protected] Lothar Spillmann, Brain Research Unit, University of Freiburg, Hansastrasse 9, D-79104 Freiburg, Germany. email: [email protected]

Contributors

Contributors Martin J. Steinbach, Toronto Western Research Institute, Toronto Western Hospital, 399 Bathurst St., Toronto, Ontario M5T 2S8 Canada. email: [email protected] R. M. Steinman, Department of Psychology, University of Maryland, College Park, MD 20742-4411 USA. email: [email protected] Duje Tadin, Vanderbilt Vision Research Center and Department of Psychology, Vanderbilt University, 111 21st Ave. South, Nashville TN 37203 USA. email: [email protected] Christopher W. Tyler Smith-Kettlewell Eye Research Institute, 2318 Fillmore Street, San Francisco, CA 94115 USA. email: [email protected] Areti Tzelepi, LPPA, CNRS - College de France, 11, place Marcelin Berthelot, Paris,75005 France. email: [email protected] Andrea J. van Doom, Industrial Design Delft, Delft University of Technology, Landbergstraat 15, 2628CE Delft, The Netherlands. email: [email protected] Zoltan Vidnydnszky, Laboratory for Neurobiology, Semmelweis University Medical School, Tuzolto u. 58,1094 Budapest, Hungary. email: [email protected]

xix

xx John S. Werner, Department of Ophthalmology and Section of Neurobiology, Physiology and Behavior, University of California, 4860 Y St., Suite 2400, Sacramento, California 95817 USA. email: [email protected] Gerald Westheimer, Division of Neurobiology, University of California, 565 Life Sciences Addition, Berkeley, CA 94720-3200 USA email: [email protected]

Contributors

1. Seeing Spatial Form Laurence R. Harris and Michael Jenkin The term "spatial form" refers to the existence in the outside world of surfaces that have a spatial extent. The problem of seeing such surfaces is the problem of deducing or constructing their existence and shape from the spatially distributed pattern and spectral content of the light reflected from them into the eyes and transduced by the retina. The level of concern of "seeing spatial form" therefore starts somewhere higher than the retina, at a point where information is integrated across and between the retinae. The upper level of its jurisdiction depends on the interpretation of the word "seeing." In this book we regard spatial form as not being perceived on its own. By the level of perception, the visual information has passed through too many constancy mechanisms and cognitive processes to allow access to the raw "spatial form." For example, it is very hard to see a coin as anything but round and a person as anything but personsized. It is hard to see the shapes of the contours of someone's cheek in terms of their geometric curvature. Spatial form has provided some of the building blocks for conscious perceptions but is no more available to unbiased conscious inspection than are absolute luminance levels or retinal distances. We therefore draw the upper bound on seeing spatial form somewhere lower than object recognition and consciousness. Looking at 'lower-level vision', the input to conscious processing, presents a methodological challenge. The tendency of the visual system to apply higher-level processes needs to be controlled so that we can look at the processing of spatial form per se. This involves designing psychophysical probes which can only be solved by extracting the particular aspect of spatial form that we wish to address. David Regan (Fig. 1.1) is a master of the design of psychophysical tests to reveal the processing of spatial form. Through his enormous output of published papers, which are listed in an appendix to this book, he has made great strides in clarifying the denning features of spatial form that the visual system is able to use. He has presented and tested many pioneering algorithms. This aspect of his work up until 2000 is summarized in his book Human Perception of Objects: Early Visual Processing of Spatial Form Defined by Luminance, Colour, Texture, Motion and Binocular Disparity (Sinauer Press, 2000; Fig. 1.2). Defining spatial form requires perceptually breaking off a surface or form from its 1

2

Seeing Spatial Form

Figure 1.1: A portrait of David Martin Regan by renowned psychological portrait composer Nick Wade titled "Sportsman, Seer, Scientist." The image combines three of Martin Regan's loves - cricket, vision, and scientific communication. He is shown as a young batsman striding towards (or away from!) the crease, and as an older man surveying the Atlantic from Halifax, Nova Scotia. Unlike many who enjoy sport and science, Martin combines them in an elegant manner. He has displayed the particular demands requked of a batsman both on the pitch and in the laboratory. In addition to his elegant strokes of the bat, those of the pen have won him many plaudits — there is art in science as well as sport. I am grateful to Marian Regan for providing the pictures of Martin which were used in producing this "perceptual portrait." The text is taken from D. Regan (1992). Visual judgements and misjudgements in cricket, and the art of flight. Percept.,21: 91-115. background. This is sometimes referred to as 'parsing' the visual scene. The term "parsing" describes the way that a sentence is grammatically analyzed into its component pieces. The connection with language is intentional and concepts such as "in front of" or "on top of" are also parts of the visual parsing process. How does the visual system parse the visual scene into forms? The system needs to detect discontinuities that mark the transition between one form and another. Candidates for such discontinuities that can be detected visually include luminance, texture,

Laurence R. Harris and Michael Jenkin

3

Figure 1.2: The cover of David Regan's book Human Perception of Objects: Early Visual Processing of Spatial Form Defined by Luminance, Colour, Texture, Motion and Binocular Disparity, (Sinauer Press, 2000). colour, distance, "stuff" (a variant of texture implying a type of material such as skin or metal), and relative motion. In addition, there are nonvisual variables such as temperature or tactile "feel," but those are beyond the scope of this book. The converse of detecting a form is to camouflage or hide it. The process of hiding something is the process of obscuring the discontinuities. Hiding might be achieved, for example, by lying flat and still against a background that appears similarly colored to the hunter. A wonderful example of camouflage in action is shown in Fig. 13.2. (The original title of the book was Seeing through Camouflage.) A major drive in the evolution of the visual system is to break camouflage in order to detect prey, other food items, or predators.

1.1 Processing by the Brain The retina and thalamic precortical cells show limited signs of coding spatial form in higher mammals. The pioneering work by Hubel and Wiesel (1959, 1962, 1968) suggested that the activity of cells in the primary visual cortex could be interpreted as extracting and identifying features. Pulling out and segragating features from the visual input implies the existence of a later stage that puts them together. This binding process is probably above the level of form extraction. Spatial form is probably one of the features that needs to be bound to the other features. Spatial form might even provide the frame onto which other features such as color are added (see chapter 6). Constructing a representation of the world by bundling separated features allows multiple solutions. The most appropriate choice of features and the emphasis placed on one rather than another feature can depend on the use to which the end product is to be

4

Seeing Spatial Form

Figure 1.3: David Martin Regan as visionary in action. put. Spatial location relative to the observer, for example, is not a centrally important part of recognizing a face but is essential when reaching out to touch, hit, or throw something (see Fig. 1.3). Two broad streams of visual processing have emerged that seem to divide this task-dependent reconstructive process between them, corresponding to a broad anatomical division of the visual system into dorsal and ventral streams (Mishkin and Ungerleider, 1982; Mishkin et al., 1983). The dorsal and ventral streams are separate right from their source where they originate in different retinal cells with different response characteristics. They are then processed by separate layers in the lateral geniculate nucleus of the thalamus, and even have distinct routes through the primary visual cortex before forming the dorsal and ventral pathways. The description of different neurological syndromes that can result from damage to one or other of these pathways has supported a "two visual systems" model (Milner and Goodale, 1993, 1995). In this influential model, one visual system, realized in the dorsal stream and the parietal cortex, processes visual information that subserves the needs of action. For this system, spatial location and movement are more important than features that might help in recognition, for example. The other visual system, the one that uses the ventral pathway, is less concerned with spatial location or movement but more with the details of pattern. It is important, however, not to overinterpret this distinction. Regan has strongly cautioned against making this mistake, pointing to evidence that connections between the dorsal stream and ventral stream (and, perhaps, subcortical nuclei) are necessary for recognizing motion-defined spatial form (Regan et al., 1992). Nevertheless, although there is no doubt that each pathway can communicate with and share information with the other, in terms of a functional division, having


5

Figure 1.4: David Martin Regan going out on a limb, as usual, this time in Wales over a 500 ft drop. two visual systems subserving the requirements of perception and action has proved a remarkably robust concept (Harris and Jenkin, 1998). Which of these two visual systems is important for seeing spatial form? Spatial form, as a lower-level building block, is actually essential to both of these uses of vision. To define the existence of any object requires knowledge about the spatial form of its surfaces.

1.2 The Structure of This Book The book is divided into five sections. I Form Vision In part I, some of the general principles involved in defining spatial form are considered. Koenderink et al. (chapter 2) look at some of the pictorial cues that are involved in defining spatial form. Pictorial cues do not include binocular or eye position cues which cannot be directly included in a conventional picture but includes cues such as shape-from-shading, luminance discontinuities and perspective cues that contribute to seeing spatial form. Defing the geometric cues is pursued further in the chapter by Westheimer (chapter 3) who extends them into three dimensions and puts them into historical perspective. Badcock and Clifford (chapter 4) introduce the processing of spatial form as a heirarchical process working from orientation selectivity to selectivity for more global patterns. Morgan et al. (chapter 5) examine the role of coincidence detectors. Tyler (chapter 6) addresses the issue of how the various scattered features can be bound back together and suggests a central role for spatial form as a representing a framework onto which other features can be bound.

6

Seeing Spatial Form

II Motion and Color The chapters in part II address some of the specific cues that can be used to define spatial form, especially color and motion. Anstis (chapter 7) compares the contributions from color, light, and motion. Giaschi (chapter 8) looks specifically at motion-defined form and how arbitrarily it relates to the magnocellular-parvocellularprocessing stream divide. Gray (chapter 9) explores the role of these low-level visual processes during human performance in flying, driving, and sport. Spillmann et al. (chapter 10) explore the use of edges in maps. Cartoon drawings, consisting exclusively of edges, can be identified with their real-world counterparts suggesting some common features in the neural representation of cartoons and real-world scenes. III Eye Movements In order to put together the spatial structure and layout of a scene, the various views obtained from sequential fixations need to be put together. This involves a knowledge of the eyes' positions in space, bearing in mind that the head, the vehicle of the eyes, is also mobile. The planning of these scanning saccadic eye movements, used to explore the visual world, is considered by Kowler in chapter 11. Steinman et al. (chapter 12) examine how eye movements may contribute to the perception of spatial form in natural (as opposed to experimental) environmental conditions IV Neural Basis of Form Vision In part IV, the neurophysiology of form vision is considered. David Regan and his wife Marian (chapter 13) provide an extensive (and historically grounded) review of the approach to investigating the processing of spacial form by recording the electrical and magnetic responses to spatial form of the human brain. This chapter brings Regan's book Human Brain Electrophysiology (1989) up-to-date. The contribution of the center-surround organization of lower-level visual cells is considered by Tadin and Lappin (chapter 14). They explore the possible link between physiological center-surround antagonism and perceptual functions in segregating figure from ground, perceiving surfaces, and perceiving 2D and 3D shape. Papathomas et al. (chapter 15) use transparent motion to model a way in which surfaces can be defined by motion. Retinal ganglion cells are most responsive to sharp changes in luminance, especially edges in the retinal image. So it is from edges or luminance gradients that the representation of form must be constructed. Sharpe et al. (chapter 16) continue a theme that runs through this book: the relative role of parvocellular and magnocellular divisions of the visual system. They report patients who have lost the ability to see motion-defined form associated with lesions in the parietal-temporal region around MT. Bodis-Wollner and Tzelepi look at dopamine's role in the retina in chapter 17. V Development Polat (chapter 18) considers the role of specific deficits of spatial form perception in creating amblyopia, and Steinbach and Gonzalez (chapter 19) explore the consequences of losing one eye.


1

Supplemental Material The chapters in this volume have two supplements. The first is the appendix found in this volume. This appendix lists, by subject, David Regan's contributions to our knowledge of spatial vision and related topics. The second is the CD-ROM associated with this text. The CD-ROM contains a number of color images, videos, presentations, and demonstrations that are associated with the various chapters. In addition to these chapter-specific components, the CD-ROM also contains a Quicktime version of David Regan's presentation at the York Vision Conference in June 2003.

References Harris, L. R. and Jenkin, M. (1998). Vision and Action. Cambridge University Press: Cambridge, UK. Hubel, D. H. and Wiesel, T. N. (1959). Receptive fields of single units in the cat's striate cortex. J. Physiol. (Lond.), 148: 574-591. Hubel, D. H. and Wiesel, T. N. (1962). Receptive fields, binocular interaction, and functional architecture in the cat's visual cortex. J. Physiol. (Lond.), 160: 106154 Hubel, D. H, Wiesel, T. N. (1968). Receptive fields and functional architecture of monkey striate cortex. J. Physiol. (Lond.) 195: 215-243. Milner, A. D. and Goodale, M. A. (1993). Visual pathways to perception and action. Prog. Brain Res., 95: 317-337. Milner, A. D. and Goodale, M. A. (1995). The Visual Brain in Action. Oxford University Press: Oxford, UK. Mishkin, M. and Ungerleider, L. G. (1982). Contribution of striate inputs to the visuospatial functions of parieto-preoccipital cortex in monkeys. Behav. Brain Res., 6: 57-77. Mishkin, M., Ungerleider, L. G. and Macho, K. A. (1983). Object vision and spatial vision: two cortical pathways. Trends Neurosci., 6: 414-417. Regan, D. (1989). Human Brain Electrophysiology: Evoked Potentials and Evoked Magnetic Fields in Science and Medicine. Elsevier: New York. Regan, D. (1992). Visual judgements and misjudgements in cricket and the art of flight. Perception, 21: 91-115. Regan, D. (2000). Human Perception of Objects: Early Visual Processing of Spatial Form Defined by Luminance, Colour, Texture, Motion and Binocular Disparity. Sinauer Press: Sunderland, MA. Regan, D., Giaschi, D., Sharpe, J. A. and Hong, X. H. (1992). Visual processing of motion-defined form: selective failure in patients with parietotemporal lesions. J. Neurosci., 12: 2198-2210.

8

Seeing Spatial Form

Regan, D. and Tansley, B. W. (1979). Selective adaptation to frequency-modulated tones: evidence for an information-processing channel selectively sensitive to frequency changes. J. Acoust. Soc. Am., 65: 1249-1257. Tansley, B. W. and Regan, D. (1979). Separate auditory channels for unidirectional frequency modulation and unidirectional amplitude modulation. Sensory Proc., 3: 132-140.

Part I

Form Vision


2. Pictorial Relief Jan J. Koenderi , Andrea J. van Doom, and Astrid M. L. Kappers 2.1 Introduction Look at a photograph: You see a flat piece of paper. Look into a photograph: You are aware of an all-but-flat pictorial space. Artists have marvelled over this since earliest times. Scientists have tried to trivialize it or explain it away. But pictorial space is there to stay and remains an enigma till this day. Pictorial space is a mental entity. You can't "see" it and it needn't be "looked at." Its existence fully coincides with your experience. Nor need a "corresponding" physical space exist. On close inspection the alleged photograph might turn out to be mere fungus overgrowth over a plain sheet of dried wood pulp. You will still be aware of the pictorial space, the "stimulus" being the same. Clearly, pictorial space is not an "image" of anything. Of course, you can look at and see the fungus growth, that has nothing to do with it. It is your "hallucination," if you want. Let the difference between a photograph and the fungus-overgrown sheet not be detectable without the use of a microscope. Then looking at or into the sheet with the unarmed eye is in no way different from such acts performed on actual photographs. But then the pictorial space evoked by a real photograph must be equally hallucinatory and the fact that the photograph was once "taken," irrelevant to your perception. Thus talk of "veridicality" in the context of pictorial spaces has nothing to do with perception.1 Of course we could go on and discuss regular seeing as "controlled hallucination" (Gibson, 1970), but we will refrain from that in this chapter. If pictorial space is controlled hallucination, a mental entity, then a study of its structure has to reveal structures of consciousness. Such thoughts were driving us when we embarked on our exploration of pictorial space about a decade ago (Koenderink, van Doom, and Kappers, 1992). 'Moreover, philosophical discussions on the intentional (Brentano, 1874) nature of pictorial spaces have nothing to do with perception proper. 11

12

2.2

Pictorial Relief

Some History

A study of the literature reveals that though artists have written much of interest, scientists have mostly tried to get rid of the phenomenon of pictorial space. This may take various forms; for example, one may simply deny the existence of the phenomenon, or explain it away. Denial has been most popular. Thus "stereopsis," which simply means "stereoscopic vision," is invariably interpreted as "binocular stereopsis" (through disparity) and "monocular stereopsis" is regarded a controdictio in terminis. The existence of pictorial space was rediscovered many times over, (e.g., when one Claparede (1904) accidentally put two equal images in a stereoscope without the perception "going flat") and led to an obscure literature on "paradoxical stereopsis" (which apparently the better journals wouldn't touch). If something believed to be impossible is actually found to be the case it is surely paradoxical! Explaining away usually takes the form of stating that pictorial space is not a perception proper at all, but merely a cognitive construction. It is mere phantasy and has nothing to do with vision. The Rorschach test (Rorschach, 1921) exploits this very notion. It was not until the early twentieth century that monocular stereopsis was acknowledged as a stubborn fact. Then the optical industry produced viewers to obtain optimum pictorial spatiality from single pictures. There exist two major types. The first type,the Zeiss "Verant" (designed by von Rohr (1904) and Gullstrand) is the generic example. The Verant uses a flat field loupe with exit pupil in the center of rotation of an eye, the other eye being occluded. Accommodation is fixed at infinity. The eye is centered at the perspective center of the picture, the visual field being about 40 °. This type of viewer is still used for viewing slides nowadays, although many modern slide viewers are of inferior design (apparently the design objectives are unclear). The second type uses optics to present the picture to both eyes, while eliminating accommodation, vergence, and disparity cues so that the observer is actually confronted with a flat picture. The Zeiss (Carl Zeiss Jena, 1907) "synopter" (also a von Rohr (1920) design) is the generic example, the late nineteenth century "zograscope" being a somewhat inferior precursor (Balzer, 1998). Monocular and binocular stereopsis yield qualitatively different results. It has been repeatedly rediscovered that true stereopsis (binocular of course) gives rise to a coulisses scene: there is indeed spectacular depth, but it is as if the objects were dissappointingly like flat stage cardboard cut-outs staggered at various depths, the depth gaps between the coulisses being well defined. In contradistinction, the depth gaps between objects are less well defined in monocular stereopsis (except when the objects are distributed over a visible ground-plane, as is usually the case), but the pictorial objects look nicely rounded and solid. In fact, the paradoxical stereopsis literature often remarks on the fact that monocular pictorial space looks "better" (more like the real thing) than binocular stereoscopic space (Koenderink, van Doom, and Kappers, 1994): truly paradoxical! Such effects are striking (people who don't see it typically know that they won't, even before they venture to look) and can easily be demonstrated with an antique Victorian stereoscopic viewer. As already noted, artists have speculated much on the topic. For centuries it was their job to evoke pictorial spaces in their clients. Thus we find advice to artists on how to do this, and to the clients on how to get the most out of it. For instance,

Jan J. Koenderink, Andrea J. van Doom, and Astrid M. L. Kappers

13

Figure 2.1: Many experiments on pictorial depth are of this type: The stimulus is a figure and the response is some data structure that captures aspects of the pictorial relief (here a map of equal depth curves). Leonardo (1804) tells you to close one eye and stand at the proper distance in front of a painting. Oblique viewing and binocular viewing have a flattening effect; wrong viewing distance leads to deformations (see below). Perhaps the culmination of such writings is Adolf Hildebrand's Das Problem der Form (The Problem of Form) of 1893. Hildebrand (1945) understands pictorial space as relief space and describes generic transformations applied by observers when looking at painting and sculpture. The notion of a relief space indeed gets at the heart of the matter; we will return to it later in this chapter.

2.3 Psychophysics: Methods When we started our investigations, there weren't really any adequate psychophysical methods to approach the problem. Thinking in terms of stimulus and response, the former is simple enough: a picture and a viewing method. The latter is more of a problem, though: in order to quantify pictorial objects one needs to measure data structures whose values are significant geometry (see figure 2.1). This means large data volumes. Consider a simple example. The description of a non-trivially curved surface might take the form of a triangulation with at least a few hundred vertices, thus over a thousand real numbers (say, 3 digit). These have to be collected in half an hour or so; thus we envisage data streams of ca. 10 bits per second. Compare that with the dozen or so yes-no answers that are the typical yield of a classical psychophysical experiment! The data stream should be many orders of magnitude improved. Clearly novel methods were needed. In the course of time we developed a number of these. For the sake of conciseness we discuss only a couple of instances in this chapter.

14

Pictorial Relief

One general principle of measurement has been repeatedly valuable to us. If you want to measure something you might compare it to a standard. Thus you need a standard and a manner of comparison. In cases of geometrical measurements the standard might be some fiducial object ("gauge figure") and the comparison may be a judgment of "fit" or "coincidence." This is the principle of measuring length with a yardstick, for instance. It occurred to us that it is possible to put gauge figures in pictorial space by superimposing pictures of gauge figures (3D Euclidian ones) on the picture. The first instance of such a method implemented by us was based upon a standard technique in drawing (Rawson, 1969): an oval suggests a circular mark on a slanted plane. We put the oval under manual control by the observer. In this way a "fit" could be obtained in a few seconds. Already in our first trial we sampled about a hundred spatial orientations of pictorial surfaces within a quarter of an hour (Koenderink, van Doom, and Kappers, 1992). Although we have been rather happy with this particular method and have used it to good advantage in quite a number of studies, the field holds a rather different opinion (fortunately with a few exceptions). We have heard frequent complaints that these methods "don't work" or are otherwise problematic. This came as a surprise to us since random visitors and many naive persons (in this respect, that is) that came by our laboratory never experienced any problems in our setups. In a few cases we were in a position to try setups that "didn't work." In such cases we had to grant that there were problems since we weren't able to perform the task ourselves! Problems we noted were of various kinds. For instance, in some cases the gauge figure was rendered in such a way that it didn't adhere to the pictorial surface. This is visually immediately obvious, and the remedy is equally obvious. (It seems amazing that people running visual experiments wouldn't notice.) In other cases the interface was such as to render the task manually impossible. Again, the remedy is obvious and it is amazing that one involved in human psychophysics would fail to notice. If you ever played with the kid's game where you write your name using knobs that control Cartesian horizontal and vertical movements, you will understand what we mean with an "impossible" interface. Finally, there were often problems (and errors) in the initial processing of the raw data. This assumes some basic knowledge of differential geometry (Do Carmo, 1976) that is apparently lacking in many laboratories involved in the study of visual form. Such problems have made our research somewhat unpopular. However, we remain firmly dedicated to this general style of approach which has led to a major step up in our rate of progress on these topics. A finding that relates to the idea that pictorial space is not a true perception but a mere thought construct is that we have encountered a few observers that appeared singular in their time taken to perform the settings. Generic observers take a few seconds, being mainly paced by the slowness of the manual task. In pilot reaction time experiments we find that pictorial space builds up in a fraction of a second and is clearly a "perception" in the sense of "presentational immediacy." Pictorial space simply happens to you, much like sneezing. There is nothing you can do about it, except from closing your eyes or looking away from the picture. No deep thoughts are required. The singular observers (maybe one out of ten; the statistics are only guesswork) take ten to a hundred times longer than typical observers. What might go on in these people? It may be that they are not performing the task in pictorial space (the very crux


15

of the method), but somehow "reason it out." Indeed, some of these people understand the task in this way: First I estimate the slant and tilt of the pictorial surface, then I adjust the oval in such a way that its perceived slant and tilt appear to have these same values. Now this is exactly what is not intended. Observers need not even know what slant and tilt is, nor do they have to estimate the spatial orientation of the pictorial surface. They simply have to make the oval "look right" (as a circle on the surface). This difference apparently cannot be explained to some people, including (quite a few) colleagues in visual perception with whom we had rather fruitless correspondences. As reviewers of papers, such people suggest that one should "calibrate the method" by requiring observers to estimate the slant and tilt of isolated ovals, and "correct" the settings in the actual runs accordingly. Notice that this immediately derives from the misrepresentation quoted above. It is indeed possible (Mingolla and Todd, 1986) to let people estimate (e.g., call out values in degrees) slant and tilt of pictorial surfaces. People hate the task, take a long time doing it, and are very unreliable on it. Such methods have no relation to the gauge figure method. Other frequently used methods involve the indication of the nearest or most remote point of a pictorial surface, either on a surface patch or constrained to a line in the image (van Doom, Koenderink, and de Ridder, 2001; Koenderink and van Doom, 2003) (thus a plane in pictorial space). Such methods clearly cannot be done on the basis of local pictorial detail, but have to be done in pictorial space. In a related method we place two dots on a picture and ask the observer which one is closer (Koenderink, van Doom, and Kappers, 1996). Such a question only makes sense because dots on the picture surface are seen in pictorial space and seem to lie on the nearest pictorial surface. Again closely related to the latter method is a method where a line is drawn over the picture surface (indicating a plane in pictorial space) and the observer is asked to indicate (a suitable interface being provided) the shape of the intersection of the pictorial relief with that plane (Koenderink, van Doom, Kappers, and Todd, 2000,2001), a "normal cut." A final method that we have frequently used is of an altogether different type. We prepare two photographs of a single object, taken from different camera positions (see figure 2.2). Thus the pictures are quite different. We show both photographs simultaneously to the observer. We place a dot on one photograph and ask the observer to place a dot on the second picture such that the dots indicate the same spot on the pictorial objects (Koenderink, Kappers, Pollick and Kawato, 1997; Koenderink, van Doom, Arend, and Hecht, 2002; van Doom, Koenderink, and de Ridder, 2001). This is a very general method indeed. For instance, the task would make sense if the two pictures were portraits of different people, perhaps even if one were replaced with a picture of a horse's head. The observer is allowed the response "no correspondence," in order to avoid conflicts. This is necessary for instance when photographs show different sides of some object. In typical cases of straight photographs of some simple object, observers find the task very easy and can set hundreds of correspondences in a twenty-minute session. Of course this is highly remarkable, since the task cannot be done at all via modern computer vision algorithms (Forsyth and Ponce, 2002). The

16

Pictorial Relief

Figure 2.2: Two pictures of the same object, photographed at different angles. Here the object was rotated by 67.5° about the vertical between exposures. In the method of correspondences an observer is asked to find the location in the right picture corresponding to a given location in the left image. Try it yourself for the white dot. task can't be done on pictorial detail, it has to be done in pictorial space (or rather in two pictorial spaces), and that is how our observers tell us they do it. These descriptions don't exhaust our repertoire of methods to quantify pictorial relief, but they are probably sufficient to convey the general idea. So far we have mainly used photographs of rigid, opaque objects, painted white, against simple backgrounds. The objects were somewhat more articulated than is typical for the field though. About 90% of the literature is on planar patches, simple polyhedra (e.g., cubes), cylinders, spheres, or triaxial ellipsoids. In our view the problem with such shapes is that they present singular cases for most pictorial cues. The generic case involves surfaces with more complicated surface articulations, patches of smoothly joined doubly curved convex, concave, and saddle shaped surfaces. We prefer such generic cases because they are conceptually simpler than singular cases and lead to results of a general instead of a mere specific (or artificial) nature. It is perhaps a reductionist trait natural to scientists to prefer "simple" stimuli; however, in this case (apparently) simple is actually more complicated! Here "simple" means generic. (See the book by Poston and Stewart (1996) on the notion of "genericity.") A lack of familiarity with the formal tools to handle all but the simplest objects from high school geometry may also have to do with the stimulus preferences of mainstream research. We used photographs instead of computer graphics renderings (as is usual in the field) because virtually all computer graphics pipelines cut corners in the interest of speed at the expense of physical realism (Koenderink, 1999). We feel that it might perhaps be advisable to start research on physically realistic cases. Of course it is somewhat of a burden to produce the stimuli, especially to produce parametric variations on stimuli. However, this is by no means impossible. For instance, moving a


17

light source in the photographic studio varies the shading parametrically (Koenderink, van Doom, Christou, and Lappin, 1996a; Koenderink, 1998) (the parameter being the location of the source). It has been somewhat difficult to find sufficiently articulated objects that can be acquired as multiple (identical) copies. For a time we have mainly used torsos of dummies sold for clothing display in fashion shops. Although these have served us well, we met with a number of initially unexpected difficulties. For one thing, we often were severely chastised for overstepping the limits of civil morals, especially in the United States, though less so in Europe. Perhaps more importantly we very frequently met with the remark "but everyone knows what people look like" - on the face of it a strange remark coming from people used to looking at ellipsoids all day! It was suggested that our observers should have been able to perform the task with their eyes closed! This is nonsense, for very few people actually know what humans look like (Hatton, 1904). It takes (academic) artists years to learn the details of the shapes of human bodies. The variation in body shape among the population is immense (Bammes, 1990). Even the dummies sold for fashion display change their shapes (according to current fashion) every year. However, in the face of all this we are looking for alternatives. We are currently experimenting with sweet peppers (painted white). No doubt this, too, will meet with unexpected opposition.

2.4 Findings 2.4.1 Veridicality A number of early results addressed the problem of veridicality. As explained earlier we don't think this is a particularly interesting or important issue. However, a few baseline results are notable: 1. different observers yield different results, and so does the same observer—though to a much lesser extent—at different times (all for a single picture); 2. the viewing mode (e.g., monocular, binocular, synoptical, etc.) has a major effect on the pictorial relief; 3. only by accident is the pictorial relief quantitatively like the object that was photographed; 4. the rendering of the picture has an influence on pictorial relief. Regarding 1 and 2, the differences tend to be mainly of a very specific type, namely, a dilatation or contraction of the depth domain (a Hildebrand relief transformation). Such differences need not be small; we note changes by factors as large as five (van Doom and Koenderink, 1996; Koenderink, van Doom and Kappers, 1994; Koenderink and van Doom, 2003). In a number of cases we find changes of a more general but very particular nature: different pictorial reliefs (for the same picture, but for different

18

Pictorial Relief

observers or different tasks) are related through a particular type of shear, that is to say, a transformation of the type:

where z and z' denote the depth before and after the transformation, x and y the picture plane coordinates, and a to d are constants. Such transformations are very precisely of the stated form and equate the reliefs to within the experimental spread (Koenderink, van Doom, and Kappers, 2000; Koenderink, van Doom, Kappers, and Todd, 2000, 2001; Cornells, van Doom, and de Ridder, in 2003). Regarding 3 and 4, one obviously expects qualitative differences when the pictorial cues are changed. These methods allow us to study the effect of cue changes around a natural "set point," which appears to be crucial ("cue conflict" situations and "cue isolation" situations lead to very artificial results that can hardly be extrapolated to real-life cases). For instance, we find very systematic deviations from shape constancy under variations of illumination direction when shading is one of the important pictorial cues (Koenderink, van Doom, Christou, and Lappin, 1996a; Koenderink, 1998; Koenderink, and van Doom, 2003). Regarding 4, we find that variation over subjects is large when pictorial cues are scarce, whereas results from different subjects come closely into step as the bouquet of available pictorial cues is expanded (Koenderink, van Doom, Christou, and Lappin, 1996b; Koenderink, van Doom, Arend, and Hecht, 2002; Koenderink and van Doom, 2003). This shows how the "controlled hallucination" can run most of the spectrum between almost fully idiosyncratic (faces in clouds) to largely cue driven (looking at pictures of a holiday on the beach.)

2.4.2

Influence of Viewing Mode

It is well known that you can influence the apparent depth of relief by changing your viewing mode (Jacobs, 1986b). This works both when looking into pictures and when looking into a real scene, albeit that the effects are opposite for the two cases. Such viewing modes typically have to be learned. It is one part of an artist's training to learn how to look (which is much simpler than to learn how to see (Jacobs, 1986b), but it has to be practiced anyway). This creates a problem for the scientific literature, because neither the scientists, nor their naive observers, typically know how to look, thus leading to many apparent conflicts in the literature. If you have never experienced strong monocular stereopsis, this is what you should do: stand in front of a realistic painting, e.g., an impressionist landscape. Paintings are good because they have a size large enough to eliminate problems with accommodation or monocular parallax. Moreover, artists are expert at placing their depth cues strategically and effectively. Stand at what you think is the intended distance (typical field of views are 40-50°) with your eye at the right position, i.e., frontally (both in height and laterally). Close one eye. Feel free to look around in the painting through eye movements. Now wait, simply look intently. Don't think, look. If you have never experienced monocular stereopsis it may take you half a minute or a minute to acquire it. Even if you are experienced it will take a few seconds. Don't worry, you will know


19

Figure 2.3: The picture on the top left led to the pictorial reliefs (depth increasing upwards) shown in the bottom row. The viewing modes were (from left to right) binocular, monocular, and synoptical. At the top right is a scatterplot of the monocular depth against the binocular (B) and synoptical (S) depth. The dashed line indicates identity (unit slope). for sure when stereopsis occurs because the whole scene changes on you, you are not looking at, but into the painting. The depth becomes real. This then is the experience that turned on many painters and led some scientists to write enthousiastically about "paradoxical stereopsis." The influence of the viewing mode is simple and fully corroborates the accounts given by artists such as Leonardo. Here are the facts: when you look at a picture frontally, with a single eye, you experience a certain pictorial relief. If you switch from monocular to binocular viewing, the relief collapses by a factor that depends on your binocular stereo vision (see figure 2.3). For typical observers the depth range typically decreases by roughly a factor of two; for stereo blind observers there is little effect. If one uses a synopter, thus nulling the disparity field, the relief becomes much deeper than for monocular vision. (We are considering only typical observers here.) The difference between binocular and synoptical viewing can be as much as a factor of five (Koenderink, van Doom, and Kappers, 1994). If you look at the picture monocularly, but obliquely, you lose pictorial relief gradually as the degree of obliqueness increases (van Doom and Koenderink, 1996). This is one reason2 why pictorial depth is very good when you use a Verant. When you change the viewing mode the pictorial cues evidently remain the same. This is no doubt the reason why one finds only Hildebrand-type relief transformations in these cases. We almost always find simple, linear scalings of the relief (Koenderink, van Doom, and Kappers, 1994; Todd, Koenderink, van Doom, and Kappers, 1996). 2 Additional factors, such as the elimination of accommodation and monocular parallax cues, increase the gain even more.

20

Pictorial Relief

Figure 2.4: The bottom row shows pictorials reliefs for a single observer for the (geometrically identical!) pictures shown in the top row. Experiments were done in the sequence of increasing cues. Different observers vary greatly on the silhouette, but yield essentially identical responses on the shaded picture.

2.4.3

Influence of Pictorial Cues

Even if the geometrical structure of a picture remains the same, a change of the pictorial cues may well turn out to change the structure of pictorial relief. This happens not only for dramatic changes (Koenderink, van Doom, Christou, and Lappin, 1996b; Koenderink and van Doom, 2003) (figure 2.4) but also for more subtle, parametric variations. One obvious example concerns shading. One can photograph a single scene under different illuminations and thus produce numerous pictures that are geometrically identical (be sure to mount the camera solidly such that it doesn't move between exposures) yet qualitatively different. Such pictures depict on cursory examination "the same scene." In one experiment we systematically varied the position of a (single) light source (Koenderink, van Doom, Christou, and Lappin, 1996a; Koenderink, 1998; Koenderink and van Doom, 2003). We fould that the pictorial relief was systematically dependent upon the light source position. At a first shot the reliefs are rather similar. This might be said to confirm the "shape constancy" hypothesis to some degree. However, the


21

residuals are significant and clearly correlate with the position of the source. At a second shot one might say that convexities in the pictorial object tend to bulge out into the direction of the source. Thus a sphere looks like an egg with the pointed side towards the light source (Koenderink, van Doom, and Kappers, 1995; van Doom, 2000). We found this effect not only in pictorial space, but also in real scenes. In fact the effect becomes very marked if you move a lamp around in the studio: All illuminated forms seem to deform and follow the source. Portrait photographers use this effect to good advantage, e.g., to put a crooked nose straight (Nurnberg, 1948).

2.4.4

Global versus Local Representation

Do observers maintain a "global representation" of pictorial space? We have reasons to doubt it. In gauge figure experiments with the oval we clearly sample local surface attitude (best represented by the local depth gradient, a vector in the picture plane) at a finite number (hundreds) of points in the picture plane (and thus on the pictorial relief). Whether these samples can be "explained" through a global, smooth surface (the "pictorial relief) is something that can be tested. The gradient field should be integrable. Technically a vector field is integrable if its curl vanishes identically, something that is amenable to statistical test (Koenderink, van Doom, and Kappers, 1992; Koenderink and van Doom, 2003). So far we have found no instance where the sampled gradient field failed to be integrable. This is an important fact in its own right. Apparently the observers sample from some smooth pictorial relief. We can perform the integration on the sampled data, and thus produce nice computer graphical renderings of the "pictorial relief." Such renderings are useful in that they summarize the data in a particularly intuitive form. Such pictures should not lead one to assume that similar pictures somehow wander around in the observers' heads though. Of course, we need not think of homunculi merrily watching internal screens. Such surfaces might be represented in the form of data structures (say, triangulations) that might be addressed in various ways to yield data such as local depth. Whether this is indeed the case is something for empirical verification. We have found that we can predict the answer to the question "which of two points is nearer?" from the integral surface with higher precision than the observers can answer the question themselves (Koenderink and van Doom, 1995). Because the integral surfaces were obtained from the observers' earlier local surface attitude judgments, we have to conclude that the observers cannot address a data structure that represents the integral surface. Apparently their representations (in the sense of abstract data structures) are more fragmentary than that. It turns out that observers are about equally good as the prediction if the two points happen to lie on a single slope of the pictorial surface, but that they are bad if the points are separated by a ridge (Koenderink and van Doom, 1993, 1994,1998) or rut in the relief (Koenderink and van Doom, 1995).

2.4.5

Influence of Method

Typically, we find good agreement when we compare results obtained via different methods. It is mainly the scatter in the data that varies, the shape of the pictorial relief remains unaffected (Koenderink, van Doom, and Kappers, 1996). We find that the

22

Pictorial Relief

Figure 2.5: For the picture at top right an observer yielded the pictorial reliefs A and B (top row). Because the method was slightly different, different results were found. A scatter plot (I) of the depths reveals very low correlation. When relief A is transformed according to z'(x, y) = ax + by + cz(x, y) + d with optimal parameter choices we obtain relief A* and the scatterplot (II) shows an excellent correlation.

scatter depends primarily on the differential order that is being sampled (zeroth order =>• depth, first order =*> surface attitude, second order =>• curvature, etc.) and the degree of locality of the method (measurement at a point, comparison at points at different location, etc.). The more local, the less the scatter. We find that observers are hard put to use the zeroth order at all, the first order being much easier and far more precise. Possibly, observers are even better at second order tasks, but we haven't tried yet. Spectacularly bad agreement can be found when tasks are being used that involve (mental) changes of orientation in the picture plane (Cornelis, van Doom, and de Ridder, 2003). Apparently observers have great difficulties in performing mental rotations in the picture plane. This may also affect the results obtained with apparently very similar interfaces rather dramatically. We have found that the bad agreement of pictorial reliefs in such cases can typically be greatly improved through very simple transformations of the data; thus there is a systematic order in the differences. An obvious way to compare pictorial reliefs is to make a scatter plot of depths at corresponding locations. For a mere change of viewing mode we find R 2 values in the 0.9-0.99 range. For particularly bad agreement the R2 value may not be significantly different from zero. If one does a multiple regression, including the Cartesian picture plane coordinates, such very low correlations often spectacularly improve and even get into the 0.9-0.99 range again (see figure 2.5)! Thus the aforementioned transformations of the type z'(x, y) = ax + by + cz(x, y) + d are indeed very special. We will refer to them as mental movements in pictorial space.


23

Figure 2.6: The figure shows physical and pictorial space as separated by the picture surface, but remember that pictorial space is merely virtual (a figment of the mind). A "pixel" could be at any depth and is represented as a taut wire of indefinite length in pictorial space. The mind assigns a depth value by "sliding a bead" along this wire.

2.5

Geometry of Pictorial Space

It is a priori clear that pictorial space has to be non-Euclidian for consider its group of congruences and similarities. For ease of reference we will distinguish the "spatial" dimensions (i.e., the extension of the picture plane or the visual field) and the "depth" dimension. The depth dimension has no immediate existence in the physical world and is thus a virtual (mental, not physical) dimension. The group actions that don't affect the depth dimension are simply the Euclidian movements and similarities of the picture plane or visual field. The transformations envisaged by Hildebrand affect only the depth dimension. Neither type poses any particular problem. The "mixed" transformations are the ones that yield the problems for a Euclidian interpretation of pictorial space. For Euclidian motions would include rotations about axes that are parallel to the picture plane. Such movements would allow you to see the back of the head in a frontal photograph of a face, clearly preposterous! Such motions should be forbidden by the geometry of pictorial space. With some geometrical insight it is easy enough to guess at the correct structure. Think of pictorial space as the picture plane with an infinitely long, taut thread attached to every pixel. All these threads are elongated in a single direction, of which we think as the depth dimension. On each thread we put a bead, thus obtaining an infinite Glosperlenspiel (see figure 2.6). A "pictorial relief" is formed by a swarm of such beads in the form of a curved surface. The position of any bead is controlled by the mind, for the pictorial relief is a mental thing. The mind can't move the threads (these are determined by the picture), but may shift the beads as it pleases. We may think the movement of the beads to be controlled by the mind's interpretation of the "pictorial cues." Any movement has to respect the threads; thus movements conserve a specific family of parallel lines. Now assume pictorial space to be homogeneous, that is to

24

Pictorial Relief

say, the same as seen from any of its points. Then you're done: there exists only one homogeneous space (Coxeter, 1989) that conserves a family of parallel lines. It is a certain Cayley-Klein space (Clifford, 1873; Klein, 1871) of zero curvature and a single isotropic direction.3 In such a space rotations about axes parallel to the picture plane are not periodic. Thus you can't "turn a pictorial object over" in order to see its backside! In retrospect the structure discussed above makes very good sense in a number of different contexts. Consider the problem of "shape from X." For many X's (e.g., shape from shading, shape from texture, etc.) this problem has been formally analyzed (albeit under typically very restrictive assumptions) by the computer vision community (Forsyth and Ponce, 2002). In all cases "solutions" fail to be unique but are specified up to certain groups of ambiguity transformations (Koenderink and van Doom, 1997). In the cases that we understand formally, these ambiguity groups coincide with the rigid motions of pictorial space (Belhumeur and Kriegman, 1998; Koenderink and van Doom, 1991). The general argument that almost all pictorial cues let one detect deviations from planarity suggests that the "full ambiguity group" (for the bouquet of all pictorial cues) must be of this type. A bold step! If true, this means that the motions of pictorial space cannot be detected through analysis of the pictorial cues. The observer has total freedom to perform such motions without ever coming into conflict with the pictorial cues (the structure of the picture). As we have found empirically, human observers indeed perform such "mental movements;" it is exactly what Gombrich (2000) aptly called "the observer's share," that is, the idiosyncratic (not picture related) part of the pictorial scene.

2.5.1

Simple Introduction to the Geometry: The 2D Case

A very simple case is that of a plane in pictorial space corresponding to a line in the picture, thus a plane extending in depth with a single spatial dimension. In this case the general similarity can be simply expressed as x' = kix + a, z' = bx + k2z + c, where x, x' denote the spatial coordinates before and after, z, z' denote depths before and after, and a, b, c, k\^, are constants. For k\^. — 1 one has "motions" (congruences); for kit2 7^ 1, similarities. Let us consider the motions first. Consider two points {x, z} and {u, w}. We see that x' — u' — x — u\ thus x — u is invariant against arbitrary motions. This makes x - u the perfect candidate for the distance function (Strubecker, 1962). This distance is simply the distance along the picture plane. Now consider the case x = u. Then the distance is zero, yet the points need not be equal because in general z will not equal w (two beads on a single string!). Such points are called "parallel." It is easy to check that for parallel points (and only for parallel points!) z' — w' = z — w. Thus z — w is a good distance measure for parallel points, called the "special distance." In general we define the distance as either the generic or (for the parallel case) the special distance (Strubecker, 1962). Clearly x' — x+a, z' = z describes a mere translation in the image plane, whereas x' — x, z' = z + c describes a depth shift. Such motions (also combinations) are fairly trivial. More interesting is the case x' = x, z' = ax + z; this is a "rotation" about the 3

Here "isotropic" means that stretches extending along an isotropic direction have zero length.


25

Figure 2.7: The Hermann von Helmholtz stamp (left) with two rotated copies: at the center the result of a Euclidian rotation; at the right a rotation in pictorial space, both rotations over one radian. Of course this illustration has to be consumed cum grano sails, for we took the vertical dimension of the stamp to represent depth: don't get confused! "angle" a. Since the parameter a can take values between ±00, we see that rotations and angles are not periodic in this plane; thus the plane is definitely non-Euclidian (see figure 2.7). The rotation only changes the depth; thus all "beads" move along their "threads" (as they should). The frontoparallel line z(x) = ZQ is no longer frontoparallel (namely, z(x) = z0 + ax) after a rotation, but has slope a. Clearly the slope is the tangent of the Euclidian angle, or, equivalently, the depth gradient (dz/dx = a). This yields a simple interpretation of the non-Euclidian angle. Notice that rotations don't affect distances between points. Because the slope of any line is changed by the same amount, the angles subtended by two lines (difference of their slopes) is also not affected by rotations. This is simply what one expects from congruences, of course (Jaglom, 1979). Lines extending purely into depth (the "threads" of the beads model) have infinite slope. They subtend infinite angle with any generic line. One says that they are normal to any line. The concept of "normal" is not very useful in this geometry, since all normals are parallel! Instead of normals one uses slopes. Next consider pure similarities, i.e, x' — k\x, z1 — kiz. We differentiate between similarities "of the first kind" with k% = 1 and similarities "of the second kind" with ki — 1. The similarities of the first kind merely scale the spatial coordinate (for x' — u' = ki (x — u)) whereas those of the second kind are seen to scale the angles (for dz'/dx' = fo dz/dx). A general similarity has two distinct magnification factors, one for the distances and one for the angles, quite unlike the Euclidian plane. In the Euclidian plane angles can't be scaled because they are periodic. Consider the "unit circle" x2 = 1, that is, the locus of all points at unit distance

26

Pictorial Relief

from the origin. It consists of the normal lines x — ±1. It can be rotated in itself and is convenient as a protractor to measure (non-Euclidian) angles. Such a circle is referred to as a circle "of the first kind," because there are other, very useful, ways to define circles. A unit circle "of the second kind" is z(x) = x2/2. It can also be moved along itself (by a rotation combined with a shift). Both types of circle satisfy many of the properties of the familiar circle in the Euclidian plane (Sachs, 1987). Notice that you have (for a circle of the second kind) dz/dx = x; thus the slope equals the arc length. Consequently, the non-Euclidian angles are simply (non-Euclidian) "radians." The rate of change of slope is d(dz/dx)/dx — d2z/dx2 = 1, which is the curvature. Likewise, a circle z(x) = x2/2R is seen to have curvature l/R, thus radius R. It is possible to turn the Euclidian plane into the non-Euclidian plane by a very simple trick (Jaglom, 1979). Let {x, y} be Cartesian coordinates of the Euclidian plane, the metric being given as ds2 = dx2+dy2. Now we dilate the y-axis by some large factor F (say). The ^-coordinate is not affected, but the y-coordinates are scaled by 1/F. Thus the metric becomes ds2 — dx2 + (dy/F)2, which tends to ds2 = dx2, the metric of the non-Euclidian plane, as we increase F beyond bounds. Thus the non-Euclidian plane is simply the "infinitesimal" neighborhood of the x-axis of the Euclidian plane. Indeed, all geometrical constructions discussed above are intuitively obvious if you regard them as limiting cases of their Euclidian equivalents. This nicely illustrates the role of the depth dimension as a "virtual" (vanishing spatial extent) dimension.

2.5.2

The 3D Case

The 3D case is very similar in spirit to the simple 2D case, but the group of proper motions and similarities is much richer (Strubecker, 1941; Sachs, 1990). Notice that the similarities that leave the pixels in place (x' = x, y' = y) transform the depth ("shift the beads") according to z' = ax + by + cz + d, i.e., exactly the "observer's share" as we have found empirically in many experiments (Cornells, van Doom, and de Ridder, 2003; van Doom, Koenderink, and de Ridder, 2001; Koenderink, van Doom and Kappers, 2000; Koenderink, van Doom, Kappers, and Todd, 2000, 2001; Koenderink and van Doom, 2003). Here the parameters (a, b) denote a (nonEuclidian) rotation that allows mental movements to turn any generic plane into a frontoparallel plane! The parameter c describes a similarity of the second kind, that is a Hildebrand-style relief transformation. The parameter d, finally, denotes a depth shift. In our experiments we cannot measure depth shifts since we tend to measure surface attitudes or curvatures. In practice observers are hard put to assign absolute distances to pictorial objects; thus the parameter d might as well be ignored altogether. It is possible to work out the complete differential geometry for this space (Sachs, 1990). This is of much interest, since it leads to definitions of "pictorial shape" as the invariants under mental movements. Notice that pictorial shape is different from Euclidian shape because "shapes" are (by definition) invariants under arbitrary displacements whereas the groups of displacements (congruences) are quite different in the two geometries.


2.5.3

27

The Panoramic Visual World

In some cases the geometry discussed above seems misplaced, namely, whenever the observer is confronted with a panoramic field of view. The situation is a complicated one though, since—at least in the case of photographs—one has to reckon with two distinct fields of view, namely that of the camera and that under which the observer views the picture. In many instances of daily life these two are very different. A "normal" sized picture on a page (say) might be 5 x 7 "; seen at normal reading distance of 10", this means a viewing angle of 28 x 38.6°. 'This conforms closely to the field of view of a miniature camera (24 x 36mm frame) fitted with a 50 mm "normal" lens (field of view 27 x 39.6°). Of course this is no accident: such pictures appear "normal" enough. Now suppose I use a very long telephoto lens, ca. 40cm focal length. Now the field of view of the camera is 3.4 x 5.2° whereas the field of view of the viewer is still 28 x 38.6°: i.e., much larger. Most people consider such images "unnatural" because the pictorial space looks extremely flattened. Next fit the camera with an extremly wide angle lens, say 15mm focal length. Now the field of view of the camera is 77.3 x 100.4° whereas the field of view of the viewer is still 28 x 38.6°, i.e., much smaller. Again, most people consider such images "unnatural" because pictorial objects look extremely deformed (Pirenne, 1970). Typically people blame the lenses for this. Yet both telephoto and wide-angle lenses deliver perfect perspective images; they are not to blame. It is simply that the "correct" viewing distances would be (roughly) 3 " for the wide angle and 80" for the telephoto lens, and nobody is willing (or even able) actually to view the pictures from these "correct" distances. A different issue is whether observers "correct" for the divergence of their visual rays. Perhaps surprisingly, we have reason to believe that they don't. It is easy enough to convince oneself of this fact: build a children's peepshow from a large cardboard box and prepare it for a visual field of about 120°. Put a picture (a long strip is best) bent along a cylinder with axis at the peephole in the box and illuminate it through a source located near the peephole (the idea is to prevent shading). Use a uniform texture (e.g., equal sized polka dots) for the picture. Now take a peep and try to judge the pictorial relief. You will need eye movements to view most of the picture because the field of view is larger. Try to avoid motion parallax cues. What we see is something most akin to %.frontoparallelplane, and not a circular cylinder about the eye! Judge for yourself. As one might have expected, the "shape from texture" is apparently reckoned with respect to the local visual directions and the mind doesn't take the divergence of visual angles into account at all. In this respect vision seems very similar to active touch, where local rod orientation seems to be referred to the hand frame instead of the body frame, leading blindfolded observers to commit judgment errors of up to 90° (Kappers and Koenderink, 1999). This is something one should experience: most people don't believe they are capable of errors like that until one demonstrates this to their satisfaction (or horror)! A geometrical model of what is happening in panoramic pictorial space is the following (Koenderink, 2003) (see figure 2.8). Describe the spatial positions 4 of points with respect to the observer in terms of radial distance from the observer g, and azimuthal angle (p. The radius runs from 0 (the observer) to oo, and the azimuth runs 4

To keep the discussion simple we only consider the horizontal plane here.

28

Pictorial Relief

Figure 2.8: Two figures of a planar, panoramic visual world before (left) and after (right) the application of a rotation in pictorial space. Here the visual field is very large, apart from a small gap about the full horizon! The "straight ahea^d" direction is towards the right. Notice that the rotation does not affect the radii (pixels!) and that the equidistance circles shear to become equiangular (constant slant) spirals. from —7T/2 (leftwards) through 0 (straight ahead) to +7T/2 (rightwards). In practice the visual field may be more limited than that, of course. Now we map points {g, (f>} on points {u, v} in a "mental space" according to the transformation u = arccos(—fccosy). When fc > 1, we have

where / = 1/fc. See figure 13.34.

13.A3.4 Half-Wave Square Root Rectifier: Response to the Sum of Two Sinusoids If the rectifier is of the form y = c^x, x > 0 and y = 0, x < 0 and for fc < 1 then, we will have the function

where x = (pt + Op) and y = (qt + 6q). Thus

but now

since /(x, y) = 0 when a; > arccos(— fccosy). When fc > 1, we have

where / = 1/fc. See figure 13.35. Similarly, we can find the response to any half-wave rectifier whose equation is y = cxn,x > 0; y = 0, x < 0, where n is any real number.

268

Processing Spatial Form

Figure 13.35: Half-wave square-root rectifier. See the caption to figure 13.33 for details. Reprinted from Regan, M. P. and Regan, D. (1988). A frequency domain technique for characterizing nonlinearities in biological systems. /. Theor. Biol., 133: 293317. Copyright 1988 with permission from Elsevier.

13.A3.5

Two Cascaded Half-Wave Rectifiers, AC Coupled

If two identical half-wave linear rectifiers are DC coupled, the output will be the same as a single linear half-wave rectifier. Indeed, if two half-wave rectifiers are DC coupled and the first of the series is a linear rectifier, the final output will be the same as the second rectifier alone. (We assume that the two rectifiers are either both positive or both negative.) If the two rectifiers are AC coupled, after the two sinusoids pass through the first rectifier, their function is given by

where x = (pt + Op) and y = (qt + 0q). This has a DC level given by A00/2, the constant term in the double Fourier series expansion of f(x, y). If our two successive rectifiers are linked by AC coupling, this DC level must be removed so that the function entering the second rectifier is given by

David Regan and Marian P. Regan

269

Figure 13.36: Two cascaded linear half-wave rectifiers. See the caption to figure 13.33 for details. Reprinted from Regan, M. P. and Regan, D. (1988). A frequency domain technique for characterizing nonlinearities in biological systems. J. Theor. BioL, 133: 293-317. Copyright 1988 with permission from Elsevier. where

After passing through the second rectifier, the output is given by

This can be represented by a double Fourier series where the coefficients A ±mn are given by

This is represented graphically in figure 13.36.

13.A3.6

Cascaded Compressive Rectifiers

Figure 13.37 shows the result for two square-root (y = c^x, x > 0; y = 0, x < 0) rectifiers in series, and figure 13.29 shows the result for three square-root rectifiers in

270


Figure 13.37: Two cascaded square-root rectifiers. See the caption to figure 13.33 for details. Reprinted from Regan, M. P. and Regan, D. (1988). A frequency domain technique for characterizing nonlinearities in biological systems. J. Theor. BioL, 133: 293-317. Copyright 1988 with permission from Elsevier. series.

13.A3.7

Two Parallel Rectifiers whose Summed Outputs Pass through a Third Linear Rectifier: The Dichoptic Case

In this situation only one frequency (Fi) passes through rectifier number 1 and only one frequency (F2) passes through rectifier number 2 in parallel with the first rectifier. Then the outputs from both rectifiers combine to form the input of the third rectifier. The output from the first rectifier is f ( x ) where

with a DC level of P/TT. The output of the second rectifier is G(y), where

whose DC level is Pk/n. To adjust for the DC level, the input to the third rectifier will be the function


271

Figure 13.38: Two parallel half-wave linear rectifiers converging onto a third half-wave linear rectifier. See the caption to figure 13.33 for details. Reprinted from Regan, M. P. and Regan, D. (1988). A frequency domain technique for characterizing nonlinearities in biological systems. J. Theor. Biol, 133: 293-317. Copyright 1988 with permission from Elsevier.

The output from the third rectifier is given by

Hence the coefficients for the double Fourier series can be found for

This rectifier combination is shown in figure 13.29 for the case that all three rectifiers have a linear characteristic and coupling is AC rather than DC. Other cases such as mixed rectifiers (e.g., where numbers 1 and 2 are cube-root rectifiers and number 3 is a square law rectifier) are amenable to the same general mathematical treatment.


272

Figure 13.39: Rectifier characteristic with accelerating segment, compressive segment, and a threshold. See the caption to figure 13.33 for details. Reprinted from Regan, M. P. and Regan, D. (1988). A frequency domain technique for characterizing nonlinearities in biological systems. J. Theor. Biol., 133: 293-317. Copyright 1988 with permission from Elsevier.

13.A3.8

Half- Wave Rectifier Combining Accelerating and Compressive Segments

For this rectifier, the curve equation is given by

whered = l/64(4c)63'/16,5 = 63/64(4c)1/16, and c is chosen suitably. Consequently

where x = (pt + Op) and y = (qt + 6q). So


273

This is shown in figure 13.38 with c = 27T/15.

Acknowledgments We thank Derek Hamanansingh for valuable technical assistance. This work was sponsored by the Air Force Office of Scientific Research under grant F49620-03-1-0114. D.R. holds the NSERC/CAR Industrial Research Chair in Vision and Aviation.

References Aseltine, J. A. (1958). Transform Methods in Linear Systems Analysis. McGraw-Hill: New York. Barlow, H. B. and Levick, W. R. (1965). The mechanism of directionally selective units in rabbit's retina. J. Physiol. (Land.), 178: 477-504. Bennet, R. W. (1933). New results in the calculation of modulation products. Bell Syst. Tech. J., 228-243. Beverley, K. I. and Regan, D. (1973). Evidence for the existence of neural mechanisms selectively sensitive to the direction of movement in space. J. Physiol. (Land.), 235: 17-29. Blaquiere, A. (1966). Nonlinear Systems Analysis. Academic Press: New York. Bracewell, R. (1965). The Fourier Transform and Its Applications. McGraw-Hill: New York. Braddick, O. J. (1974). A short-range process in apparent motion. VJs. Res., 14: 519527. Cartwright, R. F. and Regan, D. (1974). Semi-automatic, multi-channel Fourier analyzer for evoked potential analysis. Electroencephalogr. Clin. Neurophysiol., 36: 547-550. Clynes, M., Kohn, M. and Lifshitz, K. (1964). Dynamics and spatial behaviour of light-evoked potentials, their behaviour under hypnosis, and on-line correlation in relation to rhythmic components. Ann. NYAcad. Sci., 112: 468-509. Cobb, W. A. and Dawson, G. D. (1960). The latency and form in man of the occipital potentials evoked by bright flashes. J. Physiol. (Land.), 152: 108-121. Cynader, M. and Regan, D. (1978). Neurons in cat parastriate cortex sensitive to the direction of motion in three-dimensional space. /. Physiol. (Lond.), 274: 549569. DeValois, R. L., Albrecht, D. G. andThorell, L. G. (1982). Spatial frequency selectivity of cells hi macaque visual cortex. Vis. Res., 22: 545-599. Ellis, B., Burrell, G. J., Wharf, J. H. and Hawkins, T. D. F. (1975). Independence of channels in colour contrast perception. Nature, 254: 691-692.

274


Gabor,D. (1946). Theory of communication. /. Inst. Electrical and Electronic Engin., 93: 429-456. Granit, R. (1955). Receptors and Sensory Perception. Yale University Press: New Haven, CT. Hagedorn, P. (1982). Nonlinear Oscillations. Oxford University Press: Oxford. Hayashi, C. (1964). Nonlinear Oscillations in Physical Systems. McGraw-Hill: New York. Hirsch, M. and Smale, S. (1974). Differential Equations, Dynamical Systems and Linear Algebra. Academic Press: New York. Hobson, E. W. (1926). The Theory of Functions of a Real Variable and the Theory of Fourier's Series. Cambridge University Press: Cambridge. Hong, X. H. and Regan, D. (1989). Visual field defects for unidirectional and oscillatory motion in depth. Vis. Res., 29: 809-819. Jenkins, G. M. and Watts, D. G. (1968). Spectral Analysis. Holden-Day: Oakland, CA. Julesz, B. (1981). Textons: The elements of texture perception and their interactions. Nature, 290: 91-97. Kaufman, L. and Williamson, S. J. (1980). The evoked magnetic field of the brain. Ann. NYAcad. Sci., 340: 45-65. Kaufman, L. and Williamson, S. J. (1982). Magnetic location of cortical activity. Ann. NYAcad. Sci., 388: 197-213. King, L. V. (1924). On the Direct Numerical Calculation of Elliptic Functions and Integrals. Cambridge University Press: Cambridge. Legge, G. E. and Foley, J. M. (1980). Contrast masking in human vision. /. Opt. Soc. Am., 70: 1458-1471. Lennie, P., Pokorny, J. and Smith, V. C. (1993). Luminance. J. Opt. Soc. Am. A, 10: 1283-1293. Lu, Z. L. and Kaufman, L. (2003). Magnetic Source Imaging of the Human Brain. Lawrence Erlbaum Associates: Mahwah, NJ. McCree, K. J. (1960a). Colour confusion produced by voluntary fixation. Optica Acta, 7: 281-290. McCree, K. J. (1960b). Small field tritanopia and the effects of voluntary fixation. Optica Acta, 7: 317. Movshon, J. A., Thompson, I. D. and Tolhurst, D. J. (1978). Receptive field organization of complex cells in the cat's striate cortex. J. Physiol. (Lond.), 283: 79-99. Nelson, J. I., Seiple, W. H., Kupersmith, M. J. and Carr, R. E. (1984). A rapid evoked potential index of cortical adaptation. Invest. Ophthal. Vis. Sci., 59: 454—464. Norcia, A. M. and Tyler, C. W. (1985). Spatial frequency sweep VEP: Visual acuity during the first year of life. Vis. Res., 25: 1399-1408.


275

Norcia, A. M., Tyler, C. W. and Allen, D. (1986). Electrophysiological assessment of contrast sensitivity in human infants. Am. J. Optom. Physiol, Optics, 63: 12-15. Norcia, A. M., Tyler, C. W. and Hamer, R. D. (1988). High visual contrast sensitivity in the young human infant. Invest. Ophthal. Vis. Sci., 29: 44—49. Norcia, A. M., Tyler, C. W. and Hamer, R. D. (1990). Development of contrast sensitivity in the human infant. Vis. Res., 30: 1475—1486. Orban, G. A. (1985). Velocity tuned cortical cells and human velocity discrimination. In D. Ingle, M. Jeannerod, and D. N. Lee (Eds.), Brain Mechanisms and Spatial Vision, pp. 36-52. Martinus Nijhoff: Dordrecht. Pieron, H. (1932). Les lois du temps du chroma. Annals de Psychologic, 30: 277-280. Pieron, H. (1952). The Sensations. F. Mullen London. Poggio, G. F. and Talbot, W. H. (1981). Neural mechanisms of static and dynamic stereopsis in foveal cortex of rhesus monkey. /. Physiol. (Land.), 315: 469—492. Regan, D. (1964). A Study of the Visual System by the Correlation of Light Stimuli and Evoked Electrical Responses. PhD Thesis, Imperial College, University of London. Regan, D. (1966a). An effect of stimulus colour on average steady-state potentials evoked in man. Nature, 210: 1056-1057. Regan, D. (1966b). Some characteristics of average steady-state and transient responses evoked by modulated light. Electroencephalogr. Clin. NeurophysioL, 20: 238-248. Regan, D. (1968a). A high frequency mechanism which underlies visual evoked potentials. Electroencephalogr. Clin. NeurophysioL, 25: 231-237. Regan, D. (1968b). Chromatic adaptation and steady-state evoked potentials. Vis. Res., 8: 149-158. Regan, D. (1970a). Objective method for measuring the relative spectral luminosity curve in man. J. Opt. Soc. Am., 60: 856-859. Regan, D. (1970b). Evoked potential and psychophysical correlates of changes in stimulus colour and intensity. Vis. Res., 10: 163-178. Regan, D. (1973a). Evoked potentials specific to spatial patterns of luminance and colour. Vis. Res., 13: 2381-2402. Regan, D. (1973b). Rapid objective refraction using evoked brain potentials. Invest. Ophthal., 12: 669-679. Regan, D. (1974). Electrophysiological evidence for colour channels in human pattern vision. Nature, 250: 437^39. Regan, D. (1975a). Colour coding in man investigated by evoked potential feedback and direct plot techniques. Vis. Res., 15: 175-183. Regan, D. (1975b). Recent advances in electrical recording form the human brain. Nature, 253: 401^07.

276


Regan, D. (1977). Speedy assessment of visual acuity in amblyopia by the evoked potential method. Ophthalmologica, Basel, 175: 159—164. Regan, D. (1979). Electrical responses evoked from the human brain. Sci. Am., 241: 134-146. Regan, D. (1983). Spatial frequency mechanisms in human vision investigated by evoked potential recording. Vis. Res., 23: 1401-1408. Regan, D. (1989). Human Brain Electrophysiology: Evoked Potentials and Evoked Magnetic Fields in Science and Medicine. Elsevier: New York. Regan, D. (2000). Human Perception of Objects. Sinauer Press: Sunderland, MA. Regan, D. and He, P. (1995). Magnetic and electrical responses of the human brain to texture-defined form and to textons. J. NeurophysioL, 74: 1167-1178. Regan, D. and He, P. (1996). Magnetic and electrical brain responses to chromatic contrast in human. Vis. Res., 36: 1—18. Regan, D. and Heron, J. R. (1969). Clinical investigation of lesions of the visual pathway: a new objective technique. J. Neural., Neurosurgery Psychiatry, 32: 479483. Regan, D. and Heron, J. R. (1970). Simultaneous recording of visual evoked potentials from the left and right hemisphere in migraine. In A. L. Cochrane (Ed.), Background to Migraine, pp. 68-77. Heinemann: London. Regan, D. and Lee, B. B. (1993). A comparison of the human 40 Hz response with the properties of macaque ganglion cells. Vis. Neurosci., 10: 439-445. Regan, D. and Regan, M. P. (1987). Nonlinearity in human visual responses to twodimensional patterns and a limitation of Fourier methods. Vis. Res., 27: 21812183. Regan, D. and Regan, M. P. (1988). Objective evidence for phase-independent spatial frequency mechanisms in the human visual pathway. Vis. Res., 28: 187-191. Regan, D., Schellart, N. A. M., Spekreijse, H. and Van der Berg, T. J. T. P. (1975). Photometry in goldfish by electrophysiological recording. Vis. Res., 15: 799807. Regan, D. and Spekreijse, H. (1970). Electrophysiological correlate of binocular depth perception in man. Nature, 255: 92-94. Regan, D. and Spekreijse, H. (1974). Evoked potential indications of colourblindness. Vis. Res., 14: 89-95. Regan, D. and Sperling, H. (1971). A method for evoking contour-specific scalp potentials by chromatic checkerboard patterns. Vis. Res., 11: 173-176. Regan, D. and Tyler, C. W. (197 la). Wavelength-modulated light generator. Vis. Res., 11:43-56. Regan, D. and Tyler, C. W. (1971b). Some dynamic features of colour vision. Vis. Res., 11: 1307-1324.


277

Regan, D. and Tyler, C. W. (1971c). Temporal summation and its limit for wavelength changes: An analog of Bloch's law for colour vision. J. Opt. Soc. Am., 61: 1414-1421. Regan, M. P. (1986). Analysis of a Nonlinearity with Application to Visual Processing. MSc Thesis, Dalhousie University, Nova Scotia, Canada. Regan, M. P., He, P. and Regan, D. (1995). An audio-visual convergence area in human brain. Exp. Brain Res., 106: 485^87. Regan, M. P. and Regan, D. (1988). A frequency domain technique for characterizing nonlinearities in biological systems. J. Theor. Biol., 133: 293-317. Regan, M. P. and Regan, D. (1989). Objective investigation of visual function using a nondestructive zoom-FFT technique for evoked potential analysis. Can. J. Neurolog. Sci., 16: 1-12. Reichardt, W. and Poggio, T. (1981). Theoretical Approaches in Neurobiology. MIT Press: Cambridge, MA. Reichardt, W., Poggio, T. and Hausen, K. (1983). Figure-ground discrimination by relative movement in the fly. Part II. Biol. Cybern.,46: 1-15. Schiller, P. H., Sandell, J. H. and Maunsell, J. H. R. (1986). Functions of the ON and OFF channels of the visual system. Nature, 332: 824-825. Sekuler, R., Pantle, A. and Levinson, E. (1978). Physiological basis of motion perception. In R. Held, H. W. Leibowitz, and H. L. Teuber (Eds.), Perception, pp. 67-96. Springer-Verlag: New York. Spekreijse, H. (1966). Analysis of EEC Responses in Man. PhD Thesis. Dr. W. Junk Publishers: The Hague. Spekreijse, H. and van der Tweel, L. H. (1965). Linearization of evoked responses to sine wave modulated light by noise. Nature, 205: 913-914. Spekreijse, H., van der Tweel, L. H. and Regan, D. (1972). Interocular sustained suppression: Correlations with evoked potential amplitude and distribution. Vis. Res., 12: 521-526. Stoker, J. J. (1950). Nonlinear Vibrations. Plenum Press: New York. Tigges, J. and Tigges, M. (1985). Subcortical sources of direct projections to visual cortex. In A. Peters and E. G. Jones (Eds.), Cerebral Cortex, Vol. 3, pp. 351378. Plenum Press: New York. van der Tweel, L. H. and Lunel, V. (1965). Human visual responses to sinusoidally modulated light. Electroencephalogr. Clin. Neurophysiol., 18: 587-598. Wright, W. D. (1928). A trichromatic colorimeter with spectral primaries. Trans. Opt. Soc. Land., 29: 225-241. Wright, W. D. (1928-29). A re-determination of the mixture curves of the spectrum. Trans. Opt. Soc. Land., 30: 141-164. Yamamoto, T. S. and DeValois, K. K. (1996). Chromatic-spatial selectivity for luminancevarying patterns. Invest. Ophthalmol. Vis. Sci., 37 (suppl.): S1064.


14. Linking Psychophysics and Physiology of Center-Surround Interactions in isual Motion Processing Duje Tadin and Joseph S. Lappin 14.1

Introduction: Moving Image Information

The eye is stimulated and informed by continually changing patterns. Images of environmental surfaces move over the retina as the eyes move, and the images expand, contract, and deform as objects move and the observer moves through the environment. The structure of these changing patterns is a principal form of information - about the shapes, locations, and movements of environmental objects and about the location and movement of the observer (Nakayama, 1985; Andersen, 1997; Lappin and Craft, 2000). The multipurpose contribution of image motion to visual function underscores the importance of investigating not only how motion is perceived but also how motion information is exploited by the visual system to perform other related functions. The speed, reliability, and precision with which the human visual system acquires information about the environment from changing stimulus patterns might well be considered miraculous. Indeed, the visual-motor coordination of muscular output with optical input is so effective and effortless that usually it is subconscious. Training in science or art is needed to recognize these commonplace miracles of physics, biology, and computation. As Martin Regan enjoys pointing out, the efficiency of motion perception is demonstrated by cricket. To Martin's eye, one of nature's finest achievements is displayed in a batsman's ability to swing his bat to intersect the path of a cricket ball approaching at about 90 mph, bounced off the ground, and released by the bowler less than half a second earlier. Regan (1992) calculates that the bat-ball intersection occurs within a window only about 10 cm wide and 2.5 ms duration. Keep in mind that information about the ball's trajectory must be extracted from moving images, and from a complex 279

280

Linking Psychophysics and Physiology of Center-Surround Interactions

background that is also moving relative to both the eye and the ball. Keep in mind as well that this information is extracted by a neural network of billions of interconnected cells. Such impressive visual achievements can be studied at many levels of analysis, including both psychophysics and physiology. Understanding visual motion perception requires both psychophysical analyses of the optical information at the eye and physiological analyses of the neural mechanisms that detect and transform this information. Progress has occurred on both levels, but the links between these two levels of knowledge are still limited. The purpose of this chapter is to review current evidence about links between physiological characteristics and behavioral functions of center-surround interactions in visual motion perception. We focus especially on the physiological characteristics of center-surround neurons in the primate cortical area MT (V5) and on perceptual functions that may be related to these cells. Some aspects of our discussion are necessarily speculative because the experimental evidence needed to link the physiological and psychophysical analyses of this aspect of motion perception is incomplete. Nevertheless, enough has been learned that it seems timely to review current progress and gaps in describing these links between physiology and perception. We begin by discussing the logic for linking evidence about visual perception and physiology. Next, in section 14.2, we review physiological evidence about the centersurround organization of receptive fields in area MT as well as psychophysical evidence of apparent perceptual correlates of this center-surround antagonism. Sections 14.3 and 14.4 then examine evidence about two potential perceptual functions of this centersurround motion mechanism - involvingfigure-groundsegregation and perception of surface shape, respectively.

14.1.1 Linking Macroscopic and Microscopic Analyses of Visual Information Flow A fundamental but challenging problem in all areas of science is to link macroscopic and microscopic analyses of a system. Understanding how a complex system operates in a changing environment requires knowledge about both the dynamics of environmental conditions and the dynamics of the system's components. Causal relationships operate simultaneously on multiple levels and also between levels. Macroscopic and microscopic processes are interdependent; analysis of either one alone is insufficient. Functional links between visual perception and physiology involve the transmission of information — involving correspondences between (a) environmental objects and events, (b) optical stimulation of the eyes, and (c) physiological response patterns. Spatiotemporal patterns of physiological responses must maintain sufficient information about environmental stimuli to permit real-time coordination of motor actions, recognition of objects, and comprehension of meaningful scenes and events. The informational capacity of visual motion perception is suggested by athletic skills in coordinating actions with the motion of a ball and with motions of other players. Such visual-motor coordination depends on the fidelity of information transfer between the environment and the brain. The structural correspondence between these very different

Du/'e Tadin and Joseph S. Lappin

281

physical domains is not physical, of course, but based on spatial and temporal patterns. Visual information transmission occurs simultaneously on both macroscopic and microscopic levels. The multilevel nature of the information transmission from optics to perception and action enables inferences about physiological processes from psychophysical experiments and inferences about perceptual functions from observations of physiology. A well-known paper by Teller (1984) reviews the logic of a variety of "linking propositions" for relating perceptual states with physiological states. As Teller points out, such linking propositions are at least implicit if not explicit in interpreting a large body of research in visual psychophysics and physiology. The best-known linking proposition for inferring physiological processes from perceptual behavior was articulated by Brindley (1960, p. 144): "[W]henever two stimuli cause physically indistinguishable signals to be sent from the sense organs to the brain, the sensations produced by those stimuli, as reported by the subject in words, symbols or actions, must also be indistinguishable." An important implication of this proposition is its contrapositive: if an observer can behaviorally discriminate between two stimuli, then these stimuli must elicit different physiological signals at the retina and at all subsequent neural stages leading to the behavioral response. That is, behavioral discrimination implies physiological differences. This proposition might seem almost trivially obvious, but it has nontrivial applications (Brindley, 1960; Teller, 1984; Lappin and Craft, 2000). The classic experiments by Hecht et al. (1942) and by Bouman (1950) offer good illustrations, where behavioral experiments were used to demonstrate that a single photon is sufficient to excite a single photoreceptor. Moreover, most of the response variability for a given stimulus condition was attributable to physical variability of the stimulus rather than to physiological signals or cognitive decisions. Westheimer's (1979) studies of spatial "hyperacuity" also illustrate how behavioral discriminations permit important inferences about the retinal encoding of spatial position. A striking finding in these psychophysical experiments was the surprisingly small amount of information that was lost by the vast neural network between retina and motor response. The linking proposition described by Brindley and others is a special case of a more general principle related to the second law of thermodynamics: information about the input stimulation of the retina can only be lost but not created by the visual processes that lead to behavioral responses. Therefore, behavioral discriminations cannot be more precise or reliable than differences between the retinal stimuli or physiological responses to these stimuli. In general, the links between physiology and perception may be understood in terms of the flow of information about spatial and temporal patterns. Spatiotemporal information associated with moving images may be described both macroscopically and microscopically. The macroscopic perceptual level involves perceptual correlations between retinal stimulation and behavioral actions, and the microscopic physiological level involves spatiotemporal patterns of neural spike trains. This scheme is illustrated in figure 14.1. The macroscopic and microscopic quantities of transmitted information

282


Figure 14.1: Schematic illustration of the relation between two levels of analysis of the flow of information in visual motion perception. The double-pointed arrows refer to correspondences (approximate isomorphisms) between spatiotemporal patterns (relational structures) in different domains. Inferential links between processes on the macroscopic and microscopic levels are based on the general requirement that the quantities of information transmission (from optical input to behavioral output) on the two levels of analysis must equal one another. must equal one another, and the information in the neural response patterns must correspond to that in the optical stimulation of the eye. In practice, however, measurements of the physiological information are sufficiently difficult that one can seldom achieve a quantitative match between the macro and micro descriptions of information. If two different trajectories of a moving object can be behaviorally discriminated, as in most ball-playing sports, then the temporal sequence of retinal images and the resulting signals stimulated by these two motions must also differ correspondingly at the retina and at subsequent neural stages. Moreover, spatiotemporal characteristics of the ball's trajectory must also be represented with sufficient precision by physiological signals to coordinate similar actions for similar trajectories. Correlations between a ball's trajectory and an athlete's actions imply additional correlations with both the optical patterns on the eyes and the physiological response patterns leading to the behavioral actions. The information-carrying patterns in these very different physical domains must correspond to one another; indeed, they must be essentially isomorphic. The flow of information is based on approximate isomorphism of relational structures or patterns (Lappin and Craft, 2000).] 'A relational structure is defined as "a set together with one or more relations on that set" (Krantz, Luce, Suppes, and Tversky, 1971, p. 8). Sets may consist of eilher discrete elements or continua (e.g., in space, time, or real numbers), and they may contain elements formed from other elements by operations such as concatenation, addition, differentiation, and products of sets. The relations may be simple equivalence vs. nonequivalence of categories, ordinal relations, differences, ratios, distances, or other such relations

Dufe Tadin and Joseph S. Lappin

283

Visual perception of distal environmental objects and motions involve discriminations between groups of patterns denned by invariances under many transformations of the proximal images. Variations in the positions and motions of an observer's eyes and body relative to an object, variations in the background context of other objects and motions, and variations in ambient illumination yield an infinite group of proximal retinal images potentially associated with a given moving object. If visual discriminations among environmental objects and motions are robust under variations in their retinal images, then the visual signals that describe these objects and motions must also remain relatively unchanged by these image transformations. The underlying physiological signals must do more than simply discriminate between different proximal image patterns - because uncertainty about the proximal image parameters leads inevitably to reductions in detection and discrimination (Shipley, 1960; Green and Swets, 1966; Lappin and Staller, 1981). If perceived trajectories of environmental objects are robust under movements of the observer's head and eyes, under variations in background context, and so on, then physiological mechanisms must extract motion information that also is robust under such image transformations. Relative motion, for example, is invariant under rigid image translations. Perhaps, this invariant information can be extracted by center-surround mechanisms. Such mechanisms may be critical to our ability to distinguish the motion of a target object from that of the background. We explore this hypothesis in the second half of this chapter. Lappin and Craft (2000) used this line of reasoning to reach conclusions about retinal spatial primitives that specify local surface shape. They showed first that the local shape of a smooth environmental surface is isomorphic with 2D second-order spatial derivatives of the retinal images as the object rotates in depth or is viewed stereoscopically. Lower-order properties such as first-order spatial derivatives do not provide such information because they are not invariant under object motions that change the object's orientation or distance from the observer. They also concluded from psychophysical experiments that this higher-order differential image structure that specifies local surface shape must be directly represented by retinal signals. The basis for the latter conclusion was that observers maintained hyperacuities (for relative motion and among multiple elements. The classical "information theory" of Shannon and Weaver (1949) is based on statistical correspondence between relational structures involving only equivalence/nonequivalence relations. An adequate theory of the information in spatiotemporal optical patterns and physiological signals, however, requires structures with stronger relations that are at least ordinal. The fact that relational structures may have higher orders of complexity, with multiple dimensions and higher orders of differentiation or exponentiation, is especially important in applying notions of relational structures to theories of vision. This idea is implicit in Koenderink and van Doom's (e.g., 1992a, 1992b) uses of differential geometry for describing images of surfaces and analyzing local image operators. This idea was also used by Lappin and Craft (2000) to study the structural correspondence between surfaces and their images and to identify the spatial primitive for perceiving surface shape from motion and stereo. "Isomorphisms" - one-to-one correspondences - between relational structures such as environmental objects, retinal images, and physiological response patterns are only "approximate" rather than exact for two main reasons: First, the correspondences may be statistically perturbed by random optical and physiological fluctuations. Second, environmental surfaces and their retinal images are related by projective geometry - where object surfaces are often partially occluded by nearer objects and surface regions, where there are unusual but possible ambiguities associated with accidental views of objects, and where the relative scales of distances in depth and in the frontal plane are ambiguous. Despite these largely technical qualifications, the concept of isomorphism is sufficiently close that we will use it as basic to our conception of information. Lappin and Craft (2000) provide a fuller discussion of the correspondences between surfaces and their images.

284


stereoscopic disparities) for discriminating the relative position of a point on a smooth surface even under noisy perturbations of lower-order spatial relations. Empirical support for this hypothesis needs to be expanded, but the methods and rationale of this study illustrate an expanded form of linking propositions for inferring physiological relations from psychophysical discriminations. How physiological signals represent such higher-order differential relations is not yet known. 2D second-order spatial derivatives involve spatial relations among at least five points, and relations between such spatial structures in two successive images would involve a relational structure of at least ten points. The complexity and multidimensionality of such relations obviously exceeds what can be represented by spike trains of individual neurons. Neural representation of such spatiotemporal relations would require relationships among neighboring neurons. Nevertheless, receptive-field characteristics of some MT cells may provide information about spatial derivatives of moving images, and such information might be involved in perceiving local surface shape. We review this hypothesis in more detail later in this chapter.

14.1.2 Inferring Perception from Physiology Linking macroscopic perceptual processes with microscopic physiological processes requires inferential links in both directions, from physiology to perception as well as from perception to physiology. The links in both directions are difficult; typically one cannot be certain whether signals recorded in neurons in a given visual area are sufficient or even necessary for perceptual responses to the given stimulus patterns. An apparent absence of difference in specific neural responses to two different stimuli typically does not imply that these stimuli are visually indiscriminable by an observer using responses of the whole visual system. With accumulating physiological evidence about receptive field characteristics of multiple visual areas, and with accumulating evidence about comparisons between the neural and behavioral responses to particular stimulus patterns, hypotheses about links from physiology to perception and behavior have grown less speculative. Converging physiological and psychophysical evidence over the past 20 years has begun to clarify the links between physiology and visual function in perceiving motion, though we still do not have a quantitative picture of the information flow from moving images through physiological mechanisms to perceived motions of environmental objects. The present chapter examines currently available knowledge about the links between physiological and perceptual functions of motion-sensitive center-surround neurons in primate area MT. Our interest in outlining this knowledge was sparked by our recent discoveries of apparent perceptual correlates of center-surround antagonism in the responses of many cells in MT (Tadin et al., 2003). Our experiments were psychophysical but they were stimulated by physiological findings.

Diy'e Tadin and Joseph S. Lappin

285

14.2 Center-Surround Interactions in Motion Processing Center-surround receptive field organization is a ubiquitous property of visual neurons (Allman et al., 1985a). Such mechanisms are well suited for extracting information about the spatial organization of retinal images. They amplify responses to spatial differences in properties such as luminance, and suppress responses to uniform image regions. The spatial organization of image variations usually is more informative about the structure of the environment than the uniform properties of retinal images. Given the computational demands of visual motion processing, center-surround mechanisms may play an important role in motion perception. Spatial variations in image motion carry important visual information about the relative locations, orientations, and shapes of surfaces, about the trajectories of moving objects, and about the observers locomotion through the world (Nakayama, 1985; Braddick, 1993; Regan, 2000; Lappin and Craft, 2000; Warren, 1995). Uniform motion fields, however, are often caused by eye or body movements and, as such, can make the perception of object motion more difficult. Center-surround mechanisms are well suited for extracting information about the spatial structure of moving fields and for suppressing information about uniform motions.

14.2.1 Center-Surround Mechanisms Found in MT and Elsewhere Center-surround interactions are frequently observed in the neural areas sensitive to motion. In the primate cortex, center-surround receptive field organization has been observed in the primary visual cortex (VI) (Jones et al., 2001; Cao and Schiller, 2003), MT (Allman et al., 1985b), and lateral MST (Eifuku and Wurtz, 1998). Other areas and species in which center-surround neurons have been found include rabbit retina (Olveczky et al., 2003), tectum of frog (Griisser-Cornehls et al., 1963) and pigeon (Frost and Nakayama, 1983), superior colliculus of both cat (Mandl, 1985) and macaque monkeys (Davidson and Bender, 1991), area 17 of cat (Hammond and Smith, 1982; Kastner et al., 1999), and PMSL of cat (von Grunau and Frost, 1983). Among these motion-sensitive areas, center-surround mechanisms have been described in most detail in MT (Allman et al., 1985b; Tanaka et al., 1986; Born and Tootell, 1992; Lagae et al., 1989; Raiguel et al., 1995; Bradley and Andersen, 1998; Born, 2000; Borghuis et al., 2003). The function of these mechanisms has been studied both neurophysiologically (Xiao et al., 1995, 1997a, b, 1998; Bradley and Andersen, 1998; Bom et al., 2000) and by computational modeling (Nakayama and Loomis, 1974; BuraCas and Albright, 1994, 1996; Liu and Van Hulle, 1998; Gautama and Van Hulle, 2001). This substantial literature provides a foundation for describing probable perceptual roles of MT center-surround mechanisms. First described in the owl monkey by Allman and Kaas (1971), area MT is traditionally considered part of the dorsal processing stream and is believed to play a central role in motion perception (Orban, 1997). Its association with the dorsal stream emphasizes functions in perceiving space and guiding motions - "where" or "how" functions. The functions of center-surround mechanisms in MT, however, probably also involve

286


Figure 14.2: Responses of a typical center-surround (A) and wide-field neuron (B) to random-dot motion of increasing patch size. Patchy 2-deoxyglucose uptake in MT (C) was obtained by presenting a large pattern of random dot motion to a macaque monkey (Born, 2000). Dark areas show clusters of wide-field neurons. Illustrations courtesy of Richard T. Born. so-called "what" functions in shape and object perception, functions often attributed to the ventral stream. Allman et al. (1985b) found that responses of most neurons in owl monkey MT were modulated by stimulation in the region surrounding the classical receptive field (figure 14.2A). The surround regions are often described as "silent" because stimulation of the surround alone does not affect the neuron's response. Most of the observed interactions were antagonistic: the firing rate to motion in the preferred direction in the center region was reduced when the motion pattern was expanded into the surround region. That is, center-surround neurons responded poorly to large fields of uniform motion. If the motion in the surround was in the anti-preferred direction, its suppressive effect diminished, and for some neurons the response was facilitated. Center-surround neurons are found in all layers of MT but are less common in layer IV (Raiguel et al., 1995; Born, 2000), suggesting that surround inhibition is probably mediated via intrinsic MT connections. The spatial extent of the surround is usually three to five times larger than the extent of the center region (Tanaka et al., 1986; Raiguel et al., 1995), and the directional tuning of the surround is broader than that of the center (Born, 2000). Initial reports described surrounds as encircling the central region of the receptive fields (Allman et al., 1985b; Tanaka et al., 1986). Subsequent explorations, however, found that the spatial extent of most MT surrounds is nonuniform (Xiao et al., 1995,1997a, 1997b, 1998), suggesting that such surrounds may have important computational properties (explored further below). In addition to neurons with such center-surround antagonism (sometimes called "local motion" neurons), some MT neurons prefer large moving fields and show no

Du/'e Tadin and Joseph S, Lappin

287

surround suppression ("wide-field" neurons) (figure 14.2B; Altaian et al., 1985b; Born and Tootell, 1992; Raiguel et al., 1995). These two types of neurons are clustered anatomically (figure 14.2C; Born and Tootell, 1992) and make different efferent connections, with wide-field neurons projecting to ventral MST and center-surround neurons projecting to dorsal MST (Berezovskii and Born, 2000). These two types are also believed to have different functions, with center-surround neurons coding object motion and wide-field neurons signaling background motion (Born et al., 2000). Currently available descriptions of center-surround interactions in primate MT have been generally consistent. The stimulus patterns used to characterize these receptive fields, however, have been almost exclusively high-contrast random-dot patterns. As we shall see, this restriction of the methods also restricts the description of these receptive fields. For example, the spatial organization of receptive fields in primate VI has been found to vary with both contrast (Sceniak et al., 1999) and color (Solomon et al., 2003). Pack and Born (personal communication, August 2003) found that centersurround antagonism in MT neurons substantially decreases or disappears at low contrast.

14.2.2 Perceptual Correlates of Center-Surround Antagonism If center-surround antagonism is indeed an integral part of motion processing, we might expect to see the perceptual signature of this antagonism in the form of impaired motion visibility with increasing stimulus size. Existing evidence, however, shows that increasing the size of a low-contrast moving stimulus enhances its visibility, presumably owing to spatial summation. Such psychophysical estimations of the spatial properties of motion mechanisms tend to be based on low-contrast (Anderson and Burr, 1991; Watson and Turano, 1995) or noisy stimuli (Lappin and Bell, 1976), while physiological descriptions of center-surround neurons have been obtained with high-contrast patterns. Several physiological studies of visual cortex have found that center-surround interactions depend on contrast, with surround suppression stronger at high contrast and spatial summation more pronounced at low contrast (Kapadia et al., 1999, Levitt and Lund, 1997; Sceniak et al., 1999). Thus, contrast thresholds may not fully describe the spatial properties of human motion mechanisms, especially at high contrast. Tadin et al. (2003) measured the threshold exposure durations needed for human observers to accurately identify the motion direction of a drifting Gabor patch. Observers viewed foveally centered Gabor patches which varied in size and contrast. The results (Figure 14.3) showed that at low contrast, duration thresholds decreased with increasing size. This result, implying spatial summation of motion signals, is consistent with earlier reports (Anderson and Burr, 1991; Watson and Turano, 1995). At high contrast, however, duration thresholds increased fourfold as the Gabor patch width increased from 0.7° to 5°. This surprising result implies neural processes fundamentally different from spatial summation. Several psychophysical characteristics found by Tadin et al. (2003) indicate that this result is attributable to center-surround antagonism in MT: (1) Impaired visual performance with larger stimuli has been construed as the perceptual signature expected from antagonistic center-surround mechanisms (Westheimer, 1967). (2) The "critical size" at which strong suppression is first observed is large enough to impinge on the

288


Figure 14.3: Duration thresholds as a function of stimulus size at different contrasts. Adapted from Tadin et al. (2003). surrounds of MT neurons with foveal receptive fields of the macaque monkey (figure 14.4; Raiguel et al., 1995). (3) The detrimental effect of stimulus size diminished in the visual periphery, consistent with the increase of MT receptive field sizes with eccentricity (Raiguel et al., 1995). (4) Motion aftereffect (MAE), a perceptual aftereffect attributed at least partly to MT activity (Huk et al., 2001), is weaker if induced with large high-contrast stimuli. This result would be expected if such stimuli inhibit the activity of MT neurons whose adaptation normally contributes to the MAE. (5) Isoluminant motion gratings did not produce surround suppression, a characteristic that dovetails with the finding that MT neurons respond much more weakly to motion of isoluminant gratings than to motion of luminance gratings (Gegenfurtner et al., 1994). Taken together, these results suggest that impaired motion perception for large highcontrast patterns is a perceptual correlate of center-surround antagonism in MT. Contrast Dependency of Center-Surround Antagonism It is intriguing that increasing stimulus contrast dramatically changes the spatial integration of motion signals. This psychophysical result is compatible with physiological evidence that in VI neurons the relative strength and/or spatial extents of the excitatory center and inhibitory surround change with contrast (Kapadia et al., 1999; Levitt and Lund, 1997; Sceniak et al., 1999). This psychophysically observed transition from summation to suppression occurs around 5% contrast (Tadin et al., 2003), which is the contrast where an average MT neuron attains about 25% of its maximum response (Sclar et al., 1990). The contrast dependency of center-surround antagonism may have a functional role. At high contrast, the perceptual benefits of surround suppression (Born et al., 2000; BuraCas and Albright, 1996; Gautama and Van Hulle, 2001; Liu and Van Hulle, 1998; Nakayama and Loomis, 1974; Xiao et al., 1997b) probably outweigh the re-

Duje Tadin and Joseph S. Lappin

289

Figure 14.4: Psychophysically estimated "critical size" shown relative to an average foveal MT receptive field. The dark dashed circle illustrates the stimulus size beyond which an average foveal MT center-surround neuron exhibits surround suppression (Raiguel et al., 1995). Full spatial extent of the stimulus is indicated by the light dashed circle. This comparison assumes that the properties of human and macaque MT are comparable (Rees et al., 2000), and that the receptive field sizes are similar for the two species (Kastner et al., 2001). Adapted from Tadin et al. (2003). suiting decreases in neural activity and motion sensitivity. Motion sensitivity becomes more important at low contrast, so it seems functionally beneficial that receptive field organization shifts with reduced contrast from center-surround antagonism to spatial summation. Spatial integration versus differentiation of motion signals seems, therefore, to reflect an adaptive process that adjusts processing of motion signals to fit the input signal/noise characteristics. Perceptually important suppressive mechanisms seem to operate only when the sensory input is sufficiently strong to guarantee visibility. Analogous contrast dependency has also been found in other psychophysical studies of interactions among spatially separate motion signals. Lorenceau and Shiffrar have studied the perceptual integration of separated moving contours of a shape (usually a diamond) viewed through multiple apertures that occluded its vertices (Lorenceau and Shiffrar, 1992, 1999; Shiffrar and Lorenceau, 1996; Lorenceau and Alais, 2001). Perception of the partially occluded shape in these displays required integration of contours moving in different directions inside spatially separate apertures. With lowcontrast contours, the spatially separate motions usually appeared as a rigidly moving and globally connected object. At high contrast, the same patterns of local contour motion usually appeared disconnected and unrelated. Lorenceau and Shiffrar (1999) also found that motion integration was more likely to occur in noisy, eccentric viewing, and isoluminant conditions - the same conditions in which Tadin et al. (2003) found surround suppression to be weaker. Alais et al. (1998), Takeuchi (1998), and Lorenceau and Zago (1999) also found that spatially separate patches of drifting gratings were

290


more likely to be perceived as a coherently moving form at low contrast. Specifically, Takeuchi (1998) found that perception of a rigidly moving form and perception of independently moving gratings were equally likely at about 5% contrast. This finding is consistent with that of Tadin et al. (2003), who found the transition from spatial summation to spatial suppression to occur at about 5% contrast. Other Psychophysical Results Consistent With Surround Suppression Psychophysical experiments have often measured motion discriminations near threshold values of contrast or statistical coherence. Such impoverished motion signals probably promote spatial summation, precluding observations of surround suppression. Whatever hints of surround suppression might be found in the literature are likely to be found in experiments using large high-contrast patterns. Indeed, Verghese and Stone (1996) found that when a large high-contrast pattern was divided into smaller parts, speed discriminations actually improved. The authors suggested that surround suppression was one possible explanation. Derrington and Goddard (1989) found that direction discriminations of brief large gratings decreased when contrast was increased. This result is consistent with those of Tadin et al. (2003), though the authors suggested a different explanation. Murakami and Shimojo (1993) studied induced motion in stationary test stimuli presented within a large patch of moving dots. They found that induced motion was replaced by motion assimilation when the test stimulus was small, low contrast, or presented in the visual periphery - suggesting that motion antagonism changes to motion summation under these conditions. Surround suppression is also suggested by findings in several MAE studies in which large high-contrast adaptation patterns produced relatively small MAEs (Sachtler and Zaidi, 1993; Murakami and Shimojo, 1995; Tadin et al., 2003). Kim and Wilson (1997) found that when the directions of motion in center and surround differed by 45 °, the perceived direction of the central motion could shift 30-40 ° away from the surround direction. Like most of the results above, these directional interactions increased with the size and contrast of the surround. The perceived shift in the direction of the center stimulus might be a result of selective inhibition of neurons tuned to directions similar to the surround. Broad directional tuning of surround suppression (Born, 2000) may be the cause behind this rather large perceived directional shift. At high contrast, direction discriminations are improved by reducing the size of the motion pattern (Tadin et al., 2003). This trend should reverse at some small size, however, as the motion of very small stimuli should be hard to identify. Thus, at any contrast where surround suppression is observed, there must be an intermediate size at which performance is best. This optimal size marks the transition between summation and suppression of motion signals. The question of whether this optimal size varies with contrast was investigated by Lappin et al. (2003). Duration thresholds were measured for discriminating motion directions of random-dot motion patches of various sizes and contrasts. At maximum contrast the optimal size was about 1 ° in diameter, but the optimal size increased as the contrast was reduced. A related result was found by Nawrot and Sekuler (1990), who investigated how high-contrast motion at one location influences the perception of adjacent incoherent


291

random motion. Stripes of coherently moving random-dot patterns alternated with stripes of random motion. When the stripes were narrow, the perceived motion of the random stripes was assimilated to the direction of the coherent stripes, and the whole pattern appeared to move in the same direction. When the stripes were wide, the random stripes appeared to move in the opposite direction from the coherent stripes. The stripe width for this transition from motion assimilation to motion contrast occurred at about 1°, a value similar to that found by Lappin et al. (2003).

14.2.3 Interim Conclusions Center-surround antagonism in motion processing has been found in a diverse set of both physiological and psychophysical studies. The spatial interactions in these phenomena depend critically on several stimulus parameters including contrast, eccentricity, and signal/noise ratio. Thus, perceived motion can change substantially depending on the viewing conditions. The motion system has the difficult task of balancing two fundamentally conflicting processes: spatial integration and spatial differentiation (Braddick, 1993). The adaptive, contrast-dependent nature of center-surround interactions (Tadin et al., 2003), however, may allow the visual system to alternate between integration and differentiation depending on the available stimulus information. Describing these center-surround mechanisms and the conditions in which they operate is important, but it is only half the story. Understanding their functional contribution to vision is arguably more important. At first glance, center-surround antagonism may seem maladaptive - causing impaired motion perception and failure to integrate motion signals arising from a single object. One would expect such costs to be offset by significant visual benefits.

14.3

Segregating Surfaces

An important early step in visual processing is to organize the retinal image into surfaces and objects, segregating figure from ground. Objects can differ from their backgrounds in a variety of physical properties - including luminance, texture, color, motion, temporal synchrony, and binocular disparity (Regan, 2000; Lee and Blake, 1999). The extensive use of camouflage in the animal world (e.g., Thery and Casas, 2002) indicates the crucial visual role of figure-ground segregation. Even the best camouflage, however, breaks down when an animal is moving. Motion discontinuities between object and background provides important additional information for segregating images into separate surfaces (Nakayama and Loomis, 1974; Braddick, 1993; Regan, 2000). Surface segregation also defines regions within which motion signals should be integrated. Local motion signals are inherently ambiguous and often noisy. Perceiving the "veridical" motion of an object requires spatial integration of motion signals. It is critically important, however, to integrate only the motion signals arising from the same surface. Integrating motion signals from different surfaces will necessarily degrade motion perception. Constraining motion integration within object boundaries depends on figure-ground segregation.

292


14.3.1 Psychophysics of Motion-Based Figure-Ground Segregation Everyday experience suggests that we are very good at detecting objects moving against a background. We wave our arms when we want to be seen, and we stand still when we want to hide. Psychophysical observations accord with our intuitions. A single moving object immediately pops out from the background and strongly attracts attention (e.g., Dick et al., 1987). Evidence suggests that motion can be as good (Regan, 1989) and sometimes even better (Nawrot et al., 1996) than other visual cues in segregating figure from ground. Regan and his colleagues have extensively studied the perception of motion-defined form over the past two decades (reviewed in Regan, 2000). A general conclusion from Regan's research is that vision is very efficient in detecting, discriminating, and recognizing motion-defined objects. Baker and Braddick (1982) found that observers could effortlessly discriminate 2D shapes defined solely by differential motion from a random-dot background. Subsequent experiments by Regan showed that the perception of motion-defined forms is often as good as the perception of luminance-defined forms. At high contrast and with fast motions, orientation and vernier discriminations for luminance- and motiondefined forms are comparable (Regan, 1986,1989; Sary et al., 1994). Discriminations of aspect ratios of motion- and luminance-defined forms are also very similar (Regan and Beverley, 1984; Regan and Hamstra, 1991). This impressive sensitivity to motion-defined forms deteriorates quickly, however, if stimulus parameters such as speed and contrast are far from optimal values (Regan, 1989; Lappin et al., 2003). Motion-defined forms also have to be larger and longer in duration to match visual sensitivity for luminance-defined forms (Regan and Beverley, 1984). Conflicts Between Spatial Integration and Differentiation of Motion Signals Highlighting differences between local motion signals, however, is not always adaptive. Different local motion signals often belong to the same object, and should be integrated rather than differentiated. Biological motion patterns are a good illustration. Integration and segregation of motion signals can be guided by other visual cues (Rivest and Cavanagh, 1996; Croner and Albright, 1997,1998) and even form information (Lorenceau and Alais, 2001). Another solution that is independent of other visual submodalities may be to determine the spatial extent of motion integration based on the local motion signals. The strength and quality of motion signals can be substantially reduced at low contrast, by noise, or when defined by color. Apparent differences in the direction and velocity of local motion signals may be caused by noise, and the spatial segregation of such signals may lead to incorrect perception. Moreover, under such low-visibility conditions, motion patterns might require spatial integration just to be perceived at all. On the other hand, when motion signals are strong, spatial variations in the directions and speeds detected by local mechanisms are more likely to reflect the "true" motion pattern. Under conditions of good visibility, spatial differentiation should be favored. If the spatial organization of motion signals is adapted in this way to the visibility conditions, then it may also fail under some conditions. Lorenceau and Shiffrar (1992, 1999) have shown that the perceptual integration of moving contours belonging to a


293

Figure 14.5: An illustration of a 2D motion-defined shape. The shape is visible because the motion within the oval region is different from the background motion (the oval outline is only for illustration). An observer's task was to discriminate the orientation of the oval shape. single rigid form seen through separate apertures is more likely under low contrast, eccentric viewing, isoluminant, and noisy conditions. Correspondingly, however, such moving contours are less likely to be correctly integrated when the local visibility increases. Moreover, vision also makes the complementary error of failing to segregate motion signals from different objects. Regan and Beverley (1984) and Regan (1989) found that motion-defined form discrimination is impaired when the strength of the motion signals is degraded, even when motion sensitivity is relatively unaffected. Lappin et al. (2003) found that discriminations of motion directions and of motion-defined forms are oppositely affected by variations in contrast - direction discriminations decreasing and form discriminations increasing with greater contrast. The trade-off between these two tasks suggests that spatial integration and differentiation of motion signals are adaptively controlled by local visibility conditions. Psychophysical Links Between Figure-Ground Segregation and Center-Surround Mechanisms Vision is very good at detecting relative motion and segregating surfaces, but it is an open question whether these visual abilities derive from centersurround mechanisms. Intuitively, center-surround mechanisms seem likely to be involved in motion-based figure-ground segregation, but this link needs experimental support. Psychophysical evidence indicates that surround suppression varies with the stimulus conditions (Tadin et al., 2003). If figure-ground discrimination depends on center-surround mechanisms, then figure-ground discriminations should be accurate in stimulus conditions with strong surround suppression and impaired in conditions with reduced surround suppression.

294


Figure 14.6: Duration thresholds for motion direction and figure-ground discriminations as a function of stimulus contrast.

Lappin et al. (2003) found such a relationship between surround suppression and motion-defined form discriminations using the task illustrated in figure 14.5. Tadin et al. (2003) had found previously that duration thresholds for discriminating the directions of large patterns increased with contrast, indicating that surround suppression increases with contrast (figure 14.6). In the form-discrimination task of Lappin et al., the same contrast increase yielded improved performance. At the highest contrast (black arrow), form discrimination was better than the direction discrimination of the same pattern. Interestingly, the improvements in form discrimination with increasing contrast were approximately equal to the decrements in direction discrimination, so that the two functions were nearly symmetrical around the horizontal dashed line in figure 14.6. Additional experiments are needed to clarify the relation between these two aspects of motion perception. The visual ability to accurately perceive motion-defined forms may seem surprising in relation to the supposed physiological separation between mechanisms for form and motion processing. The fact that motion-defined forms pop out from the background and attract attention (typical dorsal stream functions) does not imply that shape characteristics such as orientation and aspect ratio will be accurately perceived. The finding that motion cues are sufficient for perceiving both 2D and 3D shape is remarkable and suggests an interesting interaction between motion processing and ventral stream functions. The visual complexity and diverse phenomenology of motion-based form perception all but guarantees that its neural correlates will involve multiple neural mechanisms in multiple visual areas.


295

14.3.2 Neurophysiology of Motion-Based Figure-Ground Segregation Given the visual sensitivity to relative motion and motion-defined forms, specialized neural mechanisms probably operate to detect differences in the spatial distribution of motion signals. Because of the high proportion of center-surround neurons in MT, and because of its central location within anatomical pathways of motion perception, MT seems likely to be involved in segregating figure from ground and perceiving motiondefined form. From the outset, we emphasize that MT mechanisms are unlikely to be involved in all aspects of motion-based figure-ground segregation. As is discussed below, MT is well equipped for segregating figure from ground but lacks mechanisms to directly extract detailed 2D shape of motion-defined forms. This distinction is in agreement with clinical evidence demonstrating that the detection of motion-defined forms can remain intact even when the identification of such forms is severely impaired (Regan et al., 1992; Schenk and Zihl, 1997; Cowey and Vaina, 2000). Neural mechanisms for figure-ground segregation of moving forms are discussed first, followed by the discussion of how 2D shape of such forms may be extracted. Segregation of Moving Objects from the Background Moving objects must first be detected and then- motion estimated. In principle, this can be done with little regard for detailed 2D shape. Once a moving object is detected and foveated, usually it can be recognized based on cues other than motion, as most objects are not perfectly camouflaged. Thus, the detection of moving objects is useful whether or not such objects can be recognized based on motion cues alone. The responses of center-surround MT neurons amplify the neural signature of objects moving relative to their background. The question, however, is whether such a simple mechanism is sufficient to support our ability to effortlessly segregate moving objects from the background. The observation that center-surround neurons are excited by relative motion and suppressed by uniform motion suggests a link between surround suppression and figureground segregation. That is, suppression occurs when the center and surround are stimulated by the motion of a relatively large visual feature. For this mechanism to be efficient, it should not be inhibited when different visual features stimulate the surround and center regions even if they are moving in the same direction. This may occur, for example, when two objects at different depths move with the same angular velocity or when the observer is moving and fixating at a point more distant then a moving object. In such cases, center and surround regions of some MT neurons will be stimulated by similar motion arising from different objects, resulting in response suppression. Because this suppression would be caused by object motion, it would somewhat diminish the ability of MT neurons to contribute to figure-ground segregation. Most MT cells, however, are disparity selective (DeAngelis and Newsome, 1999), a tuning property that may be exploited for "inhibiting" surround suppression if center and surround motions are at different depths. This hypothesis was investigated by Bradley and Andersen (1998), who found that the disparity tuning of center and surround regions tend to be different. That is, a neuron that is typically suppressed by a surround moving in its preferred direction becomes unsuppressed if the center and surround motions are at different depths. Surround suppression increased as either surround motion or its

296


depth became more similar to motion in the center. The disparity dependence of surround suppression indicates that MT neurons are modulated by motion fields arising from a single surface, but are unaffected by the motions of other surfaces at different depths. This "elaborated" surround suppression improves the ability of MT neurons to efficiently segregate moving objects from the background. In addition to detecting moving objects, our visual system must correctly estimate object speed and trajectory. This is critical, for example, in accurately foveating the moving object and controlling subsequent pursuit eye movements - skills essential in sports like baseball and cricket. Center-surround neurons may signal the presence of a moving object but cannot also signal its velocity because the responses of centersurround neurons are also influenced by the background speed and direction (Allman et al., 1985b). The responses of center-surround neurons, however, might be disambiguated by the neural signal representing the speed and the direction of background motion - information encoded by wide-field MT neurons (figure 14.2B). The hypothesis that center-surround and wide-field neurons jointly code object motion has received direct support from recent microstimulation experiments (Born et al., 2000). These researchers have exploited the fact that center-surround and wide-field neurons are anatomically segregated (figure 14.2C; Born and Tootell, 1992) and can be separately stimulated. Monkeys were trained to fixate a stationary target. A moving target then appeared in the periphery and the animal's task was to make a foveating saccade and visually pursue the target. On half of the trials, microstimulation was applied while the animal was estimating the direction and speed of the moving target. Microstimulation of MT sites with the center-surround neurons shifted pursuit eye movements in the direction similar to the preferred direction of the stimulated clusters of neurons. In contrast, microstimulation of the MT sites with wide-field neurons shifted pursuit eye movements in the direction opposite to the preferred direction of the stimulated neurons. These results suggest that the activity of center-surround MT neurons represents object motion whereas the activity of wide-field neurons signals background motion. Importantly, replacing microstimulation with large background motion had an effect similar to that of stimulating wide-field neurons. Neurophysiology of Motion-Defined 2D Shape Once a moving object is detected and visually segregated from its background, motion information can be used to perceive its 2D shape (Regan, 2000). Detailed motion-defined shape is conveyed by kinetic (motion-defined) boundaries - a building block (akin to edges) of motion-defined objects. One strategy for investigating neural mechanisms involved in perceiving the shapes of motion-defined objects is to look for neurons and brain areas with selectivity for kinetic boundaries. Brain imaging studies have found that MT responds strongly to kinetic boundaries, but this response does not differ from MT's response to uniform motion (Reppas et al., 1997; Van Oostende et al., 1997; Dupont et al., 1997; for an exception see Shulman et al., 1998). Reppas et al. (1997) have also shown that several early visual areas are activated by kinetic boundaries, but this activity is unlikely to be specific to motion-defined form because such areas are also activated by other types of boundaries (Leventhal et al., 1998). Orban and colleagues have suggested that the kinetic occipital area (KO)

Duje Tadin and Joseph S, Lappin

297

is an area specialized for processing of kinetic boundaries (Van Oostende et al., 1997; Dupont et al., 1997). Recently, however, KO has been shown to respond to boundaries denned by cues other than motion (Zeki et al., 2003). So far, then, imaging studies have not revealed whether MT or other cortical areas are specialized for processing kinetic boundaries. Surgical lesions of area MT (and adjacent regions) in nonhuman primates have produced conflicting results about the importance of MT in processing kinetic boundaries, with postlesion impairments ranging from mild (Schiller, 1993; Lauwers et al., 2000) to severe (Marcar and Cowey, 1992). Single cell results, however, are more consistent and show that single MT neurons are not selective for the orientation or location of kinetic boundaries (Marcar et al., 1995). MT neurons generally respond very poorly to kinetic boundaries. In fact, MT neurons respond as weakly to kinetic boundaries as they do to transparent motion (Snowden et al., 1991; Bradley et al., 1995). It should be emphasized, however, that neurons in other visual areas (primarily V2) are selective for the orientation of motion-defined boundaries (Marcar et al., 2000; Leventhal et al., 1998). V2 neurons tuned to the orientation of kinetic boundaries often exhibited similar orientation tuning to luminance edges, resulting in cue-invariant responses to visual boundaries (Marcar et al., 2000). Notably, the response to kinetic boundaries was delayed by about 40 ms (relative to the luminance boundary response), suggesting the role of cortical feedback. One possibility is that this feedback may arise from neural mechanisms sensitive to the coarse 2D shape of moving objects. The possibility that MT may contain such mechanisms is discussed next. Single MT neurons are not tuned to kinetic boundaries (Marcar et al., 1995), but the population response in MT might carry the neural signature associated with the coarse 2D shape of motion-defined objects (Snowden, 1994). Consider a population of antagonistic center-surround neurons responding to a kinetic edge (figure 14.7). Neurons with receptive fields stimulated by the kinetic edge will be suppressed due to the multiple motions within their receptive field center (Snowden et al., 1991; Marcar et al., 1995). Neurons far from the motion boundary will be suppressed due to the surround inhibition (Allman et al., 1995b). Thus, the center-surround neurons in the regions flanking the boundary will be most active within the population of neurons responding to the motion-defined edge. The emerging result is the segmentation of regions containing uniform or near-uniform motion. This coding scheme may be a part of a process that detects areas of near-uniform motion and then "draws" boundaries around such regions. Such a process is described as region- or continuity-based image segmentation (as contrasted with edge-based segmentation; Mbller and Hurlbert, 1996). We emphasize, however, that the proposed population-coding scheme (figure 14.7) is speculative. Interestingly, MT seems to rely on a population code to represent transparent motion (Treue et al., 2000) - a class of stimuli that, just as motion-defined boundaries, is composed of multiple motion directions. Furthermore, psychophysical evidence suggests that in some stimulus conditions, motion perception seems to rely on region-based segmentation algorithms. For example, reducing the salience of a motion-defined edge by introducing a gradual rather than an abrupt change in velocity was found to have very little effect on the ability to segment surfaces based on their motion (Smith and Curran, 2000; M611er

298


Figure 14.7: Illustration of how a population of hypothetical center-surround neurons would respond to a motion-defined boundary. "S" marks the receptive fields of neurons whose response would be suppressed, "LS" marks the neurons whose response would be less suppressed, and "NS" marks the neurons that would not be suppressed. Note that MT neurons with appropriately located asymmetric surrounds (see figure 14.8B; Xiao et al., 1995, 1997a) would give the strongest response to a motion boundary (marked with "NS"). and Hurlbert, 1996). M611er and Hurlbert (1996) also demonstrated that increasing the width of a motion-defined figure increased its visibility even when the detectability of its edges was kept constant. This effect was most pronounced at brief (

Seeing Spatial Form

The Idea of Spatial Form

Seeing

Seeing

Seeing

Seeing

Seeing

Seeing

Seeing

Seeing

Seeing Julia

Seeing Stars

Seeing Stars

Seeing Stars

Seeing Voices

Seeing Blind

Seeing Eye

Seeing Red

Seeing Stars

Seeing Stars

Seeing Stars

Seeing Blind

Seeing Redd

Seeing Further

True Seeing

Seeing Love

Seeing Red

Seeing I

Seeing I

Seeing Spots

Seeing Redd

Seeing Spatial Form

The Idea of Spatial Form

Seeing

Seeing

Seeing

Seeing

Seeing

Seeing

Seeing

Seeing

Seeing Julia

Seeing Stars

Seeing Stars

Seeing Stars

Seeing Voices

Seeing Blind

Seeing Eye

Seeing Red

Seeing Stars

Seeing Stars

Seeing Stars

Seeing Blind

Seeing Redd

Seeing Further

True Seeing

Seeing Love

Seeing Red

Seeing I

Seeing I

Seeing Spots

Seeing Redd

Recommend Documents