The Visual Neurosciences. Volumes 1,2

THE VISUAL NEUROSCIENCES

THE VISUAL NEUROSCIENCES Edited by Leo M. Chalupa and John S. Werner Editorial Advisory Board:

Colin Barnstable Ralph Freeman Lamberto Maffei John Maunsell Robert Shapley Murray Sherman Lothar Spillmann Mriganka Sur David I. Vaney

A BRADFORD BOOK THE MIT PRESS CAMBRIDGE, MASSACHUSETTS LONDON, ENGLAND

© 2004 Massachusetts Institute of Technology All rights reserved. No part of this book may be reproduced in any form by any electronic or mechanical means (including photocopying, recording, or information storage and retrieval) without permission in writing from the publisher. This is a work in two volumes, not sold separately. This ISBN refers to the set and is therefore used to identify both volumes. This book was set in Baskerville by SNP Best-set Typesetter Ltd., Hong Kong and was printed and bound in the United States of America. Library of Congress Cataloging-in-Publication Data The visual neurosciences / edited by Leo M. Chalupa and John S. Werner. p. ; cm. “A Bradford book.” Includes bibliographical references and index. ISBN 0-262-03308-9 1. Visual pathways. 2. Visual cortex. 3. Visual perception. 4. Neurosciences. I. Chalupa,Leo M. II. Werner, John Simon. [DNLM: 1. Vision–physiology. 2. Neurosciences– methods. 3. Visual Perception–physiology. WW 103 V83117 2003] QP475.V274 2003 612.8¢4–dc21 2003056137

CONTENTS

Preface

I

xiii

HISTORICAL FOUNDATIONS

1

1.

Vision Structure and Function: The Early History

2.

The Role of Single-Unit Analysis in the Past and Future of Neurobiology Horace Barlow 14

II

DEVELOPMENTAL PROCESSES

Mitchell Glickstein

3

31

3.

Molecular Regulation of Vertebrate Retinal Development Colin J. Barnstable 33

4.

Neurotrophins, Electrical Activity, and the Development of Visual Function Nicoletta Berardi and Lamberto Maffei 46

5.

Developmental and Genetic Control of Cell Number in the Retina Robert W. Williams and Sally A. Moody 63

6.

Development of the Vertebrate Retina Leanne Godinho 77

7.

The Development of Retinal Decussations Lynda Erskine 94

8.

The Development of Eye-Specific Segregation in the Retino-Geniculo-Striate Pathway Barbara Chapman 108

Rachel O. L. Wong and

Carol Mason and

v

9.

The Role of Neural Activity in the Development of Orientation Selectivity Chiayu Chiu and Michael Weliky 117

10.

Mechanisms of Plasticity in the Visual Cortex

11.

Ontogenesis of Cortical Connectivity Andreas Burkhalter 146

12.

Neural Limitations on Visual Development in Primates J. Anthony Movshon 159

13.

Development of Spatial Selectivity and Response Timing in Humans Anthony M. Norcia 174

14.

The Effects of Selected Forms of Early Visual Deprivation on Perception Donald E. Mitchell 189

15.

Toward a Future for Aging Eyes

III

126

Nigel W. Daw

Henry Kennedy and

Lynne Kiorpes and

205

R. A. Weale

RETINAL MECHANISMS AND PROCESSES

213

16.

Visual Transduction by Rod and Cone Photoreceptors Trevor D. Lamb 215

17.

How Retinal Circuits Optimize the Transfer of Visual Information Peter Sterling 234

18.

ON and OFF Pathways in the Vertebrate Retina and Visual System Ralph Nelson and Helga Kolb 260

19.

Retinal Synapses

20.

Retinal Neurotransmitters

21.

Excitation in the Retina: The Flow, Filtering, and Molecules of Visual Signaling in the Glutamatergic Pathways from Photoreceptors to Ganglion Cells David R. Copenhagen 320

22.

Peptide and Peptide Receptor Expression and Function in the Vertebrate Retina Nicholas C. Brecha 334

23.

Inhibition in the Retina

24.

Anatomy, Circuitry, and Physiology of Vertebrate Horizontal Cells Ido Perlman, Helga Kolb and Ralph Nelson 369

25.

Retinal Amacrine Cells

26.

Ganglion Cells in Mammalian Retinae Ulrike Grünert 410

27.

Retinal Ganglion Cell Excitability

vi



Marie E. Burns and

279

Martin Wilson

304

Robert E. Marc

Malcolm M. Slaughter

David I. Vaney

355

395 Paul R. Martin and

Andrew T. Ishida

422

28.

Direction Selectivity in Retinal Ganglion Cells

29.

Spatial Regularity among Retinal Neurons

IV

Richard H. Masland 463

Jeremy E. Cook

ORGANIZATION OF VISUAL PATHWAYS

479

30.

The M, P, and K Pathways of the Primate Visual System Ehud Kaplan 481

31.

Parallel Visual Pathways: A Comparative Perspective Xiangmin Xu 494

32.

Organization of Visual Areas in Macaque and Human Cerebral Cortex David C. Van Essen 507

33.

Communications between Cortical Areas of the Visual System 522

34.

Ventral and Dorsal Cortical Processing Streams Pasternak 541

V

SUBCORTICAL PROCESSING

451

Vivien A. Casagrande and

Jean Bullier

Leslie G. Ungerleider and Tatiana

563

35.

The Visual Relays in the Thalamus R. W. Guillery 565

36.

The Visual Functions of the Pulvinar

37.

Feedback Systems in Visual Processing Helen E. Jones 609

38.

Light Responsiveness and Photic Entrainment of the Mammalian Circadian Clock Johanna H. Meijer and Joseph S. Takahashi 625

39.

Learning from the Pupil: Studies of Basic Mechanisms and Clinical Applications John L. Barbur 641

40.

Blindsight

VI

Larry Weiskrantz

S. Murray Sherman and

Christian Casanova

592

Adam M. Sillito and

657

PROCESSING IN PRIMARY VISUAL CORTEX

671

41.

Functional Connectivity in the Pathway from Retina to Striate Cortex R. Clay Reid and W. Martin Usrey 673

42.

Cell Types and Local Circuits in Primary Visual Cortex of the Macaque Monkey Edward M. Callaway 680



vii

43.

Assembly of Receptive Fields in Primary Visual Cortex

44.

A Modern View of the Classical Receptive Field: Linear and Nonlinear Spatiotemporal Processing by V1 Neurons Gregory C. DeAngelis and Akiyuki Anzai 704

45.

Beyond the Classical Receptive Field: Contextual Modulation of V1 Responses Victor A. F. Lamme 720

46.

Contributions of Vertical and Horizontal Circuits to the Response Properties of Neurons in Primary Visual Cortex Thomas R. Tucker and David Fitzpatrick 733

47.

Nonlinear Properties of Visual Cortex Neurons: Temporal Dynamics, Stimulus Selectivity, Neural Performance Duane G. Albrecht, Wilson S. Geisler and Alison M. Crane 747

48.

Binocular Interaction in the Visual Cortex

49.

From Binocular Disparity to the Perception of Stereoscopic Depth Andrew J. Parker 779

793

50.

Formation and Acquisition of the Retinal Image Hofer 795

51.

Thresholds and Noise

52.

Ideal Observer Analysis

53.

Scotopic Vision

54.

Visual Adaptation

55.

Rod-Cone Interactions in Human Vision

825

Wilson S. Geisler

Adam Reeves

851

VIII BRIGHTNESS AND COLOR

David R. Williams and Heidi

811

Theodore E. Cohn

838

Steven L. Buck

Brightness and Lightness

57.

Color Appearance

58.

Chromatic Discrimination

59.

The Role of Color in Spatial Vision

60.

Pattern-Selective Adaptation in Color and Form Perception Michael A. Webster 936



863

879

56.

viii

695

765

Ralph D. Freeman

VII DETECTION AND SAMPLING

Walter Makous

David Ferster

Adriana Fiorentini

881

Kenneth Knoblauch and Steven K. Shevell

892 908

Joel Pokorny and Vivianne C. Smith Karen K. De Valois

924

61.

Color Constancy

62.

Comparative Color Vision

63.

Molecular Genetics of Human Color Vision and Color Vision Defects Maureen Neitz and Jay Neitz 974

64.

Linking Retinal Circuits to Color Opponency

65.

Neural Coding of Color

66.

The Processing of Color in Extrastriate Cortex Daniel C. Kiper 1017

67.

Improbable Areas in Color Vision

IX

948

David H. Brainard

962

Gerald H. Jacobs

989

David J. Calkins 1003

Russell L. De Valois

Karl R. Gegenfurtner and

1029

Semir Zeki

FORM, SHAPE, AND OBJECT RECOGNITION

1041

68.

Spatial Scale in Visual Processing

1043

69.

Spatial Channels in Vision and Spatial Pooling Frances Wilkinson 1060

70.

Contour Integration and the Lateral Connections of V1 Neurons David J. Field and Anthony Hayes 1069

71.

Shape Dimensions and Object Primitives

72.

Shape and Shading

73.

Visual Perception of Texture

74.

Visual Segmentation and Illusory Contours Dario Ringach 1119

75.

Global Yet Early Processing of Visual Surfaces Shinsuke Shimojo 1129

76.

Image Parsing Mechanisms of the Visual Cortex 1139

77.

Inferotemporal Response Properties

78.

Invariant Object and Face Recognition

79.

The Ventral Visual Object Pathway in Humans: Evidence from fMRI Nancy Kanwisher 1179

Robert F. Hess

Hugh R. Wilson and

Charles E. Connor

1080

Jan J. Koenderink and Andrea J. van Doorn

1090

Michael S. Landy and Norma Graham

1106

Robert Shapley, Nava Rubin and

Yukiyasu Kamitani and

Keiji Tanaka

Rüdiger von der Heydt

1151

Edmund T. Rolls

1165



ix

X

MOTION, DEPTH, AND SPATIAL RELATIONS

1191

80.

Motion Cues in Insect Vision and Navigation Shaowu Zhang 1193

81.

The Middle Temporal Area: Motion Processing and the Link to Perception Kenneth H. Britten 1203

82.

Merging Processing Streams: Color Cues for Motion Detection and Interpretation Karen R. Dobkins and Thomas D. Albright 1217

83.

Functional Mapping of Motion Regions Wim Vanduffel 1229

84.

Optic Flow

85.

The Cortical Analysis of Optic Flow

86.

The Perceptual Organization of Depth Barton L. Anderson 1284

87.

Stereopsis

88.

Binocular Rivalry

89.

Sensorimotor Transformation in the Posterior Parietal Cortex Hansjörg Scherberger and Richard A. Andersen 1324

XI

Charles J. Duffy

1260

Roland Fleming and

1300 1313

Randolph Blake

EYE MOVEMENTS

Guy A. Orban and

1247

William H. Warren

Clifton M. Schor

Mandyam Srinivasan and

1337

90.

Gaze Control under Natural Conditions

91.

Eye Movements in Daily Life

92.

Selection of Targets for Saccadic Eye Movements

93.

Visual Perception during Saccades M. Concetta Morrone 1391

94.

Smooth Pursuit Eye Movements: Recent Advances Edward L. Keller 1402

95.

Neural Control of Vergence Eye Movements

96.

The Primate Frontal Eye Field Charles J. Bruce, Harriet R. Friedman, Michael S. Kraus and Gregory B. Stanton 1428

97.

Changing Views of the Role of Superior Colliculus in the Control of Gaze Neeraj J. Gandhi and David L. Sparks 1449

x



Robert M. Steinman

1339

1357

Michael F. Land

Jeffrey D. Schall

1369

David C. Burr and

Stephen J. Heinen and

Lawrence E. Mays

1415

98.

The Dialogue between Cerebral Cortex and Superior Colliculus: Implications for Saccadic Target Selection and Corollary Discharge Marc A. Sommer and Robert H. Wurtz 1466

99.

Cerebellar Control of Eye Movements Mark F. Walker 1485

David S. Zee and

XII ATTENTION AND COGNITION

1499

100.

Visual Perception and Cognition in Honeybees Mandyam Srinivasan 1501

101.

A Neural Basis for Human Visual Attention

102.

Neural and Behavioral Measures of Change Detection Michael Silverman 1524

103.

The Role of Attention in Visual Cerebral Cortex 1538

104.

Volition and the Prefrontal Cortex Jonathan D. Wallis 1546

Shaowu Zhang and

Sabine Kastner

1514

Daniel J. Simons and

John H. R. Maunsell

Earl K. Miller and

XIII THEORETICAL AND COMPUTATIONAL PERSPECTIVES 1561 105.

The Evolution of the Visual System in Primates

106.

Gestalt Factors in the Visual Neurosciences Walter H. Ehrenstein 1573

107.

Neural Mechanisms of Natural Scene Perception

108.

Principles of Image Representation in Visual Cortex 1603

109.

Local Analysis of Visual Motion

110.

Visual Boundaries and Surfaces

111.

How the Visual Cortex Recognizes Objects: The Tale of the Standard Model Maximilian Riesenhuber and Tomaso Poggio 1640

Jon H. Kaas

1563

Lothar Spillmann and

Eero P. Simonocelli Stephen Grossberg

Jack L. Gallant

1590

Bruno A. Olshausen

1616 1624



xi

112.

Plasticity of Orientation Processing in Adult Visual Cortex Mriganka Sur 1654

113.

Synchrony, Oscillations, and Relational Codes

114.

The Neuronal Basis of Visual Consciousness Francis Crick 1682

List of Contributors Index

xii

I1



C1

Valentin Dragoi and

Wolf Singer Christof Koch and

1665

PREFACE Perhaps the most remarkable thing about vision is the utter simplicity of the act of seeing. We open our eyes and a three-dimensional panorama of colored images—some stationary, others in motion—unfolds before us. In most cases, the brain makes sense of this information seemingly instantaneously, allowing us to function reasonably well under a wide range of lighting conditions. The retinal image is constantly changing as we move about and yet the objects around us are perceived as stable. This seemingly effortless nature of sight beguiles the profound complexity of the processes underlying the perception of even the simplest visual stimulus. Indeed, no machine can currently perform the myriad visual recognition tasks we normally take for granted, and it is still unclear whether such technology will become available in the foreseeable future. Vision is the dominant sense in humans and other primates, with nearly 30% of our cortical surface representing information that is predominantly visual. Reflecting the importance of vision to the formation of human experience, more effort has gone into studying the visual system than any other sensory modality. As a consequence, we have accumulated an impressive amount of information about vision at many different levels, ranging from genes and molecules to theoretical computations. Our long-term objective is to explain how the brain transforms the spatiotemporal patterns defined by the photons impinging on the retina at any given moment into a coherent visual world. The information derived from understanding these basic processes will ultimately help us prevent and treat the many disorders that impair our ability to see. Almost 10% of people living today suffer from a visual disorder stemming from a defect of the retina or the visual centers of the brain. Effective treatment of these visual impairments is possible in only a few types of cases because we lack the basic knowledge to understand the dysfunction underlying these disorders. Although we have made significant progress in the visual neurosciences, much remains to be done. The scope of the overall effort has intensified in recent years, reflecting in part, the advent of new technologies, ranging from those of modern molecular biology to the functional imaging of the human brain. Such methodologies have now made it possible to pursue a host of previously unanswerable questions. There is a plethora of professional journals devoted to vision research, and a number of excellent books dealing with perception as well as the neural bases of vision. No single source, however, has attempted to provide a comprehensive and authoritative account of the visual neurosciences. In an attempt to remedy this situation, we invited 100 of the world’s leading researchers in this field to summarize their area of specialization in a manner understandable to the nonspecialist. The response by our colleagues was immensely gratifying. Virtually everyone invited agreed to participate, and some suggested the inclusion of additional chapters, so the final number of contributions was increased to 114. Each chapter was reviewed by other experts, and authors made revisions based on their feedback. As editors we strove to preserve the individual “voice” of each author, and we also agreed to tolerate a certain degree of redundancy across chapters, provided they

xiii

offered valuable insights into the topic under consideration. The Visual Neurosciences is a work in progress so some disagreement was expected among authors regarding specific issues. We made little attempt to broker a compromise between dissimilar viewpoints held by different authors, as long as these were supported by empirical evidence. Controversy is what often makes science fun, and we leave it for future generations to decide the relative merits of currently held positions. The Visual Neurosciences begins with two historical chapters and an appraisal of the prospects for single-unit approaches in neurobiology. They are followed by Chapters 3–15 on Developmental Processes. This section, as in the book as whole, is organized from molecules to pathways to systems. The section on Retinal Mechanisms and Processes (Chapters 16–29) presents the current state of knowledge on phototransduction, retinal synapses, and physiology, with authors explaining how these mechanisms ostensibly optimize the processing of visual information. These chapters set the stage for the next section on the Organization of Visual Pathways (Chapters 30–34) and the subsequent elaboration of projections for Subcortical Processing (Chapters 35–40) and for Processing in Primary Visual Cortex (Chapters 41–49). Most of the chapters in these first six sections provide an anatomical and physiological context for understanding the psychophysical, perceptual, and neurophysiological chapters that follow in the next four sections, beginning with Detection and Sampling (Chapters 50–55) and proceeding to higher-level processing of Brightness and Color (Chapters 56–67), Form, Shape, and Object Recognition (Chapters 68–79), and Motion, Depth, and Spatial Relations (Chapters 80–89). These chapters illustrate how 20th Century neuroscience unraveled many phenomenological conundrums of the 19th Century. Of course, 20th Century psychology raised still other challenges for neuroscience, including the role of nonsensory variables in perception and cognition. Sections on Eye Movements (Chapters 90–99) and Attention and Cognition (Chapters 100–104) address these questions with detailed accounts of the coordination of eye position and information processing by subcortical and cortical circuits underlying cognitive phenomena. The final section, Theoretical and Computational Perspectives (Chapters 105–114), provides an integration of ideas from neuroscience, psychology, and computer science that are likely to guide future discoveries in the visual neurosciences. For an undertaking of this scope, the entire project went remarkably smoothly. For this we thank all of the authors for adhering good-naturedly (in most cases) to the rather tight schedule. We also thank the countless anonymous reviewers, members of the Editorial Advisory Board for their input at all stages of this undertaking, and Barbara Murphy, our editor at the MIT Press, for her support and keen professional advice. It is our hope that The Visual Neurosciences will serve to motivate and inspire the next generation of researchers, whether they are currently beginning students, clinical practitioners, or established researchers in other fields of endeavor. Leo M. Chalupa and John S. Werner 7 January 2003

xiv



I HISTORICAL FOUNDATIONS

1

Vision Structure and Function: The Early History MITCHELL GLICKSTEIN

Introduction This chapter deals with the early history of the study of visual processing by the eye and brain. I begin by considering the first recognition of how images are formed in the vertebrate eye and early contributions to understanding the structure and function of the retina. I go on to discuss the connections from the retina to the cortex by way of the lateral geniculate nucleus (LGN), and the evidence that led to the recognition and spatial mapping of the visual fields on the primary visual cortex. Finally, I describe some of the studies that began to reveal the multiplicity of visual cortical areas and their functions. Because of space limitations, many interesting aspects of the history, such as color vision, visual reflexes, subcortical visual structures, and the controversies over the interpretation of macular sparing after visual cortex lesions must be beyond the scope of this chapter. The emphasis is on fundamentals of structure and its relation to visual function.

Image formation The cornea and lens form an inverted image of the visual scene at the back of the eye. The optics of image formation in the vertebrate eye were unknown until the theoretical and experimental advances of the seventeenth century. Prior to that time, scientists were troubled by the idea of an upsidedown image in the eye even though they knew of the camera obscura, a dark chamber with a pinhole aperture that admits light and forms an inverted image of an illuminated scene. As early as the eleventh century, Ibn Al-Haithem, an Arab scholar (cited in Polyak, 1941), wrote a treatise in which the principles of image formation by the camera obscura were clearly described. Even though the optics of the camera obscura appeared to be similar in some ways to that of the eye, inversion of the image troubled earlier thinkers. Leonardo da Vinci (sixteenth century; Windsor Collection) tried to construct a scheme whereby an inverted image would first be received somewhere in the lens and then reinverted to form an upright image at the back of the eye. He wrote: No image, of however small a body, penetrates into the eye without being turned upside-down and, in penetrating the crystalline sphere, it will be turned the right way again.

The true nature of image formation by the eye was first put forward by Kepler (1604) on theoretical grounds and confirmed experimentally by Scheiner (1619 and 1652), who removed some of the opaque tissue at the back of an excised eye and directly demonstrated the inverted image. The principle of image formation by the human eye was beautifully illustrated by Des Cartes (1677; cited in Polyak, 1941).

The retina Following Kepler’s analysis and Scheiner’s demonstration, the inverted image became an accepted fact. There was, however, little understanding of how the pattern of light and darkness in the image is converted into a signal by the retina. Thomas Young (1802) speculated that there must be a finite number of receptor types, say three, which would be sufficient to account for human color vision. But the actual structure of the retinal elements remained poorly understood. Invention of the compound microscope in the early nineteenth century led to an explosion of new knowledge about the structure and function of tissues in general and of the retina in particular. Among the most important of the early contributors was Max Schultze (1866), who described clearly the three major cell layers of the retina, with special attention to the distribution and morphology of the rods and cones (Fig. 1.1). He noted that there is a predominance of thin, rod-like receptors in the retina of strongly nocturnal animals and of thicker, cone-like receptors in diurnal animals. On the basis of comparative evidence and the distribution of receptors in the human eye, Schultze suggested that the two distinct classes of receptors might be associated with vision under two different conditions of illumination. Schultze and his contemporaries were vague about the connections between the successive elements of the receptors and the ganglion cells. The prevailing view of nervous tissue in general, and the retina in particular, was that it has a reticular structure in which successive elements are continuous and fused. The three prominent cell layers of the retina were thought of as swellings on optic nerve fibers. The anatomic research of Santiago Ramon y Cajal changed that view. Cajal first saw an example of the then new Golgi

3

F 1.1. Schultze’s drawings illustrating the structure of the retinal elements, with special reference to the morphological difference between rods and cones. Cones and rods are shown with their fibers up to the inner nuclear layer of the human retina. All of the figures, with the exception of Fig. 8, are drawn at a magnification of 500¥. Figs. 1–8 are taken from teased pieces of retina which were placed in osmic acid for 24 hours (1 : 700). They were from a fresh healthy eye. Figures 9–12 were from an eye with an atrophied optic nerve which was hardened in Müller’s solution. a a always refer to the external limiting membrane, b the rods, c the cones, b¢ the rod nuclei on the inner part of the external granular layer, c¢ the cone nucleus, and d the inner nuclear layer. The outer segments of the cone are incomplete since they were shriveled by the osmic acid. The outer segments of the rods are shown as they would appear in a fresh condition. Fig. 1. From the peripheral region of the retina. The space between a and d is completely filled by the rod and cone nuclei (the latter are always adjacent to the external limiting membrane). In the figure, a place has been selected in which the individual rod nuclei are removed in order to make the course of the fibers which remain apparent throughout their entire length. The cone nuclei end in a cone-shaped swelling that breaks up into fine fibers at the upper border of the inner nuclear layer. The rod fibers, which have exquisite fine varicosities, also end in the internal granular layer in an expanded varicosity at the point at which they make connections. Fig. 2. The same elements from a region outside of the macula lutea. The fibers of the cones and rods have become measurably longer, but their associated nuclei remain in the same relative position, so that now in the external granular layer d, there is a region without nuclei that consists only of the radial fibers of the external granular layer, which can reach an even greater length than the figure illustrates. It is this same place which H. Müller says appears to arise as a thickening of the inner nuclear layer and which Henle calls the external fiber layer of the retina. Fig. 3. A region of the retina which is closer to the macula. In the inner part of the external granular layer there is a change

of direction of the fibers away toward the ora serrata. With a reduction in their number, the rods have the same course as the adjacent cone fibers. Otherwise, all courses are as previously described. Fig. 4. At the border of the macula lutea. The diagonal course of the rod and cone fibers is even more marked. Figs. 5–7. These figures show the cones in the macula lutea and the fovea centralis. a is the outer limiting membrane in all cases next to the cone nuclei. As was partly seen earlier, the cone nuclei appear to follow a radial course. The fibers become so long before they reach the inner nuclear layer that a complete depiction is not possible. The one illustrated is six times longer than the one in Fig. 4. The outer segments of the cones, as stated previously, are shriveled. Fig. 8. (a) A cone from the peripheral region of the retina fixed in osmic acid and enlarged 1000-fold. The outer segment is shriveled. The inner segment and the cone nucleus have a fine fibrous structure, somewhat like that of the substance of the central ganglion cells. This apparently stops at the nuclear swelling of the cone just under the external limiting membrane, only to reappear in the cone fiber, where it is continuous with the end swelling. (b) This is an equivalently magnified rod, but without its outer segment: b¢ is the nuclear portion of the rod fiber, the so-called rod nucleus. Figs. 9–12. These figures show cones and rods of the macula lutea and its surroundings from a thin retina hardened with Müller’s fluid and then teased with needles. The preparation is shown to illustrate the fact that even if the rods and cones themselves are not present, the nuclei of the rods and cones (b¢ and c¢) can be distinguished, and those connected to the thin cones of the fovea centralis are similar to the nuclei of peripheral cones. But the preparation is not suitable to illustrate the cone fibers, which may be attributable to their long immersion in Müller’s fluid or else to a pathological condition. The eye had been excised because of intercalary staphyloma, and showed atrophy of the optic nerve and ganglion cells. (From Schultze, 1866.)

staining technique when he visited his colleague in Madrid, Don Luis Simarro, in 1878. Struck by the beauty and promise of the method, he began to apply the Golgi staining technique systematically to the study of the vertebrate retina and brain. Cajal’s classic monograph on the retina was published in French (1892) and translated into English (1972) (Fig. 1.2). Cajal was convinced that the reticular theory of organization of the nervous system was wrong. The retina, as well as the brain and spinal cord, he argued, are made up of individual elements, later called neurons by Waldeyer. Neurons may touch one another, but they do not fuse. In his monograph, Cajal described in detail the major cell types in all three retinal layers. He emphasized that the direction of conduction is from the receptors, through the horizontal, bipolar, and amacrine cells of the inner nuclear layer, ultimately to the ganglion cells, whose axons constitute the optic nerve. Cajal’s descriptions have remained the basis for all subsequent anatomical studies.

the course of the optic nerves in the optic tracts, and he was also aware of differences in the pattern of decussation in animals with laterally placed eyes. But despite Newton’s scientific authority, the true picture failed to penetrate to the medical or biological literature. Over 100 years later, William Wollaston (1824), describing his own temporary hemianopia, wrote:

On the central course of the optic nerve fibers and the pattern of decussation in the chiasma

It is plain that the cord, which comes finally to either eye under the name of the optic nerve, must be regarded as consisting of two portions, one half from the right thalamus, and the other from the left thalamus nervorum opticorum. According to this supposition, decussation will take place only between the adjacent halves of the two nerves. That portion of the nerve which proceeds from the right thalamus to the right side of the right eye, passes to its destination without interference: and in a similar manner the left thalamus will supply the left side of the left eye with one part of its fibres, while the remaining half of both nerves in passing over to the eyes of the opposite sides must intersect each other, either with or without intermixture of their fibres.

Early anatomists saw a prominent nerve exiting the back of each eye directed toward the brain. It was usually assumed that the nerves originated in the brain and extended out to the eye. The fibers arising from each eye appeared first to unite and then to cross the midline in the X-shaped optic chiasm. With earlier techniques of crude dissection, the pattern of crossing was not clear, so the true picture was not accepted until the late nineteenth century. The rearrangement of fibers in the chiasm was briefly described by Isaac Newton in his second book on optics (1704). Newton wrote: Are not the Species of Objects seen with both Eyes united where the optick Nerves meet before they come into the Brain, the fibres on the right side of both Nerves uniting there, and after union going thence into the Brain in the Nerve which is on the right side of the Head, and the fibres on the left side of both nerves uniting in the same place, and after union going into the Brain in the nerve which is on the left side of the Head, and these two Nerves meeting in the Brain in such manner that their fibres make but one entire Species or Picture, half of which on the right side of the Sensorium comes from the right side of both Eyes through the right side of both optick Nerves to the place where the Nerves meet, and from thence on the right side of the Head into the Brain, and the other half on the left side of the Sensorium comes in like manner from the left side of both Eyes. For the optick Nerves of such Animals as look the same way with both Eyes (as of Men, Dogs, Sheep, Oxen & cet.) meet before they come into the brain, but the optick Nerves of such Animals as do not look the same way with both Eyes (as of Fishes and of the Chameleon) do not meet, if I am rightly informed.

Although he incorrectly assumed that the origin of the optic nerves is within the brain, Newton described correctly

It is now more than twenty years since I was first affected with the peculiar state of vision, to which I allude, in consequence of violent exercise I had taken for two or three hours before. I suddenly found that I could see but half the face of a man whom I met; and it was the same with respect to every object I looked at. In attempting to read the name JOHNSON over a door, I saw only SON; the commencement of the name being wholly obliterated to my view.

Unaware of Newton’s suggested scheme for the course of the optic nerves, Wollaston wrote:

Wollaston rediscovered hemidecussation by observing his own transient hemianopia. Despite Newton’s scientific authority and Wollaston’s evidence, the pattern of hemidecussation was still largely unrecognized. Wollaston’s report was cited one year later by a news item in the Boston Medical and Surgical Intelligencer as an isolated curiosity in the same paragraph that described a boy in Philadelphia who allegedly saw a candle flame upside down. As late as 1880, H. Charlston Bastian (1880), Professor of Pathological Anatomy and Medicine at University College London (I blush), could still write: Although the subject is by no means free from doubt and uncertainty, the weight of the evidence seems now most in favour of the view that decussation at the Optic Commissure is as complete in Man as it is known to be in lower Vertebrates.

In spite of Dr. Bastian’s opinion, within a few years the true picture was soon clarified. By the time Gowers wrote his Textbook of Neurology (1892), the pattern of decussation and visual loss associated with lesions of the optic tract, the visual radiations, or the striate cortex was widely accepted.

: :  

5

On the termination of the optic tract fibers in the lateral geniculate nucleus Vision, like all sensory inputs, except for olfaction, is relayed to the cerebral cortex by way of the thalamus. The thalamic relay for vision is the LGN. The LGN in humans and Old World monkeys has an obvious striped appearance, with six layers of neurons separated by interleaved fiber layers. Although, by the end of the nineteenth century, it was clear that the eye projects to the LGN, the pattern of termination of optic tract fibers was not well understood. The true picture was revealed by study of transneuronal atrophy and degeneration. Cells in the LGN that are deprived of their input from the eye shrink or die. Mieczyslaw Minkowski (1920), working in Zurich, studied the LGN of a monkey that had had one eye removed 8 months earlier and that of a 75-year-old woman who had had amblyopia due to a unilateral cataract for 38 years before she died. Minkowski saw that cells in the LGN layers opposite the blind eye, layers 1, 4, and 6, were atrophied. In the ipsilateral LGN, layers 2, 3, and 5 were affected. The technique of studying transneuronal atrophy has revealed the organization of the LGN in a large number of mammals. The six-layered pattern is virtually identical in the apes and the Old World primates.

6

 

In some cases, the existence of a hidden laminar pattern can be revealed by transneuronal atrophy. For example, in the squirrel monkey, Saimiri, the dorsal parvocellular region of the LGN is not obviously laminated. One year after unilateral enucleation, a clear six-layer pattern emerges which is similar to that of the Old World primates (Doty et al., 1966).

On the representation of the visual fields in the LGN and cortex; orthograde and retrograde degeneration in the visual system In the 1920s and 1930s, anatomists (e.g., Brouwer and Zeeman, 1926) studied orthograde projections from the retina to the LGN by making restricted retinal lesions and identifying degenerating fiber terminals in the LGN using the Marchi stain. Geniculocortical projections were studied by making lesions of the primary visual cortex and mapping retrograde degeneration of cells in the LGN (e.g., Clark, 1932). These anatomical studies confirmed that there is an orderly projection from the retina to the LGN and from the LGN to the visual cortex. Neighboring points in the visual fields are represented at neighboring points on the cerebral cortex. In later studies (Van Buren, 1963), it

F 1.2. Cajal’s drawings showing the cell types in the mammalian retina. This is Cajal’s description (from Thorpe and Glickstein’s 1972 translation). All figures show cells from the mammalian retina with the exception of Fig. 1, which shows the nerve cells from the chicken retina. Fig. 1. A, ganglion cell destined for the first sublayer; B, ganglion cell destined for the second sublayer; C, small ganglion cells with granular clusters which spread in the fourth sublayer; D, multipolar cell destined for the second sublayer; E, a cell which forms two horizontal plexuses—one below the fourth sublayer and another in the third sublayer; F, small cell with two fine plexuses—one in the second sublayer and the other in the fourth sublayer; G, giant cell which forms three plexuses—in the second, third, and fourth sublayers; H, bistratified amacrine cell; J, cell with an extremely fine plexus destined for the third sublayer; K, cell which arborizes in the fourth sublayer and whose branches interlace with the end branches of an amacrine cell lying in the same layer; a, centrifugal fibers; b, another centrifugal fiber whose termination extends horizontally above the inner plexiform layer. Fig. 2. A section through the retina of an adult dog. a, cone fiber; b, cell body and fiber of a rod; c, bipolar cell with an ascending cluster destined for the rods; d, very small bipolar cell for the rods with a spare upper cluster; e, bipolar cell with a flat cluster destined for the cones; f, giant bipolar cell with a flat cluster; h, diffuse amacrine cell whose varicose branches lie, for the most part, just above the ganglion cells; i, ascending nerve fibrils; j, centrifugal fibers; g, special cells which are very rarely impregnated; they have an ascending axis cylinder; n, ganglion cell which receives the terminal cluster of a bipolar cell destined for the rods; m, nerve fiber which disappears in the inner plexiform layer; p, nerve fiber of the optic fiber layer. A, outer plexiform layer; B, inner plexiform layer. Fig. 3. Horizontal cells from the adult dog retina. A, outer horizontal cell; B, middle-sized inner horizontal cell with no descending protoplasmic processes; C, another, smaller inner horizontal cell; a, horizontal cell axis cylinder. Fig. 4. Nerve cells from the ox retina. a, bipolar cell with an ascending cluster; b, bipolar cell with a flat upper terminal cluster destined for the cones; c, d, e, bipolar cells of the same type whose lower cluster, however, arborizes in the more external sublayers of the inner plexiform layer; g, bipolar cell with a flat cluster of enormous extent; f, another bipolar cell with a giant upper cluster characterized by the rich, irregular arborization formed by the ascending processes; h, oval cells lying outside the outer plexiform layer; i, amacrine call located within the second sublayer of the inner plexiform layer; j, amacrine cell occupying the third sublayer; m, another amacrine cell whose branches apparently disappear in the third and fourth sublayers.

Fig. 5. Horizontal axis cylinder from the outer plexiform layer. a, terminal arborization as seen from the side; b, nerve fiber. Fig. 6. Another terminal arborization of the same type. Fig. 7. Nerve elements from the ox retina stained with chromium-silver according to the double impregnation method. A, semilunar amacrine cell whose enormously long branches arborize in the first sublayer; B, large amacrine cell with thick branches in the second sublayer; F, another amacrine cell, which is rather small and arborizes in the second sublayer; D, amacrine cell with a stellate cluster destined for the third sublayer; G, H, amacrine cells destined for the fourth sublayer; E, large amacrine cell destined for the fifth sublayer; C, special type of amacrine cell with very thin branches which spread preferentially in the first and fifth sublayers. a, small ganglion cell destined for the fourth sublayer; b, ganglion cell whose branches form three superimposed plexes; c, small ganglion cell with branches arborizing in the first sublayer; d, middlesized ganglion cell with branches in the fourth sublayer; f, ganglion cell which is similar to the multilayered cells (branching in three sublayers) in the reptile and bird; their branches form two plexes— one in the fourth sublayer and another in the second sublayer; e, giant ganglion cell destined for the third sublayer. Fig. 8. Amacrine cells and ganglion cells from the dog retina. A, stellate amacrine cell destined for the first sublayer and a portion of the second sublayer; B, giant amacrine cell of the third sublayer; C, G, stellate amacrine cells destined for the second sublayer; F, small amacrine cell destined for the third sublayer; E, amacrine cell destined for the fourth sublayer; D, unstratified amacrine cell; a, ganglion cell whose upper cluster spreads in the second sublayer; b, giant ganglion cell destined for the second sublayer; e, small ganglion cell whose cluster spreads in the fourth sublayer; f, middlesized ganglion cell which arborizes in the first and in a portion of the second sublayers; g, ganglion cell which arborizes in the third and a portion of the fourth sublayers; i, two-layerd cell (cellule bistratifée). Fig. 9. Ganglion cells from the dog retina. a, giant ganglion cell whose cluster spreads in the first and a portion of the second sublayers; b, small ganglion cell whose multiple processes disappear in the fifth sublayer; c, giant cell whose cluster seems to spread mainly in the second sublayer; e, giant ganglion cell of the second sublayer; d, g, small ganglion cells with clusters in the fourth sublayer; f, middle-sized ganglion cells destined for the first sublayer; h, another ganglion cell destined for the second and partially for the first sublayer; i, unstratified ganglion cell; A, B, C, spongioblasts (amacrine cells); L, lower terminal arborization of a bipolar cell. (From Cajal, 1892.)



was discovered that in addition to retrograde degeneration in the LGN that is caused by cortical lesions, there is also transneuronal degeneration in the retinal ganglion cell layer after lesion of the cerebral cortex.

On the primary visual cortex By the end of the eighteenth century, the gross structure of the cerebral cortex was beautifully illustrated in anatomical texts, but the cortex was portrayed as structurally homoge-

neous. One part of the cortex was depicted as looking like any other. The first recognition that the cerebral cortex is not uniform in structure was made by an Italian medical student, Francesco Gennari, working in the newly refounded University of Parma (Gennari, 1782; Glickstein and Rizzolatti, 1984). Gennari packed brains in ice, which allowed him to make clean, flat cuts through them. He noted a thin white line, and sometimes two lines within the cortex, running parallel to and about halfway between the pial surface above and the white matter below. The line coalesces

: :  

7

into a prominent single stripe in the caudal part of brain, “in that region near the tentorium.” Gennari first saw the stripe in 1776 and described it in his monograph De Peculiari (1782) some 6 years later (Fig. 1.3). Gennari’s monograph was published in a limited edition and he came from what was then an obscure university, so although it was cited by some authors, it was often ignored. The same cortical stripe was discovered independently a few years later by the more eminent anatomist Vicq D’Azyr. The stripe was described in his Traité D’Anatomie (1786) 3 years later. It was the Austrian anatomist Obersteiner (1888) who found Gennari’s earlier description of the white line and named it the stripe of Gennari. Although regional variability in cortical structure was soon accepted, there was no agreement about possible differences in the functions of different cortical areas. Two of the major authorities at the beginning of the nineteenth century, Gall (Gall and Spurzheim, 1810–1819) and Flourens (1824), held opposing views. Gall and his followers, the cranioscopists/phrenologists, asserted that the cerebral cortex is made up of a number of individual areas, each associated with a specific personality characteristic. If a person has a good memory, for example, the memory area of the cortex is relatively enlarged. Enlargement of a cortical area is associated with corresponding change in the shape of the skull, hence a bump on the head. Person-

F 1.3. The first recognition of the presence of a fiber layer within the cerebral cortex (labeled l in the picture) which Gennari described as being “particularly prominent in that region near to the tentorium.” (From Gennari, 1782.)

8

 

ality, ability, and character could be read by palpating the head. The earliest experimentalists failed to confirm Gall’s views. In a typical experiment, Flourens (1824) made lesions in the brains of birds and mammals and observed the resulting effects on the animals’ behavior. Although Flourens was convinced that the cerebral cortex is responsible for sensation, movement, and thought, he could find no evidence that any of these functions is localized to a particular site on the cerebral cortex. In later years, evidence began to accumulate in favor of functional localization in the cerebral cortex. A series of postmortem observations of focal injuries in the brains of patients who had lost the power of speech culminated in Broca’s (1861) description of the lesion in the left frontal lobe of the patient “Tan,” a man who had been unable to say any word other than tan for the past several years. The evidence for brain localization of speech was soon accepted, and within a few years experiments began to provide additional evidence that different areas of the cerebral cortex are specialized for different functions. The single most important experiment that led to modern understanding of the localization of motor and sensory functions in the cortex was done by Gustav Fritsch and Eduard Hitzig (1870). They electrically stimulated restricted regions of the frontal lobe of a dog and elicited movement of the face or limb on the opposite side of the body. Fritsch and Hitzig’s discovery of a specifically motor area of the cortex was instrumental in prompting a search for other functions, including vision. There had been indications (Panizza, cited by Mazzarello and Della Sala, 1993) that lesions in the caudal part of the brain are associated with visual deficits, but the clearest and most influential evidence for the visual function of the occipital lobe was provided by Hermann Munk, professor of physiology in the Veterinary Institute in Berlin. Munk (1881) made lesions in the occipital lobe of dogs and monkeys. He reported that if he destroyed one occipital lobe, the monkeys became hemianopic. Bilateral lesions caused blindness (Fig. 1.4). Munk’s discovery focused the attention of clinicians and scientists on the role of the occipital lobe in vision. Salomon Henschen (1890) summarized the postmortem findings in a group of patients who had suffered from hemianopia as a result of a stroke. He compared these patients with a similar number who had sustained a comparable loss of brain tissue that had not become hemianopic. Henschen confirmed the location of the primary visual area, and he suggested a scheme for the way in which the visual fields are mapped on the primary visual cortex. Henschen recognized that the left hemisphere receives its input from the right visual field and the upper bank of the calcarine fissure from the upper retina, hence the lower visual field. But Henschen also suggested that the periphery of the visual field is projected

F 1.4. Munk clearly illustrates the locus of the visual area of the monkey cortex. Unilateral occipital lobe lesions cause hemianopia. (From Munk, 1881.)

onto the caudal end of the striate cortex, with the fovea represented anteriorly. In this, he was in error. Henschen’s error is understandable, since the lesions in the brains that he studied were diffuse. What was needed to establish a more accurate spatial mapping was evidence of partial field defects, scotomas, caused by smaller, subtotal lesions of the striate cortex. Such lesions, regrettably, arise in wartime. One of the earliest clear pictures of the representation of the peripheral-central visual field representation was made by a young Japanese ophthalmologist, Tatsuji Inouye (Glickstein and Whitteridge, 1987; Inouye, 1909). Inouye was in the medical service of the Japanese Army during the Russo-Japanese war of 1904–1905. His responsibility was to evaluate the extent of visual loss in casualties of the war. Inouye used the opportunity to study the visual field defects caused by penetrating brain injuries. In that war the Russians used a newly developed rifle which fired smallcaliber bullets at high velocity. Unlike most bullets used in previous wars, these bullets often penetrated the skull at one point and then exited at another, making a straight path through the brain. Inouye devised a three-dimensional coordinate system for recording the entry and exit wounds. He then calibrated the course of the bullet through the brain and estimated the extent of the damage it would have caused to the primary visual cortex or the optic radiations. Based on his study of visual field defects in 29 patients, Inouye produced a map of the representation of the visual fields on the cortex. The central fields were now placed correctly in the most caudal part of the striate cortex, with the peripheral visual fields represented anteriorly, and there was an overrepresentation of the central visual fields in the primary visual cortex.

Based on his studies of the visual field defects sustained by soldiers of the First World War, Gordon Holmes (1918a) produced a more accurate and detailed map of the representation of the visual fields on the striate cortex, which still forms the basis for interpreting partial visual loss in humans.

Confirmation of the map by electrical stimulation and recording of evoked potentials In the period between the First and Second World Wars, the basic arrangement of the visual fields was confirmed in studies using electrical stimulation of the brain in neurosurgical patients. Ottfried Foerster (1929) operated under local anesthetic on patients who suffered from seizures caused by focal scarring of the brain. Electric current was applied at a specific site on the cerebral cortex, and the resultant sensation was reported by the patient. Electrical stimulation of the cortex at the occipital pole caused phosphenes that were centered in front of the patient. Stimulation of the upper lip of the calacarine fissure 5 cm anterior to the occipital pole produced a phosphene that was centered in the lower visual field opposite the side of the brain that had been stimulated. Foerster’s studies, and later those of Wilder Penfield (Penfield and Rasmussen, 1952), confirmed the representation of the visual fields on the human striate cortex that had been revealed by the analysis of the scotomas caused by focal lesions.

Electrical recording of neural activity in the primary visual cortex In the 1930s Philip Bard (1938) and his collaborators, one of whom was Wade Marshall, began to record the electrical

: :  

9

activity that is evoked on the surface of the cerebral cortex of experimental animals by stimulation of the body surface. The electrodes at the time were too large to record the activity of individual neurons but small enough to detect focal activity in a restricted group of cells. There is an orderly representation of the body surface on the primary somatosensory cortex, with neighboring points on the body represented at neighboring points on the brain. Marshall later collaborated with William Talbot in studying the activity evoked on the striate cortex of cats and Old World monkeys. They focused small spots of light on the retina of a monkey and marked the locus of maximal evoked activity on the cerebral cortex. Figure 1.5 is from their report. Talbot and Marshall’s recordings were limited to the dorsolateral surface of the macaque cortex, comprising only roughly the central 10 degrees of the visual field. The work was extended by Peter Daniel and David Whitteridge (1961), who recorded more anterior cortical areas within the calcarine fissure of baboons, extending the mapping into the peripheral visual field. The evidence from visual loss in humans and monkeys caused by cortical lesions, electrical stimulation of the cortex in humans, and recording in monkeys was all consistent. The visual fields are represented in an orderly way on the primary visual cortex.

Early electrical recording from visual areas outside of the primary visual cortex Early workers had suggested that the regions outside of the primary visual cortex might have a related visual function. Hermann Munk (1881), for example, labeled a region outside of the primary visual cortex in dogs as an area con-

cerned with the storage of visual memories. William Talbot (1942) made a brief report to the Federated Society for Experimental Biology and Medicine which initiated modern study of the way in which the visual fields are represented beyond the primary visual cortex. Talbot recorded potentials evoked by vision from the surface of a cat brain. As expected, he found that the visual field is mapped in an orderly way on the primary visual cortex, with neighboring points in the visual field represented at neighboring points on the cortex. As Talbot continued to record lateral to the representation of the vertical meridian, he found that the cortex was still activated by focused spots of light, but from increasingly peripheral regions of the visual field. Talbot had discovered a second visual area, later called Visual Area 2, which is mapped on the cortex like a mirror image of the primary representation. Talbot had started a growth industry. Some years after Talbot’s report, Margaret Clare and George Bishop (1954) described another visual area that is located on the lateral suprasylvian gyrus of cats, in which flash stimuli evoke a gross potential. Although the pioneering work of electrical mapping was done in cats, the visual cortex in these animals is not “primary” in the sense that it is in monkeys and humans. In monkeys and humans, the overwhelming majority of geniculocortical fibers terminate in the striate cortex. In cats, Visual Area 2 receives a direct and equally powerful input from the LGN (Glickstein et al., 1967). In the 1950s, techniques were developed for isolating the activity of individual neurons, and single-unit recording has since become a standard method of studying visual processing by the brain. The contribution to our understanding of vision from single-unit recording is described in this book by Horace Barlow (Chapter 2). Here we note

F 1.5. Mapping of the visual fields onto the monkey striate cortex revealed by evoked responses to small, focused spots of light. (From Talbot and Marshall, 1941.)

10

 

briefly the way in which these studies increased the number of recognized visual areas in the cerebral cortex. After Talbot’s discovery of Visual Area 2 and Clare and Bishop’s description of the lateral suprasylvian visual area, David Hubel and Tortsten Wiesel (1965) identified another area by recording from cells on the medial bank of the lateral fissure of cats, an area they called Visual Area 3. A few years later, John Allman and Jon Kaas (1971), studying the owl monkey Aotus, and Semir Zeki (1978), studying Old World macaques, identified more extrastriate visual areas, each of which appeared to be specialized for analyzing color, motion, or form. By a count made in 1992, there are no less than 32 visual areas in the monkey brain (Van Essen et al., 1992). There are doubtless at least as many in the human cortex.

demonstrate that rather than blindness, his monkeys suffered a severe impairment in guiding their movements under visual control. Virtually identical symptoms were described by Rudolf Balint (1909) in a patient who had suffered bilateral lesions of the parietal lobes and by Gordon Holmes (1918b), who studied casualties among British soldiers in the First World War. Ferrier’s monkeys, Balint’s patient, and Holmes’s soldiers were all unable to guide their movements accurately under visual control. The visual areas of the parietal lobe are principally concerned with spatial localization in the visual field (Ungerleider and Mishkin, 1982) and the visual guidance of movement (Glickstein and May, 1982).

Early behavioral evidence for the functions of extrastriate visual areas

Temporal lobe lesions cause an impairment in recognizing and remembering forms. In early studies of the effects of large temporal lobe lesions in monkeys (Brown and Schäfer,

On visual deficits following lesioning of the temporal lobe

In humans and monkeys the striate area is virtually the sole cortical target of cells in the LGN, but the cortex adjacent to the primary visual cortex is also dominated by vision. An estimated one-third or more of the monkey cerebral cortex is devoted to visual processing. There are two large groupings of visual areas outside the primary visual cortex (Glickstein and May, 1982; Ungerleider and Mishkin, 1982), a medial group centered in the parietal lobe and a lateral group centered in the temporal lobe.

The visual areas of the parietal lobe In monkeys, all of the parietal lobe cortex from the primary visual area as far rostrally as the intraparietal fissure has direct or indirect input from the primary visual cortex. The angular gyrus is a major part of this area. When David Ferrier stimulated the angular gyrus of the monkey brain electrically, he observed that the stimulation caused eye movements. When he ablated the region, the monkey appeared to be blind. Ferrier concluded that this region must be the primary visual cortex (Ferrier, 1876; Glickstein, 1985) (Fig. 1.6). Ferrier’s first experiments were done in the 1870s, prior to the widespread use of antiseptic techniques in experimental surgery. His animals were killed 3 days after he operated on them, since longer survival times inevitably led to infections. In later experiments, Ferrier adopted the sterile surgical techniques of his colleague at King’s College London, Joseph Lister. His animals could now live for weeks, months, or years after the operation. With Gerald Yeo he replicated his studies of the behavioral effect of angular gyrus lesions. They now reported that his animals were not blinded by the angular gyrus lesion but suffered a temporary loss of vision (Ferrier and Yeo, 1884). Ferrier’s protocols

F 1.6. Lesions of the angular gyrus of the parietal lobe. Ferrier initially interpreted the resultant deficit as blindness. His later study (Ferrier and Yeo, 1884) shows that the monkey was not blind but had a profound deficit in visual guidance of movement following bilateral angular gyrus lesioning. (From Ferrier, 1876.)

: :  

11

1888; Klüver and Bucy, 1938) in addition to other symptoms, there were visual defecits. The specifically visual function of the inferotemporal cortex was further clarified when animals were tested after smaller, more restricted lesions of the temporal lobes were produced. K. L Chow (1951) showed that lesions of the inferotemporal cortex cause a specific impairment in the acquisition and retention of visual discrimination learning. Mishkin (1966) demonstated that the essential input to the inferotemporal cortex is by a series of corticocortical connections originating in the striate cortex.

A note on the aims of this chapter and its sources This chapter has attempted to outline some of the major questions and discoveries that led from the first understanding of image formation by the eye, the recognition of the nature of the photoreceptors, and the connections from the eye to the primary visual cortex. The single most useful volumes for exploring the topics presented here are the two masterful scholarly works of Stephan Polyak, The Retina (1941) and The Vertebrate Visual System (1957). Both have extensive bibliographies and references which are invaluable for finding the early literature. An excellent source for the history of neuroscience in general is Clarke and O’Malley (1968). Useful also are Von Bonin’s (1950) translations of several of the papers cited here. I recognize that there are many other aspects of the history of the study of vision that are of equal interest and relevance. For example, the fascinating story of color vision has only briefly been touched on in the reference to Thomas Young (1802). There is also a tradition based on the subjective study of visual phenomena that began with Goethe, was powerfully advanced by Purkinje and Hering, and contributed to the later understanding of color coding by the brain. While apologizing for these omissions, I hope I have given an outline of some of the early contributions that formed the basis for much of the material to be covered in this volume.

REFERENCES Allman, J., and J. Kaas, 1971. Representation of the visual field in striate and adjoining cortex of the owl monkey (Aotus trivirgatus), Brain Res., 35:89–106. Balint, R., 1909. Seelenlähmung des “Schauens”, optische Ataxie, räumliche Störung der Aufmerksamkeit, Monatsschr. Psychiatr. Neurol., 25:51–81. Bard, P., 1938. Studies on the cortical representation of somatic sensibility, Harvey Lectures, 143–169. Bastian, H., 1880. The Brain as an Organ of Mind, London: Kegan. Broca, P., 1861. Remarque sur la siège de la faculté du langage articulé suive d’une observation d’aphémie (perte de la parole), Bull. Soc. Anat. Paris, 36:330–357. Brouwer, B., and W. Zeeman, 1926. The projection of the retina in the primary optic neuron in monkeys, Brain, 49:1–35.

12

 

Brown, S., and E. Schäfer, 1888. An investigation into the functions of the occipital and temporal lobes in the monkey’s brain, Phil. Trans. R. Soc. Lond (Biol), 179:303–327. Cajal, S. Ramon y, 1892. La rétine des vértebrés (La Cellule, English trans.; S. Thorpe and M. Glickstein, trans., 1972), Springfield: Thomas. Chow, K. L., 1951. Effects of partial extirpation of posterior association cortex on visually mediated behavior in monkeys, Comp. Psychol. Monogr., 20:187–218. Clare, M., and G. Bishop, 1954. Responses from an association area secondarily activated from optic cortex, J. Neurophysiol., 17:271–277. Clark, W. E. Le Gros, 1932. The structure and connections of the thalamus, Brain, 55:406–470. Clarke, E., and E. O’Malley, 1968. The Human Brain and Spinal Cord, Berkeley and Los Angeles: University of California Press. Daniel, P., and D. Whitteridge, 1961. The representation of the visual field on the cerebral cortex in monkeys, J. Physiol., 159:203–221. Des Cartes., R., 1677. L’Homme et la Formation de la Foetus, 2nd ed. Cited in Polyak, 1941. Doty, R., M. Glickstein, and W. Calvin, 1966. Lamination of the lateral geniculate nucleus in the squirrel monkey, Saimiri sciureus, J. Comp. Neurol., 127:335–340. Ferrier, D., 1876. The Functions of the Brain, London: Smith-Elder. Ferrier, D., and G. Yeo, 1884. On the effects of brain lesions in monkeys, Phil. Trans. R. Soc. Lond., 2:494. Flourens, P., 1824. Rechérches Expérimentale sur les Propriétés et les Fonctions du Système Nerveux dans les Animaux Vertébrés, Paris: Chevot, pp. 85–122. Foerster, O., 1929. Beitrage zur Pathophysiologie der Sehbahn und der Sehsphäre, J. Psychol. Neurol. Leipzig, 39:463–485. Fritsch, G., and E. Hitzig, 1870. Über die elektrische Erregbarkeit des Grosshirns, Arch. Anat. Physiol. Wissenschr. Med. Leipzig, 300– 332. Gall, F., and J. Spurzheim, 1810–1819. Anatomie et physiologie du systême nerveux et du cerveau en particulier avec des observations sur la possibilité de reconnaitre plusieurs dispositions intellectuelles et morales de l’homme et des animaux par la configuration de leurs têtes. Paris: N. Maze. Gennari, F., 1782. De peculiari structura cerebri nonnulisque ejus morbis, Parma: Regio Typographeo. Glickstein, M., 1985. Ferrier’s mistake, Trends Neurosci., 8:341–344. Glickstein, M., R. King, J. Miller, and M. Berkley, 1967. Cortical projections from the dorsal lateral geniculate nucleus of cats, J. Comp. Neurol., 130:55–76. Glickstein, M., and J. May, 1982. Visual control of movement: the circuits which link visual to motor areas of the brain with special reference to the visual input to the pons and cerebellum, in Contributions to Sensory Physiology (W. D. Neff ed.), New York: Academic Press. Glickstein, M., and G. Rizzolatti, 1984. Francesco Gennari and the structure of the cerebral cortex, Trends Neurosci., 7:464–467. Glickstein, M., and D. Whitteridge, 1987. Tatsuji Inouye and the mapping of the visual fields in the human cerebral cortex, Trends Neurosci., 9:350–353. Gowers, W., 1892. Diseases of the Nervous System, 2nd ed. vol. 2, Brain and General Functional Diseases, London: Churchill. Henschen, S., 1890. Klinische und anatomische Beiträge zur Pathologie des Gerhirns Part 1, Uppsala: Almquist and Wiksell. Holmes, G., 1918a. Disturbances of vision by cerebral lesions, Brt. J. Ophthalmol., 2:353–384. Holmes, G., 1918b. Disturbances of visual orientation, Brt. J. Ophthalmol., 2:449–468; 506–516.

Hubel, D., and T. Wiesel, 1965. Receptive fields and functional architecture in two non-striate visual areas (18 and 19) of the cat, J. Neurophysiol., 28:229–289. Inouye, T., 1909. Die Sehstörungen bei Schussverletzungen der kortikalen Sehsphäre nach Beobachtungen an Verwundeten der letzten japanische Kriege, Leipzig: Engelmann. Kepler, J., 1604. Ad Vitellionem, Frankfurt: Marnium and Haez. Klüver, H., and P. Bucy, 1938. An analysis of certain effects of bilateral temporal lobectomy in the rhesus monkey with special reference to psychic blindness, J. Psychol., 5:33–54. Leonardo da Vinci 16th Century Diagram from the Windsor Collection reproduced in Leonardo da Vinci (1989) Plate 97 in the catalog of an exhibition at the Hayward Gallery, London. London and New Haven; South Bank Centre and Yale University Press. Mazzarello, P., and S. Della Sala, 1993. The demonstration of the visual area by means of atrophic degeneration methods in the work of Bartolomeo Panizza (1855), J. Hist. Neurosci., 2:315–322. Minkowski, M., 1920. Über den Verlauf, die Endigung und die zentrale Repräsentation von gekreutzten und ungekreutzten Sehnervfasern bei einigen Säugetieren und beim Menschen, Schweiz. Arch. Neurol. Psychiatr., 6:201–252; 7:268–303. Mishkin, M., 1966. Visual mechanisms beyond the striate cortex, in Frontiers in Physiological Psychology (R. Russell, ed.), New York: Academic Press. Munk, H., 1881. Über die Funktionen der Grosshirnrinde, Berlin: Hirschwald. Newton, I., 1704. Second Book of Opticks: Treatise of Light, London: Smith and Walford. Obersteiner, H., 1888. Anleitung beim Studium des baues der Nervösen Centralorgane im gesunden und kranken Zustande, Leipzig and Vienna: Toeplitz und Deuticke.

Penfield, W., and T. Rasmussen, 1952. The Cerebral Cortex of Man, New York: Macmillan. Polyak, S., 1941. The Retina, Chicago: University of Chicago Press. Polyak, S., 1957. The Vertebrate Visual System, Chicago: University of Chicago Press. Scheiner, C. 1619, 1652. Oculus. Innsbruck: Oenoponti. Schultze, M., 1866. Zur Anatomie und Physiologie der Retina, Arch Mikroskopische Anat., 2:175–286. Talbot, S., 1942. A lateral localization in cat’s visual cortex, Fed. Proc., 1:84. Talbot, S., and W. Marshall, 1941. Physiologic studies on neural mechanisms of visual localization and discrimination, Am. J. Ophthalmol., 2:1255–1261. Ungerleider, L., and M. Mishkin, 1982. Two cortical visual systems, in Analysis of Visual Behavior (D. Ingle and M. Goodale, eds.), Cambridge, MA, MIT Press, pp. 459–486. Van Buren, J., 1963. Trans-synaptic retrograde degeneration in the visual system, Neurol. Neurosurg. Psychiatr., 26:402–409. Van Essen, D., C. Anderson, and D. Felleman, 1992. Information processing in the primate visual system: an integrated systems perpective, Science, 225:419–423. Vicq d’Azyr, F., 1786. Traité d’anatomie et de physiologie, Paris: Didot. Von Bonin, G., 1950. Essays on the Cerebral Cortex, Springfield: Thomas. Wollaston, W., 1824. On semi-decussation of optic nerves, Phil. Trans. R. Soc., 114:222–231. Young, T., 1802. On the theory of light and colours, Phil. Trans., 92:12–18. Zeki, S., 1978. Uniformity and diversity of structure and function in rhesus monkey prestriate visual cortex, J. Physiol., 277:273– 290.

: :  

13

2

The Role of Single-Unit Analysis in the Past and Future of Neurobiology HORACE BARLOW

Introduction

History

This chapter starts with a brief history of single-unit recording, biased, I am afraid, toward recounting the parts I know best, namely, those that interested me or those I took part in. It stops well short of the present, except for a brief account of some studies on MT neurons in awake behaving monkeys that I believe point the way ahead. The quantitative statistical approach of signal detection theory should enable one to follow a single quantity, the signal-to-noise ratio, through all stages from the sensory stimulus itself, through single-neuron responses at all levels, to reports of perceptual experiences or other behavioral responses. But statistical arguments have pitfalls. First, the source or sources of limiting noise must be correctly identified; second, they are good for establishing limits to what is possible but not very good as a basis for direct models because we know so little about how the brain computes statistics. I think the implications of statistical measures are most easily understood in terms of rules stating when signal-to-noise ratios are conserved, how they can be increased, and when they decrease. Using the rules, statistical arguments can give much insight into the role of single units in sensory systems, for neurons are the only elements capable of collecting together the information relevant to a particular task, which in turn is the only way to obtain high signal-to-noise ratios. I believe we need to open our eyes to the much more complex types of computation that, as cell biology is beginning to show, might be accomplished by single neurons, so I could not refrain from speculating about this at the end of the chapter. Finally, at the editor’s instigation, I have recounted in an Appendix some of my personal experiences in the remarkable department, created by Lord Adrian, in which I had the extraordinary good fortune to grow up scientifically. This, then, is a personal view. Please do not read this chapter in the hope of finding a complete historical account that leads to a balanced view of the role of single units in vision; desirable though that would be, it is not what the title proclaims and it is not what you will find.

By the beginning of the twentieth century, the basic layout of the sensory systems of the brain, including vision, was surprisingly well understood. If, for example, one reads Schäfer’s account of the cerebral cortex in his two-volume Textbook of Physiology (1900), at first one cannot fail to be amazed by how much was known. There is a good deal about cortical localization, the neuron doctrine was in place, and there were Cajal’s beautiful pictures of neurons with all sorts of shapes and sizes, sometimes with putative circuits showing how messages flowed into the dendrites through synapses and out along axons to distant destinations. But on reflection one becomes aware of what was then missing, for the nature of the activity that was localized, and the nature of the messages that passed from place to place in the brain, were quite unknown. Müller’s doctrine of specific nerve energies was based on the similar sensation produced whenever a given type of sensory nerve fiber was excited, regardless of the means employed to excite it. It was sometimes taken to imply that the messages were different in different fibers, rather than that the same message had different meanings when carried by different fibers, but with no knowledge of the nature of the messages, this misunderstanding is perhaps not surprising. The all-or-none law had been formulated for heart muscle, but it was not known to apply to nerve impulses; indeed, it was not clear that nerve messages were composed of impulses. In other words, it was well understood that nerve fibers were communication channels, but it was not understood at all how or what they communicated. The reason for this ignorance is simple: there were no methods available either for isolating the activity of an individual nerve fiber or for detecting and recording the activity if it had been isolated. Intracellular recording was unheard of, and as we now know, the potential from an impulse that can be recorded through an external electrode placed close to a nerve fiber is a brief (80% loss of ganglion cells and an increase in amacrine cells (Wang et al., 2001). In Drosophila, Notch acts as a negative regulator of atonal and it is tempting to speculate that a similar relationship holds for Notch and Math5 in the mouse. Other genes may cooperate with Math5 in triggering ganglion cell differentiation. For example, Otx-2 also shows transient expression in ganglion cells just after they become postmitotic (Baas et al., 2000). Whether Math5 and Otx-2 are acting in concert or in independent pathways is not known. Among the earliest markers of ganglion cell differentiation are the POU-domain transcription factors of the Brn3 family. These can be detected soon after the ganglion cells have become postmitotic and before they have migrated to

38

 

the ganglion cell layer. Math5 can transactivate a gene coupled to a Brn-3 promoter (Liu et al., 2001). Taking all the above results together, we can suggest a pathway in which the process of differentiation from competent precursors is initiated by the action of Shh. This factor influences the pattern of Notch activation, and in cells in which Notch is not activated Math5 is expressed. This, perhaps in conjunction with other factors like Otx-2, irreversibly triggers cells into a ganglion cell fate and promotes expression of characteristic genes such as Brn-3. This is clearly an oversimplification and ignores interactions with other cells undergoing other fate choices. A related bHLH protein, NeuroD, is vital for amacrine cell formation. Math5 can inhibit expression of NeuroD. Thus, in addition to promoting ganglion cell formation, Math5 is inhibiting amacrine cell formation. By analogy with the mechanisms of optic cup regionalization discussed earlier, we should perhaps be looking for more examples of reciprocal inhibition of bHLH genes involved in specifying retinal cell types and for other extrinsic factors that can initiate a cascade of these reciprocal interactions. Just as important as the mechanisms initiating cell differentiation in retina are those that regulate the number of each cell type produced. There is abundant evidence for a negative feedback in the production of ganglion and amacrine cells. In many species of fish and amphibians, the retina continues to grow throughout life by adding new cells at the tissue margin. Thus, the radial dimension from center to periphery recapitulates the developmental history of the retina. Dopaminergic amacrine cells are normally formed at the retinal margin at a constant density. Destruction of these cells with 6-OH-dopamine, followed by a sufficient recovery period, led to production of a higher than normal density of new dopaminergic cells (Negishi et al., 1982). Similar experiments in larval frogs gave essentially identical results (Reh and Tully, 1986). In an extension of these experiments, other retinal cell types were destroyed by treatment with kainic acid, and the number of new cells formed was increased in proportion to the damage to the differentiated neurons (Reh, 1987). The conclusion from these studies was that an inhibitory feedback signal was produced by differentiated cells that limited the number of that cell type produced. This possibility has been explored in more detail using retinal cultures. In reaggregate cultures using retinal cells of different ages, differentiated amacrine cells were found to inhibit the production of amacrine cells by embryonic cells (Belliveau and Cepko, 1999). Similarly, ganglion cell production was reduced in cultures of chick embryonic retinal cells when cultured adjacent to older retinal cells (Waid and McLoon, 1998). This type of experiment suggests that the inhibitory signal is diffusible, although its identity has yet to be established.

Differentiation of outer retinal cells We know more about the differentiation of rod photoreceptors than about any other CNS neuron. The extensive background of molecular information derived from numerous studies of phototransduction has provided an excellent set of markers, and the medical importance of photoreceptor degenerative diseases has spurred extensive studies of this cell type. As indicated earlier, most rods become postmitotic near the end of retinal development, in the first few postnatal days in mice and rats. A few rods can be labeled by thymidine injection as early as E14, although they show further signs of differentiation at the same time as the vast majority of rods born later (Morrow et al., 1998). This suggests that the events occurring during the final mitosis can be separated from those of overt differentiation. Rods are formed from late progenitors, some of which can also form bipolar cells and Müller glial cells. The visual pigment protein opsin is one of the earliest markers of rod differentiation and, because its early appearance is under transcriptional regulation, it has frequently been used to designate rod formation (Treisman et al., 1988). If retinal cells from postnatal rats or mice are plated in dilute monolayer culture, they form many opsin-expressing rods. If cells from early or midembryonic stages are plated, no opsinpositive rods are detected. On the other hand, if the embryonic tissue is maintained as an explant culture or dissociated and allowed to reaggregate, opsin-positive rods are detected at a time equivalent to the normal time of expression. This suggests that the information required to generate rods is intrinsic to the retina but that it needs cell-cell interactions for expression. Embryonic cells mixed with neonatal retinal cells (that are at their peak of rod differentiation) form more rods but only after the appropriate number of days in culture (Watanabe and Raff, 1992). These and related experiments suggest that there are factors present within retina that can promote formation of rod photoreceptors, but only when the progenitor cells are competent to respond to them. At present, the relative roles of factors located on the cell surface and secreted factors are unknown. Numerous factors including FGFs, retinoids, and even compounds like taurine can increase opsin expression in culture. There is, however, little direct evidence that any of them act directly on cell determination or opsin transcription pathways. It is more likely that they have relatively nonspecific effects that make the cells more healthy and better able to express differentiated functions. In contrast to the positive effects of the growth factors mentioned above, treatment of retinal explants with cytokines of the interleukin-6 (IL-6) family, such as ciliary neurotrophic factor (CNTF) or leukemia inhibitory factor (LIF), completely inhibits the production of rod photore-

ceptors, as judged by opsin expression (Ezzeddine et al., 1997; Neophytou et al., 1997). These cytokines are known to bind to multisubunit membrane receptors and activate a tyrosine kinase. The intracellular transduction pathways activated by cytokine binding include those characterized by MAPK and STAT proteins. Activation of either of these can lead to changes in transcription. By using pharmacological blockers of MAPK and viruses expressing either constitutively active or dominant negative STAT proteins, we have shown that the MAPK pathway is likely to be involved in Müller glial cell formation and the STAT pathway (specifically STAT3) in rod formation. Activation of STAT3 leads to a change in expression of several key transcription factors. Most relevant are increases in the levels of Otx-2 and HES1. Otx-2 is thought to play a role in the decision of progenitors to become either bipolar cells or rods. Since reduced levels of Otx-2 lead to production of more rods, it is reasonable to assume that elevated levels will help prevent rod formation. Hes-1 is an important regulator of development in a number of tissues, is a bHLH protein homologous to the hairy and enhancer of split genes of Drosophila, and generally acts as a repressor of transcription. Its expression can be increased by activation of the Notch pathway. We do not yet know whether the cytokine-induced increase in Hes-1 expression is mediated through a Notch signaling pathway or whether this is another example of regulation through two independent pathways. Positive regulators of rod formation include a number of bHLH transcription factors. The best studied of these is Neuro D, the same gene that influences amacrine cell production. Transfection of Neuro D into retinal cultures can increase the number of rods formed, and blocking of Neuro D can diminish rod production. Hes-1 inhibits Neuro D expression; thus, this pathway may be responsible for the cytokine-induced inhibition of rod formation. At present, we do not how Neuro D facilitates rod formation. It does not appear to have a direct effect on the opsin gene, suggesting that there must be several steps between Neuro D expression and opsin expression. It has also been found that inhibition of Neuro D function increases the production of Müller glial cells. This suggests that Neuro D functions at the time a retinal progenitor is making a choice of cell differentiation pathway.

Molecular regulation of opsin expression Promoter mapping studies using transgenic mice have suggested that all the information for correct temporal and spatial regulation of opsin expression is contained in about 1.5 kb of DNA 5¢ to the opsin gene. Most of the information is in the first few hundred bases 5¢ to the transcription start site. Opsin is a classical gene with a full TATA box for

:  

39

assembly of the transcription complex. In a region just upstream of this is a region of DNA that appears to become completely covered with transcription factors, several of which have now been identified (Fig. 3.4). RET-4 has been defined as a nuclear protein binding site at -39 to -52, although the identity of the binding protein is unknown. Nrl is a zinc-finger protein that binds at -55 to -70 in the rat rod opsin promoter. Crx is a homeodomain protein that probably binds to the BAT-1 site at -82 to -98 on the opsin promoter. A key promoter site for regulation of opsin expression lies at -110 to -37. This site, labeled RET-1 or PCE, is found in a number of other rod photoreceptor genes, suggesting that a common transcriptional regulatory pathway may exist (Morabito et al., 1991). Its importance is also suggested by the finding that opsin genes from species as diverse as humans and Drosophila have RET-1/PCE sites in their promoters. Although high levels of Crx can activate transcription from this site, other transcription factors act with higher efficiency. Among these are other homeodomain proteins including Rax and another member of the emx family Vax-2 (Kimura et al., 2000; Martinez and Barnstable,

1998). Since neither of these show peak expression in the mature photoreceptor layer, it is likely that either a still undescribed protein is important in opsin gene transcription or the affinity of another protein such as Crx can be altered by complex formation with another protein. The transcription factors necessary for opsin expression all seem to have different developmental profiles, suggesting that different pathways regulate their expression. What we do not have at present is evidence for a single trigger for rod photoreceptor differentiation, though Neuro D may serve this function in some way.

Maturation of retinal cells The complex set of interactions that trigger the appearance of characteristic molecules is only the first step in the formation of a fully differentiated retinal cell type. Rod photoreceptors undergo a major developmental step of outer segment formation about a week after opsin expression can first be detected. Rod outer segments represent a complex morphological specialization and contain the light-sensing

F 3.4. Comparison of the sequences upstream of the opsin gene of various vertebrates. Homologous regions are boxed, and known transcription factor or nuclear protein binding sites are labeled in order from the transcription start site. 1, TATAA box; 2, Ret-4; 3, NRE; 4, BAT-1; 5, Ret-1; 6, Ret-2.

40

 

and transduction machinery. It is not surprising, therefore, that outer segment formation is accompanied by the de novo expression of a large number of rod-specific genes. Among these are the structural proteins involved in maintaining rod disc shape, such as peripherin and Rom-1, and components of the transduction cascade, such as transducin, phosphodiesterase (PDE), and the cGMP-gated cation channel. Although most of these appear to be regulated at the transcriptional level, we have little idea of the transcription factors regulating their expression. Both in vitro and in vivo studies have shown that the presence of an RPE cell layer is vital for the full development of outer segments but not for their initiation. Explants of retinas from many species will develop normally up to the stage of forming outer segments but then stop. In the mi/mi mutant mouse, an RPE layer is present but the cells remain in an immature state. The retina adjacent to this does not develop normally. Electron microscopy has shown that the rod photoreceptors begin to make membrane expansions that are the beginning of outer segments, and molecular studies have shown that several transduction genes are turned on. Thus, it appears that the RPE does not provide a triggering signal for outer segment formation but is vital for continued growth. Whether it provides important nutritional factors or mechanical support is not yet known.

The role of glia in retinal development and function Most vertebrate retinas contain three types of glial cells. Müller glia are intrinsic glia that span the whole width of the tissue and are derived from the same set of progenitors as the retinal neurons. Astrocytes that reside in the optic nerve fiber layer migrate in through the optic stalk. Microglia also migrate into the retina, and many move from the ganglion cell layer to a position just below the outer plexiform layer. The timing of this migration coincides with the wave of developmental cell death that occurs during postnatal maturation, and these cells are thought to be intimately involved in removing dead cells from the tissue. Müller glia serve a variety of roles. They form the inner and outer limiting membranes at either side of the retina and thus serve an important barrier function for substances moving into and out of the retina. Through their glycogen stores and their ability to process transmitters such as glutamate and GABA, they play a vital role in retinal metabolism. They also clearly serve a structural role, and defects in Müller cells can lead to severe disruptions and rosette formation in the photoreceptor layer. Müller cells are also capable of producing a variety of important peptide mediators. In response to lack of oxygen, Müller cells can produce vascular endothelial growth factor (VEGF) that can act on adjacent blood vessels and stimulate their growth. Müller cells can also produce trophic factors such as CNTF

and FGF. Experimentally, this production is usually seen after an insult such as bright light exposure or excitotoxin treatment. It is thought that the Müller cell response may be part of a homeostatic mechanism to lessen the deleterious effects of these insults. Since a number of aspects of retinal development involve similar trophic factors, it is tempting to think that Müller cells, or their precursors, also release these factors as key developmental regulators.

Positional information across the retina The developmental mechanisms discussed so far in this chapter can begin to explain how distinct cell types arise from an initially uniform epithelium. They cannot, however, account for differences within the same cell class seen in central versus peripheral retina. Some differences, such as the size of the dendritic field or changes in the spacing of an orderly mosaic of cone photoreceptors, are intraretinal, but others, such as the position of synaptic targets in other visual areas, require translation of retinal position into complex patterns of growth and connectivity. It is important to try to distinguish those events in which position is reflected in expression of unique or unique amounts of particular molecules from those that arise as epiphenomena of other processes. One of the most dramatic differences seen in dendritic field morphology is in bipolar cells and retinal ganglion cells where in the human fovea a cone photoreceptor interacts with one or two bipolar cells but in the periphery hundreds of photoreceptors converge onto a single bipolar cell. We do not know whether this difference is inherent in molecular differences between the cells, a consequence of developmental processes that affect relative cell numbers, or a late event that changes from an initially more uniform pattern. Although these intrinsic retinal properties reflect positional information, the axes are more commonly defined by readout, as defined by the retinotopic mapping of ganglion cell axon terminals in target areas such as the superior colliculus. Many studies have attempted to define the molecular correlates of retinal position. A number of molecules can be found in gradients. Some surface proteins and carbohydrate groups can be found in a dorsoventral gradient (Constantine-Paton et al., 1986; Trisler et al., 1981). Some transcription factors seem to divide the retina into nasal and temporal halves (Hatini et al., 1994). To date, however, only two classes of retinal gradient are known to convey positional information. The first is retinoic acid, which is present in a gradient along the dorsoventral axis created by corresponding gradients in synthetic enzymes with different substrate affinities (Drager et al., 1998). Retinoic acid is critical for the formation of ventral retina in zebrafish (Hyatt et al., 1996). In addition, the gradient of retinoic acid may be used in mouse retina for patterning different cone types.

:  

41

The other classes of molecules found in a graded distribution are the Eph receptor tyrosine kinases and their ephrin ligands (Flanagan and Vanderhaeghen, 1998). This ligand receptor family was originally identified using a culture assay to measure topographic modulation of the growth of retinal ganglion cell axons (e.g., see Ciossek et al., 1998). Ephrins were isolated as molecules that could inhibit growth by inducing growth cone collapse. The receptor EphA3 is expressed in retinal ganglion cells as a temporal-nasal gradient, and its ligand ephrin A2 is expressed in a posterioranterior gradient in the tectum. A second tectal ligand for EphA3, ephrin A5, is also expressed in a graded fashion in the tectum. A variety of in vitro assays, knockouts, and ectopic expression studies have provided strong evidence for the role of this receptor-ligand system in retino-tectal connectivity. For example, in ephrin A5 and ephrin A2 knockouts, retinal ganglion cell axons terminate in more posterior locations exactly as predicted if a repulsive influence was removed (Feldheim et al., 2000). There is still debate whether ephrins and Eph receptors are important in both retinal axes. EphB2 and EphB3 receptors show a dorsoventral gradient of expression in the retina, and ephrin B ligands show a medial-lateral gradient in the superior colliculus. The importance of this group of ephrins and their receptors in retinotopic mapping is currently unknown. As well as interpreting molecular gradients, tissues such as the retina need a molecular mechanism for their establishment. As discussed earlier, the transcription factor Vax-2 is preferentially expressed in ventral retina. In Vax-2 mutant animals, ephrins and their receptors show altered gradients in both nasal-temporal and dorsoventral axes, suggesting that this transcription factor is one specifier of axial polarity in the retina (Mui et al., 2002). Exactly how Vax-2 can produce different levels of expression in different cells is not known. The gradients described to date cannot fully explain the formation of topographic maps. Unless there is some mechanism for producing and sensing absolute levels of a receptor or ligand, a single gradient is unlikely to be able to give a clear topographic organization. It has often been suggested that a pair of gradients in opposite directions is a more efficient mechanism because it allows the sensing of relative levels to determine position. It seems likely that this other gradient is attractive rather than repulsive, although its nature has yet to be characterized.

Retinal cell survival and cell death Understanding the ways in which retinal cells develop is very important for providing therapies for diseases that lead to retinal cell death. Use of growth factors to promote cell survival, use of transplanted retinal cells as replacement

42

 

therapy, and even use of retinal stem cells all require a knowledge of pathways that regulate retinal cell phenotypes. The most common forms of blindness occur through the death of retinal ganglion cells or photoreceptors. Loss of ganglion cells is the defining characteristic of glaucoma, which remains one of the leading causes of blindness in adults (Quigley et al., 1995). Loss of photoreceptors can involve rods, as in retinitis pigmentosa; or cones, as in macular degeneration; or both, as in rod-cone dystrophies and various syndromes. A major risk factor for glaucoma is elevated intraocular pressure, which is thought to cause mechanical deformation as the ganglion cells axons leave the eye. This deformation may prevent the retrograde transport of neurotrophic factors. Based on developmental studies showing that brain derived neurotrophic factor (BDNF) is a vital neurotrophic factor for ganglion cells, it has been suggested that BDNF, or agents that mimic BDNF at later points in its pathway of action, might have therapeutic benefit. A second major cause of ganglion cell death in glaucoma is thought to be excitotoxic injury caused by excess glutamate. This might occur as a result of ischemic damage from elevated intraocular pressure or from other retinal defects in cases where pressure seems normal. Excess glutamate has been measured in glaucomatous eyes, and experimental application of glutamate can certainly induce ganglion cell death (Dreyer et al., 1996; Otori et al., 1998). One mechanism that might account for abnormal glutamate metabolism is abnormal uptake or handling by Müller glial cells (Kawasaki et al., 2000). Although glaucomatous eyes do appear to have lower levels of the Müller cell glutamate transporter, it has yet to be shown that this is a primary cause of the disease. Defects at other steps in the glutamate pathway, including glutamine synthetase and glutamine transporters, are also candidates for causing glaucoma. Since several of these molecules show regulation by extrinsic molecules such as corticosteroids, changes in their function could arise from a number of causes. The differences between normal and diseased retinas may initially be very slight. In addition to defects in specific molecules, overall developmental differences that, for example, alter the relative numbers or spacing of ganglion cells and Müller glial cells could influence the tissue’s ability to metabolize glutamate. Retinitis pigmentosa (RP) is an inherited condition in which loss of rod photoreceptors leads to loss of night vision and, later, deleterious secondary changes in cone photoreceptors (Berson, 1993). A wide array of genes have been identified as causing RP including most of those specific to the phototransduction cascade. Some are loss of function, such as mutations in the cGMP phosphodiesterase; some are structural, such as mutations in the rod outer segment protein peripherin; and still others are “metabolic,” such as mutations in the visual pigment protein opsin that seem to

clog up the Golgi apparatus of the cell. Mutations in the CRX transcription factor gene can cause loss of rods, and later of cones as well (Freund et al., 1997). Thus, many of the molecules that are thought of as developmental regulators may also play a role in later disease. Trophic factors are thought to play an ongoing role in rod photoreceptor survival. In mouse and rat models of RP, it has been shown that some of the same trophic factors important in rod development and survival can prevent cell death for extended periods (LaVail et al., 1998). Interestingly, one of the factors that helps is CNTF, a factor that can completely block rod formation during development. Such a difference emphasizes that the same molecule can have very different functions at different stages of development. Only a few clearly inherited forms of macular degeneration (MD) have been described. MD is generally much later in onset and can probably result from many different initial causes (Stone et al., 2001). Most vision loss in MD is caused by rapid proliferation of blood vessels between the RPE and the photoreceptor layer of the retina. This is thought to occur as a secondary consequence of changes in the RPE layer, the photoreceptor layer, or both. It may turn out that the majority of cases of MD are due to defects in the RPE layer. As expected for such a late-onset disease, most forms of MD do not show clear patterns of inheritance. This is probably due to the influence of multiple genes. The beststudied examples of gene mutations that cause MD suggest that abnormal handling of retinoid metabolism in the visual cycle can lead to deposits that cause cone photoreceptor death. It remains to be seen whether mutations in transcription factor genes, such as the homologs of CRX, can also cause specific cone diseases. One of the therapies under investigation for photoreceptor degenerative disease is transplantation or cell replacement. For this to be effective, it is essential to know how to treat tissue so that photoreceptors survive and remain functional. We still need to know much more about the factors regulating formation of photoreceptors, particularly those regulating outer segment development. With the increased interest in and potential of stem cell therapies, it is essential that we understand the nature and sequence of factors necessary to turn a cell from a multipotential stem cell or progenitor into the desired phenotype of photoreceptor.

Conclusions This chapter has presented a very narrow view of retinal development in terms of a series of biochemical pathways. Prior to the completion of the sequencing of the human genome, it was often argued that the complexity of the nervous system arose because of the existence of a vast number of “brain-specific” genes. This does not appear to be the case. Many of the genes used in neural development

are also used in other organs. Many of the genes are also used in multiple regions and at multiple times. Thus, using different combinations of molecules, or the same combinations in temporally and spatially distinct compartments, can generate complexity in the organization of regions of the nervous system. A number of examples of this in retinal development have been discussed. Gene products such as Pax-6 are important in the development of multiple brain regions. Nevertheless, this molecule is critical for eye formation and, in the correct environment, can initiate the whole process of optic vesicle formation. The protein Otx-2 is used in both RPE and retinal compartments, but these are made distinct by expression of other molecules. Otx-2 is also expressed during the development of several retinal cell types but, since these develop at different times from different sets of progenitors, its functions at each stage are unique. Retinal development does not stop with the events of neurogenesis and maturation described in this chapter. There is obviously much more that could be written about the mechanisms of process growth within, and outside of, the retina and the molecular mechanisms governing synaptogenesis. These phenomena can be studied at the same level of molecular detail as earlier developmental stages. In the foreseeable future, we should have a reasonably complete description of retinal development. If the past is any guide, insights that arise from these studies will be of great value in understanding not only other parts of the visual system, but also other parts of the brain.

Acknowledgments Work from my own laboratory has been supported by NIH Grants EY 11356 and EY 13 865, as well as by the Allene Reuss Memorial Trust, the Kemper Foundation, and Research to Prevent Blindness, Inc. REFERENCES Austin, C. P., D. E. Feldman, J. A. Ida, and C. L. Cepko, 1995. Vertebrate retinal ganglion cells are selected from competent progenitors by the action of Notch, Development, 121:3637–3650. Baas, D., K. M. Bumsted, J. A. Martinez, F. M. Vaccarino, K. C. Wikler, and C. J. Barnstable, 2000. The subcellular localization of Otx2 is cell-type specific and developmentally regulated in the mouse retina, Mol. Brain Res., 78:26–37. Belliveau, M. J., and C. L. Cepko, 1999. Extrinsic and intrinsic factors control the genesis of amacrine and cone cells in the rat retina. Development—Suppl., 126:555–566. Berson, E. L., 1993. Retinitis pigmentosa. The Friedenwald Lecture, Invest. Ophthalmol. Vis. Sci., 34:1659–1676. Bumsted, K. M., and C. J. Barnstable, 2000. Dorsal retinal pigment epithelium differentiates as neural retina in the microphthalmia (mi/mi) mouse, Invest. Ophthalmol. Vis. Sci., 41:903–908. Chow, R. L., C. R. Altmann, R. A. Lang, and A. HemmatiBrivanlou, 1999. Pax6 induces ectopic eyes in a vertebrate, Development—Suppl., 126:4213–4222.

:  

43

Ciossek, T., B. Monschau, C. Kremoser, J. Loschinger, S. Lang, B. K. Muller, F. Bonhoeffer, and U. Drescher, 1998. Eph receptor-ligand interactions are necessary for guidance of retinal ganglion cell axons in vitro, Eur. J. Neurosci., 10:1574–1580. Constantine-Paton, M., A. S. Blum, R. Mendez-Otero, and C. J. Barnstable, 1986. A cell surface molecule distributed in a dorsoventral gradient in the perinatal rat retina, Nature, 324:459–462. Drager, U. C., E. Wagner, and P. McCaffery, 1998. Aldehyde dehydrogenases in the generation of retinoic acid in the developing vertebrate: a central role of the eye, J. Nutr., 128:463S– 466S. Dreyer, E. B., D. Zurakowski, R. A. Schumer, S. M. Podos, and S. A. Lipton, 1996. Elevated glutamate levels in the vitreous body of humans and monkeys with glaucoma, Arch. Ophthalmol., 114:299–305. Ezzeddine, Z. D., X. Yang, T. DeChiara, G. Yancopoulos, and C. L. Cepko, 1997. Postmitotic cells fated to become rod photoreceptors can be respecified by CNTF treatment of the retina, Development, 124:1055–1067. Feldheim, D., Y. Kim, A. Bergemann, J. Frisen, M. Barbacid, and J. Flanagan, 2000. Genetic analysis of ephrin-A2 and ephrin-A5 shows their requirement in multiple aspects of retinocollicular mapping, Neuron, 25:563–574. Flanagan, J. G., and P. Vanderhaeghen, 1998. The ephrins and Eph receptors in neural development, Annu. Rev. Neurosci., 21:309–345. Freund, C. L., C. Y. Gregory-Evans, T. Furukawa, M. Papaioannou, J. Looser, L. Ploder, J. Bellingham, D. Ng, J. A. Herbrick, A. Duncan, S. W. Scherer, L. C. Tsui, A. Loutradis-Anagnostou, S. G. Jacobson, C. L. Cepko, S. S. Bhattacharya, and R. R. McInnes, 1997. Cone-rod dystrophy due to mutations in a novel photoreceptor-specific homeobox gene (CRX ) essential for maintenance of the photoreceptor, Cell, 91:543–553. Furukawa, T., C. A. Kozak, and C. L. Cepko, 1997. Rax, a novel paired-type homeobox gene, shows expression in the anterior neural fold and developing retina, Proc. Natl. Acad. Sci. USA, 94:3088–3093. Golden, J. A., A. Bracilovic, K. A. McFadden, J. S. Beesley, JL RR, and J. B. Grinspan, 1999. Ectopic bone morphogenetic proteins 5 and 4 in the chicken forebrain lead to cyclopia and holoprosencephaly, Proc. Natl. Acad. Sci. USA, 96:2439–2444. Halder, G., P. Callaerts, and W. J. Gehring, 1995. Induction of ectopic eyes by targeted expression of the eyeless gene in Drosophila, Science, 267:1788–1792. Hallonet, M., T. Hollemann, T. Pieler, and P. Gruss, 1999. Vax1, a novel homeobox-containing gene, directs development of the basal forebrain and visual system, Genes Dev., 13:3106–3114. Hatini, V., W. Tao, and E. Lai, 1994. Expression of winged helix genes, BF-1 and BF-2, define adjacent domains within the developing forebrain and retina, J. Neurobiol., 25:1293–1309. Hill, R. E., and I. M. Hanson, 1992. Molecular genetics of the Pax gene family, Curr. Opin. Cell Biol., 4:967–972. Hirsch, N., and W. A. Harris, 1997. Xenopus Pax-6 and retinal development, J. Neurobiol., 32:45–61. Holt, C. E., T. W. Bertsch, H. M. Ellis, and W. A. Harris, 1988. Cellular determination in the Xenopus retina is independent of lineage and birth date, Neuron, 1:15–26. Hyatt, G. A., E. A. Schmitt, N. Marsh-Armstrong, P. McCaffery, U. C. Drager, and J. E. Dowling, 1996. Retinoic acid establishes ventral retinal characteristics, Development, 122:195–204. Kawasaki, A., Y. Otori, and C. J. Barnstable, 2000. Müller cell protection of rat retinal ganglion cells from glutamate and nitric oxide neurotoxicity, Invest. Ophthalmol. Vis. Sci., 41:3444 –3450.

44

 

Kimura, A., D. Singh, E. F. Wawrousek, M. Kikuchi, M. Nakamura, and T. Shinohara, 2000. Both PCE-1/RX and OTX/CRX interactions are necessary for photoreceptor-specific gene expression, J. Biol. Chem., 275:1152–1160. LaVail, M. M., D. Yasumura, M. T. Matthes, C. Lau-Villacorta, K. Unoki, C. H. Sung, and R. H. Steinberg, 1998. Protection of mouse photoreceptors by survival factors in retinal degenerations, Invest. Ophthalmol. Vis. Sci., 39:592–602. Li, H., C. Tierney, L. Wen, J. Y. Wu, and Y. Rao, 1997. A single morphogenetic field gives rise to two retina primordia under the influence of the prechordal plate, Development, 124:603–615. Liu, W., Z. Mo, and M. Xiang, 2001. The Ath5 proneural genes function upstream of Brn3 POU domain transcription factor genes to promote retinal ganglion cell development, Proc. Natl. Acad. Sci. USA, 98:1649–1654. MacDonald, R., K. A. Barth, Q. Xu, N. Holder, I. Mikkola, and S. W. Wilson, 1995. Midline signalling is required for Pax gene regulation and patterning of the eyes, Development, 121:3267–3278. Martinez, J. A., and C. J. Barnstable, 1998. Erx, a novel retinaspecific homeodomain transcription factor, can interact with Ret 1/PCEI sites, Biochem. Biophys. Res. Commun., 250:175–180. Mathers, P. H., A. Grinberg, K. A. Mahon, and M. Jamrich, 1997. The Rx homeobox gene is essential for vertebrate eye development, Nature, 387:603–607. Monaghan, A. P., D. R. Davidson, C. Sime, E. Graham, R. Baldock, S. S. Bhattacharya, and R. E. Hill, 1991. The Msh-like homeobox genes define domains in the developing vertebrate eye, Development, 112:1053–1061. Morabito, M. A., X. Yu, and C. J. Barnstable, 1991. Characterization of developmentally regulated and retina-specific nuclear protein binding to a site in the upstream region of the rat opsin gene, J. Biol. Chem., 266:9667–9672. Morrow, E. M., M. J. Belliveau, and C. L. Cepko, 1998. Two phases of rod photoreceptor differentiation during rat retinal development, J. Neurosci., 18:3738–3748. Mui, S. H., R. Hindges, D. M. D. O’Leary, G. Lemke1, and S. Bertuzzi, 2002. The homeodomain protein Vax2 patterns the dorsoventral and nasotemporal axes of the eye, Development, 129:797–804. Negishi, K., T. Teranishi, and S. Kato, 1982. New dopaminergic and indoleamine-accumulating cells in the growth zone of goldfish retinas after neurotoxic destruction, Science, 216:747– 749. Neophytou, C., A. B. Vernallis, A. Smith, and M. C. Raff, 1997. Müller-cell-derived leukaemia inhibitory factor arrests rod photoreceptor differentiation at a postmitotic pre-rod stage of development, Development, 124:2345–2354. Neumann, C. J., and C. Nuesslein-Volhard, 2000. Patterning of the zebrafish retina by a wave of sonic hedgehog activity, Science, 289:2137–2139. Otori, Y., J. Y. Wei, and C. J. Barnstable, 1998. Neurotoxic effects of low doses of glutamate on purified rat retinal ganglion cells, Invest. Ophthalmol. Vis. Sci., 39:972–981. Planque, N., C. Leconte, F. M. Coquelle, P. Martin, and S. Saule, 2001. Specific Pax-6/microphthalmia transcription factor interactions involve their DNA-binding domains and inhibit transcriptional properties of both proteins, J. Biol. Chem., 276:29330–29337. Quigley, H. A., R. W. Nickells, L. A. Kerrigan, M. E. Pease, D. J. Thibault, and D. J. Zack, 1995. Retinal ganglion cell death in experimental glaucoma and after axotomy occurs by apoptosis, Invest. Ophthalmol. Vis. Sci., 36:774–786.

Reh, T. A., 1987. Cell-specific regulation of neuronal production in the larval frog retina, J. Neurosci., 7:3317–3324. Reh, T. A., and T. Tully, 1986. Regulation of tyrosine hydroxylase–containing amacrine cell number in larval frog retina, Dev. Biol., 114:463–469. Schwarz, M., F. Cecconi, G. Bernier, N. Andrejewski, B. Kammandel, M. Wagner, and P. Gruss, 2000. Spatial specification of mammalian eye territories by reciprocal transcriptional repression of Pax2 and Pax6, Development—Suppl., 127:4325–4334. Sparrow, J. R., D. Hicks, and C. J. Barnstable, 1990. Cell commitment and differentiation in explants of embryonic rat neural retina. Comparison with the developmental potential of dissociated retina, Dev. Brain Res., 51:69–84. Stenkamp, D. L., R. A. Frey, S. N. Prabhudesai, and P. A. Raymond, 2000. Function for Hedgehog genes in zebrafish retinal development, Dev. Biol., 220:238–252. Stone, E. M., V. C. Sheffield, and G. S. Hageman, 2001. Molecular genetics of age-related macular degeneration, Hum. Mol. Genet., 10:2285–2292. Toy, J., J. M. Yang, G. S. Leppert, and O. H. Sundin, 1998. The optx2 homeobox gene is expressed in early precursors of the eye and activates retina-specific genes, Proc. Natl. Acad. Sci. USA, 95:10643–10648. Treisman, J. E., M. A. Morabito, and C. J. Barnstable, 1988. Opsin expression in the rat retina is developmentally regulated by transcriptional activation, Mol. Cell. Biol., 8:1570–1579.

Trisler, G. D., M. D. Schneider, and M. Nirenberg, 1981. A topographic gradient of molecules in retina can be used to identify neuron position, Proc. Natl. Acad. Sci. USA, 78:2145–2149. Turner, D. L., and C. L. Cepko, 1987. A common progenitor for neurons and glia persists in rat retina late in development, Nature, 328:131–136. Turner, D. L., E. Y. Snyder, and C. L. Cepko, 1990. Lineageindependent determination of cell type in the embryonic mouse retina, Neuron, 4:833–845. Waid, D. K., and S. C. McLoon, 1998. Ganglion cells influence the fate of dividing retinal cells in culture, Development, 125:1059–1066. Wang, S. W., B. S. Kim, K. Ding, H. Wang, D. Sun, R. L. Johnson, W. H. Klein, and L. Gan, 2001. Requirement for math5 in the development of retinal ganglion cells, Genes Dev., 15:24–29. Watanabe, T., and M. C. Raff, 1992. Diffusible rod-promoting signals in the developing rat retina, Development, 114:899–906. Wetts, R., and S. E. Fraser, 1988. Multipotent precursors can give rise to all major cell types of the frog retina, Science, 239:1142–1145. Zhang, L., P. H. Mathers, and M. Jamrich, 2000. Function of Rx, but not Pax6, is essential for the formation of retinal progenitor cells in mice, Genesis: J. Geneti. Dev., 28:135–142.

:  

45

4

Neurotrophins, Electrical Activity, and the Development of Visual Function NICOLETTA BERARDI AND LAMBERTO MAFFEI

Introduction Vision in mammals is very poor at birth and develops over a relatively long period (weeks, months, or years, according to the species; Fig. 4.1A) in parallel with the anatomical and functional maturation of the visual system, particularly the visual cortex. If visual experience is altered during this period, called the critical period, dramatic consequences follow both in visual cortical development and in the development of vision. For instance, if during the critical period one eye is deprived of patterned vision, as with unilateral congenital cataract, the great majority of visual cortical neurons stop responding to the deprived eye, being driven only by the normal eye, and vision for the deprived eye develops poorly (amblyopia). There seems to be a close link between visual cortical development, critical period duration, and maturation of some visual functions, as shown in Figure 4.1A: the closure of the critical period for monocular deprivation roughly coincides with completion of visual acuity development in a number of species, from rat to monkey to human. Manipulations of sensory experience also have been shown to affect the development of auditory and somatosensory systems, leading to the widely accepted assumption that the final stage of development of neural connections in sensory systems is under the irreplaceable control of sensory experience. The studies then converged into what may be the various stages for the fulfillement of the task initiated by sensory experience. The first link in the chain is likely to be electrical activity, the language into which sensory experience is translated, which is already known to guide nervous circuit rearrangements and synapse formation. Here we reexamine the role of electrical activity, both spontaneous and visually driven, in the “construction” of the visual system by analyzing the interactions between electrical activity and neurotrophins in the completion of visual function development. Neurotrophins are an essential link in the chain of events leading to maturation of visual connections, a link so necessary that electrical activity in the absence of neurotrophins fails to drive developing visual cortical circuits into their final functional state. For instance, even if visual experience is normal,

46

development of visual function is abnormal if specific neurotrophins are missing. Surprisingly, neurotrophins also seem sufficient to drive the development of some aspects of vision in the absence of visual experience. In presenting the fascinating story of the role of electrical activity and neurotrophins in the development of the visual system, studies which blossomed in the past 10–15 years, we shall follow an historical criterion, which allows us to introduce the various experimental paradigms and the experimental models used. (For recent reviews, see Berardi and Maffei, 1999; Cellerino and Maffei, 1996; McAllister et al., 1999; Pizzorusso and Maffei, 1996).

The neurotrophins Neurotrophins have usually been considered for their involvement in the development of the nervous system and for the differentiation and maintenance of specific functions of certain classes of neurons (see Box 1). The neurotrophins so far identified in addition to NGF, which is the progenitor of the entire family, are BDNF, NT4-5, NT3, and finally NT6, which has been described only in fish (Lewin and Barde, 1996). The action of neurotrophins requires their binding to particular receptors. Each neurotrophin binds to a specific tyrosine kinase receptor (trk) through which it exerts its biological functions. These specific receptors are trkA for NGF, trkB for BDNF and NT4-5, and trkC for NT3 (Bothwell, 1995). Neurotrophic factors also bind to the lowaffinity receptor p75, the role of which has been studied particularly in relation to the ligand NGF. Where NGF is concerned, one of the roles of p75 is to augment trkA function, lowering the concentration of NGF necessary for signal transduction. In addition to this role, p75 can act as an inducer of apoptosis. The classical neurotrophic hypothesis was that developing neurons compete for target-derived neurotrophic factors which are produced in limited amounts by nonneuronal targets (the skin for sensory neurons, the muscle for proprioceptive neurons); neurotrophins bind to their receptors, are internalized, and are transported retrogradely to the neuronal soma, where they promote neuronal survival. If one neurotrophin or its receptor is antagonized or missing,

F 4.1. A, The development of visual acuity for man, monkey, cat, and rat is reported as a function of age and compared with the critical period for monocular deprivation. By the end of the critical period visual acuity reaches its final value, indicating that maturation of sensory functions and decline of experiencedependent plasticity are two closely interconnected processes based on the maturation and activity-dependent stabilization of neural connenctions in the visual cortex. A variation of the critical period duration implies a variation of the rate of visual function development (see Fig. 4.8). B, Activity-dependent synapse stabilization. The crucial elements for the new neurotrophic hypothesis are depicted

here: production, release, and uptake of neurotrophins (NT) are activity dependent. The connections with electrical activity (depicted as a sequence of action potentials above the fiber) which is higher, more patterned, and richer in transients receive a greater NT supply (thick, gray arrow) and stabilize or enlarge their synaptic contacts (big, thick knob at the end of the fiber) with the target neuron, while the connections with poor electrical activity get weak neurotrophic support (thin, gray arrow) and lose their synaptic contacts (small, thin knob). Anterograde action of NT is also possible (white arrow). (Adapted from Berardi et al., 2000.)

  : , ,   

47

B 1. Classical Neurotrophic Hypothesis During development of the nervous system, neuronal populations undergo a process of naturally occurring cell death at the time their axons innervate target areas. It is believed that this process ensures a match between the size of the innervating population and the size of its target territory. The classical neurotrophic hypothesis states that production of trophic factors by target organs regulates this matching process. Limited amounts of neurotrophic factors are produced by target cells: neurotrophic factors bind specific receptors, are internalized by projecting neurons, and are transported retrogradely to the soma, where they promote survival. This hypothesis was first formulated for PNS neurons by Hamburger, Levi-Montalcini, and coworkers (reviews in LeviMontalcini, 1987; Purves, 1988) on the basis of their landmark experiments with nerve growth factor (NGF), the first neurotrophic factor discovered: they demonstrated in vivo that this factor is essential for survival of sympathetic and nociceptive sensory neurons. Subsequently, each of the major predictions of the classical neurotrophic hypothesis for NGF was confirmed:

NGF is produced by the nonneuronal targets of sympathetic and sensory neurons but not by the neurons themselves; target ablation or blockage of axonal transport results in the death of neonatal sympathetic and sensory neurons; administration of NGF prevents naturally occurring cell death; NGF binds to specific receptors (trkA) and is retrogradely transported. Now this hypothesis has been extended to the other neurotrophins: brain-derived neurotrophic factor (BDNF), neurotrophin 3 (NT3), and neurotrophin 4 (NT4). The specific receptor for BDNF and NT4 is trkB and for NT3 it is trkC; all neurotrophins, NGF included, also bind to p75. It is now clear from examination of mutant mice carrying deletions for the genes encoding neurotrophins and their receptors that specific neuronal populations require trophic support of specific neurotrophins from their final targets. A table of neuronal losses in neurotrophin and receptor deficient mice is presented below. For a review on neurotrophic factors and their receptors see Bothwell (1995), Lewin and Barde (1996), and Reichardt and Farinas (1997).

Neuronal Population Sensory Trigeminal

trkA Deletion

NGF Deletion

trkB Deletion

BDNF Deletion

NT4 Deletion

trkC Deletion

NT3 Deletion

75%

75%

60%

30%

n.s.

ND

60%

Dorsal root ganglia

70% 70% Nociceptive and thermoceptive neurons missing

30%

35% Proprioceptive fibers (1a) present

n.s.

20% Proprioceptive neurons missing

60% Proprioceptive and cutaneous mechanoreceptors missing

Sympathetic Superior >95% >95% ND n.s. n.s. n.s. 50% cervical ganglion Neuronal losses are expressed as the percentage of neurons lost in mutant compared to wild-type controls. n.s. = not significantly different. ND = not determined. Only some of the populations examined in the original papers are reported for the sake of brevity.

specific neuronal populations in the peripheral and autonomic nervous systems fail to develop or do not survive (see Box 1). For instance, in mice deficient in trkA or NGF, virtually all sympathetic neurons in the superior cervical ganglion die, while in mice deficient in trkC or NT3, proprioceptive neurons die. At variance with what is found in the peripheral or autonomic nervous system, no central nervous system (CNS) population is solely dependent on one neurotrophin for its survival. Indeed, in neurotrophin knockout mice, there is no loss of any specific population of neurons in the CNS (Thoenen, 1995).1 With hindsight, this observation already hinted at the possibility that the role of neurotrophins in the CNS was not neuronal survival. However, it took a completely dif-

48

 

ferent series of experiments to show clearly that neurotrophins have an important role in the plasticity of the CNS, leading to the formulation of a new neurotrophic hypothesis. These experiments were performed in the developing visual system.

1

Note: Several authors have employed neurotrophins to save neurons from lesion-induced death. Paradigmatic is the effect of NGF on the survival of cholinergic basal forebrain neurons after axotomy. Retinal ganglion cell survival is also increased by exogenous neurotrophins. This, however, does not mean that the exogenous neurotrophin can substitute for the loss of the target-derived endogenous one. Indeed, blockage of axonal transport does not cause appreciable death of retinal ganglion cells.

A new neurotrophic hypothesis Functional properties of mammalian visual cortical neurons are immature at eye opening and develop gradually during the first months of postnatal life (Fagiolini et al., 1994). Development of the visual system is strongly influenced by depriving one eye of patterned vision during a short period of postnatal development called the critical period for the effects of monocular deprivation (which, from now on, will be referred to as simply the critical period). Modifications of cortical circuitry in response to an imbalance between the inputs from the two eyes are extremely rapid; for instance, a few hours of monocular deprivation during the critical period are sufficient to shift the ocular dominance distribution of visual cortical cells toward the nondeprived eye, and a few days are enough to produce a shift which is equal to that induced by a deprivation lasting for the entire critical period. What are the mechanisms leading to such dramatic modifications of cortical connections? Wiesel and Hubel introduced in visual physiology the important concept of binocular competition (Wiesel and Hubel, 1963). The two eyes compete for functional possession of the binocular cortical neurons, and the competition takes the form of electrical activity. If electrical activity in the two sets of thalamic fibers, those driven by the contralateral eye and those driven by the ipsilateral eye, is temporally correlated, then both sets of fibers will be allowed to maintain connections with the same cortical neuron. If, however, the activity in the two sets of fibers is not temporally correlated, only one set of fibers will be allowed to keep its hold on the postsynaptic neuron, the one whose activity is more able to drive it. In normal development, where the activity in the two sets of fibers driven by either eye is equally strong and temporally patterned, this process of activity-dependent competition leads to the existence of binocular neurons and to a balanced ocular dominance distribution: neurons in the visual cortex have very similar probabilities of being dominated by either eye. During monocular deprivation the competition between the two eyes becomes uneven, because electrical activity in the afferent fibers driven by the deprived eye is both uncorrelated with that of the fibers driven by the undeprived eye and weaker; as a result, the closed eye loses the fight at cortical level, leaving the dominance of cortical neurons to the undeprived eye. It is not clear what the two eyes compete for at a molecular level. A reasonable hypothesis is that they compete for a reward important for their function. A reward can be thought of as chemical messages that strengthen nervous connections; therefore, it can be said that the two eyes, during development, compete for eating. Our initial hypothesis (see Fig. 4.1B) was that the fibers driven by either eye compete for a neurotrophic factor available in only a limited amount at cortical level (Maffei et al., 1992).

This introduced the neurotrophic hypothesis in the CNS, transforming neurotrophins from survival factors for neurons, derived from nonneuronal targets, to survival factors for neural connections, exchanged from neuron to neuron as they establish functional connections. This new neurotrophic hypothesis envisaged two broad fields of action for neurotrophins in the CNS. The first stage could influence the probability of formation of synaptic contacts between incoming fibers and target neurons. The second stage could be the regulation of synaptic efficacy, maintenance of connections, and development of a function, as in binocular vision development. This hypothesis implies that the production and uptake of the neurotrophic factor are functions of the quantity and pattern of electrical activity at both presynaptic and postsynaptic levels, and that neurotrophic factors, in turn, can enhance synaptic transmission at both the functional and morphological levels, thus firmly linking together neurotrophins and electrical activity in the control of visual development. It should be noted that at the time this new hypothesis concerning the role of neurotrophins in activity-dependent synaptic plasticity was put forth, the reciprocal control between neurotrophins and electrical activity, now well characterized, was totally unknown. The first demonstrations that neurotrophin production was under the control of electrical activity occurred around 1990 (Ernfors et al., 1991; Zafra et al., 1990), showing that artificially increasing the electrical activity in the hippocampus or neocortex increased both the mRNA and protein of neurotrophins; later on, it was shown that this also promoted their release (Blochl and Thoenen, 1995). Also, protocols inducing long-term potentation (LTP) in the hippocampus were then shown to increase neurotrophin mRNA (Castren et al., 1993). Complementary to this, a decrease of activity by tetradotoxin (TTX) decreased neurotrophin mRNA (Castren et al., 1992). In addition to the two main differences already pointed out between the classical neurotrophic hypothesis for the PNS and the new one for the CNS (formation and survival of connections and not of neurons, produced by neurons and not by nonneuronal targets), another difference is emerging from the literature, namely, the possibility of an anterograde action of neurotrophins as opposed to the classical target-derived action. This significantly changes the frame of thought: in addition to thinking that cortex-derived factors guide, in concert with electrical activity, stabilization of thalamic afferents on cortical neurons, we may have to consider that thalamic fibers themselves release factors which promote and guide the formation and maintenance of their synapses on cortical neurons and that corticothalamic afferents may contribute to the development of the pattern of thalamocortical connectivity The evidence for anterograde actions is illustrated in Box 2.

  : , ,   

49

The rationale for most of the experiments performed to assess the role of neurotrophins in the changes induced at

the visual cortical level in monocularly deprived and strabismic animals is the following: if one provides exogenously the neurotrophic factor to the neural competitors, they should not have any reason to fight and even the geniculocortical axons driven by the eye unstimulated or not properly stimulated by vision might achieve enough neurotrophic factor at the level of its cortical projections to ensure its physiological development. In short, the working hypothesis is that the effects of monocular deprivation or strabismus should be prevented by an exogenous supply of NGF. This hypothesis may now appear somewhat simpleminded, but it turned out to be very appropriate operationally. Indeed, an intraventricular exogenous supply of NGF in the rat prevents all the tested effects of monocular deprivation and (convergent) strabismus (Domenici et al., 1991, 1992; Maffei et al., 1992). Interestingly, these results have been recently confirmed in another rodent, the mouse, by Fagiolini and Stryker (1996). In cats, an intraventricular supply of NGF was effective in attenuating monocular deptivation effects up to the behavioral level; visual acuity of the deprived eye developed normally in kittens treated with NGF (Fiorentini et al., 1995). A synopsis of the main results concerning NGF and monocular deprivation is presented in Figure 4.2. More recently, NGF was directly provided to cortical neurons by intracortical infusion; this also prevented monocular deprivation effects on ocular dominance (Lodovichi et al., 2000), suggesting that the receptors for NGF should be

F 4.2. Intraventricular NGF administration counteracts the effects of monocular deprivation on ocular dominance and visual acuity. Top: Ocular dominance distributions for normal rats, rats monocularly deprived during the critical period and rats monocularly deprived and treated with NGF throughout the deprivation period (postnatal days 15–45, P15–P45). Cells were classified according to the Hubel and Wiesel criteria, with cells in classes 1 and 7 being monocular and driven exclusively by the contralateral and ipsilateral eye, respectively; cells in ocular dominance classes 2/3 and 5/6 being binocular and driven preferentially by the contralateral and ipsilateral eye, respectively; and cells in class 4 being binocular and driven equally by the two eyes. It is evident that the shift in ocular dominance distribution in monocularly deprived rats is absent in NGF-treated rats. (Data replotted from Maffei et al., 1992.) Middle: A, Visual evoked potentials (VEP) recorded in the visual cortex contralateral to the deprived eye in response to sinusoidal gratings of increasing spatial frequencies (examples of sinusoidal gratings with spatial frequency in the ratio 1 : 2 : 4 are reported on the left of the graphs). Mean relative VEP amplitude (normalized to the amplitude of the signal recorded for 0.2 cycle/deg) is reported as a function of stimulus spatial frequency for monocularly deprived control animals and monocularly deprived NGFtreated animals. Vertical bars represent standard deviations. The noise level is indicated by the dotted line. As the spatial frequency of the stimulus increases, the signal amplitude decreases until it falls

to the noise level. The highest spatial frequency still evoking a reliable signal with a maximal stimulus contrast is taken as the visual acuity. The symbols just above the noise level correspond to estimated mean visual acuity for the deprived and nondeprived eyes, which is 0.4 ± 0.7 vs. 1 ± 0.04 cycles/deg in monocularly deprived control animals and 0.97 ± 0.1 vs. 1 ± 0.08 cycles/deg in monocularly deprived NGF-treated animals. The difference between visual acuity in the deprived and undeprived eyes is significant for monocularly deprived animals but not for monocularly deprived animals treated with NGF (replotted from Domenici et al., 1991). To visualize the difference in acuity brought about by NGF in deprived animals, the ratio between the spatial frequency of the gratings below the abscissa is 1 : 2.5, which is the ratio between visual acuity for the deprived eye in control animals and for the deprived eye in NGF-treated animals. Bottom: B, Percentage of correct responses in a behavioral forced choice discrimination of a grating versus a uniform gray (N = 85 trials per point) plotted as a function of the spatial frequency of the grating for a kitten monocularly deprived from P30 to P45 and for a littermate deprived for the same period but treated with NGF. The estimated visual acuities (corresponding to 70% correct) for the deprived and nondeprived eyes are indicated by the arrows on the abscissa (adapted from Fiorentini et al., 1995). It is clear that the beneficial effects of NGF in preventing deprivation amblyopia (loss of visual acuity in the deprived eye) are present at the level of visual behavior.

B 2. Anterograde and Retrograde Actions of Neurotrophins Following the experiments in the PNS, the concept had been accepted that neurotrophins are transported retrogradely and that this is the basis of their action. This idea has been passively extended to the interpretation of CNS data. In some instances, particularly for NGF and the cholinergic projection from the basal forebrain to the hippocampus, this assumption seems to hold. In other cases, it is supported only by the observation that exogenous neurotrophins injected into specific brain regions are retrogradely transported. However, recent experiments studying the transport of BDNF/NGF in the optic nerve of the rat have shown that the situation is somewhat more complicated. If one ligates the optic nerve and observes the accumulation of neurotrophins at both sides of the ligature, only accumulation of BDNF on the retinal side is seen; by contrast, if NGF and BDNF are injected in the superior colliculus and lateral geniculate nucleus, both promptly accumulate at the distal side of the ligature. This suggests that transport of exogenous neurotrophins is not proof of the transport of endogenous ones and that anterograde actions are more important than was previously thought (Caleo et al., 2000). Evidence for anterograde actions has also been obtained in the visual cortex (Kohara et al., 2001) and in the chick visual system (von Bartheld et al., 1996).

NGF and monocular deprivation



50

 

present at the cortical level. Indeed, trkA and p75 are present in the visual cortex during the critical period, and activation of cortical trkA receptors is sufficient to allow normal development of ocular dominance in monocularly deprived rats, mimicking the effects of NGF (Pizzorusso et al., 1999). Thus, even in the presence of an abnormal visual experience, an increase in NGF availability to cortical neurons or activation of NGF cortical receptors allows normal development of visual cortical connectivity and, ultimately, of vision.

Administration of neurotrophins during dark rearing can replace the lack of visual experience Without visual experience the visual cortex does not develop normally, and it remains largely immature even after the end of the critical period (Fagiolini et al., 1994; Timney et al., 1978). The classical signs of dark-rearing effects on cortical development are reported in Figure 4.3 and include abnormal habituation of cortical responses and decreased visual acuity.

  : , ,   

51

Following the experiments on the effects of NGF in monocularly deprived animals, the hypothesis was advanced that visual experience during development promotes normal maturation of the visual cortex by regulating the availability of neurotrophins to visual cortical circuits or to structures projecting to them (Fagiolini et al., 1997; Pizzorusso et al., 1997). The physiological alterations induced by lack of visual experience in dark-reared animals could be due simply to an alteration in the level of expression of neurotrophins in the visual cortex. If so, increasing the availability of neurotrophins could replace, at least partly, the lack of visual experience during dark rearing. This hypothesis has been tested for NGF and BDNF. Supplying animals with suitable doses of neurotrophins during the whole period of dark rearing poses technically difficult problems. It is known that neurotrophins do not cross the blood-brain barrier and therefore must be administered directly to the brain. Daily administration of neurotrophins, for instance, into the ventricles, is practically impossible or troublesome during dark rearing. To overcome this problem, three different methods have been used. All of them have been successful and could have a bearing on clinical applications of neurotrophins in humans. Conceptually the first two methods are very similar, in that they aim at implanting in the lateral ventricles biological minipumps delivering sufficient doses of neurotrophins (in this case NGF) throughout the whole period of dark rearing. As biological minipumps, either Schwann cells or polymer-encapsulated cells genetically engineered to release NGF have been employed (for details of the methods and results, see Fagiolini et al., 1997; Pizzorusso et al., 1997). The results, summarized in Figure 4.3, show that NGF allows a normal or nearly normal development in darkreared animals with respect to the tested parameters. In particular, Figure 4.3 shows that development of visual acuity is normal in dark-reared rats treated with NGF. More recently, the problem of supplying neurotrophins to the CNS has been solved by taking advantage of BDNF overexpression in the postnatal forebrain of mice (Huang et al., 1999; see Fig. 4.7). Also in these animals, all tested parameters were normal despite dark rearing (Fig. 4.3). In addition, in dark-reared BDNF-overexpressing mice we have observed that the critical period, which is normally prolonged by dark rearing, ends at the same time as in normally reared wild-type mice (P45). Thus, development of visual function can proceed almost normally even in the absence of visual experience, provided that neurotrophins are supplied. A possible interpretation of these results is that visual experience controls development of visual function by controlling the expression of neurotrophins. Indeed, expression and function of neurotrophins are altered in dark-reared animals (Castren et al., 1992; Pollock et al., 2001; Viegi

52

 

et al., 2002), suggesting that the effects of dark rearing could, at least in part, be attributed to the lack of appropriate neurotrophin action. This allows us to speculate that neurotrophins engage in an innate program of visual development which is normally triggered by visual experience. This program is dependent on activity in the visual pathways, in particular on spontaneous activity. Blockage of spontaneous activity of retinal ganglion cells, at least in the case of monocular deprivation, caused a failure of NGF action (Caleo et al., 1999).

Blockage of endogenous NGF affects development of the visual system If neurotrophic factors are a crucial link between activity and development of appropriate connectivity in the visual system, either by acting as retrograde messages from neurons or dendrites to nerve terminations, to selectively reinforce active inputs, as Donald Hebb envisaged, or by acting anterogradely, then the blockage of endogenous neurotrophins during the critical period should interfere with development of the visual system: connections should remain immature and the system plastic—for instance, sensitive to monocular deprivation—beyond the critical period. These predictions were tested by the intraventricular transplant of cells secreting antibodies against NGF during development (Berardi et al., 1994; Domenici et al., 1994), and it was shown that they were fulfilled (Fig. 4.4). Visual acuity was reduced, receptive fields remained large, and the critical period for monocular deprivation was abnormally prolonged, effects very similar to those produced by dark rearing. Very similar results have been obtained in cats blocking endogenous trkB ligands; infusion of BDNF and NT4 scavengers (trkB-IgG fusion proteins) into the visual cortex (Cabelli et al., 1997) disrupts ocular dominance column formation (Fig. 4.5), an effect reminiscent of that produced by binocular deprivation (Crair et al., 1998). Thus, not only can the deleterious effects of an abnormal or absent visual experience on development be counteracted by increasing neurotrophin availability to visual neurons, but a normal visual experience is unable to normally drive development and closure of the critical period if the action of endogenous neurotrophins is blocked. This last result strongly suggests that neurotrophins may be the crucial link in the chain of events linking visual experience with development of vision.

Do BDNF, NGF, NT4, and NT3 play similar or different roles in cortical plasticity and development? The question now arose, what were the roles played by the different neurotrophins in developmental visual cortical plasticity? This question occurred because, over the years,

F 4.3. Top left: summary of the functional properties affected by dark rearing (DR). Habituation is the progressive attenuation of cell responses to repetitive visual stimulation: typically, cell responses disappears after three or four passages of a drifting bar over the receptive field in DR animals (DR from birth to P60). Habituation is absent in normally reared animals. Middle and right: the deficits induced by DR are absent (no significative differences with respect to normal) or strongly attenuated (orientation selectivity, small difference with respect to normal) in rats subjected to intraventricular implant of NGF-producing cells (DR + NGF) or

in mice overexpressing BDNF in the telencephalon (DR + BDNF). Bottom: Summary of the results obtained for visual acuity in DR rats. The shaded area represents the range (mean ± one standard deviation) of visual acuities for normal rats during development from P20 to adulthood. At each age and for each animal visual acuity was estimated by VEP, as in Figure 4.2. DR up to P45 strongly decreases visual acuity ( filled triangle); visual acuity of DR rats with intraventricular implant of NGF-producing cells is, however, normal ( filled circle).

some differences in the actions of the neurotrophins became apparent and a debate developed about the nature of the active neurotrophins(s). As for the effects on cortical development, only BDNF and NT4 disrupt ocular dominance columns in kittens (Cabelli et al., 1995); however, all neurotrophins influence dendritic growth in developing ferret visual cortex (McAllister et al., 1995, 1997). As for the

effects on plasticity, injection into the visual cortex of microbeads conjugated with neurotrophins has shown that NT4, and only NT4, is effective in preventing the shrinkage of lateral geniculate nucleus (LGN) neurons induced by monocular deprivation in ferrets (Riddle et al., 1995). On the other hand, BDNF was found to be able to prevent the shift of the ocular dominance distribution of cortical

  : , ,   

53

Effects of Blocking Endogenous NGF During Development of Rat Visual System

F 4.5. An example of the effects of trkB-IgG infusion on ocular dominance columns. trkB-IgG was infused into cat visual cortex from P21 to P42 at a rate of 2.5 mg/ml. To label ocular dominance patches in layer IV, geniculocortical afferents were labeled with the transneuronal tracer [3H]-proline injected into the vitreous chamber of one eye. Top: darkfield image from an infused brain; bottom: corresponding profile of silver grain density in layer 4. The position of the tip of the infusion cannula is marked with an X. Scale bar, 1 mm. It is evident that the regular alternation of labeled patches (axons of LGN neurons driven by the injected eye) and of gaps (axons of LGN neurons driven by the noninjected eye) is disrupted in the portion of visual cortex infused with trkB-IgG (zone above the cannula tip). (Adapted from Cabelli et al., 1997.)

F 4.4. Top: Summary of the main effects of the blockage of endogenous NGF action by implantation in the lateral ventricle of hybridoma cells secreting antibodies to NGF. The implant was done at the beginning of the critical period (P15). Rats implanted with parental myeloma cells were used as controls. Animals were left to develop normally, with both eyes open. LGN soma size, visual acuity, cell ocular dominance, and receptive field size were measured at P45. Bottom: The effects of NGF blockage on the critical period. Animals were left to develop normally, with both eyes open, up to P45, which is past the end of the rat critical period, and then were monocularly deprived; the effects of monocular deprivation were assessed electrophysiologically 1 month later. It is evident that monocular deprivation is still able to shift the ocular dominance distribution toward the nondeprived eye in rats with blockage of NGF (MD P45, blockage of NGF ) but not in control rats (MD P45). (Adapted from Domenici et al., 1994).

neurons induced by monocular deprivation in cats (Galuske et al., 1996). It became necessary, therefore, to assess whether the differences observed were simply due to different experimental conditions, like the use of different animals, different methods of drug administration, and different ages of the animal, or whether different neurotrophins played different roles. Another important question concerned the effects of neurotrophins on visual cortical cell electrical activity. Many studies had shown that neurotrophins, in particular BDNF, strongly modulate synaptic transmission and electrical activity of cortical neurons in vitro (see McAllister et al., 1999,

54

 

for review). The most striking effect of BDNF was documented by Kafitz et al. (1999) in cultured hippocampal neurons: BDNF was as fast and potent as glutamate, a classical excitatory neurotransmitter, in exciting the neurons onto which it was puffed (Kafitz et al., 1999). Knowledge of neurotrophin effects on cortical cell activity in vivo is essential to understand more thoroughly their mechanisms of action in regulating visual cortical plasticity. Any strong direct effect on visual cortical cell electrical activity is bound to influence developmental cortical plasticity (Ramoa et al., 1988; Reiter and Stryker, 1988; Shaw and Cynader, 1984). A recent study has compared in vivo the actions of all four neurotrophins—NGF, BDNF, NT3, and NT4—on cortical plasticity and on electrical activity in the same species, in animals of the same age, and in the same experimental conditions. This has been done in the rat, employing monocular deprivation as a plasticity test (Lodovichi et al., 2000). NGF and NT4 were found to be very effective in counteracting the shift in the ocular dominance distribution of rat visual cortical neurons induced by monocular deprivation. This protective effect was not accompanied by any detectable changes in cell responsiveness or in orientation selectivity. BDNF, even at much higher doses than NGF and NT4, was less effective in counteracting monocular deprivation effects and, in addition, was the only neurotrophin which altered visual cortical cell electrical activity, both spontaneous and evoked. These results suggest that the partial effect of BDNF on monocular deprivation could stem from its ability to dramatically alter the electrical activity of cor-

tical neurons. NT3 is ineffective in preventing ocular dominance shift, both at a dose comparable to that of NGF and NT4 and at a much higher dose. Thus, the differences found in the literature for neurotrophin action in visual cortical plasticity cannot be attributed solely to species differences: different neurotrophins play their roles in visual cortical plasticity through different mechanisms and, in particular, through a different interplay with electrical activity. This could be due to a difference in the cellular targets of neurotrophins (see below for the peculiar link between BDNF and intracortical inhibitory circuitry) and/or to differences in the intracellular signaling cascades activated. Diversity in the postreceptor transduction pathways is likely to be a necessary explanation for the differences between NT4 and BDNF actions, since they both bind trkB.

Neurotrophins and the modulation of synaptic transmission in the visual cortex A possible mechanism of action of neurotrophins on neural plasticity is the modulation of synaptic efficacy (see McAllister et al., 1999, for review). To investigate whether neurotrophins can modulate synaptic transmission in the visual cortex, a very convenient in vitro preparation, visual cortex synaptosomes, has recently been used. Synaptosomes contain mainly the presynaptic component of synapses, with all the machinery for neurotransmitter release. The advantage of this preparation is that the effectiveness of one neurotrophin in modulating each neurotransmitter system can be investigated in isolation (Fig. 4.6). For instance, an effect of BDNF on acetylcholine release can be attributed to a direct action of BDNF on cholinergic terminals, and not secondary to release of another neurotransmitter acting on cholinergic terminals. Both NGF and BDNF potentiate glutamate (Glu) and acetylcholine (ACh) release, while only BDNF does so for GABA release (Sala et al., 1998) (Fig. 4.6). trkA plays the major role in mediating NGF effects, with p75 playing a small facilitatory role. This suggests a direct trkA-mediated effect of NGF on synaptic terminals of glutamatergic neurons in the visual cortex. More recently, the effects of NT4 on synaptic release were investigated in the same preparation. Like BDNF, NT4 potentiated GABA and Glu release but was much less effective than BDNF in potentiatiating acetylcholine release. Two conclusions can be drawn at this point. First, modulation of synaptic transmission is an important mechanism of action for neurotrophins in the developing visual cortex. Second, different neurotrophins have different targets: NGF modulates synaptic release of cholinergic and glutamatergic terminals but not of GABAergic interneurons; BDNF is active on all three neurotransmitter systems and NT4 on glutamatergic and GABAergic terminals. Putting this informa-

tion together with data on the expression of trk receptors in the visual cortex and with data on retrograde transport of cortically injected NGF (Domenici et al., 1994), it can be concluded that NGF is likely to act directly on cholinergic afferents from the basal forebrain and on a population of glutamatergic cortical neurons and, indirectly, by modulation of the cholinergic function, on cortical inhibitory interneurons (Xiang et al., 1998); BDNF targets are cortical pyramidal cells (glutamatergic), inhibitory interneurons, cholinergic afferents, and serotonergic afferents; NT4 acts on thalamic afferents (glutamatergic), probably pyramidal neurons, and certainly inhibitory interneurons. The likely action of NT4 on thalamic afferents coud explain its effectiveness in preventing monocular deprivation effects without disturbing visual cortical neuron electrical activity. The same ability could be conferred on NGF by its combined action on a neuromodulatory system such as the cholinergic system (which also provides indirect control of some interneuronal populations) and on a selected population of glutamatergic cortical neurons. BDNF emerges as the neurotrophin with the largest spectrum of targets and the strongest effectiveness on cortical neurons, both pyramidal and interneurons.

BDNF overexpression accelerates the functional development of the visual cortex The experiments on the role of neurotrophins in modulating synaptic release show that BDNF has a strong effect in potentiating GABA release which is not shared by NGF. These results show that neurotrophins (in particular BDNF and NT4) also act on the inhibitory circuitry. trkB is present on cortical interneurons, and BDNF regulates the development of at least one class of inhibitory interneurons. The relationship between neurotrophins and the development of inhibitory processes has been investigated, using an elegant transgenic mouse with postnatal overexpression of BDNF in the forebrain (Huang et al., 1999), as shown in Figure 4.7A,B. In these animals, the levels of BDNF expression typical of adult age is reached in the second postnatal week (Fig. 4.7B). This precocious expression of BDNF is accompanied by a precocious development of inhibitory synapses (Fig. 4.7C, GAD staining) and of inhibitory currents (Fig. 4.7C, insert). This accelerated development of inhibition is paralleled by changes in the functional development of the visual system: there is precocious development of visual acuity with respect to the wild type and precocious closure of the critical period, possibly accompanied by precocious opening. Figure 4.8 clearly illustrates that in BDNF mice there is a shift toward younger ages of the curve describing the developmental time course of visual acuity and of the curve describing the decline of monocular deprivation effective-

  : , ,   

55

F 4.6. Top: Sketch of the experiment. Left: A synaptosome containing synaptic vesicles filled with neurotransmitter and presenting receptors for neurotrophins is subjected to depolarizing stimuli (high K+ concentrations); this causes release of transmitter (molecules outside the synaptosome). Right: A synaptosome subjected to depolarization is also exposed to neurotrophins. Neurotrophins activate their receptors, and the ensueing transduction

56

 

signals cause an increased release of neurotransmitter. Bottom: NGF and BDNF effects on K+-induced release of neurotransmitter from synaptosomes isolated from P23 rat visual cortices. Each point represents the mean ± SEM of five to seven experiments run in triplicate. The small increase in GABA release is not statistically significant. (Adapted from Sala et al., 1998.)

F 4.7. Precocious expression of BDNF in the telencephalon causes accelerated development of inhibition in the visual cortex. A, Spatial expression of the BDNF transgene revealed by in situ hybridization. Coronal brain sections from 4-week-old transgenic (right) and wild-type mice (left) hybridized with a BDNF oligonucleotide probe that detects both the endogenous and transgenic BDNF mRNA. Expression is restricted to the telencephalon. B, Quantification of total BDNF mRNA levels in the cerebral cortex at different developmental ages (data from Northern blotting experiments) for wild-type and transgenic mice. It is evident that the levels of expression normally reached after the third postnatal week

in wild-type mice are reached at P7 in BDNF mice. C, Development of GAD65, the synthetic enzyme for the inhibitory neurotransmitter GABA, in the visual cortex of wild-type and transgenic mice. Quantification of GAD65 expression in the presynaptic boutons of GABAergic interneurons was done around the soma of the target neurons. In BDNF mice there is accelerated maturation of GABAergic synapses. In the insert, examples of maximal inhibitory postsynaptic currents (IPSCs) recorded at P23–P26 from visual cortical slices of wild-type and transgenic mice. IPSCs are larger in transgenic mice. (Adapted from Huang et al., 1999.)

  : , ,   

57

F 4.8. Critical periods for monocular deprivation and development of visual acuity are reported for wild-type mice, A and transgenic mice with precocious expression of BDNF, B. For BDNF mice, both the critical period (dotted line) and the visual acuity (solid

line) curves are shifted to the left (see crossing points, dashed lines). This indicates that precocious expression of BDNF induces both accelerated development of visual function and an early closure of critical period.

ness during the critical period. This reinforces the close link between the time course for visual acuity development and the time course of the critical period discussed in the Introduction. The data presented in Figures 4.7 and 4.8 suggest the following considerations, which are relevant for the studies on development of neocortex: 1. Transgenic animals can be extremely useful in studies on development of visual function. 2. Modulation of a single molecule, in this case BDNF, can have dramatic consequences for the development of visual processes and the duration of the critical period. The observation that in the presence of normal visual experience overexpression of BDNF causes precocious development of vision can be considered the mirror finding to that obtained with blockage of endogenous neurotrophins: in that case, normal visual experience was unable to drive development, suggesting that neurotrophins are crucial effectors through which visual experience drives development of visual function. In BDNF-overexpressing mice, as in dark-reared animals with an increased supply of neurotrophins, neurotrophins seem to “substitute” for experience in driving development. 3. There is a new player on the stage of visual development, namely, inhibition. That inhibition is important for plasticity was previously suggested by experiments showing

that monocular deprivation is less effective in mice with reduced inhibitory transmission (GAD65 knockout mice) (Hensch et al., 1998). The data on the BDNF mouse, with precocious closure of the critical period and normal monocular deprivation effectiveness, clearly indicate that development of inhibition is important for the time course of the critical period. This point is further strengthened by the elegant experiment of Fagiolini and Hensch (2000) showing that precocious enhancement of inhibitory tone accelerates the opening of the critical period. 4. It seems that both NGF and BDNF are important in determining the timing of the critical period and of visual cortical development, pointing toward a more complex role for neurotrophins: they do not seem to be only retrograde, activity-dependent rewards for active thalamic afferents, but also important determinants of development for populations of cortical neurons.

58

 

Experience and neurotrophins: intracellular mechanisms Several intracellular mechanisms are likely to be important in mediating the action of electrical activity and neurotrophins on structural and functional plasticity. Amongst these we have chosen to illustrate the involvement of a particular cascade, the ERK 1,2, also called p42/p44 MAPK.

F 4.9. MAPK activation is required for visual cortical plasticity. A, Pathways of MAPK activation. MAPK is activated both by neurotrophins (NT) binding to their trk receptors via the transduction cascade of ras and by the influx of calcium through voltage-gated (V-gated) calcium channels and through glutamate NMDA receptors, also acting on ras. Electrical activity can therefore activate MAPK via two different converging pathways, and MAPK is a a crucial converging point for the integrated action of electrical activity and neurotrophins. Activated MAPK can both act on local targets (synaptic proteins, adhesion molecules) and translocate to the nucleus, where, via kinases such as RSK, it can activate CREB, a transcription factor crucial for many plasticity phenomena. Other pathways impinging on MAPK are omitted for simplicity. The site of action of the two MAPK blockers employed in the experiment is indicated by the arrow. B, MAPK inhibitors

U0126 and PD98059 block LTP induction visual cortical slices. Average time course of layer III field potential amplitude before and after TBS of the white matter in the presence of U0126, PD98059, or vehicle. Field potentials recorded in layer III can be potentiated by TBS in control slices but not in U0126- and PD98059-treated slices. C, Blockage of MAPK activation prevents experience-dependent plasticity in the visual cortex. Ocular dominance distributions for normal P28 animals (NOR) and for animals monocularly deprived from P21 to P28, either untreated (MD) or treated with U0126, vehicle, PD98059, or SB203580 (an inhibitor of a kinases of the same family as MAPK). It is evident that blockage of MAPK activation by U0126 or PD98059 prevents the ocular dominance shift produced by monocular deprivation; the effect is specific for MAPK blockage, since inhibition of the related p38 kinase is uneffective. (Adapted from Di Cristo et al., 2001.)

The MAPK cascade has been involved in phenomena of synaptic plasticity and learning and memory from Aplysia to mammals (see Grewal et al., 1999, for review), but the characteristic which attracted our attention is that this biochemical cascade is sensitive to both electrical activity and neurotrophins, thus being a crucial converging point for the integrated action of the two. In the visual cortex, evidence that neurotrophins activate MAPK has been obtained only recently (Pizzorusso et al., 2000). The authors also show that MAPK activation is crucial for BDNF-induced cAMP response element binding protein (CREB) phosphorylation, which is likely to be an important step in BDNF action on synaptic plasticity. In Figure 4.9A the principal pathways of MAPK activation are shown. It is clear from the figure that MAPK is a

hub linking neurotrophins and elecrical activity with both cytoplasmatic and nuclear targets, an ideal position for a candidate player in visual cortical plasticity. Recently, it has been shown that MAPK is important for experience-dependent plasticity in the visual cortex (Di Cristo et al., 2001). MAPK activation has been blocked with two specific inhibitors. The arrow in Figure 4.9A indicates the exact point of the transduction cascade where the block is performed (the molecule immediately upstream of MAPK, MEK). Two approaches have been used, one in vivo, employing the paradigm of monocular deprivation, to investigate experience-dependent plasticity and one in vitro, using the paradigm of long-term potentiation (LTP) to investigate synaptic plasticity.

  : , ,   

59

Patterned electrical stimulation (theta burst stimulation, TBS) of the white matter, which readily triggers LTP of field potentials recorded in layers II–III, rapidly triggers MAPK activation. Blockage of MAPK activation abolishes induction of LTP in the visual cortex (Fig. 4.9B). The link between MAPK activation and synaptic plasticity in the visual cortex is further confirmed by the observation that abolishing the ability of TBS to trigger LTP with an -methyl--aspartate (NMDA) receptor antagonist also inhibits the ability of TBS to activate MAPK (Di Cristo et al., 2001). In vivo, it has been found that visual experience rapidly triggers MAPK activation. In turn, MAPK activation is crucial for experience-dependent plasticity: the ocular dominance shift induced by monocular deprivation is prevented by both MAPK blockers employed U0126 and PD98050 (Fig. 4.9C). It is important to note that blockage of MAPK activation prevents experience-dependent plasticity without affecting the development of the visual cortex. Visual acuity, receptive fields, and orientation selectivity were normal in PD98050- and U0126-treated animals (Di Cristo et al., 2001). The conclusion is that MAPK, acting on both local and nuclear targets, promotes strengthening of synaptic transmission under the control of electrical activity and neurotrophins. Thus, the effects of neurotrophins in preventing the ocular dominance shift induced by monocular deprivation could be attributed to activation of MAPK, which promotes stabilization of synaptic contacts even for the fibers driven by the deprived eye possessing only spontaneous electrical activity.

Conclusions

F 4.10. Possible sites of action of neurotrophins on visual cortical plasticity. Synergistic and antagonistic effects of neurotrophins, however clearly possible, are not included in the model. Even so, it is clear that several neural circuits are involved. Some targets of neurotrophin action could be common, such as the basal

forebrain cholinergic neurons or the intracortical excitatory circuitry. Some could be specific targets for specific neurotrophins; for instance, NT4 seems to be most effective on thalamic afferents, and BDNF/NT4 seem to be specifically active on inhibitory intracortical circuitry.

60

 

The results reported in this chapter show that visual experience and neurotrophins cooperate in guiding the development of vision. The actions of neurotrophins and electrical activity are linked in a functional chain which is often reciprocal: activity affects neurotrophin production and uptake, neurotrophins regulate synaptic transmission and may affect electrical activity in visual cortical neurons; visual experience is not sufficient to drive visual cortical development if endogenous neurotrophin action is antagonized, and exogenous neurotrophins become uneffective if electrical activity is blocked. At the other end, neurotrophins seem to be able to promote visual development even in the absence of visual experience, as suggested by the experiments with darkreared animals. The neurotrophins active in visual cortical development and plasticity—BDNF, NT4, and NGF—seem to play their roles by acting on different targets: each neurotrophin has its particular subset of targets among the intracortical neurons and the cortical afferents. Some targets are direct, as with the inhibitory circuitry and the neurotrophins BDNF and NT4; some are indirect, in that the action of the neurotrophin is mediated by a neuromodulatory system, such as the cholinergic system. The possible sites of action of neurotrophins in visual cortical plasticity are sketched in Figure 4.10. It is evident that neurotrophin action is not limited to thalamic afferents but involves both cortical circuitry and subcortical afferents.

Up to 10 years ago, neurotrophins were known for their involvement in survival, differentiation, and maintenance of specific classes of neurons in the PNS. Now the scenario has completely changed, and their involvement is at least equally prominent in the CNS. In the CNS, however, neurotrophins are not survival factors for neurons, but rather survival factors for neural connections. In the developing CNS, neurotrophins are essential for the formation, maintenance, and plasticity of synaptic contacts and are crucial for the development of sensory functions. In adult life, they are likely to be still at work in the control of cortical plasticity, and modifications of their activity could be responsible for the slow decline of cognitive functions with age. REFERENCES Berardi, N., A. Cellerino, L. Domenici, M. Fagiolini, T. Pizzorusso, A. Cattaneo, and L. Maffei, 1994. Monoclonal antibodies to nerve growth factor affect the postnatal development of the visual system, Proc. Natl. Acad. Sci. USA, 91:684–688. Berardi, N., and L. Maffei, 1999. From visual experience to visual function, J. Neurobiol., 41(1):119–126. Berardi, N., T. Pizzorusso, and L. Maffei, 2000. Critical periods during sensory development, Curr. Opin. Neurobiol., 10(1):138– 145. Blochl, A., and H. Thoenen, 1995. Characterization of nerve growth factor (NGF) release from hippocampal neurons: evidence for a constitutive and an unconventional sodiumdependent regulated pathway, Eur. J. Neurosci., 7(6):1220–1228. Bothwell, M., 1995. Functional interactions of neurotrophins and neurotrophin receptor, Annu. Rev. Neurosci., 18:223–253. Cabelli, J., A. Hohn, and C. J. Shatz, 1995. Inhibition of ocular dominance column formation by infusion of NT-4/5 or BDNF, Science, 267:1662–1666. Cabelli, R. J., D. L. Shelton, R. A. Segal, and C. J. Shatz, 1997. Blockade of endogenous ligands of TrkB inhibits formation of ocular dominance columns, Neuron, 19:63–76. Caleo, M., C. Lodovichi, and L. Maffei, 1999. Effects of nerve growth factor on visual cortical plasticity require afferent electrical activity, Eur. J. Neurosci., 11(8):2979–2984. Caleo, M., E. Menna, S. Chierzi, M. C. Cenni, and L. Maffei, 2000. Brain-derived neurotrophic factor is an anterograde survival factor in the rat visual system, Curr. Biol., 10(19):1155– 1161. Castren, E., M. Pitkanen, J. Sirvio, A. Parsadanian, D. Lindholm, H. Thoenen, and P. J. Riekkinen, 1993. The induction of LTP increases BDNF and NGF mRNA but decreases NT-3 mRNA in the dentate gyrus, Neuroreport., 4(7):895–898. Castren, E., F. Zafra, H. Thoenen, and D. Lindholm, 1992. Light regulates expression of brain-derived neurotrophic factor mRNA in the rat visual cortex, Proc. Natl. Acad. Sci. USA, 89:9444–9448. Cellerino, A., and L. Maffei, 1996. The action of neurotrophins in the development and plasticity of the visual cortex, Prog. Neurobiol., 49:53–71. Crair, M. C., D. C. Gillespie, and M. P. Stryker, 1998. The role of visual experience in the development of columns in cat visual cortex, Science, 279(5350):566–570. Di Cristo, G., N. Berardi, L. Cancedda, T. Pizzorusso, E. Putignano, G. M. Ratto, and L. Maffei, 2001. Requirement

of ERK activation for visual cortical plasticity, Science, 292(5525): 2337–2340. Domenici, L., G. Fontanesi, A. Cottaneo, P. Bagroli and L. Maffei, 1996. Nerve growth factor (NGF) uptake and transport following injection in the developing rat visual cortex, Vis. Newosci., 11:1083–1102. Domenici, L., A. Cellerino, N. Berardi, A. Cattaneo, and L. Maffei, 1994. Antibodies to nerve growth factor (NGF) prolong the sensitive period for monocular deprivation in the rat, Neuroreport, 5:2041–2044. Domenici, L., N. Borardi, G. Comignolo, G. Vantini, and L. Maffei, 1991. Nerve growth factor promis the amblyopic effects of monocutar deprivation, Proc. Natt. Acad. Sci. USA, 88(19): 8811–8815. Domenici, L., V. Parisi, and L. Maffei, 1992. Exogenous supply of NGF prevents the effects of strabismus in the rat, Neuroscience, 51:19–24. Ernfors, P., J. Bengzon, Z. Kokaia, H. Persson, and O. Lindvall, 1991. Increased levels of messenger RNAs for neurotrophic factors in the brain during kindling epileptogenesis, Neuron, 7(1):165–176. Fagiolini, M., and T. K. Hensch, 2000. Inhibitory threshold for critical-period activation in primary visual cortex, Nature, 404(6774):183–186. Fagiolini, M., T. Pizzorusso, N. Berardi, L. Domenici, and L. Maffei, 1994. Functional postnatal development of the rat primary visual cortex and the role of visual experience: darkrearing and monocular deprivation, Vis. Res., 34:709–720. Fagiolini, M., T. Pizzorusso, V. Porciatti, M. C. Cenni, and L. Maffei, 1997. Transplant of Schwann cells allows normal development of the visual cortex of dark-reared rats, Eur. J. Neurosci., 9:102–112. Fagiolini, M., and M. P. Stryker, 1996. Delayed onset of NGF effects on ocular dominance plasticity in mice, Soc. Neurosci. Abs., 22:682.11. Fiorentini, A., N. Berardi, and L. Maffei, 1995. Nerve growth factor preserves behavioral visual acuity in monocularly deprived kittens, Vis. Neurosci., 12(1):51–55. Galuske, R. A. W., D.-S. Kim, E. Castren, H. Thoenen, and W. Singer, 1996. Brain-derived neurotrophic factor reverses experience-dependent synaptic modifications in kitten visual cortex, Eur. J. Neurosci., 8:1554–1559. Grewal, S. S., R. D. York, and P. J. Stork, 1999. Extracellular-signalregulated kinase signalling in neurons, Curr. Opin. Neurobiol., 9(5):544–553. Hensch, T. K., M. Fagiolini, N. Mataga, M. P. Stryker, S. Baekkeskov, and S. F. Kash, 1998. Local GABA circuit control of experience-dependent plasticity in developing visual cortex, Science, 282(5393):1504–1508. Huang, Z. J., A. Kirkwood, T. Pizzorusso, V. Porciatti, B. Morales, M. F. Bear, L. Maffei, and S. Tonegawa, 1999. BDNF regulates the maturation of inhibition and the critical period of plasticity in mouse visual cortex, Cell, 98(6):739–755. Kafitz, K. W., C. R. Rose, H. Thoenen, and A. Konnerth, 1999. Neurotrophin-evoked rapid excitation through TrkB receptors, Nature, 401(6756):918–921. Kohara, K., A. Kitamura, M. Morishima, and T. Tsumoto, 2001. Activity-dependent transfer of brain-derived neurotrophic factor to postsynaptic neurons, Science, 291(5512):2419–2423. Levi-Montalcini, R., 1987. The nerve growth factor 35 years later, Science, 237:1154–1162. Lewin, G. R., and Y.-A. Barde, 1996. Physiology of the neurotrophins, Annu. Rev. Neurosci., 19:289–317.

  : , ,   

61

Lodovichi, C., N. Berardi, T. Pizzorusso, and L. Maffei, 2000. Effects of neurotrophins on cortical plasticity: same or different? J. Neurosci., 20(6):2155–2165. Maffei, L., L. Berardi, L. Domenici, V. Parisi, and T. Pizzorusso, 1992. Nerve growth factor (NGF) prevents the shift in ocular dominance distribution of visual cortical neurons in monocularly deprived rats, J. Neurosci., 12:4651–4662. McAllister, A. K., D. C. Lo, and L. C. Katz, 1995. Neurotrophins regulate dendritic growth in developing visual cortex, Neuron, 15:791–803. McAllister, A. K., L. C. Katz, and D. C. Lo, 1997. Opposing roles for endogenous BDNF and NT-3 in regulating cortical dendritic growth, Neuron, 18:767–778. *McAllister, A. K., L. C. Katz, and D. C. Lo, 1999. Neurotrophins and synaptic plasticity, Annu. Rev. Neurosci., 22:295–318. Pizzorusso, T., N. Berardi, F. M. Rossi, A. Viegi, K. Venstrom, L. F. Reichardt, and L. Maffei, 1999. TrkA activation in the rat visual cortex by antirat trkA IgG prevents the effect of monocular deprivation, Eur. J. Neurosci., 11(1):204–212. Pizzorusso, T., and L. Maffei, 1996. Plasticity in the developing visual system, Curr. Opin. Neurol., 9:122–125. Pizzorusso, T., V. Porciatti, J. L. Tseng, P. Aebischer, and L. Maffei, 1997. Transplant of polymer-encapsulated cells genetically engineered to release nerve growth factor allows a normal functional development of the visual cortex in dark-reared rats, Neuroscience, 80:307–311. Pizzorusso, T., G. M. Ratto, E. Putignano, and L. Maffei, 2000. Brain-derived neurotrophic factor causes cAMP response element-binding protein phosphorylation in absence of calcium increases in slices and cultured neurons from rat visual cortex, J. Neurosci., 20(8):2809–2816. Pollock, G. S., E. Vernon, M. E. Forbes, Q. Yan, Y. T. Ma, T. Hsieh, R. Robichon, D. O. Frost, and J. E. Johnson, 2001. Effects of early visual experience and diurnal rhythms on BDNF mRNA and protein levels in the visual system, hippocampus, and cerebellum, J. Neurosci., 21(11):3923–3931. Purves, D., 1988. Body and Brain. A Trophic Theory of Neural Connections. Cambridge: Harvard University Press. Ramoa, A. S., M. A. Paradiso, and R. D. Freeman, 1988. Blockade of intracortical inhibition in kitten striate cortex: effects on receptive field properties and associated loss of ocular dominance plasticity, Exp. Brain. Res., 73(2):285–296.

62

 

Reichardt, L. F., and I. Farinas, 1997. Neurotrophic factros and their receptors, in Molecular and Cellular Approaches to Neural Development (W. Maxwell Cowan, T. M. Jessel, and S. L. Zipursky, eds.), New York: Oxford University Press. Reiter, H. O., and M. P. Stryker, 1988. Neural plasticity without postsynaptic action potentials: less-active inputs become dominant when kitten visual cortical cells are pharmacologically inhibited, Proc. Natl. Acad. Sci. USA, 85(10):3623–3627. Riddle, D. R., D. C. Lo, and L. C. Katz, 1995. NT-4 mediated rescue of lateral geniculate neurons from effects of monocular deprivation, Nature, 378:189–191. Sala, R., A. Viegi, F. M. Rossi, T. Pizzorusso, G. Bonanno, M. Raiteri, and L. Maffei, 1998. NGF and BDNF increase transmitter release in the rat visual cortex, Eur. J. Neurosci., 10:2185–2191. Shaw, C., and M. Cynader, 1984. Disruption of cortical activity prevents ocular dominance changes in monocularly deprived kittens, Nature, 308(5961):731–734. Thoenen, H., 1995. Neurotrophins and neuronal plasticity, Science, 270:593–598. Timney, B., D. E. Mitchell, and F. Giffin, 1978. The development of vision in cats after extended periods of dark-rearing, Exp. Brain. Res., 31:547–560. Viegi, A., T. Cotrufo, N. Berardi, L. Mascia, and L. Maffei, 2002. Effects of chark rearing on phosphorylation of neurotrophin Trk receptors, J. Neurosci., 16:1925–1930. von Bartheld, C. S., M. R. Byers, R. Williams, and M. Bothwell, 1996. Anterograde transport of neurotrophins and axodendritic transfer in the developing visual system, Nature, 379(6568):830– 833. Wiesel, T. N., and D. H. Hubel, 1963. Single cell responses of striate cortex of kittens deprived of vision in one eye, J. Neurophysiol., 26:1003–1017. Xiang, Z., J. R. Huguenard, and D. A. Prince, 1998. Cholinergic switching within neocortical inhibitory networks, Science, 281(5379):985–988. Zafra, F., B. Hengerer, J. Leibrock, H. Thoenen, and D. Lindholm, 1990. Activity dependent regulation of BDNF and NGF mRNAs in the rat hippocampus is mediated by non-NMDA glutamate receptors, EMBO J., 9(11):3545–3550.

5

Developmental and Genetic Control of Cell Number in the Retina ROBERT W. WILLIAMS AND SALLY A. MOODY

Now, nearly a century after Weismann, it is self-evident that the missing chapters of the Modern Synthesis—the merging of genetics with development and the merging of development with evolution—remain the major tasks before us. Buss (1987)

Introduction The retina is one of the most highly conserved parts of the central nervous system (CNS), with unambiguous homology of major cell types among most chordates. These cells and their progenitors therefore share common patterns of gene activity. This widespread commonality is illustrated most dramatically by the Pax6 genes—a small family of transcription factors that trigger the formation of retinas, ommatidia, and photoreceptors in animals as diverse as fruit flies, squid, ascidians, fish, frogs, mice, and humans (Gehring and Ikeo, 1999; Onuma et al., 2002). Retinas differ strikingly, however, in terms of the number and distribution of cells. Structural diversity of the vertebrate retina is generated using a relatively constant palette of cell types (Walls, 1942). Downstream effects mediated by transcription factors such as Pax6, and a rapidly expanding list of trophins, hormones, cytokines, and other signaling molecules, can produce significant changes in the number and ratio of cells and their interconnections. Evolution of the retina has therefore involved the episodic modulation of developmental processes that often adjust quantities, not quality. The contrast between universal mechanisms and structural diversity highlights the need to study development using complementary approaches. The first approach focuses on fundamental qualitative features that are common denominators; the second approach focuses on quantitative adaptations of genetic networks that underlie functional and evolutionary transformations, as well as individual differences. This review is divided into sections that consider developmental, genetic, and functional factors that modulate the types and numbers of cells in the vertebrate retina from these two complementary viewpoints. There are approximately 65 to 70 distinct cell types in the retina (Marc and Jones, 2002; Chapter 20, Retinal Neurotransmitters), each with its own idiosyncratic developmental history, and we only consider key themes and a few exem-

plar cell types. In Part 1, we cover the initial stages of visual system development, in which cell proliferation and differentiation produce the cellular substrates common to all vertebrate retinas. Most of our examples are drawn from studies of the South African clawed frog, Xenopus laevis, a longfavored experimental subject for developmental analyses. In Part 2, we consider the relation between cell number and visual system performance, addressing the question of how much variation is tolerated within species and why. Finally, we describe a novel approach called complex trait analysis that is being used to uncover the genetic basis of individual differences in retinal development. Combined developmental and genetic approaches should eventually provide a much better understanding of how evolutionary modifications have produced such amazing arrays of almost perfectly adapted eyes and retinas, a subject that gave Darwin serious pause (and a cold chill) a century and a half ago (Darwin, 1899; Dawkins, 1996).

PART 1: THE GENERATION OF A MENAGERIE OF RETINAL CELLS A wide variety of retinal cell types are generated in appropriate numbers by sequential and cooperative changes in patterns of gene expression that are coupled with cell division, cell differentiation, and cell death. These changes start at very early stages, even before gastrulation and the establishment of the neural plate, and changes continue through to late stages of fetal development. Differentiation and cell division are directed by factors that act at different spatial scales and stages, but both ultimately affect the activity of enhancer and promoter response elements in single cells. Broadly speaking, these factors include (1) inherited maternal determinants, (2) regional molecular gradients, (3) cell autonomous lineage factors, and (4) local intercellular signals. In this section we consider some of the key mechanisms involved in selecting the embryonic progenitors of the retina, partitioning part of the forebrain neuroectoderm into a field of specified retinal stem cells, and the progressive differentiation of the many cell types common to all vertebrate retinas that are eventually derived from these stem cells.

63

Molecular complexity of the retina The adult retina expresses at least 25,000 transcripts from over 20,000 genes (Sharon et al., 2002). Even more genes are likely to be expressed throughout development. The majority of these genes undoubtedly have common roles in cellular differentiation and metabolism, but a significant fraction—including many so-called housekeeping genes— are likely to contribute to unique features of retinal development and function. This is demonstrated clearly by the fact that over 10% of 12,422 transcripts quantified in triplicate using microarrays (Affymetrics U74Av2) differ more than twofold in abundance between retina and other brain regions such as cerebellum, brainstem, and forebrain (see the databases at www.nervenet.org/array). Even with modest numbers of samples, the expression levels of well over 20% of retinal transcripts differ significantly from forebrain transcripts using standard statistical criteria at the p < 0.05 level. Given the imposing molecular complexity of development of the eye and retina, the reliability of the process is astonishing. The robust molecular systems that ensure the production of a functional retina clearly are built on a principle of genetic redundancy and feedback interactions. While a few genes are essential for eye development (e.g., Pax6 ), most genetic processes are sensitive to context and can adapt to significant perturbations. The inactivation of important genes is often associated with only mild ocular and retinal effects. For example, the loss of the retinoic acid alpha receptor early in retinal development has essentially no effect on the inner retina despite early and intense expression among the pool of progenitors (Mori et al., 2001; Zhou et al., 2001). A process that is controlled by a complex developmental network involving genetic and epigenetic interactions is likely to produce greater variance and flexibility, and is also likely to be more resistant to environmental perturbations (Waddington and Robertson, 1966). As presented below, flexibility is built into the retinal developmental program from beginning (progenitor competence) to end (cell type specification).

The first step: selecting competent retinal progenitors Retinal development can be divided into several steps, some of which occur even before overt morphogenesis of the eye cup (Fig. 5.1). First, a subset of the pluripotent embryonic cells acquires the competence to contribute to the retina. From this competent pool, a smaller subset is biased to become the retina-forming cells. Later, a specified retinal stem cell population emerges from the descendants of the biased embryonic cells to form the eye field in the anterior neural plate. Further interactions during neural tube morphogenesis segregate this population into three major com-

64

 

F 5.1. Early steps of retinogenesis. Retinal development begins with several steps that set aside embryonic precursors. First, a subset of embryonic cells (green) becomes competent to contribute to the retina. The blue-hatched cell is inhibited from forming retina. Next, a smaller subset (yellow) is biased to become the retina-forming cells. Then a specified retinal stem cell population (red) emerges from the descendants of the biased embryonic cells (yellow) to form the eye field in the anterior neural plate (green). Finally, during neural tube morphogenesis, this specified population is segregated into three major compartments: neural retina (red), pigmented retina (black), and optic stalk (blue). (See color plate 1.)

partments: neural retina, pigmented retina, and optic stalk. Each of these compartments goes on to produce different subsets of specialized cells with distinct functions in the mature eye. An initial question is, how and when during development do pluripotent embryonic cells acquire the differential competence to contribute to the retina? Is this a partly stochastic process, or do early factors bias or restrict the selection of retina-forming embryonic progenitors? Xenopus embryos have been crucial for studying the roles of maternal determinants, lineage effects, and cell-cell inductions at this early stage (even before gastrulation) because eggs and blastomeres are large and easily manipulated (Moody, 1999). Each retina in Xenopus descends from a stereotypic subset of 9 animal blastomeres at the 32-cell stage (Fig. 5.2), and each of these blastomeres produces characteristic proportions of the cells that make up the mature retina (Huang and Moody, 1993). Are these nine cells the only cells that are competent to form the retina, and just how fixed is their commitment to a retinal fate? These questions have been largely answered by transplanting single cells to novel positions at a very early stage when only maternally inherited transcripts are expressed. The first key finding is that even at this stage, not all blastomeres are equally competent to contribute to the retina. For example, vegetal blastomeres transplanted to the most retinogenic coordinates never contribute progeny to the retina (Huang and Moody, 1993). This developmental restriction could not be overcome by providing components of the known maternal signaling pathways involved in neural and dorsal fate specification (see below), even in situations in which ectopic heads and eyes were induced successfully (Moore and Moody, 1999).

F 5.2. Retina competent and biased blastomeres in Xenopus laevis embryos. Left: animal pole view of a 32-cell embryo showing the major (orange) and minor (yellow) blastomeres that contribute to the retina. The numbers within each cell indicate the percentage of retinal cells that derive on average from each blastomere. Right: side view of a 32-cell embryo showing the retina-forming blastomeres (yellow), the blastomeres that do not make retina but are competent to do so (green), and the vegetal blastomeres that are inhibited by maternal factors from making retina (blue). Data are from Huang and Moody (1993). An, animal pole; D, dorsal; V, ventral; Veg, vegetal pole. (See color plate 2.)

These results indicate that vegetal blastomeres contain one or more maternal molecules that repress the transcription of genes that are critical in the initial steps of retinal differentiation. Other blastomeres are not so refractive. Both ventral animal blastomeres and equatorial blastomeres that normally do not contribute progeny to the retina can reprogram when transplanted to the center of the retinogenic zone (Huang and Moody, 1993). Furthermore, if the most retinogenic blastomere is deleted, a ventral implant will be respecified in response to interactions with neighboring cells to help produce a normal-sized retina (Huang and Moody, 1993). Complementing these findings, the retina-forming blastomeres are biased but not completely committed. When transplanted to a ventral vegetal site that normally produces gut and tail, they retain their neural fate but fail to make retina (Gallagher et al., 1991). Collectively, these embryonic manipulations demonstrate that the correct cellular coordinates within the animal hemisphere are necessary from the earliest stage for a blastomere to produce its normal cohort of mature retinal cells. The position-specific selection of retina-forming blastomeres from the competent pool appears to be mediated by the local signaling environment within the blastula. The ectopic expression of components of several growth factor pathways involved in embryo patterning, such as activin, fibroblast growth factor (FGF), and bone morphogenetic protein (BMP), demonstrates that competent blastomeres acquire the ability to express a retinal fate by being located in an environment in which BMP signaling is repressed (Moore and Moody, 1999). Expression of BMP4 in a

blastomere that normally is a major contributor to the retina inhibits its retinal fate, whereas repressing BMP signaling in a blastomere that normally gives rise to epidermis induces retinal cells within that lineage. Consistent with this model in frog, Bmp7 null mice are anophthalmic (Dudley et al., 1995) and rat embryos cultured in anti-BMP7 antibodies have reduced or absent eyes (Solursh et al., 1996). The inhibition of BMP signaling in the dorsal part of the embryo defines the domain of presumptive retina, just as it defines the domain of presumptive neural plate (Harland, 2000). Although fate maps show that there are nine retinaforming blastomeres in Xenopus (Fig. 5.2), not every one of these cells contributes to retina in every embryo. In the initial map (Moody, 1987), the major progenitors contributed to retina 85–100% of the time, whereas the minor progenitors contributed only 20–50% of the time. Thus, there is surprising individual variation in the number of embryonic cells that produce the retinal lineage. There also is individual variation in the number of retinal cells descended from each retina-forming blastomere (Huang and Moody, 1993). For the major progenitor, there is a 2-fold range in descendants between embryos, but for the others there is as much as a 10-fold range. These data have been verified in many subsequent studies using similar techniques. They demonstrate that the ultimate numbers of progeny produced by each lineage are not predetermined and that individual variation is well tolerated within consistent limits. This is apparently just as true in mammals as in Xenopus (Goldowitz et al., 1996; Williams and Goldowitz, 1992a).

The second step: specifying stem cells The next step in retinogenesis is to select from the competent pool those cells that will become specified as retinal stem cells (Fig. 5.1). After the neural plate is established, it is subdivided into fields that give rise to different CNS domains. Both retinas arise from a common anterior midline group of cells called the eye field, which is specified as the sole source of retinal cells when the neural plate is established (Perron and Harris, 1999; Saha and Grainger, 1992). A number of transcription factors regulate pattern formation in the anterior neural plate and eye field, and they are crucial in determining whether neuroepithelial cells express a retinal fate as opposed to another anterior neural fate. Otx2, a vertebrate homolog of the Drosophila orthodenticle gene, has at least two potential functions in the formation of the eye field. Early Otx2 expression in the anterior endoderm appears necessary for the induction of the anterior neural plate (Acampora et al., 1995; Ang et al., 1996; Rhinn et al., 1998), and its continued expression is required to initiate or maintain the expression of several other key neurogenic factors (Rhinn et al., 1998), leading to the

  :  

65

regionalization of the neural plate that allows the eye field to form. Bf1, a fork head/winged helix transcription factor, is expressed in the forebrain and in the future nasal retina (Hatini et al., 1994; Tao and Lai, 1992). The Xenopus homolog Fkh4 is expressed in the anterior margin of the neural plate and then in the eye field (Dirksen and Jamrich, 1995). Pax6, the potent paired box and homeodomain transcription factor mentioned in the Introduction, is highly expressed in the eye field, and its ectopic expression causes the formation of ectopic eye structures in fly and frog (Altmann et al., 1997; Chow et al., 1999; Halder et al., 1995; Kenyon et al., 2001). Rx1, a novel homeobox gene, is expressed exclusively in the eye field, retina, and pineal gland (Casarosa et al., 1997; Furukawa et al., 1997; Mathers et al., 1997). Its overexpression in Xenopus causes enlargement of native eye tissue and ectopic patches of pigmented retina (Kenyon et al., 2001; Mathers et al., 1997). Six3, related to the Drosophila sine oculis gene, is expressed in the eye field and optic vesicle (Oliver et al., 1995). Overexpression of mouse Six3 in fish or mouse causes ectopic retinal tissues to form (Lagutin et al., 2001; Loosli et al., 1999). Another six family gene, Optx2, is also expressed in the eye field and optic vesicle (Toy et al., 1998), and when Optx2 is overexpressed, it transforms more caudal neural plate into eye field (Bernier et al., 2000). Loss-offunction mutations of Bf1, Otx2, Pax6, and Rx1 each lead to loss of retinal tissue, indicating that all are likely to have important roles during establishment of the eye field. The segregation of the anterior neural plate into eye field versus non–eye field domains is likely to result from the localized expression of several transcription factors, including those described above, which then initiate eye-specific transcriptional programs. In addition, some of these genes are involved in even earlier retinal fate decisions. Otx2, Pax6, and Rx1 are expressed more widely and diffusely at stages just before the formation of the eye field, and they can cause ventral epidermal-forming blastomeres to contribute to retina by altering gastrulation movements that allow their descendants access to the eye field (Kenyon et al., 2001). Thus, these genes appear to regulate the selection of retinal stem cells at multiple levels: transcriptional modulation, signaling, and cell movements. This complex program of selecting retina-specific stem cells provides several opportunities for adjustments in cell number. From the pregastrula competent pool of cells, only a subset forms the eye field. Expressing the correct genetic program increases the probability of competent cells being selected to become retinal stem cells. For example, in the experiments of Kenyon et al. (2001), not every cell that ectopically expressed the exogenously supplied retinal gene expressed a retinal fate. Some cells migrated into the eye field and others did not. Some of the ventrally derived retinal clones were composed of hundreds of cells, and some were composed of fewer than ten cells. These data indicate that

66

 

there are local cues that also influence these fate decisions. Subtle epigenetic variations in these cues have significant effects on final retinal cell numbers by altering the size of the initial stem cell pool.

The third step: specification of different cell types After the compartmental domains of the future eye are defined during neural plate and neural tube stages, cells within the optic cup begin to differentiate into the many different cell types that characterize the functional vertebrate retina. They exit the cell cycle in a defined spatiotemporal pattern, migrate to appropriate layers, turn on the correct neurotransmitters or visual pigments, and develop the correct pre- and postsynaptic connections. The ganglion cells project long axons to very distinct targets, whereas intrinsic cells make highly specialized connections locally. Intricate choreography! A large number of factors and interactions affect the diversity and number of distinct retinal cell types that are generated during this period of retinogenesis (reviewed in Cepko, 1999; Harris, 1997; Livesey and Cepko, 2001; Vetter and Brown, 2001; Vetter and Moore, 2001). R  E  L  C F The optic cup was one of the first vertebrate tissues in which the issue of lineage regulation of cell phenotype specification was addressed. Vertebrate embryos, with their enormous numbers of cells, are difficult subjects for complete lineage studies because it is not technically feasible to map every mitotic division. Nonetheless, parts of retinal lineages have been defined by intracellular injection of tracers into proliferating cells of the optic vesicle/cup (Holt et al., 1988; Wetts and Fraser, 1988) or by injecting a recombinant retrovirus that infects proliferating cells and marks their clones (Turner and Cepko, 1987; Turner et al., 1990). Both techniques showed that dividing cells of the optic vesicle/cup are multipotent; labeled clones spanned the entire thickness of the retina and were composed of various combinations of cells. They also retain striking pluripotency right up to their last division, often producing daughters consisting of two very different cell types—for example, Müller glial cells and bipolar cells (Turner and Cepko, 1987). The great majority of retinal stem cells are not committed to produce only a single phenotype, nor do they appear to be restricted to an identifiable invariant combination of cell types. With the exception of retinal astrocytes, which originate from separate progenitors in the optic stalk (Chang-Ling and Stone, 1991), it was initially thought that lineage restriction plays little if any role in determining types or numbers of retinal cells. However, the amazing diversity in the cellular composition and size of clones labeled at very early stages (Williams and Goldowitz, 1992a) suggested the presence of

matched heterogeneity among retinal progenitors, which implies some form of differential commitment of the progenitors (Goldowitz et al., 1996; Williams and Goldowitz, 1992b). The original lineage tracing techniques could not label single identified progenitors across different animals— a criterion by which lineage mechanisms are usually identified—and this left the original data open to alternative interpretations. M R S C T We now know that the eye field stem cells are a heterogeneous population (Dyer and Cepko, 2001; Moody et al., 2000). There are at least two kinds of cells, those that produce radial clones and those that produce layer-restricted clones. In Xenopus these two progenitors differ in the layer distribution of their constituents, their cellular complexity, and the number of cell divisions remaining in their respective lineages. These two different eye field stem cell types may represent either two different determinative states or separate lineages for the early-formed primary (layered) versus later-formed secondary (radial) retina, similar to what has been described for neural plate progenitors of primary and secondary spinal neurons (Hartenstein, 1989). Two kinds of clones (single cells and radial columns) also have been described in the early chick and mouse retina (Fekete et al., 1994; Reese et al., 1999). Cellular diversity in type and number could easily be generated by changes in the processes that separate the original stem cell pool into different subsets of progenitors that have different ultimate fates. This could result in correlated shifts in populations that participate in common visual function—a primary example being scotopic and photopic subsystems. Although lineage studies show that progenitor cells in the optic vesicle and cup are multipotent, recent in vitro studies also suggest that this pool contains differentially biased precursors (Alexiades and Cepko, 1997; Belleveau and Cepko, 1999; Jensen and Raff, 1997; Marrow et al., 1998). In fact, the concept that retinal lineages are not homogeneously multipotent but are differentially biased was elegantly demonstrated by a statistical analysis (Williams and Goldowitz, 1992b) of published data (Turner et al., 1990); the frequency of clones containing only two cell types was much higher than expected, and the frequency of clones extending across all layers was much lower than expected. Furthermore, in vitro studies showed that some rat optic cup progenitors are biased to produce amacrine and horizontal cells (Alexiades and Cepko, 1997). Other progenitors appear biased to produce either rod or bipolar cells (Marrow et al., 1999). One model put forward to reconcile the strong evidence of extrinsic control of retinal cell differentiation and the strong evidence of fate-biased determination acting via committed progenitors is simply that retinal progenitors pass

through a series of determinative states, each of which is intrinsically specified to respond to particular environmental cues that influence the cell types produced (Cepko, 1999). For example, embryonic retinal progenitor cells differ in many characteristics from neonatal ones (Alexiades and Cepko, 1997; Lillien and Cepko, 1992; Marrow et al., 1998; Waid and McLoon, 1995; Watanabe and Raff, 1990), and these states can be influenced by the presence of other cell populations (Austin et al., 1995; Belleveau and Cepko, 1999; Reh, 1992; Reh and Tully, 1986; Waid and McLoon, 1998) and cytokines (Harris, 1997). An important aspect of this process is a cell-autonomous “clock” that regulates the competence of progenitors (Rapaport et al., 2001). Subtle changes in the timing of the determinative states and/or progenitor competence could be significant contributors to individual variation in cell numbers and types. L B  S A S One approach used to eliminate the problem of labeling progenitors at random was to map the origins of specific cell types from the “identified” retina-forming blastomeres of the 32-cell frog embryo (Fig. 5.2). When specific progeny within retinal clones were identified by neurotransmitter expression, certain subtypes of amacrine cells—dopamine (DA), neuropeptide Y (NPY), and serotonin (5-hydroxytryptamine, 5-HT) were observed to arise from different subsets of blastomeres, whereas others (GABA, glycine) did not (Huang and Moody, 1995, 1997). This novel information likely was revealed because blastomeres can be consistently identified, unlike single progenitors in the optic cup, and each subtype of cell may follow its own subprogram of developmental instructions that would not be obvious from studying the broad population. This proposed lineage bias of DA, NPY, and 5-HT amacrine cell specification could result from the asymmetric distribution of intrinsic (i.e., maternal) factors that autonomously influence the different amacrine fates, or from inductive signaling related to the position in which the blastomere descendants differentiate. To test whether amacrine fate is intrinsically biased, blastomeres with distinct amacrine fates were transplanted to a position that normally expresses a different amacrine fate (Moody et al., 2000). A lateral animal blastomere expressed its normal, large number of DA amacrine cells after it was transplanted into a position that normally produces few of these cells, suggesting an intrinsic bias in amacrine fate within this lineage. However, another blastomere changed amacrine fate in accord with its new position. These experiments illustrate that the retina-producing blastomeres are already a mosaic of intrinsically biased and positionally specified progenitors. At later stages of development, there also is evidence for lineage bias in the specification of amacrine cell subtypes.

  :  

67

When DA, NPY, and 5-HT amacrine cell subtypes first differentiate, they are arranged as single cells scattered across the inner nuclear layer (Huang and Moody, 1995, 1997). As more cells differentiate, the first cells are joined by likeexpressing cells to form small clusters (Huang and Moody, 1995, 1997). If amacrine cell clusters are induced to express the same neurotransmitter by local interactions with their peers, then clusters should form even in the absence of cell division and after the emergence of the first amacrine cell. This is true of 5-HT clusters (Moody et al., 2000). In sharp contrast, DA and NPY clusters are virtually eliminated by blocking mitosis, indicating that cells in these clusters are generated by continued asymmetrical divisions of a strongly biased lineage. An important point is that even within a single cell class, specification of subtypes can be modulated by surprisingly different processes and at different stages of development. L I S Local signaling in the developing retina is critical in the progressive changes in retinal progenitors and in the subsequent differentiation of many cell types (Adler and Hatlee, 1989; Gan et al., 1996, 1999; Harris, 1997; Reh, 1992; Repka and Adler, 1992). A growing number of secreted factors have effects on cellular differentiation, usually as assayed in tissue culture and explants. Ciliary neurotrophic factor (CNTF) (Ezzeddine et al., 1997; Fuhrmann et al., 1995; Kirsch et al., 1996), brain-derived neurotrophic factor (BDNF) (Rickman and Rickman, 1996), (GDNF) (Politi et al., 2001), (TGFa) (Lillien, 1995), retinoic acid and thyroid hormone (Kelley et al., 1994, 1995), sonic hedgehog (Shh) (Levine et al., 1997), FGFs (Hyer et al., 1998; McFarlane et al., 1998; Opas and Dziak, 1994; Pittack et al., 1991, 1997), and BMPs (Belecky-Adams and Adler, 2001) all appear to have important roles in discriminating among different retinal class fates, in some cases also shifting relative numbers of earlyand late-generated cell types. Activin signaling, for example, decreases the number of photoreceptors and increases the number of nonphotoreceptor cells present in low-density retinal cultures in a dose-dependent manner (Davis et al., 2000). Retinoic acid can do the opposite (Kelley et al., 1994). Chick ganglion cells express BMP receptors, and retinal explants exposed to BMPs contain enhanced numbers of ganglion cells (Carri et al., 1998). FGF2 has been reported to either stimulate (Hicks and Courtois, 1992) or suppress (Perron and Harris, 1999) photoreceptor numbers. There is convincing evidence that in the intact embryo, FGF2 may not alter photoreceptor number per se but may affect the ratio of rods to cones (Patel and McFarlane, 2000). Increased FGF2 signaling also reduces the number of Müller glial cells and increases the number of retinal ganglion cells (Patel and McFarlane, 2000). These examples highlight the fact that signaling factors have diverse effects on cell fate specifica-

68

 

tion. Local signaling can profoundly influence the cell types generated in culture. R T C A large number of transcriptional regulators are involved in retinal cell fate decisions. Several of the same genes that establish the eye field also are expressed in distinct patterns in the developing retina (Perron and Harris, 1999). Otx2, Pax6, Rx1, and Six3 are expressed by the stem cells at the margins of the retina, and it has been proposed that their collective expression maintains a retinal identity in these cells (Perron et al., 1998). Optx2 has been implicated in transforming pigmented retina into neural retina (Toy et al., 1998). Many of these genes also are implicated in regulating the proliferation of the optic vesicle and cup progenitor cells (Loosli et al., 2001; Martinez-Morales et al., 2001). For example, retinoblasts transfected with Optx2 produce clones that are twofold larger that control clones, and its overexpression early in embryogenesis results in giant eyes (Zuber et al., 1999). Furthermore, eye field genes are expressed differentially as retinal cells differentiate (Perron and Harris, 1999). Ganglion and amacrine cells express Pax6 and Six3; cells in the outer zone of the inner nuclear layer express Six3, Rx1, and Otx2; and photoreceptor cells express Rx1. Added to this complexity is the differential expression of a large number of basic helix-loop-helix (bHLH) differentiation factors (Perron and Harris, 1999; Vetter and Brown, 2001). Different combinations of these latter genes are expressed as cells transit from the purely stem cell zone at the margin, through more restricted progenitor zones, and finally into their retinal layers. Photoreceptors require NeuroD and Ngn2; bipolar cells require Ash1, Ash3, and Ngn2; amacrine cells require NeuroD and Ath3; and ganglion cells require Ath5. There is compelling evidence that Pax6 directly regulates several of the bHLH genes (Marquardt et al., 2001) and that Pax6, Six3, and Rx1 modulate a cell’s responsiveness to the bHLH genes (Inoue et al., 2002). Many of these genes also cross-regulate each other: when a gene manipulation eliminates one cell class, other classes are significantly enlarged. This indicates that the transcriptional program that specifies one cell type is interrelated with programs that specify other cell types, forming transcriptional webs. This suggests that individual and evolutionary changes in the expression of single genes, the main topic of the next section, will frequently have pleiotropic effects that lead to coordinate changes in several cell types.

PART 2: FUNCTIONAL AND GENETIC ANALYSIS OF INDIVIDUAL DIFFERENCES The types and numbers of cells that are generated and distributed across the retina represent an explicit, although still incompletely deciphered, summary of a species’ relationship

with its visual world (Hayes and Brooke, 1990; Østerberg, 1935; Walls, 1942). These cellular parameters can be modified relatively rapidly in response to changes in selective pressure ( Jeffery and Martasian, 1998; Jeffery et al., 2000; Williams et al., 1993). Variation is by no means limited to retina, and there are even more striking quantitative differences in the size and structure of the complex quilt of visual areas that extend from occipital pole into temporal and parietal neocortex of mammals (Chapter 32, Organization of Visual Areas in Macaque and Human Cerebral Carter; Chapter 105, the Evolution of the Visual System in Primates). In this part, we first consider the functional relevance of variation in numbers of neurons and photoreceptors and review what we are beginning to learn about the genetic basis of individual differences within species. As illustrated by a few specific examples as the end of this part, variation in retinal structure traces back to specific allelic differences that affect developmental processes.

Functional perspective on cell population size N  L N An enormous number of cells and synapses are needed to generate and interpret multiple neural representations of the visual world. The advantage of having large numbers of cells is most obvious in the vertebrate fovea (e.g., Collin and Collin, 1999; Wikler et al., 1990). Cones are often packed together in a tight triangular mosaic with peak densities that reach up to 300,000/mm2 in humans (Curcio et al., 1987, 1990). Trios of photoreceptors, midget bipolar cells, and retinal ganglion cells are organized with nearly pixel-like precision to recapture and transmit the high spatial frequencies into the central visual system. Several remarkable developmental events help to generate large numbers of these cells in central retina (Hendrickson, 1994; LaVail et al., 1991; Lia et al., 1987; Provis et al., 1998; Springer, 1999), the most surprising being a shift of cones toward fovea from surrounding retina late in retinal development (Packer et al., 1990). The need for large numbers cascades through the dorsal lateral geniculate nucleus (LGN) to the primary visual cortex. Ratios of retinal ganglion cells and projection neurons in the LGN are close to 1:1 in both rhesus monkeys and humans (Spear et al., 1996; Williams and Rakic, 1988). Cell densities are about 25% higher in the posterior region that represents fovea (Ahmad and Spear, 1993). The numbers increase 300- to 400-fold in visual cortex. An average of 350 million neurons are packed into area 17 of both macaques and humans (Suner and Rakic, 1996; Tolhurst and Ling, 1988). Cell densities may be twice as high as those in other neocortical areas (Powell and Hendrickson, 1981; Rockel et al., 1980). A staggering 2–3 trillion synapses are involved in the first stage cortical analysis of the visual world (Colonnier and O’Kusky, 1981; O’Kusky and Colonnier,

1982). Further amplifying these numbers, area 17 devotes disproportionate computational resources to the foveal representation. The central 1 mm2 of both retinas contains a total of about 150,000 cones that view the central 12 square degrees of visual space. These receptors feed signals bilaterally to 400–500 mm2 of visual cortex that contains approximately 50 million neurons and 150 billion synapses (estimate computed for both hemispheres by combining data in O’Kusky and Colonnier, 1982; Tolhurst and Ling, 1988; and van Essen et al., 1984). F C A behavioral corollary of high numbers in fovea and the visual cortex is extremely high acuity, that is, an ability to resolve spatial frequencies of 60–100 cycles per degree (Banks et al., 1991). Even more remarkable, humans can detect misalignments of short line segments finer than the grain of the photoreceptor mosaic. Detection thresholds can be as low as 10 arc-seconds. This computational feat, referred to as vernier hyperacuity (Kumar and Glaser, 1993; Westheimer, 1981), depends in part on maintaining high spatiotemporal precision across large numbers of coactivated neurons (Kandil and Fahle, 2001). N  N N Impressive visual capabilities of this type depend on minimizing noise at all levels of the retinogeniculocortical system. Noise is suppressed roughly as a function of the square root of n, where n is numbers of active neurons, synapses, and synaptic vesicle fusion events (Sterling, 1998). Small gains in signal-to-noise ratios may appreciably improve fitness due to increased acuity, and this in turn may have fueled steep evolutionary increases in numbers of cells in several vertebrate lineages. For example, the prominent expansion of the peristriate visual cortex in hominids may have been driven both by the complexity of simultaneously processing multiple streams of visual information (Dacey, 2000) and by selective pressure to improve the signal-to-noise ratios of cortical circuits. E L-L V R L N  C Vision in low light also demands high numbers of cells, but for reasons other than visual acuity. A wide belt of the retina that usually surrounds the fovea at 10–15 degrees of eccentricity has rod densities that can rise to 500,000–850,000 cells/mm2 in primates (Dkhissi-Benyahya et al., 2001; Wikler and Rakic, 1990), cats (Williams et al., 1993), and mice ( Jeon et al., 1998). Even in the far periphery of humans, rod density is still typically above 40,000 cells/mm2 (Williams, 1991). An explanation for the high density and small size of rods is that they must be able to convert the absorption of a single photon into a distinct reduction in the transmitter release (Rodieck, 1998). This requires an exceedingly slender rod

  :  

69

outer segment in which a single photon will generate a sufficient reduction in cGMP concentration to shut cation channels. The high impedance of rods converts small fluctuations in outer segment currents into corresponding voltage fluctuations at the synaptic terminal. The great majority of the outer surface of the retina is tiled by rod outer segments to ensure efficient capture of the few photons available under low light. Their signals are averaged and integrated by AII amacrine cells, and these cells appear to define the acuity limit under low-light conditions (Mills and Massey, 1999), much as ganglion cell densities define the acuity limit in daylight. Because of the inherent noise of the sporadic arrival of photons and receptor noise generated by spontaneous isomerization of rhodopsin and Poisson variance in vesicle release, detecting and interpreting objects in the dark is a challenging computational task. The consequence of these particular environmental and photochemical characteristics is that the population of rods in the human retina is incredibly high—typically ranging from 100 to 150 million (Curcio et al., 1990). Even a mouse retina has approximately 6 million rods. Having very large photoreceptor populations ensures high acuity (cones) in daylight or higher sensitivity and lowered noise levels (rods) in the dark.

rates that depend on their functional contributions to visual performance. For example, populations of rods are higher in domestic cats than in ancestral wildcats (Williams et al., 1993), and populations of retinal ganglion cells are lower in domestic cats and dogs than in wildcats and wolves (Peichl, 1992; Williams et al., 1993); in addition, both domestic species are likely to have lower acuity than their wild peers. Numbers of neurons will reach asymptotes at which further changes are blunted by countervailing selective pressure, such as the metabolic load associated with increased population size (Franco et al., 2000). The final cell number is a compromise between the marginal utility of adding yet more cells and the metabolic costs that these extra neurons inevitably incur. An interesting consequence of reaching a functional asymptote is that cell populations within a species will typically be maintained at levels around which significant numerical deviation is surprisingly well tolerated by individual animals (see the list below). Near the asymptote, the relation between number and function is nearly flat, and visual system performance will be robust even in the face of surprisingly large deviations in numbers and ratios of cells that may be introduced by rare alleles and mutations, population admixture, developmental noise, environmental perturbation, or disease.

C F  C N The functional relationship between cell number and visual system performance is not always as clear-cut as suggested by these examples. There are approximately 60 cell types in the retina and certainly just as many or more in the central part of the visual system, each of which, by definition, makes somewhat different contributions to vision. Some cells, such as horizontal cells and the population of photosensitive retinal ganglion cells that project to hypothalamic circadian pacemaker cells, have computational roles that can best be summarized as averaging mean illuminance (Hattar et al., 2002). In the case of horizontal cells, this averaged signal provides a negative feedback to photoreceptors. Losing all horizontal cells has severe consequences, reducing retinal transduction efficiency and increasing the latency of visual evoked potentials in the cortex (Peachey et al., 1997). However, the crucial biometric parameter in this case is not numbers of channels but uniform field coverage. Small numbers of cells with extensive dendritic and axonal arbors may provide equivalent or even superior feedback compared to larger numbers of cells with smaller fields. Cells with this sort of role will have a number : performance function that is relatively flat or even negative when their numbers exceed the optimum.

V  S  G Ranges of normal variation in absolute numbers of cells and in ratios of cells within a species are often large. There is usually no obvious functional effect. The estimates of variation that are listed below come from small samples, and these examples therefore are likely to underestimate the actual range of variation. In most cases, the examples illustrate variation among individuals.

E F  N Given these considerations, populations of neurons in evolving species will ratchet up or down in numbers at somewhat idiosyncratic

70

 

1. Foveal cone densities vary at least threefold among humans without retinal disease (Curcio et al., 1990; n = 8). 2. Ratios of L and M cones (red and green) in human central retina can range from 1:1 to 3.8:1 without evident (or with only subtle) differences in the categorization of colors (Roorda and Williams, 1999; n = 2). 3. Even within a single retina, the ratio of rods and cones can vary twofold. In one case, adjacent large fields in the periphery of a normal human retina had rod : cone ratios of 1:2 and 2:1 (Williams, 1991). 4. Twofold differences in horizontal cell number and density are common among normal inbred strains of mice (Williams et al., 1998a). A single strain of inbred mouse does not provide a good index of a typical ratio of cell types (see Williams et al., 1998b). For example, the ratio of ganglion cells to horizontal cells in mice varies from 3.2:1 to 6.7:1. 5. The population of retinal ganglion cells—the information bottleneck of the entire visual system—varies from 1,000,000 to 1,600,000 in humans and macaques (Provis et

al., 1985; Rakic and Riley, 1983; Spear et al., 1996). This population varies from 45,000 to 75,000 among strains of mice (Williams et al., 1996). 6. The population of projection neurons in the dorsolateral geniculate varies as much as twofold within cat, mouse, and macaque (Ahmad and Spear, 1993; Williams and Rakic, 1988; n < 10 in cat and macaque; n = 100 in mouse). For example, in mouse, the population of projection neurons varies from 12,000 in CE/J to 22,000 in AKR/J (Kulkarni et al., 2000). 7. The surface area and cell population of primary visual cortex in both humans and macaques vary two- or even threefold (Gillisen and Zilles, 1996; Leuba and Kraftsik, 1994; Suner and Rakic, 1996; van Essen et al., 1984; n £ 20). 8. Ocular dominance column numbers along the vertical meridian range from 101 to 154 in macaques. The width of columns ranges from 400 to nearly 700 mm (Horton and Hocking, 1996; n = 12 hemispheres). T P  H V The magnitude of this variation may be puzzling, particularly in species that are highly dependent on vision. Stabilizing selection, a process that normally trims away the tails of a distribution of phenotypes, would normally limit the range of variation that is tolerated in an interbreeding population. But the examples above provide ample demonstration that wide variation is well tolerated even in species, such as macaques and humans, that are utterly dependent on vision. The apparent paradox is resolved if we come back to the point that the visual system is overengineered and that there is much functional and developmental redundancy built into the retinas of most individuals. This also means that directional selection has plenty of variation with which to work on an evolutionary scale. Selection is highly effective in changing the cellular demographics of the retina and other parts of the visual system. Rapid changes in cell number have occurred in at least two carnivore lineages—wolves and wildcats— over an interval of less than 20,000 years (Peichl, 1992; Williams et al., 1993). In cats, the population of retinal ganglion cells has eroded from about 240,000 in wildcats to 160,000 in domestic cats. Even more extreme changes have been uncovered in response to rapid changes in habitat—for example, isolation in lightless cave systems in which the eyes regress due to the accumulation of deleterious mutations in genes critical at several stages of eye development ( Jeffery and Martasian, 1998).

Genetic basis of individual difference in retinal structure G P  S The striking differences in the retinal architecture of the mouse and the

human arose from natural genetic variation generated and selected within single species. Roughly one-fifth of the genes in most species have functional polymorphisms that produce changes in the electrophoretic mobility of the corresponding proteins. This reservoir of genetic variation often translates into correspondingly high heritability estimates for quantitative retinal traits (Williams, 2000; Zhou and Williams, 1999). For example, variation in retinal ganglion cell number in mice has a heritability above 80% (Williams et al., 1996), demonstrating strong genetic modulation of this population under typical laboratory conditions. The goal of a new field of genetics called complex trait analysis is to convert these bland estimates of heritability into the much more interesting and informative gene variants. These gene variants or polymorphisms can, in turn, be used to explore the genetic control of retinal development. M  Q G Mendelian methods of gene analysis, with which most readers will be familiar, assume that single gene variants—usually rare alleles or mutations—generate essentially all interesting differences. For example, mutation of the tyrosinase gene (albinism) causes a 35% reduction in the fraction of retinal ganglion cells with ipsilateral projections in mice, regardless of genetic background (Rice et al., 1995). In contrast to Mendelian genetics, complex trait analysis begins with the assumption that differences in cell populations are generated by the combined effects of a large number of polymorphic genes scattered across the genome. The goal is to map this set of genes to precisely delimited chromosomal locations and then proceed to identify these genes and the molecular pathways of which they are part. The types of genes that are part of these multigenic systems are known as quantitative trait loci (QTLs). Overwhelming technical difficulties prevented QTLs from being mapped in vertebrates even a decade ago. However, with the introduction of efficient polymerase chain reaction (PCR)based methods to genotype animals and sophisticated statistical programs to discover associations between gene variants and phenotypes, the prospects for QTL mapping are now greatly improved. It is possible to target virtually any heritable quantitative trait for what is called genetic dissection. This is particularly the case if one can exploit strains of inbred mice. U  I L An inbred strain is essentially an isogenic clone of fully homozygous animals that are able to reproduce sexually. They are a remarkable tool with which it is possible to resample the same genetic individual at different stages and after different treatments. They can also be used to obtain reliable quantitative estimates of traits that have high noise caused either by technical error or by stochastic developmental variation. If two or more inbred lines

  :  

71

differ substantially in a set of retinal traits under a common environment, then the difference traces back to genetic differences between those lines. These characteristics have been exploited to map and characterize novel gene loci that modulate cell populations in the retina. The population of retinal ganglion cells has been counted in over 1000 mice belonging to 60 isogenic lines. Numbers range from 45,000 in strain CAST/Ei to 75,000 in BXD32 (Williams et al., 1996). While individual values have a normal distribution, when cases are pooled by strain the histogram has a characteristic multimodal structure in which the different modes represent different combinations of genotypes at several QTLs. The most prominent pair of modes is present in neonatal mice before ganglion cells have been eliminated by apoptosis (Strom and Williams, 1998). This indicates that the QTL responsible for these modes, the Nnc1 locus, has a modulatory effect on the proliferation or differentiation of retinal ganglion cells. M N G Crosses between strains with high and low ganglion cell numbers have been used to map Nnc1 and three other QTLs responsible for much of the variation in ganglion cell number (Strom, 1999; Williams et al., 1998b). The gene mapping method can be distilled to statistical tests of association between differences in cell number and differences in genotype (genotypes are often scored in a numerical format: AA = 1, Aa = 0, aa = -1). For example, the correlation between ganglion cell number and genotypes at Nnc1 reaches a peak of 0.72 on chromosome (Chr) 11. The other three QTLs that modulate ganglion cell number map to Chr 7 at 65 cM (Nnc2, likelihood adds ratio [LOD] score of 5.9), Chr 1 at 82 cM (Nnc3, LOD of 9.3), and Chr 16 at 42 cM (Nnc4, LOD of 6.0). Nnc1 has the largest effect, and this QTL maps to a 6 million base pair stretch of DNA on Chr 11 between Hoxb and Krt1. This region contains two obvious candidate genes—the retinoic acid alpha receptor and the thyroid hormone alpha receptor. The addition of exogenous retinoic acid to tissue cultures increases rod production at the expense of amacrine cells (Kelley et al., 1994). However, knockouts of the retinoic acid receptor have no detectible effect on retinal ganglion number, a surprising result given the high expression of this gene in ganglion cells from very early stages (Zhou et al., 2001). In contrast, inactivation of the thyroid hormone alpha receptor neatly reproduces a 15% predicted difference in cell number. This result suggests that the QTL on Chr 11 corresponds to a polymorphism of the thyroid hormone receptor gene (Strom, 1999), but this idea has not yet been confirmed by sequence comparison. However, the hypothesis is attractive because thyroid hormone has long been known to control retinal ganglion cell proliferation during Xenopus metamorphosis—the stage in this species’ development when a population of ipsilater-

72

 

ally projecting ganglion cells is first generated (Hoskins, 1985). The thyroid receptor is a transcription factor that often pairs with a retinoid X receptor to control gene expression. It may not be coincidental that the QTL on Chr 1 that modulates ganglion cell number overlaps the position of the Rxrg gene (Strom, 1999).

Conclusion It is already apparent that many developmental processes are intertwined in such a way as to maintain general conformity in retinal structure and cellular content while still permitting substantial latitude for individual variation. A growing number of molecules and processes are now known to affect the diversity and number of retinal cell types generated in multiple steps during retinogenesis (Cepko, 1999; Harris, 1997; Livesey and Cepko, 2001; Vetter and Brown, 2001; Vetter and Moore, 2001). Experimental tests to sort out their relative contributions also demonstrate that there are numerous stages at which the final numbers and relative proportions of the different retinal phenotypes can be changed. Understanding this process of differentiation at a quantitative level will require an analysis of the transcriptome and proteome of the lineages of the major cell types, and an understanding of the processes that occur during the very early embryonic stages to bias and then select the retinal stem cells. Although we are still far from being able to navigate through the molecular labyrinths that generate different retinal cell types in appropriate numbers, the pace of research is now so rapid that it is hard not to be optimistic. REFERENCES Acampora, D. S., S. Mazan, Y. Lallemand, V. Avantaggiato, M. Maury, A. Simeone, and P. Brulet, 1995. Forebrain and midbrain regions are deleted in Otx2-/- mutants due to a defective anterior neuroectoderm specification during gastrulation, Development, 121:3279–3290. Adler, R., and M. Hatlee, 1989. Plasticity and differentiation of embryonic retinal cells after terminal mitosis, Science, 243:391– 393. Ahmad, A., and P. D. Spear, 1993. Effects of aging on the size, density, and number of rhesus monkey lateral geniculate neurons, J. Comp. Neurol., 334:631–643. Alexiades, M. R., and C. L. Cepko, 1997. Subsets of retinal progenitors display temporally regulated and distinct biases in the fates of their progeny, Development, 124:1119–1131. Altmann, C. R., R. L. Chow, R. A. Lang, and A. HemmatiBrivanlou, 1997. Lens induction by Pax-6 in Xenopus laevis, Dev. Biol., 185:119–123. Ang, S. L., O. Jin, M. Rhinn, N. Daigle, L. Stevenson, and J. Rossant, 1996. A targeted mouse Otx2 mutation leads to severe defects in gastrulation and formation of axial mesoderm and to deletion of rostral brain, Development, 122:243–252. Austin, C. P., D. E. Feldman, J. A. Ida, and C. L. Cepko, 1995. Vertebrate retinal ganglion cells are selected from competent

progenitors by the action of Notch, Development, 121:3637– 3650. Banks, M. S., A. B. Sekuler, and S. J. Anderson, 1991. Peripheral spatial vision: limits imposed by optics, photoreceptors, and receptor pooling, J. Opt. Soc. Am. A, 8:1775–1787. Belecky-Adams, T., and R. Adler, 2001. Developmental expression patterns of bone morphogenetic proteins, receptors, and binding proteins in the chick retina, J. Comp. Neurol., 430:562–572. Belleveau, M. J., and C. L. Cepko, 1999. Extrinsic and intrinsic factors control the genesis of amacrine and cone cells in the rat retina, Development, 26:555–566. Bernier, G., F. Panitz, X. Zhou, T. Hollemann, P. Gruss, and T. Pieler, 2000. Expanded retina territory by midbrain transformation upon overexpression of Six6 (optx2) in Xenopus embryos, Mech. Dev., 93:59–69. Buss, L. W., 1987. The Evolution of Individuality, Princeton: Princeton University Press. Carri, N. G., H. Bengtssonn, M. F. Charette, and T. Ebendal, 1998. BMPR-II expression and OP-1 effects in developing chicken retinal explants, Neuroreport, 9:1097–1101. Casarosa, S., M. Andreazzoli, A. Simeone, and G. Barsacchi, 1997. Xrx1, a novel Xenopus homeobox gene expressed during eye and pineal gland development, Mech. Dev., 61:187–198. Cepko, C. L., 1999. The roles of intrinsic and extrinsic cues and bHLH genes in the determination of retinal cell fates, Curr. Opin. Neurobiol., 9:37–46. Chang-Ling, T., and J. Stone, 1991. Factors determining the migration of astrocytes into the developing retina: migration does not depend on intact axons or patent vessels, J. Comp. Neurol., 303:375–386. Chow, R. L., C. R. Altmann, R. A. Lang, and A. HemmatiBrivanlou, 1999. Pax6 induces ectopic eyes in a vertebrate, Development, 126:4213–4222. Collin, S. P., and H. B. Collin, 1999. The foveal photoreceptor mosaic in the pipefish, Corythoichthyes paxtoni (Syngnathidae, Teleostei), Histol. Histopathol., 14:369–382. Colonnier, M., and J. O’Kusky, 1981. Number of neurons and synapses in the visual cortex of different species, Rev. Can. Biol., 40:91–99. Curcio, C. A., K. R. Sloan, R. E. Kalina, and A. E. Hendrickson, 1990. Human photoreceptor topography, J. Comp. Neurol., 292:497–523. Curcio, C. A., K. R. Sloan, O. Packer, A. E. Hendrickson, and R. E. Kalina, 1987. Distribution of cones in human and monkey retina: individual variability and radial asymmetry, Science, 236:579–582. Dacey, D. M., 2000. Parallel pathways for spectral coding in primate retina, Annu. Rev. Neurosci., 23:743–775. Darwin, C., 1899. The Life and Letters of Charles Darwin, vol. 2 (Francis Darwin, ed.), New York: D. Appleton and Co., pp. 66– 67. Davis, A. A., M. M. Matzuk, and T. A. Reh, 2000. Activin A promotes progenitor differentiation into photoreceptors in rodent retina, Mol. Cell. Neurosci., 15:11–21. Dawkins, R., 1996. Climbing Mount Improbable, New York: W. W. Norton. Dirksen, M. L., and M. Jamrich, 1995. Differential expression of fork head genes during early Xenopus and zebrafish development, Dev. Genet., 17:107–116. Dkhissi-Benyahya, O., A. Szel, W. J. Degrip, and H. M. Cooper, 2001. Short and mid-wavelength cone distribution in a nocturnal Strepsirrhine primate (Microcebus murinus), J. Comp. Neurol., 438:490–504.

Dudley, A. T., K. M. Lyons, and E. J. Robertson, 1995. A requirement for BMP7 during development of the mammalian kidney and eye, Genes Dev., 9:2795–2805. Dyer, M. A., and C. L. Cepko, 2001. p27Kip1 and p57Kip2 regulate proliferation in distinct retinal progenitor cell populations, J. Neurosci., 21:4259–4271. Ezzeddine, Z. D., X. Yang, T. DeChira, G. Yancopoulos, and C. T. Cepko, 1997. Postmitotic cells fated to become rod photoreceptors can be respecified by CNTF treatment of the retina, Development, 124:1055–1067. Fekete, D. M., J. Perez-Miguelsanz, E. F. Ryder, and C. L. Cepko, 1994. Clonal analysis in the chicken retina reveals tangential dispersion of clonally related cells, Dev. Biol., 166:666– 682. Franco, E. C., B. L. Finlay, L. C. Silveira, E. S. Yamada, and J. C. Crowley, 2000. Conservation of absolute foveal area in New World monkeys. A constraint on eye size and conformation, Brain Behav. Evol., 56:276–286. Fuhrmann, S., M. Kirsch, and H. D. Hofmann, 1995. Ciliary neurotrophic factor promotes chick photoreceptor development in vitro, Development, 121:2695–2706. Furukawa, T., C. A. Kozak, and C. L. Cepko, 1997. rax, a novel paired-type homeobox gene, shows expression in the anterior neural fold and developing retina, Proc. Natl. Acad. Sci. USA, 94:3088–3093. Gallagher, B. C., A. M. Hainski, and S. A. Moody, 1991. Autonomous differentiation of dorsal axial structures from and animal cap cleavage stage blastomeres in Xenopus, Development, 112:1103–1114. Gan, L., S. W. Wang, Z. Huang, and W. H. Klein, 1999. POU domain factor Brn-3b is essential for retinal ganglion cell differentiation and survival but not for initial cell fate specification, Dev. Biol., 210:469–480. Gan, L., M. Xiang, L. Zhou, D. S. Wagner, W. H. Klein, and J. Nathans, 1996. POU domain factor Brn-3b is required for the development of a large set of retinal ganglion cells, Proc. Natl. Acad. Sci. USA, 93:3920–3925. Gehring, W. J., and K. Ikeo, 1999. Pax 6: mastering eye morphogenesis and eye evolution, Trends Genet., 15:371–377. Gilissen, E., and K. Zilles, 1996. The calcarine sulcus as an estimate of the total volume of human striate cortex: a morphometric study of reliability and intersubject variability, J. Hirnforsch., 37:57–66. Goldowitz, D., D. S. Rice, and R. W. Williams, 1996. Clonal architecture of the mouse retina, Prog. Brain Res., 108:3–15. Halder, G., P. Callaerts, and W. J. Gehring, 1995. Induction of ectopic eyes by targeted expression of the eyeless gene in Drosophila, Science, 267:1788–1792. Harland, R., 2000. Neural induction, Curr. Opin. Genet. Dev., 10: 357–362. Harris, W. A., 1997. Cellular diversification in the vertebrate retina, Curr. Opin. Genet. Dev., 7:651–658. Hartenstein, V., 1989. Early neurogenesis in Xenopus: the spatiotemporal pattern of proliferation and cell lineages in the embryonic spinal cord, Neuron, 3:399–411. Hatini, V., W. Tuo, and E. Lai, 1994. Expression of winged helix genes, BF-1 and BF-2, define adjacent domains within the developing forebrain and retina, J. Neurobiol., 25:1293– 1309. Hattar, S., H. W. Liao, M. Takao, D. M. Berson, and K. W. Yan, 2002. Melanopsin-containing retinal ganglion cells: architecture, projections, and intrinsic photosensitivity, Science, 295:1065– 1070.

  :  

73

Hayes, B. P., and M. D. Brooke, 1990. Retinal ganglion cell distribution and behaviour in procellariiform seabirds, Vision Res., 30:1277–1289. Hendrickson, A. E., 1994. Primate foveal development: a microcosm of current questions in neurobiology, Invest. Ophthalmol. Vis. Sci., 35:3129–3133. Hicks, D., and Y. Courtois, 1992. Fibroblast growth factor stimulates photoreceptor differentiation in vitro, J Neurosci., 12:2022–2033. Holt, C. E., T. W. Bertsch, H. M. Ellis, and W. A. Harris, 1988. Cellular determination in the Xenopus retina is independent of lineage and birthdate, Neuron, 1:15–26. Horton, J. C., and D. R. Hocking, 1996. Intrinsic variability of ocular dominance column periodicity in normal macaque monkeys, J. Neurosci., 16:7228–7239. Hoskins, S. G., 1985. Control of the development of the ipsilateral retinothalamic projection in Xenopus laevis by thyroxine: results and speculation, J. Neurobiol., 17:203–229. Huang, S., and S. A. Moody, 1993. Retinal fate of Xenopus cleavage stage progenitors is dependent upon blastomere position and competence: studies of normal and regulated clones, J. Neurosci., 13:3193–3210. Huang, S., and S. A. Moody, 1995. Asymmetrical blastomere origin and spatial domains of dopamine and neuropeptide Y amacrine subtypes in Xenopus tadpole retina, J. Comp. Neurol., 360:442–453. Huang, S., and S. A. Moody, 1997. Three types of serotonincontaining amacrine cells in tadpole retina have distinct clonal origins, J. Comp. Neurol., 387:42–52. Hyer, J., T. Mima, and T. Mikawa, 1998. FGF1 patterns the optic vesicle by directing the placement of the neural retina domain, Development, 125:869–877. Inoue, T., M. Hojo, Y. Bessho, Y. Tano, J. E. Lee, and R. Kageyama, 2002. Math3 and NeuroD regulate amacrine cell fate specification in the retina, Development, 129:831–842. Jeffery, W. R., and D. P. Martasian, 1998. Evolution of eye regression in the cavefish Astyanax: apoptosis and the pax6 gene, Am. Zool., 38:685–696. Jeffery, W. R., A. G. Strickler, S. Guiney, D. Heyser, and S. I. Tomarev, 2000. Prox1 in eye degeneration and sensory compensation during development and evolution of the cavefish Astyanax, Dev. Genes Evol., 210:223–230. Jensen, A. M., and M. C. Raff, 1997. Continuous observation of multipotential retinal progenitor cells in clonal density culture, Dev. Biol., 188:267–279. Jeon, C.-J., E. Strettoi, and R. H. Masland, 1998. The major cell populations of the mouse retina, J. Neurosci., 18:8936–8946. Kandil, F. I., and M. Fahle, 2001. Purely temporal figure-ground segregation, Eur. J. Neurosci., 13:2004–2008. Kelley, M.W., J. K. Turner, and T. A. Reh, 1994. Retinoic acid promotes differentiation of photoreceptors in vitro, Development, 120:2091–2102. Kelley, M. W., J. K. Turner, and T. A. Reh, 1995. Ligands of steroid/thyroid receptors induce cone photoreceptors in vertebrate retina, Development, 121:3777–3785. Kenyon, K. L., N. Zaghloul, and S. A. Moody, 2001. Transcription factors of the anterior neural plate alter cell movements of epidermal progenitors to specify a retinal fate, Dev. Biol., 240:77– 91. Kirsch, M., M. Y. Lee, V. Meyer, A. Wiese, and H. D. Hofmann, 1996. Evidence for multiple, local functions of ciliary neurotrophic factor (CNTF) in retinal development: expression of CNTF and its receptors and in vitro effects on target cells, J. Neurochem., 68:979–990.

74

 

Kulkarni, A. L., D. C. Airey, and R. W. Williams, 2000. Genetic architecture of the mouse retinogeniculate system: a QTL analysis of numerical matching, Neurosci. Abstr., 26:1087. Kumar, T., and D. A. Glaser, 1993. Initial performance, learning and observer variability for hyperacuity tasks, Vision Res., 33: 2287–2300. Lagutin, O., C. C. Zhu, Y. Furuta, D. H. Rowitch, A. P. McMahon, and G. Oliver, 2001. Six3 promotes the formation of ectopic optic vesicle-like structures in mouse embryos, Dev. Dyn., 221: 342–349. LaVail, M. M., D. H. Rapaport, and P. Rakic, 1991. Cytogenesis in the monkey retina, J. Comp. Neurol., 309:86–114. Leuba, G., and R. Kraftsik, 1994. Changes in volume, surface estimate, three-dimensional shape and total number of neurons of the human primary visual cortex from midgestation until old age, Anat. Embryol., 190:351–366. Levine, E. M., H. Roelink, J. Turner, and T. A. Reh, 1997. Sonic hedgehog promotes rod photoreceptor differentiaiton in mammalian retinal cells in vitro, J. Neurosci., 17:6277–6288. Lia, B., R. W. Williams, and L. M. Chalupa, 1987. Formation of retinal ganglion cell topography during prenatal development, Science, 236:848–851. Lillien, L., 1995. Changes in retinal cell fate induced by overexpression of EGF receptor, Nature, 377:158–162. Lillien, L., and C. L. Cepko, 1992. Control of proliferation in the retina: temporal changes in responsiveness to FGF and TGF alpha, Development, 115:253–266. Livesay, F. J., and C. L. Cepko, 2001. Vertebrate neural cell-fate determination: lessons from the retina, Nature Neurosci., 2:109– 118. Loosli, F., S. Winkler, C. Burgtorf, E. Wurmbach, W. Ansorge, T. Henrich, C. Grabher, D. Arendt, M. Carl, A. Krine, E. Grzebisz, and J. Wittbrodt, 2001. Medaka eyeless is the key factor linking retinal determination and eye growth, Development, 128:4035–4044. Loosli, F., S. Winkler, and J. Wittbrodt, 1999. Six3 overexpression initiates the formation of ectopic retina, Genes Dev., 13:649–654. Marc, R. E., and B. W. Jones, 2002. Molecular phenotyping of retinal ganglion cells, J. Neurosci., 22:413–427. Marquardt, T., R. Ashery-Padan, N. Andrejewski, R. Scardigli, F. Guillemot, and P. Gruss, 2001. Pax6 is required for multipotent state of retinal progenitor cells, Cell, 105:43–55. Marrow, E. M., M. J. Belliveau, and C. L. Cepko, 1998. Two phases of rod photoreceptor differentiation during rat retinal development, J. Neurosci., 18:3738–3748. Marrow, E. M., T. Furukawa, J. E. Lee, and C. L. Cepko, 1999. NeuroD regulates cell fate determination in the developing neural retina, Development, 126:23–26. Martinez-Morales, J. R., M. Signore, D. Acampora, A. Simeone, and P. Bovolenta, 2001. Otx genes are required for tissue specification in the developing eye, Development, 128:2019–2030. Mathers, P. H., A. Grinberg, K. A. Mahon, and M. Jamrich, 1997. The Rx homeobox gene is essential for vertebrate eye development, Nature, 387:603–607. McFarlane, S., M. E. Zuber, and C. E. Holt, 1998. A role for the fibroblast growth factor receptor in cell fate decisions in the developing vertebrate retina, Development, 125:3967–3975. Mills, S. L., and S. C. Massey, 1999. AII amacrine cells limit scotopic acuity in central macaque retina: an analysis with calretinin labeling, confocal microscopy and intracellular dye injection, J. Comp. Neurol., 411:19–34. Moody, S. A., 1987. Fates of the blastomeres of 32-cell stage Xenopus embryo, Dev. Biol., 122:300–319.

Moody, S. A., 1999. Testing the cell fate commitment of single blastomeres in Xenopus laevis, in Advances in Molecular Biology: A Comparative Methods Approach to the Study of Oocytes and Embryos ( J. Richter, ed.), New York: Oxford University Press. Moody, S. A., I. Chow, and S. Huang, 2000. Intrinsic bias and lineage restriction in the phenotype determination of dopamine and neuropeptide Y amacrine cells, J. Neurosci., 20:3244–3253. Moore, K. B., and S. A. Moody, 1999. Animal-vegetal asymmetries influence the earliest steps in retina fate commitment in Xenopus, Dev. Biol., 212:25–41. Mori, M., N. B. Ghyselinck, P. Chambon, and M. Mark, 2001. Systematic immunolocalization of retinoid receptors in developing and adult mouse eyes, Invest. Ophthalmol. Vis. Sci., 42:1312–1318. O’Kusky, J., and M. Colonnier, 1982. A laminar analysis of the number of neurons, glia, and synapses in the adult cortex (area 17) of adult macaque monkeys, J. Comp. Neurol., 210:278–290. Oliver, G., A. Mailhos, R. Wehr, N. G. Copeland, N. A. Jenkins, and P. Gruss, 1995. Six3, a murine homologue of the sine oculis gene, demarcates the most anterior border of the developing neural plate and is expressed during eye development, Development, 121:4045–4055. Onuma, Y., S. Takahashi, M. Asashima, S. Kurata, and W. J. Gehring, 2002. Conservation of Pax 6 function and upstream activation by Notch signaling in eye development of frogs and flies, Proc. Natl. Acad. Sci. USA, 99:2020–2025. Opas, M., and E. Dziak, 1994. bFGF-induced transdifferentiation of RPE to neuronal progenitors is regulated by the mechanical properties of the substratum, Dev. Biol., 161:440–454. Østerberg, G., 1935. Topography of the layer of rods and cones in the human retina, Acta Ophthalmol., 13(Suppl)6:1–103. Packer, O., A. E. Hendrickson, and C. A. Curcio, 1990, Development redistribution of photoreceptors across the Macaca nemestrina (pigtail macaque) retina, J. Comp. Neurol., 298:472–493 Patel, A., and S. McFarlane, 2000. Overexpression of FGF-2 alters cell fate specification in the developing retina of Xenopus laevis, Dev. Biol. 222:170–180. Peachey, N. S., L. Roveri, A. Messing, and M. A. McCall, 1997. Functional consequences of oncogene-induced horizontal cell degeneration in the retinas of transgenic mice, Vis. Neurosci., 14:627–632. Peichl, L., 1992. Topography of ganglion cells in the dog and wolf retina, J. Comp. Neurol., 324:303–320 Perron, M., and W. A. Harris, 1999. Cellular determination in amphibian retina, in Cell Lineage and Fate Determination (S. A. Moody, ed.), San Diego: Academic Press, pp. 353–368. Perron, M., S. Kanekar, M. L. Vetter, and W. A. Harris, 1998. The genetic sequence of retinal development in the ciliary margin of the Xenopus eye, Dev. Biol., 199:185–200. Pittack, C., G. B. Grunwald, and T. A. Reh, 1997. Fibroblast growth factors are necessary for neural retina but not pigmented epithelial differentiation in chick embryos, Development, 124:805–816. Pittack, C., M. Jones, and T. A. Reh, 1991. Basic fibroblast growth factor induces retinal pigment epithelium to generate neural retina in vitro, Development, 113:577–588. Politi, L. E., N. P. Rotstein, and N. G. Carri, 2001. Effect of GDNF on neuroblast proliferation and photoreceptor survival: additive protection with docosahexaenoic acid, Invest. Ophthalmol. Vis. Sci., 42:3008–3015. Powell, T. P., and A. E. Hendrickson, 1981. Similarity in number of neurons through the depth of the cortex in the binocular and monocular parts of area 17 of the monkey, Brain Res., 216:409–413.

Provis, J. M., C. M. Diaz, and B. Dreher, 1998. Ontogeny of the primate fovea: a central issue in retinal development, Prog. Neurobiol., 54:549–580. Provis, J. M., D. van Driel, F. A. Billson, and P. Russell, 1985. Human fetal optic nerve: overproduction and elimination of retinal axons during development, J. Comp. Neurol., 238:92–100. Rakic, P., and K. P. Riley, 1983. Overproduction and elimination of retinal axons in the fetal rhesus monkey, Science, 219:1441– 1444. Rapaport, D. H., S. L. Patheal, and W. A. Harris, 2001. Cellular competence plays a role in photoreceptor differentiation in the developing Xenopus retina, J. Neurobiol., 49:129–141 Reese, B. E., B. D. Necessary, P. P. Tam, B. Faulkner-Jones, and S. S. Tan, 1999. Clonal expansion and cell dispersion in the developoing mouse retina, Eur. J. Neurosci., 11:2965–2978. Reh, T. A., 1992. Cellular interactions determine neuronal phenotypes in rodent retinal cultures, J. Neurobiol., 23:1067–1083. Reh, T. A., and T. Tully, 1986. Regulation of tyrosine hydroxylasecontaining amacrine cell number in larval frog retina, Dev. Biol., 114:463–469. Repka, A., and R. Adler, 1992. Differentiation of retinal precursor cells born in vitro, Dev. Biol., 153:242–249. Rhinn, M., A. Dierich, W. Shawlot, R. R. Behringer, M. Le Meur, and S. L. Ang, 1998. Sequential roles for Otx2 in visceral endoderm and neuroectoderm for forebrain and midbrain induction and specification, Development, 125:845–856. Rice, D. S., R. W. Williams, and D. Goldowitz, 1995. Genetic control of retinal projections in inbred strains of albino mice, J. Comp. Neurol., 354:459–469. Rickman, D. W., and C. B. Rickman, 1996. Suppression of trkB expression by antisense oligonucleotides alters a neuronal phenotype in the rod pathway of the developing rat retina, Proc. Natl. Acad. Sci. USA, 93:12564–12569. Rockel, A. J., R. W. Hiorns, and T. P. Powell, 1980. The basic uniformity in structure of the neocortex, Brain, 103:221–244. Rodieck, R. W., 1998. The First Steps in Seeing. Sunderland, MA: Sinauer. Roorda, A., and D. R. Williams, 1999. The arrangement of the three cone classes in the living human eye, Nature, 397:520– 522. Saha, M. S., and R. M. Grainger, 1992. A labile period in the determination of the anterior-posterior axis during early neural development in Xenopus, Neuron, 8:1003–1014. Sharon, D., S. Blackshaw, C. L. Cepko, and T. P. Dryja, 2002. Profile of the genes expressed in the human peripheral retina, macula, and retinal pigment epithelium determined through serial analysis of gene expression (SAGE), Proc. Natl. Acad. Sci. USA, 99:315–320. Solursh, M., R. M. Langill, J. Wood, and T. K. Sampath, 1996. Osteogenic protein-1 is required for mammalian eye development, Biochem. Biophys. Res. Commun., 21:438–443. Spear, P. D., C. B. Kim, A. Ahmad, and B. W. Tom, 1996. Relationship between numbers of retinal ganglion cells and lateral geniculate neurons in the rhesus monkey, Vis. Neurosci., 13:199– 203. Springer, A. D., 1999. New role for the primate fovea: a retinal excavation determines photoreceptor deployment and shape, Vis. Neurosci., 16:629–636. Sterling, P., 1998. Retina, in Synaptic Organization of the Brain, 4th ed., New York: Oxford University Press, pp. 205–253. Strom, R. C., 1999. Genetic analysis of variation in neuron number. Ph.D. dissertation, University of Tennessee. http://www.nervenet.org/ papers/Strom99/Chapter5.html

  :  

75

Strom, R. C., and R. W. Williams, 1998. Cell production and cell death in the generation of variation in neuron number, J. Neurosci., 18:9948–9953. Suner, I., and P. Rakic, 1996. Numerical relationship between neurons in the lateral geniculate nucleus and primary visual cortex in macaque monkeys, Vis. Neurosci., 13:585–590. Tao, W., and E. Lai, 1992. Telencephalon-restricted expression of BF-1, a new member of the HNF-3/fork head gene family, in the developing rat brain, Neuron, 8:957–966. Tolhurst, D. J., and L. Ling, 1988. Magnification factors and the organization of the human striate cortex, Hum. Neurobiol., 6:247–254. Toy, J., J. M. Yang, G. S. Leppert, and O. H. Sundin, 1998. The optx2 homeobox gene is expressed in early precursors of the eye and activates retina-specific genes, Proc. Natl. Acad. Sci. USA, 95:10643–10648. Turner, D. L., and C. L. Cepko, 1987. A common progenitor of neurons and glia persists in rat retina late in development, Nature, 328:131–136. Turner, D. L., E. Y. Snyder, and C. L. Cepko, 1990. Lineage independent determination of cell type in the embryonic mouse retina, Neuron, 4:833–845. van Essen, D. C., W. T. Newsome, and J. H. Maunsell, 1984. The visual field representation in striate cortex of the macaque monkey: asymmetries, anisotropies, and individual variability, Vision Res., 24:429–448. Vetter, M. L., and N. L. Brown, 2001. The role of basic helix-loophelix genes in vertebrate retinogenesis, Semin. Cell Dev. Biol., 12:491–498. Vetter, M. L., and K. B. Moore, 2001. Becoming glial in the neural retina, Dev. Dyn., 221:146–153. Waddington, C. H., and E. Robertson, 1966. Selection for developmental canalization, Genet. Res., 7:303–312. Waid, D. K., and S. C. McLoon, 1995. Immediate differentiation of ganglion cells following mitosis in the developing retina, Neuron, 14:117–124. Waid, D. K., and S. C. McLoon, 1998. Ganglion cells influence the fate of dividing retinal cells in culture, Development, 125:1059–1066. Walls, G. L., 1942. The Vertebrate Eye and Its Adaptive Radiation, Bloomfield Hills MI: Cranbook Institute of Science. Watanabe, T., and M. C. Raff, 1990. Rod photoreceptor development in vitro: intrinsic properties of proliferating neuroepithelial cells change as development proceeds in the rat retina, Neuron, 2:461–467. Westheimer, G., 1981. Visual hyperacuity, Prog. Sensory Physiol., 1:1–37.

76

 

Wetts, R., and S. E. Fraser, 1988. Multipotent precursors can give rise to all major cell types of the frog retina, Science, 239:1142–1145. Wikler, K. C., and P. Rakic, 1990. Distribution of photoreceptor subtypes in the retina of diurnal and nocturnal primates, J. Neurosci., 10:3390–3401. Wikler, K. C., R. W. Williams, and P. Rakic, 1990. Photoreceptor mosaic: number and distribution of rods and cones in the rhesus monkey retina, J. Comp. Neurol., 297:499–508. Williams, R. W., 1991. The human retina has a cone-enriched rim, Vis. Neurosci., 6:403–406. Williams, R. W., 2000. Mapping genes that modulate brain development: a quantitative genetic approach, in Mouse Brain Development (A. F. Goffinet and P. Rakic, eds.), New York: Springer Verlag, pp. 21–49. Williams, R. W., C. Cavada, and F. Reinoso-Suarez, 1993. Rapid evolution of the visual system: a cellular assay of the retina and dorsal lateral geniculate nucleus of the Spanish wildcat and the domestic cat, J. Neurosci., 13:208–228. Williams, R. W., and D. Goldowitz, 1992a. Structure of clonal and polyclonal cell arrays in chimeric mouse retina, Proc. Natl. Acad. Sci. USA, 89:1184–1188. Williams, R. W., and D. Goldowitz, 1992b. Lineage versus environment in embryonic retina: a revisionist perspective, Trends Neurosci., 15:368–373. Williams, R. W., and P. Rakic, 1988. Elimination of neurons from the rhesus monkey’s lateral geniculate nucleus during development, J. Comp. Neurol., 272:424–436. Williams, R. W., R. C. Strom, and D. Goldowitz, 1998b. Natural variation in neuron number in mice is linked to a major quantitative trait locus on Chr 11, J. Neurosci., 118:138– 146. Williams, R. W., R. C. Strom, D. S. Rice, and D. Goldowitz, 1996. Genetic and environmental control of variation in retinal ganglion cell number in mice, J. Neurosci., 16:7193–7205. Williams, R. W., R. C. Strom, G. Zhou, and Z. Yan, 1998a. Genetic dissection of retinal development, Semin. Cell Dev. Biol., 9:249–255. Zhou, G., R. C. Strom, V. Giguere, and R. W. Williams, 2001. Modulation of retinal cell populations and eye size in retinoic acid receptor knockout mice, Mol. Vis., 7:253–260. Zhou, G., and R. W. Williams, 1999. Eye1 and Eye2: gene loci that modulate eye size, lens weight, and retinal area in mouse, Invest. Ophthalmol. Vis. Sci., 40:817–825. Zuber, M. E., M. Perron, A. Philpott, A. Bang, and W. A. Harris, 1999. Giant eyes in Xenopus laevis by overexpression of XOptx2, Cell, 98:341–352.

6

Development of the Vertebrate Retina RACHEL O. L. WONG AND LEANNE GODINHO

Introduction The neural retina, the light-sensitive part of the eye, converts light into electrical signals that are relayed to visual centers in the brain. The retina is part of the central nervous system (CNS), originating from the neural ectoderm during development. In vertebrates, the retina begins as an apparently homogeneous collection of cells in a single-layered neuroepithelium. With maturation, however, it develops into a laminated tissue with seven major cell classes, each of which occupies characteristic positions within the retina. The structural and functional development of the retina has long been studied because of its importance in vision, and because its highly organized structure is ideally suited for studying cell fate, differentiation, and patterning of synaptic networks in the nervous system.

Generation of cellular components C G Like other regions of the CNS, the retina is derived from the neural tube (Pei and Rhodin, 1970). Each eye begins as an outgrowth on either side of the neural tube. These outgrowths, called optic vesicles, invaginate to form a double-layered, cup-shaped structure. The thin outer wall becomes the pigment epithelium, while the inner wall develops into the neural retina. At this stage, the neural retina is a single-layered, pseudostratified epithelium. An extended period of cell proliferation begins to furnish the structure that will become multilayered. Cell proliferation in the retinal neuroepithelium occurs in a manner peculiar to the CNS. With their cytoplasmic processes extending from the pigment epithelial end to the internal limiting membrane, the nuclei of individual progenitor cells engage in a back-and-forth movement, undergoing different phases of the cell cycle at specific depths within the retinal neuroepithelium (Sidman, 1961; Young, 1985a). This nuclear movement, termed interkinetic migration, was first observed in the developing chick neural tube (Sauer, 1935). In the retina, mitosis (M-phase) occurs at the outermost surface, near the pigment epithelium. Each daughter cell resulting from a mitotic division extends a process toward the opposing vitreal surface while maintaining contact with the ventricular side. The daughter nuclei then enter G1, a resting phase, while migrating within their cytoplasmic pro-

jections toward the vitreal surface. At their destination, nuclei replicate their DNA (S-phase) and subsequently enter the G2 resting phase on their inward journey to the ventricular surface. Upon arriving, cells retract their cytoplasmic processes from the opposing end and remain poised for the next mitotic division (Robinson, 1991; Sidman, 1961; Young, 1985a). The resulting daughter cells then either reenter the cell cycle or leave it to become postmitotic neuroblasts. Cohorts of progenitors leave the cell cycle throughout the period of retinal development to differentiate into one of seven cell classes (Carter-Dawson and LaVail, 1979; Sidman, 1961; Young, 1985a, 1985b). The developmental time point at which this occurs for a particular cell type can be referred to as its birth date. The “birth dates” of each of the seven retinal cell classes have been deduced by exposing retinal cells to 3H-thymidine at known time points during development and examining the identity of the most heavily labeled nuclei in the mature retina (Carter-Dawson and LaVail, 1979; Sidman, 1961; Young, 1985a, 1985b). Each retinal cell type has a characteristic period during which its entire population is generated. However, there is considerable temporal overlap between the generation periods of most cell classes. Across species, retinal neuroblasts destined to become ganglion cells are the first to become postmitotic. Amacrine, horizontal, and cone photoreceptor cells are generated next; bipolar and Müller glial cells then follow. There is still some debate as to whether an early and transient population of Müller glial cells is generated during the period when ganglion and amacrine cells are born (Robinson, 1991). Rod photoreceptors are generated almost throughout the entire period of cytogenesis in the retina (Carter-Dawson and LaVail, 1979; Sidman, 1961; Young, 1985a, 1985b). Figure 6.1 summarizes the temporal order of appearance and placement of the cell types in the retina. In addition to the sequential addition of cells across the depth of the retina, cytogenesis also proceeds in a strict centroperipheral sequence. The first postmitotic retinal cells are found in a region (see the section “Cell Distribution”) close to the optic nerve head. Cell genesis occurs last, near the ora serrata (Carter-Dawson and LaVail, 1979; Sidman, 1961; Young, 1985a, 1985b). In many species, the centroperipheral sequence of cell genesis holds true for every retinal cell type.

77

A

B

ELM

E

R

NBL

C

R

ONL OPL

H

GCL

G

INL

B

ILM

C

M

A

D

IPL C H

H

M A

R H

ONL H

B

OPL

GCL

INL

A

IPL G

G

G

GCL

A= Amacrine cell G= Ganglion cell H= Horizontal cell

B= Bipolar cell C=cone, R=rod M=Müller cell

F 6.1. Basic structure and temporal order of cell differentiation of the vertebrate retina. A, The neural retina begins as a sheet of neuroepithelial cells (NBL, neuroblastic layer) with processes that contact the internal (ILM) and external (ELM) limiting membranes. B, Ganglion cells are the first cell type to differentiate and migrate toward the ILM to form the ganglion cell layer (GCL). C, Amacrine cells and horizontal cells differentiate next. Cone photoreceptors are born at this stage but do not differentiate until later. The first network of connections, between amacrine cells and ganglion cells, is established as a continuous inner plexiform layer (IPL)

forms across the retina. D, In the inner nuclear layer (INL), bipolar cells and Müller glial cells are generated and differentiate. At this stage, both cone and rod photoreceptors comprising the outer nuclear layer (ONL) begin to mature, bearing few outer segments or discs. The outer plexiform layer (OPL) emerges, and connections between photoreceptors, horizontal cells, and bipolar cells are formed. Cell genesis (mainly rods) continues at this stage but to a greatly reduced degree. E, At maturity, all cell types are present, connections in both plexiform layers are established, and photoreceptor outer segments are well developed.

C F Cell fate determination in the vertebrate retina is complex. Separate progenitors, each exclusively dedicated to the production of a single retinal cell type, do not exist. Instead, progenitor cells in the retina are regarded as multipotent (Holt et al., 1988; Turner and Cepko, 1987; Turner et al., 1990; Wetts and Fraser, 1988), that is, they are capable of producing more than one retinal cell class. How they adopt distinct fates during development has therefore been an area of intense research. The findings of numerous studies have resulted in a proposed model of retinal development in which cues both intrinsic and extrinsic to progenitors contribute to the determination of the cell fate (Cepko et al., 1996; Livesey and Cepko, 2001; Marquardt and Gruss, 2002). In this model, progenitor cells are believed to move sequentially through a series of stages, during which they are capable of producing only a limited repertoire of cell types (Cepko et al., 1996). For example, progenitor cells taken from chick retina at an early age when ganglion cells are exclusively produced continue to produce this cell type alone even when transplanted into a later retinal environment when signals conducive to a rod cell fate are present (Austin

et al., 1995). This limited competence of progenitor cells to produce only a small number of cell types at any given stage is thought to result from their genetic makeup. Progenitor cells destined for ganglion cell fates have been shown to express transcription factors such as Brain-3 (Brn-3) and retina-derived POU-domain factor-1 (RPF-1) (reviewed in Harris, 1997). Subsets of progenitors in the rat retina that express markers of mature amacrine and horizontal cells (syntaxin 1a and VC1.1) are biased toward producing these cell types (Alexiades and Cepko, 1997). Thus, the progenitor cell population is a heterogeneous one that is biased toward producing different cell fates. It is on this heterogeneous population of progenitors that extrinsic signals exert their influence. Extrinsic signals that have been implied in cell fate determination include neurotrophic factors such as nerve growth factor (NGF) and ciliary neurotrophic factor (CNTF), as well as other factors such as transforming growth factor a and b (TGF-a and -b), insulin-like growth factor (IGF), retinoic acid, thyroid hormone, and the amino acid taurine (reviewed in Harris, 1997). A direct role in cell fate determination is more evident for some factors such as CNTF (promotes

78

 

bipolar cell fates) and retinoic acid and taurine [rod photoreceptor cell fate (Altshuler et al., 1993)], while for others it remains unclear whether they are also involved in neurogenesis, differentiation, and cell survival. Of course, extrinsic signals that promote a particular cell fate are only positively received by cells competent to respond to them. Environmental signals could also act to suppress cell fates. Postmitotic neurons might be the source of such factors providing feedback to progenitors to cease the production of a particular cell type. This has been shown for ganglion cells; a still unidentified diffusible factor produced by ganglion cells limits the further production of this cell type (Waid and McLoon, 1998). While diffusible factors are a means of signaling between retinal cells during cell fate determination, communication also occurs by contact-mediated lateral inhibition. The receptor Notch and its ligand Delta have been shown to play a role in controlling cell fate. All cells start off with equal amounts of Notch and Delta. Activation of Notch blocks differentiation of the cell, at the same time leading to a reduction in its Delta levels. Expression of an activated form of Notch in progenitor cells from Xenopus and rat essentially inhibited cell differentiation and caused cells to remain in a progenitor-like state (Bao and Cepko, 1997; Dorsky et al., 1995). Conversely, when antisense oligonucleotides were used to reduce Notch activity in the retina, the number of ganglion cells produced was greatly increased (Austin et al., 1995). By misexpressing Delta in the Xenopus retina, Dorsky et al. (1997) demonstrated how the NotchDelta pathway regulates cellular differentiation in the retina. At early stages of development, cells with high levels of Delta adopt a ganglion cell or cone photoreceptor fate when surrounded by wild-type cells because their neighbors failed to suppress their differentiation. Misexpression at older stages leads to a high proportion of photoreceptor fate. Delta overexpressors, however, failed to differentiate when they were surrounded by cells with similarly high Delta levels. In contrast, reduction of Delta levels by expression of a dominant negative form of Delta resulted in an increase in the percentage of cells with an earlier fate. Thus the NotchDelta pathway is important for regulating the competence of progenitors to differentiate and respond to signals biasing their choice of fate at each stage of development. C M All cells are effectively “born” at the outer surface of the retina, apposed to the pigment epithelium (Sidman, 1961). Postmitotic cells must therefore migrate some distance to occupy positions characteristic of their phenotype within the retina. Unlike the cerebral cortex, another highly laminated structure, cell positioning in the vertical dimension of the retina is not strictly related to cell genesis. Cells born at similar times, for example, horizontal cells and amacrine cells, can end up in different layers.

In the cerebral cortex, postmitotic neuroblasts migrate radially along radial glial fibers from the ventricular zone toward the pia to take up their final positions (Rakic, 1971, 1990). Retinal neuroblasts are also believed to disperse radially from their point of origin. However, whether such radially migrating neuroblasts use glial guides, like their counterparts in the cortex, remains contentious. The presence of a radial glial scaffold in the immature retina was suggested by studies based on electron microscopy and immunohistochemistry with glial-specific antibodies (Meller and Tetzlaff, 1976; Wolburg et al., 1991). However, the positive identification of such glial structures by electron microscopy remains unclear, and glial markers such as vimentin may not be specific to such cells during development (Bennett and DiLullo, 1985; Lemmon and Rieser, 1983). Perikaryal translocation has been proposed as an alternative mechanism to account for radial migration within the retina (Book and Morest, 1990; Morest, 1970; Snow and Robson, 1994, 1995). This theory suggests that a newly postmitotic cell, located at the scleral surface of the neuroepithelium, extends a process toward the vitreal surface, its nucleus moving to its final destination within this process. The cell maintains an attachment with the scleral side of the neuroepithelium as its nucleus translocates, losing this attachment and that from the vitreal surface only when migration is complete. Evidence in support of perikaryal translocation in the retina originally came from studies of the morphology of retinal neuroblasts by Golgi impregnations (Morest, 1970; Prada et al., 1981). Observations from the retrograde labeling of ganglion cells (Dunlop, 1990; Snow and Robson, 1994, 1995) and immunohistochemical studies using ganglion cell–specific antibodies (McLoon and Barnes, 1989) support this hypothesis. All these studies reported postmitotic cells with a bipolar morphology, attached by processes to both surfaces of the retinal epithelium, and somata located at various depths within the neuroepithelium. Whatever mechanism is employed, it is now well accepted that newborn retinal cells migrate to their final positions in a radial fashion. When single or small numbers of progenitor cells were marked early in development by the injection of retroviral constructs or fluorescent dyes (Fekete et al., 1994; Turner and Cepko, 1987; Turner et al., 1990), and the distribution of their progeny was examined in the mature retina, they were seen to be distributed radially across the entire depth of the retina in tightly organized columns. Such a cell dispersion pattern was suggestive of the way in which retinal neuroblasts move from their point of origin in the germinal zone. Not all cells move strictly in the radial axis, however. When larger numbers of progenitors were labeled using transgenic (Fig. 6.2; Reese et al., 1995, 1999) or chimeric mice (Williams and Goldowitz, 1992), a small

  :  

79

XX XX

percentage of clonally related cells were found to stray outside the boundaries of the radial columns. Such cells are regarded to have dispersed tangentially from their point of origin in the germinal zone. Importantly, whether a cell dispersed radially or tangentially from its birthplace appears to be linked to its ultimate phenotype. Cells destined for rod photoreceptor, bipolar, and Müller glial cell fates were always found to disperse radially; radial columns are always composed of these cell types. Cells that disperse tangentially include cone photoreceptors and horizontal, amacrine, and ganglion cells (Reese et al., 1995).

XX XX

Both X-chromosomes active

X-inactivation

X

X

X

X

One X-chromosome active

ONL

INL

GCL X-Gal stained section of adult retina F 6.2. Schematic depicting the creation of X-inactivation transgenic mosaic retinas. A transgenic mouse line was created in which the lacZ transgene was inserted into one of the X-chromosomes (depicted as a blue-colored X). Early in development, all progenitor cells express b-galactosidase, the protein product of the lacZ transgene, as both chromosomes are active. In female transgenic mice, through the natural phenomenon of X-inactivation, one of the two X-chromosomes is randomly inactivated. This results in roughly equal numbers of transgene-expressing and -nonexpressing cells. In the adult, only the progeny of transgeneactive cells express b-galactosidase. When histochemically detected, these cells appear blue. Alternating radial columns of blue and white can be seen in a section of retina from such a transgenic mouse, indicative of their mode of dispersion. Occasionally however, individual blue cells can be seen tangentially removed from blue radial columns (arrow). ONL, outer plexiform layer; INL, inner nuclear layer; GCL, ganglion cell layer. (L. Godinho and S. S. Tan, unpublished.) (See color plate 3.)

80

 

C D At maturity, retinal neurons are often densely packed in a small region, typically referred to as central retina, which is actually located temporal to the optic nerve head. In primates this corresponds to the fovea, the center of which is a deep pit-like structure where cone photoreceptor density is at its highest (Chapter 26). Not all animals have a fovea, however; in some species, cell densities peak at a small patch called the area centralis. Spatial resolution of the retina is limited by the densities of cone photoreceptors and ganglion cells at the fovea and the area centralis (Rodieck, 1998). As the retina grows, cell density in the peripheral retina declines steeply, but it is reduced by a relatively smaller amount in the central retina (reviewed by Robinson, 1991). In primates, a further step takes place in fovea development, in which ganglion cell bodies become displaced from the center of the fovea with maturation, giving rise to the foveal pit. This lateral movement occurs after synapses have formed on the ganglion cells and may involve active migration of the cell bodies while their connections are “anchored” in place (reviewed by Kirby and Steineke, 1996). How do retinal cells become differentially distributed? Selective neurogenesis and naturally occurring cell death could contribute to the sharpening of the centroperipheral gradients in cell density. While there is little evidence for selective cell addition across the retina, cell death, particularly in the ganglion cell layer, is common and widespread (Robinson, 1991). In fact, about half the population of ganglion cells dies by maturity. But there is no well-defined spatiotemporal pattern of cell death across the retina, although this process, like other maturation events, follows a centroperipheral gradient. Differential expansion of the retina in which the peripheral region becomes stretched, like the surface of a balloon, has been suggested to be an important mechanism which shapes the density gradients of retinal neurons (Lia et al., 1987; Mastronarde et al., 1984; Robinson, 1991). This is because cell gradients continue to sharpen even after the period of neurogenesis and cell death. No one mechanism, however, appears to account for the final centroperipheral density gradients of each cell type (Robinson, 1991; Wikler and Rakic, 1996). Understanding how each gradient of cell density is determined,

and whether there is coordination in setting up these neuronal gradients, will reflect on how the convergence or divergence of information is established in the retina during development.

Cell differentiation and formation of the plexiform layers Cell differentiation in the developing retina is clearly reflected by the emergence of the inner and outer plexiform layers. These laminar synaptic regions appear before functional synaptic inputs are formed, as retinal neurons extend neurites toward each other in an apparently coordinated fashion. Here we will discuss what is known about process outgrowth in retinal neurons and what mechanisms might direct and coordinate their morphological differentiation, as well as produce laminar organizations of their connections. C P  P O The patterns of early neurite outgrowth in the retina have largely been investigated for retinal ganglion cells, mainly because of interest in the organization of their axonal projections to visual targets in the brain. Upon becoming postmitotic, retinal ganglion cells become polarized, generating an axon which is directed toward the vitreal surface and dendrites which project toward the neuroblastic layer (Fig. 6.3). Comparing the morphology of dye-labeled cells at different stages in development suggests that the axon may arise de novo from the cell body or is derived from the vitreally directed ventricular process (Hinds and Hinds, 1974; Maslim et al., 1986; Thanos and Mey, 2001). Axon outgrowth begins even prior to the arrival of the cell body to take a position in the forming ganglion cell layer. From static images of cells at different ages, it appears that dendritic outgrowth begins only when the cell body reaches the vitreal surface and when the

process attached to the ventricular surface retracts (Fig. 6.3). Axonal growth cones reach the optic fissure to form the optic nerve, and in some species they arrive at their central targets even before dendrites elaborate within the retina (Thanos and Mey, 2001). Coculture studies demonstrate that ganglion cell axonal and dendritic growth is directed or promoted by the microenvironment of the inner and outer retina, respectively (Bauch et al., 1998; Stier and Schlosshauer, 1998). Radial glial cells, thought to be Müller glia, influence the outgrowth of axons or dendrites in a highly selective way; their vitreal endfeet provide cues that encourage axon outgrowth, whereas cues from the region of the glial cell body direct dendritic, rather than axonal, outgrowth. The molecular interactions that polarize retinal ganglion cells are not yet fully understood, but some key players have been identified. For example, in the absence of Brn-3b, a POU-domain transcription factor, most retinal ganglion cells fail to extend axons, and instead prefer to generate dendrites (Wang et al., 2000). Compared to retinal ganglion cells, there is relatively little information concerning differentiation and process outgrowth of other inner retinal neurons. The majority of amacrine interneurons do not have polarized axons and dendrites; this makes these cells interesting candidates for studying how a single neurite can achieve this dual function. Neurite outgrowth from amacrine cells has been studied by serially reconstructing these cells by electron microscopy (Hinds and Hinds, 1983). These studies suggest that amacrine cells may be derived from retinal ganglion cells that have lost their axons, or that they may originate directly from ventricular neuroblasts. However, direct evidence demonstrating how amacrine cells differentiate and extend neurites to form the inner plexiform layer is still lacking. The observations of Hinds and Hinds do suggest that neurite outgrowth from amacrine cells is unlikely to be directed exclusively toward the ganglion cell–amacrine ELM

P

dendrites

axon

ILM

F 6.3. Presumed sequence of retinal ganglion cell differentiation. P, postmitotic ganglion cell; ELM and ILM, external and internal limiting membranes, respectively. Ganglion cells are first recognized when their axons appear.

  :  

81

cell border. Instead, it is possible that neurite outgrowth in amacrine cells is initially random or multidirectional, but that in time only processes growing laterally are maintained and elaborate to form the inner plexiform layer (Hinds and Hinds, 1983). How bipolar cells become polarized has not been investigated, but it is possible that the cues that direct ganglion cell polarization may be involved. The difficulty in following bipolar cell development has been the lack of bipolar cell–specific markers that are needed to identify these cells early in development. The earliest available marker is recoverin, which labels a subset of bipolar cells (Milam et al., 1993). However, recoverin also labels photoreceptors before they differentiate. Markers that distinguish different bipolar cell classes in the mature retina, such as protein kinase C or calbindin, appear relatively late, that is, when bipolar axonal terminals are already stratified in the inner plexiform layer (Miller et al., 1999). In the future, the use of molecular approaches to drive fluorescent protein expression in subsets of bipolar cells will be invaluable once suitable promoters are identified. In the outer retina, process outgrowth begins even before an outer plexiform layer is apparent (Robinson, 1991). Interestingly, photoreceptor terminals which in the mature retina stratify only within this outer lamina may do so only after retracting processes from the inner plexiform layer. In the ferret, recoverin-immunoreactive rods and cones transiently project an axon to the inner plexiform layer before bipolar cells differentiate and the outer plexiform layer forms ( Johnson et al., 1999). The function of this early projection is not known, but if connections are made with inner retinal neurons, these early projections would suggest the presence of a novel circuit in the developing retina. The first indica-

tion of a forming outer plexiform layer is when horizontal cells elaborate processes laterally, demarcating a border between the inner and outer retina. Initially, these cells put out processes in multiple directions, some even reaching the inner plexiform layer (Schnitzer and Rusoff, 1984). Eventually, horizontal cells maintain only processes that elaborate laterally. What signals direct the lateral growth of horizontal cell processes at the relevant position and time remain a mystery. C T–S A Shortly after retinal neurons extend neurites, they undergo a period of extensive process outgrowth. In some cells, a great deal of structural remodeling takes place before the final morphologies are attained. Factors, intrinsic and extrinsic to the cells, that may influence the final patterning of neurites in retinal neurons have largely been studied for the inner retinal neurons, particularly for ganglion cells. Dendritic development of retinal ganglion cells is complex, requiring each cell to achieve a specific pattern of branching, arbor size, and stratification within the inner plexiform layer. As dendrites emanate from the cell body, a rudimentary tree is formed, which elaborates by branching and process extension (Dunlop, 1990; Maslim et al., 1986). However, dendrites also retract, suggesting that the final spatial arrangement results from selective elaboration and elimination of processes (Fig. 6.4). A combination of genetic factors, environmental cues, and cell-cell interactions is likely to govern dendritic elaboration and retraction in retinal ganglion cells. Evidence for intrinsic control of dendritic patterning in ganglion cells comes from studies in which neonatal cat ganglion cells, dissociated and cultured without contacting neighboring cells, t= 0 hr

t= 3 hr

10 µm a el

el

ex

r

F 6.4. The dendritic arbor of retinal ganglion cells is sculpted by the addition and elimination of processes over time. Shown here is a ganglion cell from an embryonic chick retina labeled after transfection with plasmid encoding green fluorescent

82

 

ex

r 5 µm

protein. Images obtained 3 hours apart indicate that whereas some processes are stable (arrowhead), others are eliminated (el), have extended (ex), or have retracted (r). New processes are also added (a). (W. T. Wong and R. O. L. Wong, unpublished.)

regrow dendritic arbors that are diverse in morphology but resemble that of mature cells (Montague and Friedlander, 1989). Whether the neonatal cells reexecute an earlier plan to elaborate specific patterns of dendritic arborizations is unknown, but the observations suggest that patterning of the arbor can emerge in the absence of direct cell-cell contact. Also, cells with distinctive patterns such as large-field alphalike cells can be found across several species (Peichl, 1991), implying that the basic “dendritic plan” of ganglion cell classes may be specified intrinsically. Of course, whether there is a complement of genes issuing branching instructions to generate “alpha-like” patterns remains unknown, but it is tempting to speculate about such a possibility. Finally, the dendritic terminals of alpha-like cells in the ferret form dendro-dendritic contacts or fascicles early in development only if they are destined to be of the same ON or OFF subtype (Lohmann and Wong, 2001). Together, these observations encourage investigation into the genetic makeup of retinal ganglion cell classes. There is also substantial support for a role for cell-cell interactions or environmental cues that regulate the final dendritic patterning of retinal ganglion cells. Neurotrophins such as BDNF and NT-4 affect the patterning of retinal ganglion cell dendrites (Lom and Cohen-Cory, 1999); the source of these neurotrophins may be outside or inside the retina. In addition, neighbor relationships influence the size and symmetry of the ganglion cell dendritic arbor. In all species examined, dendritic field size varies inversely with cell density across the retina. This relationship is present before the retina matures (Sernagor et al., 2001). Experimental manipulations to increase or decrease eye size, and thus affect cell density distributions, result in adjustments in dendritic field area (Troilo et al., 1996). A direct relationship between cell density and the patterning of the dendritic arbor is evident from studies in which ganglion cells have been ablated in some regions of the retina (Deplano et al., 1994; Perry and Linden, 1982). Ganglion cells reorient their dendrites toward the lesion zone apparently in an attempt to cover the cell-depleted area. It is still unclear whether such rearrangements are due to a loss of neighboring ganglion cells of the same class or subclass (Ault et al., 1993; Weber et al., 1998). This knowledge will provide more insight into whether cell-cell interactions between specific cell types are needed to maintain, for example, coverage of the retina by a single cell class (see Chapter 29) or whether it is the overall availability of presynaptic terminals that is key to regulating dendritic patterns. What is apparent, however, is that neurotransmission is involved in regulating the dendritic patterning of retinal ganglion cells (Sernagor and Grzywacz, 1996; Wong et al., 2000a, 2000b). Much less is known about how the many classes of amacrine and bipolar cells achieve their characteristic arborizations. Cholinergic amacrine cells which differentiate

early in development (Galli-Resta et al., 1997) have radially arranged arbors that undergo some structural remodeling with maturation (Wong and Collin, 1989; Fig. 6.5). In contrast, process outgrowth in another amacrine cell with a radiating arbor, the so-called fireworks cell, may be more “deterministic.” The characteristic nonbranching, radiating processes of this type of amacrine cell are evident even during embryonic development (Fig. 6.5). However, studies in the rodent retina suggest that cell-cell interactions are also important for defining the arborization patterns of amacrine cells. Signaling involving neurotrophins (Rickman, 2000) and reelin (Rice et al., 2001) shape the projection patterns of AII amacrine cells. L From a developmental point of view, it is intriguing to find that the retinal layers are generated sequentially, and normally, without error in their order of appearance. For instance, in all species studied, the inner plexiform layer appears before the outer plexiform layer. As noted in the section on “Cell Genesis,” apart from ganglion cells, when a retinal cell type is generated does not necessarily predict where it will ultimately reside. Thus, environmental factors must come into play to organize the layering of the retina in terms of cell body location and where their processes eventually elaborate. Another intriguing aspect of lamination concerns how the layers become contiguous across the retina. Not only do the major lamination patterns extend across the retina, but for each type of inner retinal neuron, such as amacrine cells, processes of the same kind ramify in one or a few distinct sublaminae, forming a contiguous plexus (or plexuses) across the retinal surface. How does this happen? Zebrafish mutants have provided some insights into how this might occur for both the inner and outer plexiform layers. In lakritz mutants, a continuous inner plexiform player still develops, suggesting that the formation of this synaptic lamina does not require the presence of ganglion cells, which fail to differentiate in this mutant (Kay et al., 2001). In other zebrafish mutants, however, such as young, there are no obvious plexiform layers; this may be because of the perturbed final stages of morphological differentiation of retinal neurons (Link et al., 2000). The outer plexiform layer forms in the absence of transmission from photoreceptors, implying that elaboration of processes within this layer does not require this form of cell-cell communication (Allwardt et al., 2001). Although our knowledge of the mechanisms that regulate layering in the retina is still sketchy, it is likely that this situation will change when more lamination mutants and the genes responsible for these mutations are identified. The actions of multiple well-coordinated mechanisms come to mind when we also realize that within the inner plexiform layer, up to 10 or more functionally distinct sublaminae are established by maturity (see Chapter 18). Of

  :  

83

A

C

B

neonate

adult

50 µm

F 6.5. Development of amacrine cell processes. A, A “fireworks” type of amacrine cell in the embryonic chick retina sends out processes radially without much branching. Each process is tipped with a growth cone (examples, arrows). Even at this immature stage, the arbor morphology of this cell type is already patterned. Moreover, the processes are already confined to two distinct

strata in the inner plexiform layer (B, side view of the cell). The cell was labeled after transfection with plasmid encoding green fluorescent protein. C, In contrast, the processes of cholinergic amacrine cells (from the rabbit retina) undergo remodeling with maturation. Many small branches are eliminated with age (Wong and Collin, 1989). Cells labeled by intracellular dye filling.

most interest is how ON and OFF sublamination takes place, an event that commences well before eye opening and requires the sorting out of three sets of neuronal processes— the dendrites of ganglion cells, processes of amacrine cells, and axons of bipolar cells. Morphological studies show that much structural reorganization takes place before the dendrites of ganglion cells become confined to the ON or OFF sublamina (Fig. 6.6; Bodnarenko and Chalupa, 1993;

Bodnarenko et al., 1999; Kirby and Steineke, 1996; Lohmann and Wong, 2001; Maslim and Stone, 1988). Retinal ganglion cells which at maturity have either ON- or OFF-center responses receive converging inputs from ON and OFF bipolar cells when their arbors have not yet stratified (Wang et al., 2001). Thus, importantly, developmental rearrangements in the dendritic structure of the ganglion cells underlie their functional maturation.

84

 

How ganglion cell dendritic stratification is achieved is not yet known, but it is a subject of intense investigation. The first indication that transmission from presynaptic interneurons is involved in dendritic stratification of retinal ganglion cells came from studies in which 2-amino-4phosphonobutyric acid (APB), an agonist of mGluR6 receptors, was injected intraocularly during the period in which dendritic arbors of ganglion cells became stratified (Bodnarenko et al., 1995, 1999). Retinal ganglion cells failed to stratify in the APB-treated eyes. Although APB treatment disrupts the stratification of ganglion cell dendrites, it is unlikely that glutamatergic transmission initiates this process. This is because the process of stratification begins before bipolar cells have differentiated. Amacrine cells, which differentiate earlier than bipolar cells and around the same time as ganglion cells, may mediate the early stratification process. Knockout mice in which populations of amacrine cells are ablated or their function perturbed will be useful for assessing the role of these interneurons. Examination of mice lacking the beta2 subunit of the cholinergic receptor indicated that ganglion cell stratification still occurs, although there may be some abnormalities (Bansal et al., 2000). However, it may be that not all ganglion cell dendritic arbors are affected in beta2 knockout mice because not all ganglion cells may be major postsynaptic targets of cholinergic amacrine cells. Because there are many other popula-

tions of amacrine cells, determining whether ganglion cell dendritic stratification is affected by amacrine transmission will require detailed consideration of subcircuits within the inner retina. This will involve determining which amacrine cells form the major input onto each type of ganglion cell, a task that is extremely challenging. Finally, bipolar cell interactions are likely to be important for the maintenance of stratification in the ganglion cells. Selective perturbation of transmission from ON or OFF bipolar cells should help determine whether bipolar cell inputs affect structural reorganization of the ganglion cell dendritic arbor. Ganglion cell dendritic stratification in mGluR6 knockout mice appears normal, but this may be because activity is enhanced in the ON bipolar cells rather than suppressed (Tagawa et al., 1999). How could neurotransmission help determine the stratification of ganglion cell arbors? One possibility is that initially, synaptic inputs are not equally distributed across the dendritic arbor. Through competitive interactions that favor the retention of more “active” or collectively “stronger” inputs, dendrites in one sublamina may be selectively maintained or eliminated. To date, however, the initial distributions and synaptic strengths of inputs onto the dendritic arbor of retinal ganglion cells are unknown, but their measurement would be very helpful in understanding how the inner retinal circuitry is established. Mature

Immature

A

IPL

OFF ON

IPL

B OFF

IPL

ON GCL

F 6.6. The dendritic arbors of retinal ganglion cells undergo structural reorganization during the period of synapse formation and maturation. A, Reconstructions from confocal images of alpha-like ganglion cells in the newborn (left) and 3-week-old (right) ferret retina. Below, orthogonal view of the cells. Scale bars,

ON

OFF

20 mm. The distribution of dendrites within the inner plexiform layer (IPL) becomes gradually restricted to either the ON or OFF sublamina with age. B, Schematic representation of the stratification process. GCL, ganglion cell layer. (A adapted from Lohmann and Wong, 2001.)

  :  

85

Even though interactions between amacrine cells and bipolar cells may influence the segregation of ON and OFF pathways within the retina, other nonactivity-dependent mechanisms must be considered. This is because there are other aspects of dendritic development in ganglion cells that may not depend on neurotransmission alone. Notably, mechanisms which regulate the arbor size of retinal ganglion cells during mosaic formation are likely to include molecular cues (Sernagor et al., 2001). Thus, the remodeling of dendritic arbors during development of retinal ganglion cells possibly requires several mechanisms that act together to shape their lateral and vertical distributions in a coordinated way. While contact between retinal interneurons and ganglion cells might direct their stratification according to depth within the inner plexiform layer, contact between retinal ganglion cells themselves may organize the lateral extent of their dendritic territory (see also Chapter 9). If amacrine and bipolar cell interactions effect the dendritic stratification of ganglion cells, what controls the stratification of these interneurons? Although an answer is still far off, deletion of one of the three major inner retinal cell classes during development should provide important insights. What is known so far is that in rodents, ganglion cells are not needed for amacrine or bipolar cell processes to stratify in the inner plexiform layer (Gunhan-Agar et al., 2000).

Wiring up the retina N  R The functional maturation of retinal circuits depends on the expression of neu-

rotransmitters and their receptors, as well as the formation of synaptic contacts. We will discuss briefly the timing and patterns of expression of the major excitatory and inhibitory transmitters in the retina and summarize when receptors for these transmitters appear (see Fig. 6.7). Glutamate is the major excitatory transmitter in the adult retina, present in photoreceptors, bipolar cells, and retinal ganglion cells. Immunolabeling for glutamate in the developing retina demonstrates that expression of this amino acid is first detected in undifferentiated cells in the ventricular zone of the retina before the plexiform layers appear. Surprisingly, cell bodies in the ganglion cell layer are immunonegative for glutamate until bipolar cells and Müller glial cells have differentiated. This may be because ganglion cells synthesize glutamate from glutamine that is normally provided by the Müller glia. In the outer retina, photoreceptors express glutamate long before they differentiate and form connections. Functional receptors for glutamate are expressed by retinal ganglion cells and amacrine cells before they receive synaptic inputs (Liets and Chalupa, 2001; Wong, 1999). The distribution of the various ionotropic glutamate receptor subunits has been assessed by immunocytochemistry: both Alpha-amino-3-hydroxy-5methylisoxazole-4-proprionic acid (AMPA) and -methyl-aspartate (NMDA) receptor subunit staining becomes restricted to distinct bands within the inner plexiform layer by the time of eye opening (Grunder et al., 2000a, 2000b). These banding patterns appear in retinas immunolabeled for AMPA receptor subunits before those of NMDA receptor subunits.

B

A

E

R

NBL

ONL

C

OPL H

GCL

G

M

INL

B

C

D

A C

H

H H

M A a G

B

R

OPL GCL

G

INL

A G

IPL

ONL

IPL

GABA

ACh

GCL

Glycine

Glutamate

F 6.7. Expression patterns of major excitatory and inhibitory neurotransmitters in the developing and adult retina. Labeling of cell types and retinal layers are provided in Figure 6.1. GABA, gamma-aminobutyric acid; ACh, acetylcholine. (See color plate 4.)

86

 

The other major excitatory fast neurotransmitter in the retina is acetylcholine. Immunolabeling for choline acetyltransferase (ChAT) has been carried out in a large range of species. ChAT expression is present early in development, often prenatally, and cholinergic transmission takes place before glutamatergic transmission is apparent (see the section “Spontaneous Activity”). In mammal and chick, retinal neuroblasts in the ventricular cell layer demonstrate an increase in intracellular calcium upon stimulation with muscarinic receptor agonists (Sakaki et al., 1996; Wong, 1995a). Whether the expression of these receptors has a major functional role in the differentiation of these cells remains to be determined. Amacrine and ganglion cells, prior to the formation of synapses, are also able to respond to applied nicotine (Wong, 1995b). Thus, like ionotropic glutamate receptors, retinal neurons express functional cholinergic receptors prior to the emergence of synaptic networks. There has been much interest in the development of the inhibitory systems of the retina, primarily because GABA and glycine have been linked to cell survival and differentiation in many neuronal systems (Cherubini et al., 1991). GABA immunoreactivity is first detected in amacrine cells and in some ganglion cells just before birth in the rabbit and in rodents. Ganglion cell expression disappears at maturity. In mammals, horizontal cells are transiently immunoreactive for GABA at the time when cone photoreceptors can respond to this amino acid (Huang and Redburn, 1996). Glycine immunoreactivity is dominant in the ganglion cell layer and the fiber layer in the immature retina. With maturation, glycine immunoreactivity disappears from the ganglion cells and becomes restricted to amacrine cells and a few bipolar cells (Pow et al., 1994). In contrast to cells at maturity, retinal neurons, including ganglion cells, are depolarized by GABA and glycine early in development (Wong et al., 2000b; Yamashita and Fukuda, 1993). This is because, like other neurons in the developing CNS, immature retinal ganglion cells have a relatively high intracellular chloride concentration. This is probably because the K-Cl cotransporter (KCC2) that extrudes chloride from cells increases in expression only after birth (Vu et al., 2000). Interestingly, KCC2 expression in the inner retina precedes that in the outer retina, suggesting that prior to eye opening, GABA action in the outer plexiform layer may differ from that in the inner retina. A common feature, then, of the developing neurotransmitter system in the retina is that expression of the transmitters and their receptors occurs before visual processing, and even before electrical activity can be detected. It is possible that communication between retinal neurons still occurs, perhaps through release of transmitter from growth cones (Gordon-Weeks et al., 1984). Such communication could be important for maturation events in the retina,

although direct evidence for a developmental role for transmitter-mediated interactions in cell differentiation is still lacking. During the period of synaptogenesis, however, neurotransmission appears to be important for the maintenance of dendritic structure, as discussed earlier (Wong et al., 2000a). Furthermore, early transmission, before vision, is also necessary for establishing the correct patterns of connections between the retina and visual centers in the brain (see the section “Spontaneous Activity”). The availability of knockout mice lacking synthetic enzymes for various neurotransmitters and the ability to generate conditional knockouts of these mice will no doubt be useful in the future for assessing the importance of the early expression of neurotransmitter systems in the retina and in other parts of the nervous system. S F Synapse formation in the retina has largely been assessed by electron microscopy (see Robinson, 1991, for a review). In many species, synaptogenesis in both plexiform layers begins before eye opening, occurring first in the inner plexiform layer about two-thirds of the way through gestation. In particular, amacrine cells form synapses with ganglion cells, and between themselves, shortly after these populations of neurons extend neurites into the inner plexiform layer (Maslim and Stone, 1986; Nishimura and Rakic, 1985, 1987). Membrane thickenings are observed in the inner plexiform layer before synaptic vesicles can be recognized. Studies of primate retina suggest that these thickenings or specializations occur first in postsynaptic processes, the dendrites of ganglion cells, before they are apposed by amacrine cell processes (Nishimura and Rakic, 1985, 1987). Maturation of conventional synapses involves the acquisition of an increasing number of synaptic vesicles. Bipolar synapses, which are characterized by the presence of a ribbon-like structure, appear after conventional synapses are formed in the retinal periphery of primates. Initially, serial electron microscopy indicates that putative bipolar terminals are observed without ribbons. Then ribbon-containing terminals form dyads, making contact with a single postsynaptic process. With maturation, the typical triad arrangement is seen when each bipolar terminal contacts two processes, which could belong to amacrine or ganglion cells. In contrast to the retinal periphery, in the fovea bipolar synapses appear before conventional synapses are observed (Crooks et al., 1995). It may be that in cone-dominated regions of the retina, bipolar synapses develop before amacrine synapses (see Hendrickson, 1996). Synapses in the outer plexiform layer are first formed between photoreceptors and horizontal cells (McArdle et al., 1977). The last cellular element to enter into the circuitry is the bipolar cell that appears to be contacted first by cones. Rod bipolar cell differentiation may occur later, and

  :  

87

synapses from rods are therefore established after those of cones, although there is substantial overlap in timing. Synaptogenesis in the outer plexiform layer continues after eye opening, raising the possibility that the properties of this outer synaptic layer may be more susceptible to sculpting by visual stimulation compared to synapses in the inner plexiform layer (Robinson, 1991). The ultrastructural observations provide a broad view of the order of appearance of synapses within the retina, as well as provide clues as to how various connections may arise. To gain an understanding of the nature of the initial interactions that result in the contact and maturation of synapses between retinal neurons, it will be necessary to watch how pre- and postsynaptic processes behave in living tissue. Although this has not yet been achieved, recent imaging experiments of dendritic motility in the isolated developing retina suggest that contact between inner retinal neurons may involve transient filopodia-like structures that are highly dynamic in nature (Wässle, 1988; Wong and Wong, 2000; Wong et al., 2000b). These filopodia-like structures, which extend and retract over the course of seconds to minutes, are most abundant during the period of synaptogenesis in the inner retina. Because of their high density and motility, dendritic filopodia effectively increase the volume of dendrite that can be contacted by incoming amacrine or bipolar afferents. Although this potential role for dendritic filopodia has yet to be demonstrated, their regulation by neurotransmission supports the notion that afferent signaling encourages contact with dendrites by promoting their motility (Wong and Wong, 2001). Many questions, of course, remain. How do the many classes and subclasses of retinal neurons find their appropriate synaptic partners? How much remodeling of the initial connectivity occurs in retinal circuits? What determines the lamination and sublamination of the plexiform layers? How is synapse formation coordinated across the plexiform layers and within each lateral and vertical network? Is the mature mammalian retina capable of regenerating connections?

Physiology of the developing retina S A Neurotransmission occurs in the developing retina even before photoreceptors mature, and begins around the time that amacrine inputs to ganglion cells are formed. This early transmission between amacrine and ganglion cells sustains a unique pattern of retinal activity that has important developmental roles. Retinal ganglion cells fire action potentials during the period when their connections with central targets undergo reorganization. Recordings from rat fetuses in vivo demonstrated that immature ganglion cells fire spikes in bursts that occur periodically, about once every minute (Galli and Maffei, 1988;

88

 

Maffei and Galli-Resta, 1990). Simultaneous recordings from neighboring ganglion cells using multielectrode arrays (Fig. 6.8) later showed that bursts of spikes from the ganglion cells are temporally synchronized (Meister et al., 1991). This is because activity propagates across the retinal surface as waves, which are clearly seen using optical imaging techniques (Fig. 6.8). All major classes of retinal ganglion cells and subpopulations of amacrine cells participate in the wave activity. The waves are generated by mechanisms requiring synaptic input from amacrine cells, and later with maturation, transmission from bipolar cells (reviewed in Wong, 1999). Exactly how this inner retinal circuitry generates the wave pattern of activity is not fully known, but transmission from cholinergic amacrine cells is important early on (Feller et al., 1996; Sernagor et al., 2001). Waves are abolished in the presence of neuronal nicotinic receptor antagonists and are absent in mice which lack specific subunits of these receptors (Bansal et al., 2000). The spontaneous activity patterns of the immature ganglion cells contain several spatiotemporal cues that could be important for the refinement of their connectivity with their central targets. First, spiking in bursts appears necessary for long-term potentiation of synapses between retinal ganglion cells and the lateral geniculate nucleus (Mooney et al., 1996). Second, the propagating waves ensure that activity of neighboring cells is better correlated compared to cells that are more distant. This feature provides a means for geniculate neurons to gauge intercellular distance within the retina based on the timing of their spikes. Such information could be important in helping refine the retinotopic map, which becomes sharpened in many species with maturation. Third, because waves occur randomly and are relatively infrequent, waves from the two eyes are unlikely to be synchronized. Such asynchronous activity from the two eyes is thought to drive the segregation of connections from the two eyes which initially converge onto single geniculate neurons. Indeed, the projections from the left and right eyes are abnormal in nicotinic receptor knockout mice and in ferrets that have been injected intraocularly with epibatidine, a nicotinic receptor antagonist, during the period of axonal segregation (Penn et al., 1998; Bansal et al., 2000). Although activity from the retinas prior to visual stimulation is important for establishing the projection patterns of the ganglion cells during development, direct evidence for patterned activity underlying these events has yet to be obtained. This would require maintaining spontaneous activity but altering the spatiotemporal relationship between spiking of neighboring ganglion cells. Prior to eye opening in the ferret, the spontaneous spiking patterns of retinal ganglion cells are altered in their temporal characteristics in a cell-type-specific manner. Although waves are still present and each wave synchronizes the spiking of neighboring ON and OFF cells, OFF cells spike

A

Immature

Mature

100s

B

fura-2

1 mm

t=0 s

2s

4s

6s

F 6.8. Patterns of spontaneous activity in the developing retina. A, Spike recordings from the mouse retinal ganglion cell layer using a multielectrode array before (immature) and after (mature) eye opening. B, Calcium imaging of the immature mouse

retina showing labeling of the retina with the calcium indicator, fura-2, and the spread of activity, indicated by the white pixels, across the retinal surface over time. (A, J. Demas and R. O. L. Wong, unpublished; B, R. O. L. Wong, unpublished.)

more frequently between waves than ON cells (Wong and Oakley, 1996). This change in the activity patterns of the ON and OFF ganglion cells occurs during the period when the axonal projections of these ganglion cells to the geniculate nucleus becomes refined. Theoretical studies suggest that the decrease in correlated activity between ON and OFF ganglion cells is sufficient to cause their inputs onto geniculate neurons to segregate over time (Lee et al., 2002). By the time of eye opening, waves disappear and the spontaneous spike activity of retinal ganglion cells is no longer patterned, instead appearing randomly (Fig. 6.8). However, different rates of spontaneous firing occur even in the adult retina, although their correlation with retinal ganglion cell type has yet to be established. The mechanisms that underlie the progressive change in spontaneous activity patterns of the ganglion cells are not yet fully uncovered. It is likely that the developmental maturation of the spike patterns is shaped by both intrinsic properties of the ON and OFF ganglion cells and by their circuitry (Myhr et al., 2001). It is also possible that maturation of the outer retina, particularly the emergence of light sensitivity in photoreceptors, affects the patterns of spontaneous spiking of ganglion cells.

days later, the concentric center-surround organization of the receptive fields, as well as ON and OFF center responses, are already present (Bowe-Anders et al., 1975; Masland, 1977; Tootle, 1993). Whether the connectivity that underlies these physiological properties is established before photoreceptors are present is unknown. Determining how surround inhibition appears in the retinal ganglion cells has not been straightforward. Some immature rabbit retinal ganglion cells have silent surrounds that, when stimulated, can suppress the response to center stimulation, but direct stimulation of the surround does not evoke a response (Masland, 1977). In the cat, however, the strength of the antagonistic surround relative to that of the center does not seem to change with postnatal maturation (Tootle, 1993). Recordings from ferrets, however, clearly demonstrate that connectivity in the inner retina is remodeled with maturation. In the postnatal ferret, alpha- and beta-like retinal ganglion cells have convergent ON and OFF inputs prior to maturity (Wang et al., 2001). In these cells, maturation of the receptive field center responses thus involves the loss of one type of input. Specialized receptive field properties such as direction selectivity also develop before eye opening (Masland, 1977; Sernagor and Grzywacz, 1995), although the synaptic basis for this property remains to be determined. How then is the ganglion cell receptive field established during development? Visual experience after eye opening does not appear to alter the receptive field properties of mammals that were raised in an environment with unidirectionally moving stimuli (Daw and Wyatt, 1974). But this may be because, in rabbits, ganglion cell receptive fields are fairly mature by the time of eye opening (Masland, 1977). In contrast, the peak firing rate of retinal ganglion cells

L R Light responses emerge as the photoreceptor-bipolar pathway begins to mature shortly before eye opening (Dacheux and Miller, 1981a, 1981b; Masland, 1977; Tootle, 1993). Electrophysiological recordings from retinal ganglion cells show several major trends in the maturation of their responses to light. The early response of retinal ganglion cells to light stimulation is weak and the cells adapt rapidly (Masland, 1977; Tootle, 1993). But when robust responses to light become detectable a few

  :  

89

in response to light stimulation is decreased in dark-reared mice (Tian and Copenhagen, 2001). Cells also respond more sluggishly in dark-reared animals. The spatial organizations of the receptive fields have not yet been assessed after dark rearing of mice. In turtles, which become light responsive prior to hatching, dark rearing causes an increase in receptive field size (Sernagor and Grzywacz, 1996). However, this study also suggested that it is spontaneous activity rather than visual stimulation that regulates the receptive field size. Clearly, much remains to be done to fill our knowledge gaps concerning how the light responses of retinal neurons are established in ways that are characteristic of each cell type. While in the past such properties were largely studied using electrophysiological methods, it is likely that the combination of physiology with molecular and live-imaging techniques will now aid investigations in this important area of retinal development. REFERENCES Alexiades, M. R., and C. L. Cepko, 1997. Subsets of retinal progenitors display temporally regulated and distinct biases in the fates of their progeny, Development, 124:1119–1131. Allwardt, B. A., A. B. Lall, S. E. Brockerhoff, and J. E. Dowling, 2001. Synapse formation is arrested in retinal photoreceptors of the zebrafish nrc mutant, J. Neurosci., 21:2330–2342. Altshuler, D., J. J. Lo Turco, J. Rush, and C. Cepko, 1993. Taurine promotes the differentiation of a vertebrate retinal cell type in vitro, Development, 119:1317–1328. Ault, S. J., K. G. Thompson, Y. Zhou, and A. G. Leventhal, 1993. Selective depletion of beta cells affects the development of alpha cells in cat retina, Vis. Neurosci., 10:237–245. Austin, C. P., D. E. Feldman, J. A. Ida, Jr., and C. L. Cepko, 1995. Vertebrate retinal ganglion cells are selected from competent progenitors by the action of Notch, Development, 121:3637–3650. Bansal, A., J. H. Singer, B. J. Hwang, W. Xu, A. Beaudet, and M. B. Feller, 2000. Mice lacking specific nicotinic acetylcholine receptor subunits exhibit dramatically altered spontaneous activity patterns and reveal a limited role for retinal waves in forming ON and OFF circuits in the inner retina, J. Neurosci., 20:7672–7681. Bao, Z. Z., and C. L. Cepko, 1997. The expression and function of Notch pathway genes in the developing rat eye, J. Neurosci., 17:1425–1434. Bauch, H., H. Stier, and B. Schlosshauer, 1998. Axonal versus dendritic outgrowth is differentially affected by radial glia in discrete layers of the retina, J. Neurosci., 18:1774–1785. Bennett, G. S., and C. DiLullo, 1985. Transient expression of a neurofilament protein by replicating neuroepithelial cells of the embryonic chick brain, Dev. Biol., 107:107–127. Bodnarenko, S. R., and L. M. Chalupa, 1993. Stratification of ON and OFF ganglion cell dendrites depends on glutamatemediated afferent activity in the developing retina, Nature, 364:144–146. Bodnarenko, S. R., G. Jeyarasasingam, and L. M. Chalupa, 1995. Development and regulation of dendritic stratification in retinal

90

 

ganglion cells by glutamate-mediated afferent activity, J. Neurosci., 15:7037–7045. Bodnarenko, S. R., G. Yeung, L. Thomas, and M. McCarthy, 1999. The development of retinal ganglion cell dendritic stratification in ferrets, Neuroreport, 10:2955–2959. Book, K. J., and D. K. Morest, 1990. Migration of neuroblasts by perikaryal translocation: role of cellular elongation and axonal outgrowth in the acoustic nuclei of the chick embryo medulla, J. Comp. Neurol., 297:55–76. Bowe-Anders, C., R. F. Miller, and R. Dacheux, 1975. Developmental characteristics of receptive organization in the isolated retina-eyecup of the rabbit, Brain Res., 87:61–65. Carter-Dawson, L. D., and M. M. LaVail, 1979. Rods and cones in the mouse retina. II. Autoradiographic analysis of cell generation using tritiated thymidine, J. Comp. Neurol., 188:263–272. Cepko, C. L., C. P. Austin, X. Yang, M. Alexiades, and D. Ezzeddine, 1996. Cell fate determination in the vertebrate retina, Proc. Natl. Acad. Sci. USA, 93:589–595. Cherubini, E., J. L. Gaiarsa, and Y. Ben-Ari, 1991. GABA: an excitatory transmitter in early postnatal life, Trends Neurosci., 14:515–519. Crooks, J., M. Okada, and A. E. Hendrickson, 1995. Quantitative analysis of synaptogenesis in the inner plexiform layer of macaque monkey fovea, J. Comp. Neurol., 360:349–362. Dacheux, R. F., and R. F. Miller, 1981a. An intracellular electrophysiological study of the ontogeny of functional synapses in the rabbit retina. I. Receptors, horizontal, and bipolar cells, J. Comp. Neurol., 198:307–326. Dacheux, R. F., and R. F. Miller, 1981b. An intracellular electrophysiological study of the ontogeny of functional synapses in the rabbit retina. II. Amacrine cells, J. Comp. Neurol., 198:327– 334. Daw, N. W., and H. J. Wyatt, 1974. Raising rabbits in a moving visual environment: an attempt to modify directional sensitivity in the retina, J. Physiol., 240:309–330. Deplano, S., G. M. Ratto, and S. Bisti, 1994. Interplay between the dendritic trees of alpha and beta ganglion cells during the development of the cat retina, J. Comp. Neurol., 342:152– 160. Dorsky, R. I., D. H. Rapaport, and W. A. Harris, 1995. Notch inhibits cell differentiation in the Xenopus retina, Neuron, 14:487–496. Dorsky, R. I., W. S. Chang, D. H. Rapaport, W. A. Harris, 1997. Regulation of neuronal diversity in the Xenopus retina by Delta signalling, Nature, 385:67–70. Dunlop, S. A., 1990. Early development of retinal ganglion cell dendrites in the marsupial Setonix brachyurus, quokka, J. Comp. Neurol., 293:425–447. Fekete, D. M., J. Perez-Miguelsanz, E. F. Ryder, and C. L. Cepko, 1994. Clonal analysis in the chicken retina reveals tangential dispersion of clonally related cells, Dev. Biol., 166:666–682. Feller, M. B., D. P. Wellis, D. Stellwagen, F. S. Werblin, and C. J. Shatz, 1996. Requirement for cholinergic synaptic transmission in the propagation of spontaneous retinal waves, Science, 272:1182–1187. Galli, L., and L. Maffei, 1988. Spontaneous impulse activity of rat retinal ganglion cells in prenatal life, Science, 242:90–91. Galli-Resta, L., G. Resta, S. S. Tan, and B. E. Reese, 1997. Mosaics of islet-1-expressing amacrine cells assembled by short-range cellular interactions, J. Neurosci., 17:7831–7838. Gordon-Weeks, P. R., R. O. Lockerbie, and B. R. Pearce, 1984. Uptake and release of [3H]GABA by growth cones isolated from neonatal rat brain, Neurosci. Lett., 52:205–210.

Grunder, T., K. Kohler, and E. Guenther, 2000a. Distribution and developmental regulation of AMPA receptor subunit proteins in rat retina, Invest. Ophthalmol. Vis. Sci., 41:3600–3606. Grunder, T., K. Kohler, A. Kaletta, and E. Guenther, 2000b. The distribution and developmental regulation of NMDA receptor subunit proteins in the outer and inner retina of the rat, J. Neurobiol., 44:333–342. Gunhan-Agar, E., D. Kahn, and L. M. Chalupa, 2000. Segregation of on and off bipolar cell axonal arbors in the absence of retinal ganglion cells, J. Neurosci., 20:306–314. Harris, W. A., 1997. Cellular diversification in the vertebrate retina, Curr. Opin. Genet. Dev., 7:651–658. Hendrickson, A. E., 1996. Synaptic development in macaque monkey retina and its implications for other developmental sequences, Perspect. Dev. Neurobiol., 3:195–201. Hinds, J. W., and P. L. Hinds, 1974. Early ganglion cell differentiation in the mouse retina: an electron microscopic analysis utilizing serial sections, Dev. Biol., 37:381–416. Hinds, J. W., and P. L. Hinds, 1983. Development of retinal amacrine cells in the mouse embryo: evidence for two modes of formation, J. Comp. Neurol., 213:1–23. Holt, C. E., T. W. Bertsch, H. M. Ellis, and W. A. Harris, 1988. Cellular determination in the Xenopus retina is independent of lineage and birth date, Neuron, 1:15–26. Huang, B. O., and D. A. Redburn, 1996. GABA-induced increases in [Ca2+]i in retinal neurons of postnatal rabbits, Vis. Neurosci., 13:441–447. Johnson, P. T., R. R. Williams, K. Cusato, and B. E. Reese, 1999. Rods and cones project to the inner plexiform layer during development, J. Comp. Neurol., 414:1–12. Kay, J. N., K. C. Finger-Baier, T. Roeser, W. Staub, and H. Baier, 2001. Retinal ganglion cell genesis requires lakritz, a zebrafish atonal homolog, Neuron, 30:725–736. Kirby, M. A., and T. C. Steineke, 1996. Morphogenesis of retinal ganglion cells: a model of dendritic, mosaic, and foveal development, Perspect. Dev. Neurobiol., 3:177–194. Lee, C. W., S. J. Eglen, and R. O. L. Wong, 2002. Segregation of On and Off retinogeniculate connectivity directed by patterned spontaneous activity, J. Neurophysiol., 88:2311–2321. Lemmon, V., and G. Rieser, 1983. The development distribution of vimentin in the chick retina, Brain Res., 313:191–197. Lia, B., R. W. Williams, and L. M. Chalupa, 1987. Formation of retinal ganglion cell topography during prenatal development, Science, 236:848–851. Liets, L. C., and L. M. Chalupa, 2001. Glutamate-mediated responses in developing retinal ganglion cells, Prog. Brain. Res., 134:1–16. Link, B. A., J. M. Fadool, J. Malicki, and J. E. Dowling, 2000. The zebrafish young mutation acts non-cell-autonomously to uncouple differentiation from specification for all retinal cells, Development, 127:2177–2188. Livesey, F. J., and C. L. Cepko, 2001. Vertebrate neural cell-fate determination: lessons from the retina, Nat. Rev. Neurosci., 2:109– 118. Lohmann, C., and R. O. L. Wong, 2001. Cell-type specific dendritic contacts between retinal ganglion cells during development, J. Neurobiol., 48:150–162. Lom, B., and S. Cohen-Cory, 1999. Brain-derived neurotrophic factor differentially regulates retinal ganglion cell dendritic and axonal arborization in vivo, J. Neurosci., 19:9928–9938. Maffei, L., and L. Galli-Resta, 1990. Correlation in the discharges of neighboring rat retinal ganglion cells during prenatal life, Proc. Natl. Acad. Sci. USA, 87:2861–2864.

Marquardt, T., and P. Gruss, 2002. Generating neuronal diversity in the retina: one for nearly all, Trends Neurosci., 25:32– 38. Masland, R. H., 1977. Maturation of function in the developing rabbit retina, J. Comp. Neurol., 175:275–286. Maslim, J., and J. Stone, 1986. Synaptogenesis in the retina of the cat, Brain Res., 373:35–48. Maslim, J., and J. Stone, 1988. Time course of stratification of the dendritic fields of ganglion cells in the retina of the cat, Brain Res. Dev. Brain. Res., 44:87–93. Maslim, J., M. Webster, and J. Stone, 1986. Stages in the structural differentiation of retinal ganglion cells, J. Comp. Neurol., 254:382– 402. Mastronarde, D. N., M. A. Thibeault, and M. W. Dubin, 1984. Non-uniform postnatal growth of the cat retina, J. Comp. Neurol., 228:598–608. McArdle, C. B., J. E. Dowling, and R. H. Masland, 1977. Development of outer segments and synapses in the rabbit retina, J. Comp. Neurol., 175:253–274. McLoon, S. C., and R. B. Barnes, 1989. Early differentiation of retinal ganglion cells: an axonal protein expressed by premigratory and migrating retinal ganglion cells, J. Neurosci., 9:1424–1432. Meister, M., R. O. L. Wong, D. A. Baylor, and C. J. Shatz, 1991. Synchronous bursts of action potentials in ganglion cells of the developing mammalian retina, Science, 252:939–943. Meller, K., and W. Tetzlaff, 1976. Scanning electron microscopic studies on the development of the chick retina, Cell. Tissue. Res., 170:145–159. Milam, A. H., D. M. Dacey, and A. M. Dizhoor, 1993. Recoverin immunoreactivity in mammalian cone bipolar cells, Vis. Neurosci., 10:1–12. Miller, E. D., M. N. Tran, G. K. Wong, D. M. Oakley, and R. O. L. Wong, 1999. Morphological differentiation of bipolar cells in the ferret retina, Vis. Neurosci., 16:1133–1144. Montague, P. R., and M. J. Friedlander, 1989. Expression of an intrinsic growth strategy by mammalian retinal neurons, Proc. Natl. Acad. Sci. USA, 86:7223–7227. Mooney, R., A. A. Penn, R. Gallego, and C. J. Shatz, 1996. Thalamic relay of spontaneous retinal activity prior to vision, Neuron, 17:863–874. Morest, D. K., 1970. The pattern of neurogenesis in the retina of the rat, Z. Anat. Entwicklungsgesch., 131:45–67. Myhr, K. L., P. D. Lukasiewicz, and R. O. L. Wong, 2001. Mechanisms underlying developmental changes in the firing patterns of ON and OFF retinal ganglion cells during refinement of their central projections, J. Neurosci., 21:8664– 8671. Nishimura, Y., and P. Rakic, 1985. Development of the rhesus monkey retina. I. Emergence of the inner plexiform layer and its synapses, J. Comp. Neurol., 241:420–434. Nishimura, Y., and P. Rakic, 1987. Development of the rhesus monkey retina: II. A three-dimensional analysis of the sequences of synaptic combinations in the inner plexiform layer, J. Comp. Neurol., 262:290–313. Pei, Y. F., and J. A. Rhodin, 1970. The prenatal development of the mouse eye, Anat. Rec., 168:105–125. Peichl, L., 1991. Alpha ganglion cells in mammalian retinae: common properties, species differences, and some comments on other ganglion cells, Vis. Neurosci., 7:155–169. Penn, A. A., P. A. Riquelme, M. B. Feller, and C. J. Schatz, 1998. Competition in retinogeniculate patterning driven by spontaneous activity, Science, 279:2108–2112.

  :  

91

Perry, V. H., and R. Linden, 1982. Evidence for dendritic competition in the developing retina, Nature, 297:683–685. Pow, D. V., D. K. Crook, and R. O. L. Wong, 1994. Early appearance and transient expression of putative amino acid neurotransmitters and related molecules in the developing rabbit retina: an immunocytochemical study, Vis. Neurosci., 11:1115– 1134. Prada, C., L. Puelles, and J. M. Genis-Galvez, 1981. A Golgi study on the early sequence of differentiation of ganglion cells in the chick embryo retina, Anat. Embryol., 161:305–317. Rakic, P., 1971. Guidance of neurons migrating to the fetal monkey neocortex, Brain Res., 33:471–476. Rakic, P., 1990. Principles of neural cell migration, Experientia, 46:882–891. Reese, B. E., A. R. Harvey, and S. S. Tan, 1995. Radial and tangential dispersion patterns in the mouse retina are cell-class specific, Proc. Natl. Acad. Sci. USA, 92:2494–2498. Reese, B. E., B. D. Necessary, P. P. Tam, B. Faulkner-Jones, and S. S. Tan, 1999. Clonal expansion and cell dispersion in the developing mouse retina, Eur. J. Neurosci., 11:2965–2978. Rice, D. S., S. Nusinowitz, A. M. Azimi, A. Martinez, E. Soriano, and T. Curran, 2001. The reelin pathway modulates the structure and function of retinal synaptic circuitry, Neuron, 31:929–941. Rickman, D. W., 2000. Neurotrophins and development of the rod pathway: an elementary deduction, Microsc. Res. Tech., 50:124–129. Robinson, S. R., 1991. Developmental of the mammalian retina, in Neuroanatomy of the Visual Pathways and Their Development (B. Dreher and S. R. Robinson, eds.), London: U.K. Macmillan. Rodieck, R. W., 1998. The First Steps in Seeing, Sunderland, MA: Sinauer. Sakaki, Y., Y. Fukuda, and M. Yamashita, 1996. Muscarinic and purinergic Ca2+ mobilizations in the neural retina of early embryonic chick, Int. J. Dev. Neurosci., 14:691–699. Sauer, F. C., 1935. Mitosis in the neural tube, J. Comp. Neurol., 62:377–405. Schnitzer, J., and A. C. Rusoff, 1984. Horizontal cells of the mouse retina contain glutamic acid decarboxylase–like immunoreactivity during early developmental stages, J. Neurosci., 4:2948–2955. Sernagor, E., S. J. Eglen, and R. O. L. Wong, 2001. Development of retinal ganglion cell structure and function, Prog. Retin. Eye. Res., 20:139–174. Sernagor, E., and N. M. Grzywacz, 1995. Emergence of complex receptive field properties of ganglion cells in the developing turtle retina, J. Neurophysiol., 73:1355–1364. Sernagor, E., and N. M. Grzywacz, 1996. Influence of spontaneous activity and visual experience on developing retinal receptive fields, Curr. Biol., 6:1503–1508. Sidman, R. L., 1961. Histogenesis of mouse retina studies with thymidine-H3, in Structure of the Eye (G. K. Smelser, ed.), New York: Academic Press, pp. 487–506. Snow, R. L., and J. A. Robson, 1994. Ganglion cell neurogenesis, migration and early differentiation in the chick retina, Neuroscience, 58:399–409. Snow, R. L., and J. A. Robson, 1995. Migration and differentiation of neurons in the retina and optic tectum of the chick, Exp. Neurol., 134:13–24. Stier, H., and B. Schlosshauer, 1998. Different cell surface areas of polarized radial glia having opposite effects on axonal outgrowth, Eur. J. Neurosci., 10:1000–1010.

92

 

Tagawa, Y., H. Sawai, Y. Ueda, M. Tauchi, and S. Nakanishi, 1999. Immunohistological studies of metabotropic glutamate receptor subtype 6-deficient mice show no abnormality of retinal cell organization and ganglion cell maturation, J. Neurosci., 19: 2568–2579. Thanos, S., and J. Mey, 2001. Development of the visual system of the chick. II. Mechanisms of axonal guidance, Brain Res. Brain. Res. Rev., 35:205–245. Tian, N., and D. R. Copenhagen, 2001. Visual deprivation alters development of synaptic function in inner retina after eye opening, Neuron, 32:439–449. Tootle, J. S., 1993. Early postnatal development of visual function in ganglion cells of the cat retina, J. Neurophysiol., 69:1645– 1660. Troilo, D., M. Xiong, J. C. Crowley, and B. L. Finlay, 1996. Factors controlling the dendritic arborization of retinal ganglion cells, Vis. Neurosci., 13:721–733. Turner, D. L., and C. L. Cepko, 1987. A common progenitor for neurons and glia persists in rat retina late in development, Nature, 328:131–136. Turner, D. L., E. Y. Snyder, and C. L. Cepko, 1990. Lineageindependent determination of cell type in the embryonic mouse retina, Neuron, 4:833–845. Vu, T. Q., J. A. Payne, and D. R. Copenhagen, 2000. Localization and developmental expression patterns of the neuronal K-Cl cotransporter (KCC2) in the rat retina, J. Neurosci., 20:1414– 1423. Waid, D. K., and S. C. McLoon, 1998. Ganglion cells influence the fate of dividing retinal cells in culture, Development, 125: 1059–1066. Wang, G. Y., L. C. Liets, and L. M. Chalupa, 2001. Unique functional properties of on and off pathways in the developing mammalian retina, J. Neurosci., 21:4310–4317. Wang, S. W., L. Gan, S. E. Martin, and W. H. Klein, 2000. Abnormal polarization and axon outgrowth in retinal ganglion cells lacking the POU-domain transcription factor Brn-3b, Mol. Cell. Neurosci., 16:141–156. Wässle, H., 1988. Dendritic maturation of retinal ganglion cells, Trends Neurosci., 11:87–89. Weber, A. J., R. E. Kalil, and L. R. Stanford, 1998. Dendritic field development of retinal ganglion cells in the cat following neonatal damage to visual cortex: evidence for cell class specific interactions, J. Comp. Neurol., 390:470–480. Wetts, R., and S. E. Fraser, 1988. Multipotent precursors can give rise to all major cell types of the frog retina, Science, 239:1142–1145. Wikler, K. C., and P. Rakic, 1996. Development of photoreceptor mosaics in the primate retina, Perspect. Dev. Neurobiol., 3:161–175. Williams, R. W., and D. Goldowitz, 1992. Structure of clonal and polyclonal cell arrays in chimeric mouse retina, Proc. Natl. Acad. Sci. USA, 89:1184–1188. Wolburg, H., E. Willbold, and P. G. Layer, 1991. Müller glia endfeet, a basal lamina and the polarity of retinal layers form properly in vitro only in the presence of marginal pigmented epithelium, Cell. Tissue. Res., 264:437–451. Wong, R. O. L., 1995a. Cholinergic regulation of [Ca2+]i during cell division and differentiation in the mammalian retina, J. Neurosci., 15:2696–2706. Wong, R. O. L., 1995b. Effects of glutamate and its analogs on intracellular calcium levels in the developing retina, Vis. Neurosci., 12:907–917. Wong, R. O. L., 1999. Retinal waves and visual system development, Annu. Rev. Neurosci., 22:29–47.

Wong, R. O. L., and D. M. Oakley, 1996. Changing patterns of spontaneous bursting activity of on and off retinal ganglion cells during development, Neuron, 16:1087–1095. Wong, R. O. L., and S. P. Collin, 1989. Dendritic maturation of displaced putative cholinergic amacrine cells in the rabbit retina, J. Comp. Neurol., 287:164–178. Wong, W. T., B. E. Faulkner-Jones, J. R. Sanes, and R. O. L. Wong, 2000a. Rapid dendritic remodeling in the developing retina: dependence on neurotransmission and reciprocal regulation by Rac and Rho, J. Neurosci., 20:5024–5036. Wong, W. T., K. L. Myhr, E. D. Miller, and R. O. L. Wong, 2000b. Developmental changes in the neurotransmitter regulation of correlated spontaneous retinal activity, J. Neurosci., 20:351–360.

Wong, W. T., and R. O. L. Wong, 2000. Rapid dendritic movements during synapse formation and rearrangement, Curr. Opin. Neurobiol., 10:118–124. Wong, W. T., and R. O. L. Wong, 2001. Changing specificity of neurotransmitter regulation of rapid dendritic remodeling during synaptogenesis, Nat. Neurosci., 4:351–352. Yamashita, M., and Y. Fukuda, 1993. Calcium channels and GABA receptors in the early embryonic chick retina, J. Neurobiol., 24:1600–1614. Young, R. W., 1985a. Cell proliferation during postnatal development of the retina in the mouse, Brain Res., 353:229–239. Young, R. W., 1985b. Cell differentiation in the retina of the mouse, Anat. Rec., 212:199–205.

  :  

93

7

The Development of Retinal Decussations CAROL MASON AND LYNDA ERSKINE

Introduction In the vertebrate visual system, retinal ganglion cell (RGC) axons from each eye grow toward the brain and meet at the midline of the ventral diencephalon. Here they establish an X-shaped pathway, the optic chiasm. This major brain decussation, or crossing, carries fibers from the retina to targets in the thalamus (the lateral geniculate nuclei, LGN) and the midbrain (superior colliculus, or optic tectum in lower vertebrates). The projection through the chiasm establishes connections in central targets for an orderly topographic map of the retina. During development of the optic chiasm, RGC axons from each eye diverge from one another to grow to the optic tract on the same and opposite sides of the brain, a projection pattern that subserves binocular vision in higher animals. Here we will review optic chiasm development, including the axon paths and behaviors of RGCs during their growth as they form the chiasm, and the relationships of RGC growth cones to specialized cells positioned in and around the optic chiasm. Where relevant, we will highlight differences across species in terms of degree of binocularity and the developmental principles underlying the plan of retinal projection. We will then discuss the recent progress on uncovering the cellular and molecular mechanisms directing chiasm formation. Finally, we will discuss aberrations in chiasm development, such as in albinos, in which reduced ocular pigment is linked to a decrease in uncrossed projections, and in a rarer condition in which complete failure of optic chiasm development occurs.

Principles of optic chiasm organization T B P The patterns of RGC axon projection at the optic chiasm in different species range from complete crossing and segregation of fibers from each eye to partial decussation and formation of an ipsilateral projection with complex intermingling of the fibers from the two eyes. The presence and relative size of the ipsilateral projection depend on the degree of binocular overlap in the visual field. In lower vertebrates, such as fish, that lack binocular vision, the fibers from each eye are segregated and the mature visual system contains an entirely crossed projection (Fig. 7.1). Amphibian tadpoles also have an entirely crossed projection. However, in some species, for example, Xenopus

94

laevis, an ipsilateral projection develops at metamorphosis when the eyes change position and binocularity develops to subserve their now predatory lifestyle (Grant and Keating, 1986). In chickens, as in fish, there is nearly complete crossing but the fibers intercalate as they cross and, early in development, there is a transient ipsilateral projection (Drenhaus and Rager, 1992; O’Leary et al., 1983; Thanos et al., 1984). In mammals, partial decussation occurs, the extent of which varies widely among species. In humans and primates, all RGC axons originating from the nasal retina cross the midline to project into the contralateral optic tract and all RGC axons from the temporal retina project into the ipsilateral optic tract (Chalupa and Lia, 1991; Polyak, 1957; Stone et al., 1973). By contrast, in mouse, only a small proportion of RGCs (10,000 td) are reached. For foveal stimuli, threshold is near 1 td and the limited sensitivity regulation is achieved by 3 to 8 td. This pathway difference is evident both in psychophysical studies (Swanson et al., 1987) and in physiological studies (Lee et al., 1990). Now, Weber’s law requires n = 1.0 in equation (1). The finding that Weber’s law occurs only in the MC pathway implies that sensitivity regulation is hierarchical. Some regulation is common to both pathways and may be in the cones themselves. Intracellular recordings from primate horizontal cells, one synapse removed from the receptors, show slopes of 0.6 to 0.7 (Smith et al., 2001) and independent sensitivity regulation in each cone type (Lee et al., 1999). The MC and PC pathways are differentiated at the first synapse, the cone-bipolar synapse. The additional regulation of the MC pathway thus occurs in the bipolar or ganglion cell complexes. The linear slopes of 0.7 or 1.0 are sometimes referred to as multiplicative regulation (Hood and Finkelstein, 1986). The sensitivity regulation mechanism scales the response both to the background and to the test stimulus. The neural mechanisms that underlie multiplicative regulation are not delineated. In the photoreceptors, sensitivity regulation probably involves the complex interactions of the outer segment

Sensitivity regulation in S cones. Sensitivity regulation in S cones differs considerably from that in L and M cones. In some ways, S-cone regulation is more similar to that in rods in that the S cones do show saturation (Mollon and Polden, 1977). They do not obey Weber’s law. They may have some multiplicative regulation but with a limiting slope of 0.6 to 0.7. The S-cone system does show subtractive regulation. This has been termed second-site regulation in the literature (Pugh and Mollon, 1979). However, the range of S-cone stimulation relative to an adapting white stimulus is high, 60 to 1. Thus, the subtractive mechanism does not protect the S-cone system from saturation. There is still no final understanding of sensitivity regulation in the S-cone system.

Studies of chromatic discrimination Chromatic discrimination is usually investigated at a constant luminance level. Originally, this tactic was chosen to ensure that the discrimination depended only on the presence of a spectral reflectance difference in the discrimination field. Classically, parametric chromatic discrimination data have been collected under three conditions: wavelength variation, purity variation, and chromaticity variation. A recent advance has involved sampling chromaticity along the theoretically significant axes shown in Figure 58.2, the two postreceptoral spectral opponent axes (Boynton and Kambe,

25

Delta Lambda (nm)

photocurrent. Additionally time-dependent mechanisms, in which temporal resolution is traded for sensitivity, probably play a role both in the photoreceptor and in the MCpathway mechanism. There is another form of regulation in which the mechanism scales only the background (Hood and Finkelstein, 1986). This is called subtractive regulation. Subtractive regulation by itself is usually not considered an effective mechanism of light adaptation since it provides only a single scaling of the background light level. As the sole method of sensitivity regulation, it will delay but not protect the postreceptoral neurons from saturation. However, in combination with other mechanisms (e.g., following partial multiplicative regulation), it can be effective. Subtractive feedback is a possible neural substrate for subtractive sensitivity regulation. There is evidence that the PC pathway shows subtractive feedback (Krauskopf and Gegenfurtner, 1992; Smith et al., 2000). Subtractive feedback as a second form of hierarchical regulation in the PC pathway is attractive. It was stated that the L- and M-photopigment sensitivities are highly correlated. As a result, at a given luminance level the range of differential spectral stimulation is limited. There is only 0.3 log unit differential in M-cone to L-cone stimulation between 480 nm and 700 nm. This small range is easily handled by a subtractive feedback mechanism in place following the spectral opponent receptive field.

20 15 10 5 0 400

450

500

550

600

650

700

Wavelength (nm) F 58.4. Wavelength discrimination. The threshold step Dl is plotted versus. the standard l. The data are for four observers of Pokorny and Smith (1970). The solid curves are predictions based on the model given in the section “A Modern Discrimination Theory Based on Retinal Physiology.”

1980; Krauskopf and Gegenfurtner, 1992; Smith et al., 2000). W D For wavelength discrimination, the observer adjusts the test wavelength to achieve a just noticeable difference from a standard wavelength. In a typical experiment, the observer is instructed to view a stimulus field composed of two half-fields. The standard field is filled with a homogeneous light of narrow spectral band (l), and the comparison field is filled with light that is slightly different from the standard (l + Dl). The fields are equiluminant so that the discrimination is not dependent on luminance information. The experiment is repeated for many standard wavelengths throughout the visible spectrum. Data obtained from several laboratories reveal similar results (e.g., Bedford and Wyszecki, 1958; Pokorny and Smith, 1970; Wright and Pitt, 1934, 1935). Figure 58.4 shows the measured Dl plotted as a function of the standard wavelength for four observers (Pokorny and Smith, 1970) viewing a 16 td field. The field was present for 5 seconds alternating with a 16 td white-appearing field. The typical wavelength discrimination curve has minima near 490 nm and 580 nm. Individual variance of the discrimination function appears greater below 460 nm (Wright and Pitt, 1934). The solid lines represent a model of wavelength discrimination detailed in the section “A Modern Discrimination Theory Based on Retinal Physiology.” C P D Colorimetric purity refers to the continuum of lights that arise from the mixture of a spectral wavelength with an achromatic-appearing light of the same luminance. There are two important measure-

  :  

913

15

15 14

14

13

13

SG

SG 11

11

10

10 9

9

CL -LOG (1- Pc)

-LOG Pc

CL 8 7 6

8 7 6

FS

FS

5

5

4

4 3

3

TY

TY 2

2

1

1 0

0

110 TD

110 TD -1 350

450

550

650

WAVELENGTH

750

B

-1 350

450

550

650

750

WAVELENGTH

F 58.5. Colorimetric purity discrimination. A, The first step from white is plotted versus wavelength. B, The step from the spectrum is plotted versus wavelength. The data are for five observers of Yeh et al. (1993). The curves are predictions based on the model given in the section “A Modern Discrimination Theory Based on Retinal Physiology.”

ments. For least colorimetric purity, the observer compares a white-appearing light of fixed luminance with a mixture of the same white and a spectral wavelength to judge when the mixture is just tinged with color (Kraft and Werner, 1999; Yeh et al., 1993; see Pokorny and Smith, 1986, for a review of the earlier literature). Least colorimetric purity is defined as Pc = L l (Lw + L l )

(2)

where Ll is the luminance of a spectral light and Lw is the luminance of the white-appearing light. For example, if 1 td of Ll plus 99 td of Lw is discriminated from 100 td of Lw, the least colorimetric purity is 1%. Data are represented as sensitivity (-log(Pc)), and show best discriminative ability at 400 nm and poorest ability at 570 nm. Figure 58.5A shows data for five observers of Yeh et al. (1993) collected at 100 td. For each observer, the range from best to poorest discrimination spans about 1.6 log units. Colorimetric purity discrimination can also be estimated by adding a small amount of white-appearing light to a spectral light in comparison with the spectral light. For example, 20 td of Lw + 80 td of Ll may be discriminated from 100 td of Ll. This is the step from the spectrum, Ps.

914

(3)

The spectral colorimetric purity function, (-log(1 - Pc)), shows little wavelength dependence (Kaiser et al., 1976; Yeh et al., 1993; see Pokorny and Smith, 1986, for a review of the earlier literature). Figure 58.5B shows data for five observers of Yeh et al. (1993) collected at 100 td. The solid lines in Figure 58.5 represent a model described in the section “A Modern Discrimination Theory Based on Retinal Physiology.”

12

12

A

Ps = Lw (Lw + L l ) = (1 - Pc )

JH

JH

  

C D An alternate representation of wavelength and colorimetric purity discrimination is on a chromaticity diagram such as that of Figure 58.2A. Wavelength discrimination will appear plotted as line segments on the horseshoe-shaped spectrum locus. Least colorimetric purity will be line segments radiating from the EES chromaticity toward the spectrum locus. Purity discrimination from the spectrum will appear as line segments pointing from the spectrum locus toward the EES chromaticity. The next logical advance was to assess chromatic discrimination from any point of the chromaticity diagram and in any direction. The production of arbitrary chromaticities and directions was not easy using a traditional colorimeter. Nonetheless, Wright (1941) recorded chromatic discrimination steps for various directions in the CIE space. The results were plotted as a series of short lines in the chromaticity space; the orientation of the line segment indicated the direction for discrimination, and the length of the line represented the just noticeable differences in chromaticity. Wright’s data showed that a given distance on the CIE chromaticity diagram represents different chromatic changes in discrimination ability. The discrimination step size is smallest near the short-wavelength region and largest near the 500 to 550 nm region in the CIE diagram. MacAdam (1942) developed another method to represent chromatic discrimination. He measured the standard deviations of repeated color matches at a set of arbitrary chromaticities and derived discrimination ellipses within the chromaticity diagram. Starting from the center of the ellipse, each ellipse represented the discrimination distance in all directions. Figure 58.6 shows some of the ellipses fitted to MacAdam’s data. The ellipses on the figure are 10 times larger than their actual size. MacAdam’s data generally agreed with the Wright (1941) data. A number of studies have investigated the effect of luminance level on chromatic discrimination. The wavelength discrimination function is stable in the range of 100 to 2000 td (Bedford and Wyszecki, 1958). At lower luminance levels, wavelength discrimination deteriorates in the shortwavelength region (McCree, 1960; Stabell and Stabell, 1977; Weale, 1951). With decreasing field luminance, the major axes of chromatic discrimination ellipses rotate toward the

520

0.8

530

540

510

550

560

0.6 570 500 580

y 590

0.4 600 610 490

650

0.2 480

470 460

0 0

450

0.4

0.2

0.6

x F 58.6. The MacAdam (1942) discrimination ellipses plotted in the (x,y) chromaticity diagram of the 1931 CIE observer. The ellipses are represented at 10 times their actual scale.

blue corner of the chromaticity diagram (Brown, 1951; Brown and MacAdam, 1949; Wyszecki and Fielder, 1971). Generally, it can be concluded that the discrimination sensitivity based on S cones declines more than that based on M and L cones at low luminance levels. A clinical method of estimating chromatic discrimination is the Farnsworth-Munsell 100 hue test. The test was designed to measure hue discrimination among people with normal color vision and to measure the areas of color confusion in color-defective observers. The observer is required to arrange color samples by similarity in a sequential color series. The color samples are mounted in caps, which are numbered on the back and can be moved about freely during performance. The samples were chosen to represent perceptually equal steps of hue and to form a natural hue circle. Dean Farnsworth (1943) originally suggested that observers could be specified as showing only superior, average, or inferior chromatic discriminative ability, but more recent analysis techniques allow quantitative evaluation (Kitahara, 1984; Knoblauch, 1987; Smith et al, 1985; Victor, 1988). M A Modern models include aspects of the classical approaches such as independent sensitivity regulation in the receptor types and opponent processes, but additionally account for discrimination under conditions where the discrimination stimuli are similar to the adapting chromaticity and under conditions where the discriminative

stimuli differ substantially from the adapting chromaticity. In the luminance domain, Craik (1938) found that discrimination was best when the discriminated stimuli were similar in luminance to the adaptation field, and that discrimination worsened as the luminance difference between the discriminated stimuli and the adaptation field increased. Krauskopf and Gegenfurtner (1992) extended this idea to the chromatic domain. They measured discrimination for a test field briefly displaced in chromaticity from a steady adapting field. When thresholds were measured in the same direction as the displacement DL for an L displacement or DS for an S displacement, thresholds rose in proportion to the difference between the test and adaptation chromaticities. When the threshold and the displacement were in different directions, DL for an S displacement or DS for an L displacement, thresholds were unchanged. A variety of experiments support the hypothesis that for discrimination threshold, the L- and S-cone axes are independent, or nearly independent, including habituation (Krauskopf et al., 1982) and noise masking (Sankeralli and Mullen, 1997) studies. In a pioneering analysis, Le Grand (1949) analyzed the MacAdam (1942) data in terms of cone excitation. In making this calculation, he assumed that luminance was determined only by the sum of L- and M-cone responses. This simplification allowed him to calculate only the S- and L-cone excitation axes passing through each ellipse center; the M-cone excitation level was directly calculable from L. He then plotted the size of the arc through the ellipse as a function of the excitation at the center. The data for S cones had the shape of an increment threshold function. However, the data for L and M cones did not. Instead, there was a trade-off of L- and M-cone excitation. The data could be summarized by plotting the discrimination step as a function of the L/M ratio; discrimination was optimal when the cone excitations were balanced. Thus, for discriminations mediated by L and M cones, there is an intrinsic normalization near the EES chromaticity even when chromatic stimuli are presented continuously in an otherwise dark field, while for discriminations mediated by S cones, the data have the appearance of an increment threshold function. Boynton and Kambe (1980) confirmed and extended Le Grand’s conclusions. They defined their stimuli in terms of S and L td. This allowed them to replot their chromatic discrimination data in a TVI format (log threshold versus log illuminance), where the abscissa is the amount of S-cone or L-cone stimulation and the ordinate is the discrimination threshold (DS and DL), at equiluminance. Discrimination thresholds were gathered with a dark surround. At 120 td, discrimination ability dependent on S cones changed slowly with S-cone stimulation, then accelerated at high S-cone stimulation levels. On the other hand, discrimination ability along the L/(L + M) axis (Figs. 58.2A and 58.2B) showed a V shape with a minimum near white.

  :  

915

Test Paradigm Adaptation

Surround: 14.8• x11.2• 0.5

Test

0.07∞ 0.07

Log Delta L

1∞ X 1∞

Pulse

0

-0.5

-1 1.8

Pedestal

With a white-appearing surround, discriminations mediated by S cones show a minimum at the background (Miyahara et al., 1993). Smith et al. (2000) measured chromatic discrimination on the L/(L + M) axis for equiluminant stimuli in chromatic surrounds. They used two different stimulus presentation paradigms. The first was with pulsed stimuli like those of Krauskopf and Gegenfurtner (1992), in which there was a large temporal contrast step of the entire stimulus array. The second paradigm employed steadily presented discrimination stimuli, in which the stimulus array was continuously presented and only one of the test squares changed. Figure 58.7 shows the display sequences for the two paradigms. The paradigms differed only in the display preceding the trial; the trials presented identical stimuli. Chromatic discrimination was measured for steadily presented stimuli in order to evaluate conditions more nearly approximating those encountered in everyday experience. Figure 58.8 shows data collected under adaptation to three chromaticities along a constant S-cone line intersecting EES. The open circles represent thresholds for pulsed stimuli; the closed squares represent steadily presented test stimuli. The solid lines represent predictions from the model detailed in the section “A Modern Discrimination Theory Based on Retinal Physiology.” Arrows on the abscissa indicate the adapting chromaticities. For all conditions, discrimination is best when the test and adapting chromaticities are the same. There is little difference between thresholds for the pulsed and steadily presented test stimuli. Additional measurements showed that the discrimination steps were unchanged with variation of the background size. The pattern of results indicates that discrimination is determined at the border between test and surround. Chromatic alternation generates large signals in

916

  

1.9

1.95

2

1.85

1.9

1.95

2

1.9

1.95

2

Log Delta L

0.5

0

-0.5

-1 1.8 0.5

Log Delta L

F 58.7. Display sequence for the Pulse (upper) and Pedestal (lower) Paradigms. The left figures show the display appearance during adaptation and intertrial intervals; the right figures show the appearance during a trial. (From Smith et al., 2000.) (See color plate 34).

1.85

0

-0.5

-1 1.8

1.85

Log Test L

F 58.8. Discrimination data for a large rectangular surround comparing the Pulse and Pedestal Paradigms. The surround lchromaticities are shown by arrows on the abscissa of each plot: 0.62 (upper panel), 0.665 (middle panel), and 0.74 (lower panel). Open circles show data for the Pulse Paradigm; closed squares show data for the Pedestal Paradigm. Icons on the left show the display appearance (not to scale) during adaptation and the intertrial interval for the Pedestal Paradigm. (From Smith et al., 2000.) (See color plate 35).

PC ganglion cells that do not adapt (Lee et al., 1990), and this situation may mimic that of blinking or of small eye movements sweeping a target back and forth across a small receptive field. In this case the adaptation state is maintained by the surround, and there is a continuous chromatic contrast across the border. H D  L  S A C? Boynton et al. (1986) developed a strategy to evaluate how response differences from the two chromatic mechanisms add to determine threshold. They plotted discrimination data in a normalized cone excitation space by setting the discrimination steps on the L-M- and S-cone axes equal. If signals from the two pathways were independent and combined by probability summation, data would have plotted as a circle. Diverse data sets plot as ellipses, with the major axis oriented at 135 degrees

in most cases (Nagy et al., 1987; Yebra et al., 1994). Though the ellipses typically did not deviate greatly from a circle, they did indicate an interaction. The data sets were gathered with diverse methodologies, so the finding is unlikely to be due to a procedural bias. When S- and L-cone excitation increase or decrease in tandem, there is some facilitation. When Sand M-cone excitation increase or decrease in tandem, there is independence or perhaps inhibition. On the other hand, studies designed to evaluate the interaction directly (Mullen et al., 1997) find only probability summation. Therefore, the presence of an interaction is controversial.

Models of chromatic discrimination There have been two major traditions in theoretical accounts of color vision and color discrimination. One approach arose within the classical views of trichromacy. Color discrimination was thought to arise within the three independent cone pathways. This approach is called the line element theory. The second approach arose with Hurvich and Jameson’s (1955) account of the Hering opponent process concept. Their model was proposed as an account of discrimination and color appearance. This opponent colors theory turned out to have the characteristics of so-called stage or zone theories that attempted to combine receptoral trichromacy with postreceptoral opponency. There were numerous stage or zone theories (reviewed by Judd, 1951a), but most were linear transforms of color mixture data. Vos and Walraven (1972a,b) proposed a well-developed stage theory that combined a trichromatic cone input stage with a subsequent cone ratio model of opponency. Although this model showed great predictive value, it has received little recent attention since it is inconsistent with modern electrophysiology. Both the line element and the opponent colors approaches have severe limitations, but it is nonetheless interesting to review them briefly. L E T The line element was first proposed by Helmholtz (1891, 1892, 1896) and was developed by Schrödinger (1920) and later by Stiles (1946), among others (reviewed by Graham, 1965, and Stiles, 1972). In line element theory, the signals generated by three independent cone types, the S, M, and L cones, are subject to weights reflecting the radiance and quantal catch rate, the adaptational state, threshold noise, and a number of other considerations such as spatiotemporal parameters. Most versions of line element theory assume that the cones are in the Weber region at photopic levels of discrimination measurement. The cone excitations form three independent vectors. A discrimination threshold between two patches of color was envisioned as distance (dG) in this three-dimensional space. Helmholtz proposed the following equation to characterize a threshold difference between any two stimuli:

[

2

2

2 12

dG = (dL s L ) + (dM s M ) + (dS s S )

]

(4)

where dG is the discrimination threshold, dL, dM, and dS are the response differences in the three cone systems to the two colored patches, and sL, sM, and sS are weighting factors associated with sensitivity regulation. Starting at moderate luminance levels, the response of each cone sensitivity function was assumed to obey Weber’s law and the Weber fraction of the three cone mechanisms was assumed to be equal. The Stiles (1946) modification of the Helmholtz line element allowed differences in the limiting Weber fractions of the cone mechanisms in the ratio (r:g:b) = (0.78:1:4.46). The attraction of the line element approach lay in the fact that it treated chromatic discrimination in the same manner as increment thresholds in the luminance domain, thus providing a unified theory of detection and discrimination for the entire luminance range of photopic vision. The line element approach was proved wrong by the above-described Le Grand (1949) analysis, which showed that there was an antagonism between L- and M-cone responses. Subsequently Stiles (1972) noted that the existence of this minimum was inconsistent with line element theory even if a subtractive opponent process was added following a stage of cone-specific Weber sensitivity regulation. A major conclusion of the Stiles analysis was that Weberian sensitivity regulation (n, equation (1) is not solely a property of the cone photoreceptors). O C T Hurvich and Jameson (1955) updated the Hering concept of the opponent process. Their theory was based on some aspects of color appearance of spectral lights viewed in dark surrounds (see Chapter 57). Hurvich and Jameson pointed out that perceptually there are four unique hues—red, green, yellow, and blue—in the spectrum that represent unitary entities. Intervening hues appear to be mixtures of only pairs of these unique hues. Red could be paired with yellow (to give orange percepts) or blue (to give purple percepts) but not with green. In turn, green could be paired with yellow (to give lime percepts) or blue (to give aqua percepts) but not with red. Hurvich and Jameson measured what they called chromatic opponent valences to assess the spectral sensitivity of the two proposed chromatic opponent processes. They used a cancellation paradigm in which the amount of a unique hue needed to cancel its opposite was measured. For example, if the test was 620 nm (appearing orange), its redness content could be canceled by adding unique green and its yellowness content could be canceled by adding unique blue. The data could be plotted as energy-based red-green and blueyellow valence curves with negative values (arbitrarily) assigned to the green and blue valences. It was necessary to use supplementary information to scale the red-green

  :  

917

and blue-yellow valence curves relative to each other. This was done by noting where the percept “orange” occurred in the spectrum. A valence chromaticity space at equiluminance with red-green and blue-yellow as the major axes could also be plotted. Although the paradigm sounds like an experiment in color appearance, it can also be conceived as memory color matching to an internally generated appearance standard. Hurvich and Jameson (1955) recognized that the data could be described by a linear transformation of color matching previously derived by Judd (1951a). Hurvich and Jameson showed that the data of wavelength and least colorimetric purity discrimination were predicted by the chromaticity space formed by the valence curves. Wavelength discrimination was predicted by finding the wavelength excursion corresponding to constant angles around white. Least colorimetric purity was predicted by calculating the proportional distance of a fixed step from white to the spectrum locus. In other work, Jameson and Hurvich (1964) showed that many aspects of color appearance, such as hue scaling and color contrast, could be explained by their model (see Chapter 5). Although the opponent colors model was successful in describing discrimination, it has problems as a color theory. The model assumes linearity, based in part on the supposition that hue cancellation data can be characterized as a linear transform of color matching data. However, this is not a critical test. Although the starting colors are spectral wavelengths, the end points expressed as chromaticity coordinates form clusters within the chromaticity diagram (Burns et al., 1984). There is no explicit statement of luminance or sensitivity regulation. The valence curves can be weighted to predict effects of luminance, but at the expense of generality of the model (Wyszecki and Stiles, 1982). In addition, the redgreen valence has three lobes with positive red valence at short wavelengths. Although this aspect reflects color appearance, it is not seen in retinal spectral opponency. Aspects of color discrimination that primarily reflect the retinal opponency do not agree with the opponent colors model. A M D T B  R P Here we present a model of spectral processing based on current physiological knowledge of the retinal pathways. The goal of the model is to incorporate both luminance gain and chromatic discrimination. It is based on modern physiology of retinal ganglion cells. The model for L- and M-cone discrimination, sketched in Figure 58.9A, is based on physiological data of the spectral opponent PC pathway of primates (Derrington et al., 1984; Lee et al., 1990, 1994). The model for S-cone discrimination is based on physiological data for the spectral opponency of the KC pathway (Fig. 58.9B). A conceptually similar version of this approach was proposed for by Zaidi et al. (1992).

918

  

A

Spatiotemporal Contrast Gain control L cone L

G(L)

M cone M

B

Static Saturation

Subtractive Process S S OPP -(+)

+(-)

R Contrast

OPPc

G(M)

S cone S

Static Saturation

G(S) +(-)

L cone L

S G(L)

+(-)

S

S OPPc

R Contrast

-(+)

M cone M

OPP

G(M)

F 58.9. Schematic of a retinal model for equiluminant chromatic discrimination. A, the L/M cone system. B, the S-cone system.

Both models postulate an early stage of cone-specific multiplicative adaptation followed by a stage of spectral opponency between L and M cones for the L/M pathway or between S and L + M cones for the S pathway. Since gain control is not complete, the spectral opponent signal varies with wavelength and luminance. This signal is further controlled by subtractive feedback by the background. The signal then depends on the contrast between background and test chromaticity. All ganglion cells respond to their preferred contrast with a negatively accelerating function of contrast. This response follows a hyperbolic saturation function. Thus, four components determine contrast detection and discrimination: the absolute threshold, the gain function, the subtraction at the surround chromaticity, and the saturation function. A given retinal ganglion cell responds best to a chromaticity change in its preferred direction (Kremers et al., 1993; Lee et al., 1994). Change in the nonpreferred direction will drive the cell below its resting level. The resting level is usually only about 15% of the maximal response rate. This is an intrinsic nonlinearity that renders the cell response asymmetric: the cell behaves as if partially rectified. For equiluminant chromatic pulses, (+L - M) and (-M + L) give redundant information, responding positively to “redward” changes from their adaptation point; similarly, (+M - L) and (-L + M) give redundant information, responding positively to “greenward” changes from their adaptation point (Lee et al., 1994). To achieve a response for the entire chromatic contrast range we require pairs of cells of opposite chromatic signatures, such as (+L - M) and (+M - L). A (+L -

M) cell predicts chromatic discrimination for pulses with redward direction (L > LA) from the adaptation chromaticity, and a (+M - L) cell predicts chromatic discrimination for pulses with greenward direction (L < LA) from the adaptation chromaticity. A similar argument may be made for the S-cone discriminations. All ganglion cells respond to their preferred contrast with a negatively accelerating function of contrast (Kaplan and Shapley, 1986). The predictions were optimized for a condition in which the observer views a steady background field metameric to the EES (Smith et al., 2000). The discrimination target replaces the background for a trial. Both background and discrimination stimuli are maintained at equiluminance. The trial creates a chromatic spatiotemporal contrast event in the test array from the fixed adaptation level. An optimal test stimulus for chromaticity detection is of low spatiotemporal frequency, for example, a 2-degree diameter, 1 second duration pulse with blurred spatiotemporal edges. Use of a complex spatial pattern, a pseudoisochromatic plate design, with continuous view similarly raises thresholds by a constant (Watanabe et al., 1998). The equations used to predict chromatic contrast detection and discrimination are detailed in the appendix. A simplified version of equation (A13) is log (DLC ) = log (L0 ) - log (G ) + log (O )

A

B

(5)

where L 0 depends on the criterion response, G refers to the multiplicative gain, and O refers to the opponent saturation function. The absolute threshold LTh is given by log (L Th ) = log (L0 ) + log (S )

(6)

where S is a constant, characteristic of the parameters of the hyperbolic saturation function. When chromatic contrast detection is measured at the adapting chromaticity, the opponent signal is very small, and the chromatic contrast threshold approaches the shape of a TVI function rising from threshold: log (L ) ~ log (L0 ) - log (G ) + log (S )

(7)

where G refers to the multiplicative gain. This function is shown as a dashed line in Figure 58.10. Finally, chromatic contrast discrimination at various chromaticities (equation 5) describes a V-shaped function riding on the TVI function. The output of this model is shown in Figure 58.10A for fixed illuminations between 0.1 and 10,000 td. The chromatic contrast discriminations are shown by V shapes. A parallel equation can be derived for the S-cone pathway. A simplified version of equation (A17) is log (DSC ) = log (S 0 ) - log (G ) + log (O )

(8)

The output of this model is plotted in Figure 58.10B with the assumption of adaptation to the EES. Again, the dashed

F 58.10. Discriminations predicted by the retinal model plotted as a function of retinal illuminance in trolands. A, The L/M cone system. B, The S-cone system. (From Smith and Pokorny, 1996.)

line shows the function predicted for a chromatic threshold measured on a “white” background. Both models predict a V-shaped discrimination function at a constant retinal illuminance level. For the PC pathway, the V shapes are narrow, reflecting the small range of differential chromatic signal available at each illuminance. In comparison, the S-cone system shows broad functions since the range of S-cone stimulation relative to the “white” background is large. The V shapes are constant on the linear rising portion of the increment threshold function but gradually become shallow as illuminance drops. This model can describe chromatic discriminations of many kinds. The model’s predictions for wavelength and colorimetric purity discrimination are shown in Figures 58.4 and 58.5. In these figures, the L/M and S-cone predictions are plotted independently. Model predictions for chromatic contrast discrimination are shown in Figure 58.7. The models allow prediction of the effect of using chromatic

  :  

919

backgrounds. The position of the V shape depends on the chromaticity of the background. Although the number of free parameters seems high at five, at a fixed retinal illuminance the threshold and gain terms are constant determining the vertical scaling constant. Thus, the V shape can be fit with only two parameters, a vertical scaling constant and the factor determining the slope of the V shape.

where OPPT is the spectral opponent term at the test chromaticity, OPPA represents the spectral opponent term at the adapting chromaticity, and k4 represents the subtractive feedback strength. Psychophysical studies of luminance thresholds have suggested that subtractive feedback can be as high as 0.9 (Hood and Finkelstein, 1986). The response of a spectral opponent cell to a chromaticity change C from a fixed adapting chromaticity A follows the form of a hyperbolic saturation function (Kaplan and Shapley, 1986; Lee et al., 1990). This is given by ROPP = Rmax [OPPC (OPPC + SAT )]

APPENDIX Here we describe the equations used to derive the threshold predictions given in the main text as equations (5) and (6).

PC pathway The L- and M-cone responses to a light of a given chromaticity and luminance are given by R1 = 1I l max = L l max

(A1)

Rm = mI mmax = M mmax

(A2)

where l and m are the L and M cone chromaticities as in Figure 58.2B, I is the illumination level, and lmax and mmax are the maximal values of the L and M spectral sensitivities, scaled so that they sum to the total retinal illuminance. The L- and M-cone trolands sum to the total luminance level, which is usually constant in any measurement of chromatic discrimination. The cone sensitivities are subject to the multiplicative sensitivity regulation at the adapting chromaticities LA and MA provided by the background. Gain is described by a illuminance-dependent equation that is unity at absolute threshold and is a fraction at high illuminances. The terms G(LA/lmax) and G(MA/mmax) are multiplicative gain terms at the adapting chromaticity, given by an equation of the form G ( L ) = 1 (1 + k1L A l max )

k2

G ( M ) = 1 (1 + k1 M A mmax )

(A3) k2

(A4)

where LA is in cone trolands and k1, k2 are constants. Independent estimation of k1 and k2 requires evaluation of chromaticity discrimination as a function of radiance. The value of k1 is about 1/3 tds and the value of k2 is about 0.65. The value k1 is such that in a simple gain system, a threshold at illuminance 1/k1 is twice the absolute threshold. At high illuminances, the product of the response and the gain term approaches a limiting value. For LA >> 1: R.G ( L A ) = ( L A l max ) (1 + k1L A l max )

k2

(A5)

The cone spectral opponent term can be derived for each of the four subtypes of PC-pathway cells, (+L - M-), (+M L), (-L+M), and (-M+L). For a (+L - M) cell, the spectral opponent term at the test chromaticity would be given by OPP(+L - M) = [ LT l maxG ( L A l max ) - k3 MT mmaxG ( M A mmax )]

(A6)

(A8)

where OPPC is a spectral opponent term and SAT is the static saturation. If the subtractive term does not remove the entire effect of the adapting chromaticity, there will be some residual signal at the adapting chromaticity. Provided that the criterion for a threshold d is small relative to Rmax, the chromatic discrimination threshold for an optimal spatiotemporal stimulus at the adapting chromaticity can be written based on the derivative to equation (5): log(DL A ) = log(d Rmax ) - log[1 (G ( L ) l max ) + 1 (G ( M ) mmax )] + log[(OPPA + SAT ) SAT ] 2

(A9)

This equation includes three terms: a threshold term, a gain term, and a saturation term. At absolute threshold, the gain is unity and the saturation term is zero. The first term, log(d/Rmax), represents the absolute threshold. Above threshold at the adapting chromaticity, the value of OPPA is very small and the third term approaches log[SAT]. The equation then describes a regular TVI function as retinal illuminance is raised. The threshold at other test chromaticities will depend on the size of the contrast step, DOPP, between the adapting chromaticity and the new test chromaticity. This will introduce an additional term, DOPP, modifying equation (A9). log(DLC ) = log(d Rmax ) - log[1 (G ( L A ) l max ) + 1 (G ( M A ) mmax )] + log[(DOPP + OPPA + SAT ) SAT ] 2

(A10)

A simplified version of this equation appears as equation (5) in the main body of the chapter.

S-cone pathway A similar approach can be used to describe chromatic discrimination based on physiological data of the spectral opponent KC pathway of primates (Derrington et al., 1984; Lee et al., 1990, 1994). A major simplification occurs because at equiluminance, the (L+M) surround is constant. Discrimination is determined only by the S-cone stimulation. The initial equations are unchanged. Rs = sI smax = S smax

(A11)

where s is the S-cone chromaticities, as in Figure 58.2B, I is the illumination level, and smax is the maximal values of the S spectral sensitivities, scaled S tds (Boynton and Kambe, 1980). The cone term is subject to multiplicative sensitivity regulation:

where the constant k3 represents the surround strength of the spectral opponency. In retinal ganglion cell data, the surround strength for PC-pathway cells varies from 0.7 to 1.0 (Smith et al., 1992). Next, we assume that the spectral opponent term is subject to subtractive feedback determined by the opponent signal at the adapting chromaticity.

where SA is the adapting signal in cone trolands and k1 and k2 are constants. The spectral opponent term can be derived for each of the two subtypes of S-cone-pathway cell, (+S (L+M) and (-S+(L-)+M). For a (+S-(L+M)) cell, the opponent term is given by

OPPC = OPPT - k 4OPPA

OPP(+S -) = SC SmaxG (S A Smax ) - k3

920

  

(A7)

G (S ) = 1 (1 + k1S A Smax )

k2

(A12)

(A13)

The constant k3 represents the surround strength of the spectral opponency. For the S-cone pathway, k3 is set to give a null response for the cell at 500 nm (Pugh and Larimer, 1980). The opponent term is again subject to subtractive feedback: OPPC = OPP - k 4OPPA

(A14)

where OPPA represents the spectral opponent term at the adapting chromaticity and k4 represents the subtractive feedback strength. The response of a given cell to a chromaticity, C, at a fixed adapting chromaticity is R = Rmax [OPPC (OPPC + SAT)]

(A15)

where OPPC is the opponent signal at the adapting chromaticity. Provided that the criterion for a threshold, d, is small relative to Rmax, the chromatic contrast detection threshold at the adapting chromaticity can be written based on the derivative to equation (10): log(DS A ) = log(Sth ) - log[(G (S ) Smax )] + log[(OPPA + SAT ) SAT] 2

(A16)

where the first term, Sth, represents d/Rmax, the second term represents the luminance gain response, and the third term represents the opponent term. The thresholds for other starting chromaticities DOPP then include the extra term as described for the L/M pathway. log(DSC ) = log(Sth ) - log[(G (S ) Smax )] + log[(OPPA + DOPP + SAT) SAT] 2

(A17)

A simplified version of this equation appears as equation (8) in the main body of the chapter.

Acknowledgments Preparation of this chapter was supported by U.S. Public Health National Eye Institute Research Grant EY00901. We thank Patrick Monnier for assistance with the figures. REFERENCES Aguilar, M., and W. S. Stiles, 1954. Saturation of the rod mechanism of the retina at high levels of illumination, Optica Acta 1:59–65. Barlow, H. B., 1958. Intrinsic noise of cones, in: Visual Problems of Colour, Her Majesty’s Stationery Office, London, pp. 617–630. Bedford, R. E., and G. W. Wyszecki, 1958. Wavelength discrimination for point sources, J. Opt. Soc. Am., 48:129–135. Boynton, R. M., and N. Kambe, 1980. Chromatic difference steps of moderate size measured along theoretically critical axes, Color Res. Appl., 5:13–23. Boynton, R. M., A. L. Nagy, and R. T. Eskew, Jr., 1986. Similarity of normalized discrimination ellipses in the constant-luminance chromaticity plane, Perception, 15:755–763. Brainard, D. H., 1995. Colorimetry, in The Handbook of Optics, 2nd ed., vol. I (M. Bass, E. W. V. Stryland, D. R. Williams, and W. L. Wolfe, eds.), New York: McGraw-Hill, pp. 26.1–26.54. Brown, W. R. J., 1951. The influence of luminance level on visual sensitivity to color differences, J. Opt. Soc. Am., 41:684–688. Brown, W. R. J., and D. L. MacAdam, 1949. Visual sensitivities to combined chromaticity and luminance differences, J. Opt. Soc. Am., 39:808–834. Burns, S. A., A. E. Elsner, J. Pokorny, and V. C. Smith, 1984. The Abney effect: chromaticity coordinates of unique and other constant hues, Vis. Res., 24:479–489.

Craik, K. J., 1938. The effect of adaptation on differential brightness discrimination, J. Physiol. (Lond.), 92:406–421. Dacey, D. M., 2000. Parallel pathways for spectral coding in primate retina, Annu. Rev. Neurosci., 23:743–775. Dacey, D. M., and B. B. Lee, 1994. The “blue-on” opponent pathway in primate retina originates from a distinct bistratified ganglion cell type, Nature, 367:731–735. Derrington, A. M., J. Krauskopf, and P. Lennie, 1984. Chromatic mechanisms in lateral geniculate nucleus of macaque, J. Physiol. (Lond.), 357:241–265. Derrington, A. M., and P. Lennie, 1982. The influence of temporal frequency and adaptation level on receptive field organization of retinal ganglion cells in cat, J. Physiol. (Lond.), 333:343–366. De Valois, R. L., I. Abramov, and G. H. Jacobs, 1966. Analysis of response patterns of LGN cells, J. Opt. Soc. Am., 56: 966–977. Farnsworth, D., 1943. The Farnsworth-Munsell 100 hue and dichotomous tests for color vision, J. Opt. Soc. Am., 33:568–578. Graham, C. H., 1965. Color: data and theories, in Vision and Visual Perception (C. H. Graham, ed.), New York: Wiley. Helmholtz, H. von, 1891. Versuch einer erwerterteen Anwendung des Fechnerschen Gesetzes im Farbensystem, Z. Psychol. Physiol. Sinnesorgane, 2:1–30. Helmholtz, H. von, 1892. Versuch das psychophysische Gesetz auf die Farbenunterschiede trichromatischer. Augen anzuwenden, Z. Psychol. Physiol. Sinnesorgane, 3:1–20. Helmholtz, H. von, 1896. Handbuch der Physiologischen Optik, 2nd ed., Hamburg: Voss. Hendry, S. H., and R. C. Reid, 2000. The koniocellular pathway in primate vision, Annu. Rev. Neurosci., 23:127–153. Hood, D. C., and M. A. Finkelstein, 1986. Sensitivity to light, in Handbook of Perception and Human Performance, vol. I, Sensory Processes and Perception (K. R. Boff, L. Kaufman, and J. P. Thomas, eds.), New York: Wiley, pp. 5-1–5-66. Hurvich, L. M., and D. Jameson, 1955. Some quantitative aspects of an opponent-colors theory. II. Brightness, saturation and hue in normal and dichromatic vision, J. Opt. Soc. Am., 45:602– 616. Jameson, D., and L. M. Hurvich, 1964. Theory of brightness and color contrast in human vision, Vis. Res., 4:135–154. Judd, D. B., 1951a. Basic correlates of the visual stimulus, in Handbook of Experimental Psychology (S. S. Stevens, ed.), New York: Wiley, pp. 811–867. Judd, D. B., 1951b. Colorimetry and artificial daylight, in Technical Committee No. 7 Report of Secretariat, United States Commission, International Commission on Illumination, Twelfth Session, Stockholm, pp. 1–60. Kaiser, P. K., J. P. Comerford, and D. M. Bodinger, 1976. Saturation of spectral lights, J. Opt. Soc. Am., 66:818–826. Kaiser, P. K., B. B. Lee, P. R. Martin, and A. Valberg, 1990. The physiological basis of the minimally distinct border demonstrated in the ganglion cells of the macaque retina, J. Physiol. (Lond.), 422:153–183. Kaplan, E., and R. M. Shapley, 1986. The primate retina contains two types of ganglion cells, with high and low contrast sensitivity, Proc. Natl. Acad. Sci. USA, 83:2755–2757. Kitahara, K., 1984. An analysis of the Farnsworth-Munsell 100-hue test, Doc. Ophthalmol. Proc. Ser., 39:233–246. Knoblauch, K., 1987. On quantifying the bipolarity and axis of the Farnsworth-Munsell 100-Hue test, Invest. Ophthalmol. Vis. Sci., 28:707–710. König, A., and E. Brodhun, 1889. Experimentalle Unterschungen uber die psychophysische Fundamentalformel in Bezug auf den

  :  

921

Gesichtssinn, Sitzungsberichte Preuss. Akad. Wissenschaften, Berl., 27:641–644. Kraft, J. M., and J. S. Werner, 1999. Aging and the saturation of colors. 1. Colorimetric purity discrimination, J. Opt. Soc. Am. A, 16:223–230. Krauskopf, J., and K. Gegenfurtner, 1992. Color discrimination and adaptation, Vis. Res., 32:2165–2175. Krauskopf, J., D. R. Williams, and D. W. Heeley, 1982. Cardinal directions of color space, Vis. Res., 22:1123–1131. Kremers, J., B. B. Lee, J. Pokorny, and V. C. Smith, 1993. Responses of macaque ganglion cells and human observers to compound periodic waveforms, Vis. Res., 33:1997–2011. Kuffler, S. W., 1952. Neurons in the retina: organization, inhibition and excitation problems, Cold Spring Harbor Symp. Quant. Biol., 17:281–292. Le Grand, Y., 1949. Les seuils différentiels de couleurs dans la théorie de Young Rev. ‘Opt., 28:261–278. (Color difference thresholds in Young’s theory, Color Res. Appl., 19:296–309, 1994; K. Knoblauch, trans.). Lee, B. B., 1996. Receptive field structure in the primate retina, Vis. Res., 36:631–644. Lee, B. B., D. M. Dacey, V. C. Smith, and J. Pokorny, 1999. Horizontal cells reveal cone type–specific adaptation in primate retina, Proc. Natl. Acad. Sci. USA, 96:14611–14616. Lee, B. B., P. R. Martin, and A. Valberg, 1988. The physiological basis of heterochromatic flicker photometry demonstrated in the ganglion cells of the macaque retina, J. Physiol. (Lond.), 404:323–347. Lee, B. B., P. R. Martin, and A. Valberg, 1989. Sensitivity of macaque retinal ganglion cells to chromatic and luminance flicker, J. Physiol. (Lond.), 414:223–243. Lee, B. B., J. Pokorny, V. C. Smith, and J. Kremers, 1994. Responses to pulses and sinusoids in macaque ganglion cells, Vis. Res., 34:3081–3096. Lee, B. B., J. Pokorny, V. C. Smith, P. R. Martin, and A. Valberg, 1990. Luminance and chromatic modulation sensitivity of macaque ganglion cells and human observers, J. Opt. Soc. Am. A, 7:2223–2236. Lennie, P., J. Pokorny, and V. C. Smith, 1993. Luminance, J. Opt. Soc. Am. A, 10:1283–1293. MacAdam, D. L., 1942. Visual sensitivities to color differences in daylight, J. Opt. Soc. Am., 32:247–274. MacLeod, D. I. A., and R. M. Boynton, 1979. Chromaticity diagram showing cone excitation by stimuli of equal luminance, J. Opt. Soc. Am., 69:1183–1185. Martin, P. R., A. J. White, A. K. Goodchild, H. D. Wilder, and A. E. Sefton, 1997. Evidence that blue-on cells are part of the third geniculocortical pathway in primates, Eur. J. Neurosci., 9:1536–1541. McCree, K. J., 1960. Colour confusion produced by voluntary fixation, Opt. Acta, 7:281–291. Miyahara, E., V. C. Smith, and J. Pokorny, 1993. How surrounds affect chromaticity discrimination, J. Opt. Soc. Am. A, 10:545–553. Mollon, J. D., and P. G. Polden, 1977. An anomaly in the response of the eye to light of short wavelengths, Philos. Trans. R. Soc. Lond. Ser. B: Biol. Sci., 278:207–240. Mullen, K. T., S. J. Cropper, and M. A. Losada, 1997. Absence of linear subthreshold summation between red-green and luminance mechanisms over a wide range of spatio-temporal conditions, Vis. Res., 37:1157–1165. Nagy, A. L., R. T. Eskew, and R. M. Boynton, 1987. Analysis of color-matching ellipses in a cone-excitation space, J. Opt. Soc. Am. A, 4:756–768.

922

  

Pokorny, J., and V. C. Smith, 1970. Wavelength discrimination in the presence of added chromatic fields, J. Opt. Soc. Am., 69:562–569. Pokorny, J., and V. C. Smith, 1986. Colorimetry and color discrimination, in Handbook of Perception and Human Performance, vol. I, Sensory Processes and Perception (K. R. Boff, L. Kaufman, and J. P. Thomas, eds.), New York: Wiley, pp. 8-1–8-51. Pugh, E. N. J., and J. Larimer, 1980. Test of the identity of the site of blue/yellow hue cancellation and the site of chromatic antagonism in the pi 1 pathway, Vis. Res., 20:779–788. Pugh, E. N. J., and J. D. Mollon, 1979. A theory of the p-1 and p-3 color mechanisms of Stiles, Vis. Res., 19:293–312. Rushton, W. A. H., 1972. Visual pigments in man, in Handbook of Sensory Physiology, vol. VII/I (H. J. A. Dartnall, ed.), Berlin: Springer, pp. 364–394. Sankeralli, M. J., and K. T. Mullen, 1997. Postreceptoral chromatic detection mechanisms revealed by noise masking in threedimensional cone contrast space, J. Opt. Soc. Am. A, 14:2633– 2646. Schrödinger, E., 1920. Grundlinien einer Theorie der Farbenmetrik im Tagessehen, Ann. Physik (Leipzig), 63:134–182. English translation in Sources of Color Science (D. L. MacAdam, ed.), Cambridge, MA: MIT Press, 1970. Shapiro, A. G., J. Pokorny, and V. C. Smith, 1996. An investigation of scotopic threshold-versus-illuminance curves for the analysis of color-matching data, Color Res. Appli., 21:80–86. Sharpe, L. T., C. Fach, K. Nordby, and A. Stockman, 1989. The incremental threshold of the rod visual system and Weber’s law, Science, 244:354–356. Smith, V. C., B. B. Lee, J. Pokorny, P. R. Martin, and A. Valberg, 1992. Responses of macaque ganglion cells to the relative phase of heterochromatically modulated lights, J. Physiol. (Lond.), 458:191–221. Smith, V. C., and J. Pokorny, 1975. Spectral sensitivity of the foveal cone photopigments between 400 and 500 nm, Vis. Res., 15: 161–171. Smith, V. C., and J. Pokorny, 1996. The design and use of a cone chromaticity space, Color Res. Appl., 21:375–383. Smith, V. C., and J. Pokorny, 2003. Color matching and color discrimination, in The Science of Color (S. Shevell, ed.), Optical Society of America, pp. 103–148. Smith, V. C., J. Pokorny, B. B. Lee, and D. M. Dacey, 2001. Primate horizontal cell dynamics: an analysis of sensitivity regulation in the outer retina, J. Neurophysiol., 85:545–558. Smith, V. C., J. Pokorny, and A. S. Pass, 1985. Color-axis determination on the Farnsworth-Munsell 100-hue test, Am. J. Ophthalmol., 100:176–182. Smith, V. C., J. Pokorny, and H. Sun, 2000. Chromatic contrast discrimination: data and prediction for stimuli varying in L and M cone excitation, Color Res. Appl., 25:105–115. Stabell, U., and B. Stabell, 1977. Wavelength discrimination of peripheral cones and its change with rod intrusion, Vis. Res., 17:423–426. Stiles, W. S., 1946. A modified Helmholtz line-element in brightness-colour space, Proc. Phys. Soc. (Lond.), 58:41–65. Stiles, W. S., 1972. The line element in colour theory: a historical review, in Color Metrics ( J. J. Vos, L. F. C. Friele, and P. L. Walraven, eds.), Soesterberg: AIC/Holland, pp. 1–25. Swanson, W. H., T. Ueno, V. C. Smith, and J. Pokorny, 1987. Temporal modulation sensitivity and pulse detection thresholds for chromatic and luminance perturbations, J. Opt. Soc. Am. A, 4:1992–2005.

Valberg, A., B. B. Lee, and D. A. Tigwell, 1986. Neurons with strong inhibitory S-cone inputs in the macaque lateral geniculate nucleus, Vis. Res., 26:1061–1064. Victor, J. D., 1988. Evaluation of poor performance and asymmetry in the Farnsworth-Munsell 100-hue test, Invest. Ophthalmol. Vis. Sci., 29:476–481. Vos, J. J., and P. L. Walraven, 1972a. An analytical description of the line element in the zone-fluctuation model of colour vision— I. Basic concepts, Vis. Res., 12:1327–1365. Vos, J. J., and P. L. Walraven, 1972b. An analytical description of the line element in the zone-fluctuation model of color visionII. The derivation of the line element, Vis. Res., 12:1345–1365. Watanabe, A., J. Pokorny, and V. C. Smith, 1998. Red-green chromatic discrimination with variegated and homogeneous stimuli, Vis. Res., 38(21):3271–3274. Weale, R. A., 1951. Hue-discrimination in para-central parts of the human retina measured at different luminance levels, J. Physiol. (Lond.), 113:115–122. Wiesel, T., and D. H. Hubel, 1966. Spatial and chromatic interactions in the lateral geniculate body of the rhesus monkey, J. Neurophysiol., 29:1115–1156.

Wright, W. D., 1941. The sensitivity of the eye to small colour differences, Proc. Phys. Soc. (Lond.), 53:93–112. Wright, W. D., and F. H. G. Pitt, 1934. Hue-discrimination in normal colour-vision, Proc. Phys. Soc. (Lond.), 46:459–473. Wright, W. D., and F. H. G. Pitt, 1935. The colour-vision characteristics of two trichromats, Proc. R. Soc. (Lond.), 47:205–217. Wyszecki, G., and G. H. Fielder, 1971. New color-matching ellipses, J. Opt. Soc. Am., 61:1135–1152. Wyszecki, G., and W. S. Stiles, 1982. Color Science—Concepts and Methods, Quantitative Data and Formulae, 2nd ed., New York: Wiley. Yebra, A., J. A. Garcia, and J. Romero, 1994. Color discrimination data for 2-degrees and 8-degrees and normalized ellipses, J. Opt. Nouv. Rev. Opt., 25:231–242. Yeh, T., V. C. Smith, and J. Pokorny, 1993. Colorimetric purity discrimination: data and theory, Vis. Res., 33:1847–1857. Zaidi, Q., 1997. Decorrelation of L- and M-cone signals, J. Opt. Soc. Am. A, 14:3430–3431. Zaidi, Q., A. Shapiro, and D. Hood, 1992. The effect of adaptation on the differential sensitivity of the S-cone color system, Vis. Res., 32:1297–1318.

  :  

923

59

The Role of Color in Spatial Vision KAREN K. DE VALOIS

C   common in nature, as is color vision in the animal kingdom. The ability to discriminate color differences without regard to luminance variations has apparently evolved independently multiple times. This, as well as the massive neural investment that primates make in color vision, suggests that it must provide significant advantages to the species that possess it. Color vision has traditionally been studied in splendid isolation, with only minimal attempts to examine the significance of spatial and temporal factors in color processing or the role of color in spatial or temporal vision. There have been compelling practical reasons for separating color and spatial vision experimentally, because before the advent of video displays it was difficult to produce complex spatiotemporal stimuli that varied in color without associated variations in luminance. Since the middle of the twentieth century, however, there has been an explosion of research on the characteristics of spatial vision based on color alone and on the role that color may play in spatial vision. There are two general approaches to the study of color spatial vision. One is to determine the characteristics of spatial vision that is based solely upon color. This requires eliminating luminance variations from the stimuli to be used, then studying sensitivity to and analysis of spatial patterns when only color differences are present. This is technically quite demanding, in part because of the chromatic aberrations produced by the optical system of the eye. These introduce unwanted and sometimes unrecognized luminance artifacts into otherwise isoluminant stimuli. When it is successfully accomplished, however, using isoluminant stimuli has the advantage of isolating the color system so that its characteristics can be determined without the intrusion of the visual mechanisms that analyze luminance variations. This approach is based on the traditional assumption that the visual system can be neatly separated into a color vision subsystem and a quite separate luminance vision mechanism. If that assumption is unwarranted, and it may be, then interpretation of the results of such studies is not so straightforward. Nonetheless, a great deal of research has investigated the characteristics of spatial vision when only color variations are present. The general conclusion is that color differences alone can subserve reasonable spatial vision, though with somewhat lower resolution than is found with luminance variations. This work will be discussed further below.

924

The second major approach to studying the role of color in spatial vision is to consider the joint variation of color and luminance. The luminance mechanism as traditionally conceived is assumed to be insensitive to variations in chromaticity (see Lennie et al., 1993, for discussion). The interactions between the color vision system and the (presumed) single luminance mechanism have been widely studied. Alternative conceptions suggest that there may be additional mechanisms that are responsive to luminancevarying patterns but that are also selective for hue. Some research has been devoted to the study of the possible existence and characteristics of hue-selective mechanisms that respond to variations in effective intensity. This work will be discussed further below. The ability to analyze the spatial variation of color across a scene confers certain marked advantages. Although color and luminance variations are often correlated in nature, they do not covary perfectly. Color provides a separate and often more reliable indication of the presence of reflectance differences in a visual scene, and thus may aid substantially in segregating the perceptual world into discrete objects. Color vision is also useful in foraging for food, for conspecific communication, and for identifying possible mates. A particular advantage for humans is that color can be readily categorized and thus efficiently coded in memory for later retrieval. This chapter will consider only psychophysical approaches to the study of color spatial vision. For more information concerning the in this volume relevant physiological substrates, see Chapter 65.

Some important constraints on color spatial vision There are many constraints on color vision. Two of these are especially significant in considering the role that color vision might play in spatial processing and will be briefly described here. First, color vision can operate only at relatively high light levels. There are three classes of cone photoreceptors, each with its own photopigment. The S cones are maximally sensitive in the short visible wavelengths; the M cones in the medium wavelengths; and the L cones in the longer wavelengths. The cones are responsible for transduction at moderate to high light levels and thus for color vision. The great majority of all cones are either L or M cones. Both absorb light over the entire

visible spectrum, and their spectral absorption functions differ only slightly. A different class of photoreceptors, rods, is responsible for vision at low light levels. They are not active at the high light levels at which color vision is prominent, and the cones are not responsive at very low light levels. To estimate the amount of light present at a given position (more properly, the light coming from a given direction), the visual system need only count the number of photons it absorbs in the corresponding retinal region. (Here we ignore the added complexities resulting from differential sensitivity to different wavelengths and from nonlinear intensityresponse functions.) This estimate can be based on the output of a single receptor type (the rods at low, or scotopic, light levels, for example) or on a combination of multiple receptor types in a specific region. To judge the chromaticity at a single point, however, the system must compare the number of photons absorbed in each of at least two photoreceptors that contain photopigments that differ in spectral sensitivity. To estimate luminance, thus, the critical information lies in the sum of the responses of the various photoreceptors present, but to judge color, it is the difference between receptors’ outputs that is critical. Insofar as the receptor spectral sensitivities overlap, the signal from the sum will be larger than that from the difference. The L and M cones, which constitute some 90% or more of the cone population, have closely spaced spectral sensitivity peaks, only about 30 nm apart, and thus give a much larger sum than the difference signal. In order to have a difference signal large enough to work with, the light level must be high and there must be at least two photoreceptor types with different spectral sensitivities. Thus, color vision cannot operate at all at scotopic levels and only poorly at low photopic light levels. Consequently, a spatial vision system that depended solely on color discriminations would be a major handicap for any species that is active in low light levels. Spatial discriminations based on color differences have other constraints as well. A local luminance judgment can be based on the output of a single L or M cone. Detecting a variation in luminance across space can be accomplished by comparing the outputs of two such receptors at different retinal positions. Since the two receptors can be adjacent (representing adjacent visual directions), spatial resolution can in principle be as fine as the receptor spacing. To judge the color at any single point, however, requires a comparison between the outputs of two or more different photoreceptor types near one another in retinal location. To detect a variation in color across space, it is necessary to make at least two local color judgments, each of which requires a comparison of the outputs of at least two different cone types. This perforce limits spatial acuity for patterns that vary only in color.

Spatial vision at isoluminance S C S One of the most basic measures of spatial vision is the spatial contrast sensitivity function (CSF). To characterize the spatial CSF, the minimum amount of contrast required to detect a sinusoidal grating is measured at each of several spatial frequencies (where spatial frequency is defined in terms of cycles per degree visual angle, and contrast sensitivity is the reciprocal of the minimum detectable contrast). The spatial CSF for luminance, measured at photopic light levels in the central retina, is a bandpass function of spatial frequency (see De Valois and De Valois, 1988, for references). Sensitivity is highest for spatial frequencies in the range of about 2 to 5 c/deg and falls to zero by about 50 to 60 c/deg. There is a pronounced decline in sensitivity at the lower spatial frequencies as well. When the grating to be detected contains only color variations, however, with luminance held constant, the spatial CSF is quite different. Contrast sensitivity is a low-pass function of spatial frequency, with no decline in sensitivity at lower spatial frequencies (Granger and Heurtley, 1973; Kelly, 1983; McKeefry et al., 2001; Mullen, 1985; van der Horst and Bouman, 1969; van der Horst et al., 1967). The fall-off in sensitivity at high frequencies is rapid. Contrast sensitivity falls to zero at lower spatial frequencies than is found with luminance-varying gratings. Both the highfrequency cutoff and the overall sensitivity level depend on the chromatic axis studied. When the pattern varies along a color axis in which there are equal and opposite variations in absorption by L and M cones, the high-frequency cutoff is lower than that seen for luminance-varying gratings, but it is reliably higher than that found with S-cone-varying gratings (McKeefry et al., 2001). When the grating is defined solely by variations in absorption in the S cones (i.e., it lies along a tritanopic confusion axis), the high-frequency cutoff occurs below 10 c/deg (e.g., McKeefry et al., 2001). This can be understood by considering the small number of S cones (only about 7% of the total number of cones in the retina; Curcio et al., 1991) and their distribution. S cones are quite rare in the foveola, reach their peak density at an eccentricity of about 1 degree, and fall off very gradually as eccentricity increases. At no eccentricity is their density high enough to support fine spatial resolution. Figure 59.1 illustrates the differences among the spatial CSFs for luminance and for two chromatic axes, one defined by the difference in absorption along an L-M cone axis (labeled 0 deg) when S-cone absorption is held constant and the other by differences along an S-cone axis (labeled 90 deg) when LM-cone absorption is held constant. In this illustration, the logarithm of contrast sensitivity is normalized such that all three functions are equated at their points of

 :    

925

LUM 0 deg 90 deg

0.6 0.4 0.2

1

10 Spatial Frequency (c/deg)

F 59.1. Normalized spatial contrast sensitivity functions. Filled diamonds show data from tests with luminance-varying gratings. Open squares represent an isoluminant axis differencing the output of L and M cones (labeled 0 deg). Open triangles show data from tests in which gratings varied only in S-cone absorption (labeled 90 deg). Note that the luminance function shows attenuation at the lowest spatial frequency, and that the S-cone function loses sensitivity at high spatial frequencies more rapidly than either of the other two functions.

maximum sensitivity. This allows a more direct comparison of their shapes. Note three things. First, the luminance CSF shows attenuation at the lowest spatial frequency. Had the function been extended to still lower frequencies, sensitivity would continue to decrease. Second, the two chromatic CSFs are low-pass. Both continue to increase in constrast sensitivity as the spatial frequency falls. Third, the fall-off in sensitivity at high spatial frequencies is more rapid for the S-cone axis than for the LM-cone axis. The fact that we cannot detect isoluminant color-varying patterns of high spatial frequencies implies that the color system cannot contribute greatly to our perception of fine spatial detail. Reading fine print, for example, can only be done when there is sufficient luminance contrast between the letters and their background. This is less of a hindrance than might be supposed, since much of the most critical information in vision is carried in the medium and low spatial frequency components of scenes (e.g., Marron and Bailey, 1982). Thus, the color vision system that underlies vision at isoluminance could participate significantly in the detection and analysis of spatial pattern variations, despite its inability to encode high spatial frequency information. The lack of a reduction in contrast sensitivity in the low spatial frequencies for isoluminant color-varying patterns can be understood by reference to the receptive field structure of color-opponent neurons in the retinogeniculate pathway (De Valois and De Valois, 1975; also see Chapter 65). The consequence is that as spatial frequency decreases beyond about 2 c/deg, the signal carried by the color system becomes increasingly strong while that from the luminance system becomes weaker. This can be best understood by reference to a diagram of the cone input map for a coloropponent cell.

926

  

Opponent Cell Cone Input Map 12 +

Relative Response

0.8

The upper panel of Figure 59.2 illustrates the cone inputs to a typical color-opponent cell in the retina or the lateral geniculate nucleus (LGN) of the thalamus. In this neuron, there is an excitatory input from L cones in the receptive field (RF) center and an inhibitory input from M cones in the RF surround. The lower panel shows the separate RFs for color variations and for luminance variations. When the color changes from, say, white to an isoluminant red, the L cones absorb relatively more (producing excitation in the center mechanism), while the M cones absorb relatively less. Since the amount of inhibition produced by the M cones is a function of the number of photons they absorb, as the light

8

Excitatory L-cone input

4 0 Inhibitory M-cone input

-40

20

40

60

Spatial Position

80

100

Luminance and Chromatic Receptive Fields

Relative Response

Normalized Log Contrast Sensitivity

Spatial Contrast Sensitivity 1

16 + 12 Color RF

8 4 0 Luminance RF

-4 0

20

40 Position 60 Spatial

80

10

F 59.2. Color and luminance receptive fields. The upper panel shows the cone-input map from a +L center, -M surround opponent cell in the LGN. The L-cone input to the center mechanism is more powerful but restricted to a smaller region. The Mcone input to the concentric surround is weaker but covers a larger region. The center input here is excitatory to light increments, while the surround input is inhibitory to light increments. The lower panel illustrates the receptive fields for luminance variations (thin line) and for isoluminant color variations (thick line). Note that the color RF is excitatory across its entire extent. A color change produces the same kind of response at all locations. The luminance RF is excitatory for an increment in the center but inhibitory for an increment in the surround.

S F  O C At early levels in the visual system, the luminance system performs parallel local spatial analyses using mechanisms (channels, for short) that are selective for spatial frequency (see De Valois and De Valois, 1988, for discussion). The luminance channels are bandpass for spatial frequency and orientation, as can be demonstrated with pattern adaptation (Blakemore and Campbell, 1969) or spatial masking (Legge and Foley, 1980). Adaptation to a luminance-varying grating of a single spatial frequency, for example, produces a temporary reduction in contrast sensitivity for similar gratings whose spatial frequencies lie within about ± 0.75 octave of the adaptation grating. A high-contrast mask of one spatial frequency will reduce the detectability of a simultaneously presented grating of a spatial frequency that is within about ± 1.25 octaves of the mask frequency. When stimuli vary only in color, not in luminance, similar adaptation and masking phenomena are seen. Adaptation to an isoluminant red-green sinusoidal grating of one spatial frequency produces a tem-

Log Contrast Sensitivity

Red-Green Contrast Sensitivity 2 1.75 Unadapted

1.5

Adapted

1.25 1 0 0.5

Difference in Log Contrast Sensitivity

shifts from white to an isoluminant red, the amount of inhibition produced by the M-cone surround decreases. The net response of the neuron is the sum of its excitatory center response and its inhibitory surround response. Thus, anything that either increases the center response or decreases the surround response will cause a net increase in the neuron’s excitatory output, and increasing the width of a red stimulus bar centered on the RF will increase total excitation all across the RF. When the stimulus is a luminance increment, not a color change, however, the spatial receptive field is quite different. When a narrow bright bar is flashed on the RF center, it produces more excitation from the center mechanism and more inhibition from the surround mechanism. Since the center mechanism is more sensitive in the RF center, however, the net change in the neuron’s response will be positive. Once the size of the bar exceeds that of the center mechanism, however, increasing its width even more will produce still more inhibition from the surround with no compensatory increase in excitatory response. Thus, the net response will decrease. Mapping the entire RF with a small spot would thus produce a center-surround function for a luminance change. This neuron would show its peak response to a luminance grating at some intermediate frequency, but its response to isoluminant chromatic gratings would be low-pass. Although it is not entirely clear how best to compare color contrast and luminance contrast, Chaparro et al. (1993) have shown that on the basis of cone contrast comparisons, we are significantly more sensitive to color variations at low spatial frequencies than to luminance variations. Color vision thus assumes greater relative importance in the analysis of low spatial frequency information.

21 4 64 2 Spatial Frequency

88

0.6 0.4 0.2 0 0 0.5

12

4 2

6 4

88

Spatial Frequency

F 59.3. Pattern adaptation for an isoluminant grating. The upper panel shows spatial contrast sensitivity functions for isoluminant red-green gratings before and after adaptation to a 2 c/deg red-green grating. The lower panel plots the difference between the functions above. Note that the effect of adaptation is band-limited and centered on the adaptation frequency.

porary loss in contrast sensitivity to red-green test gratings of the same or similar frequencies but no loss in sensitivity to frequencies further removed (Bradley et al., 1988). The frequency spread of the adaptation effect for color is only slightly greater than that seen following adaptation to luminance-varying gratings. Similarly, adaptation to a vertical red-green isoluminant grating is followed by a temporary reduction in contrast sensitivity for similar vertical and near-vertical gratings but not for horizontal gratings (Bradley et al., 1988). As with spatial frequency, the spread in orientation is somewhat broader for adaptation to isoluminant chromatic gratings than to luminance-varying gratings, but the difference is not great. The upper panel of Figure 59.3 shows spatial contrast sensitivity functions for a red-green grating measured before (filled symbols) and after (open symbols) adaptation to a 2 c/deg red-green grating. In the lower panel, the difference between the two functions is plotted. Note that the loss in sensitivity is band-limited and centered on the adaptation frequency. Pattern masking results show a similar relationship between luminance and color vision. When a sinusoidal test grating is detected in the presence of a masking grating that is identical in every respect save contrast, the detection

 :    

927

contrast threshold for the test can be markedly affected. If the mask is at or below its own detection threshold, the contrast required to detect the presence of the test will be reduced (Legge and Foley, 1980), a result referred to as subthreshold summation. If the mask contrast is well above threshold, however, the contrast required to detect the test will be increased. A similar relationship between mask contrast and test threshold holds when the mask and test are isoluminant chromatic gratings, identical except in contrast (Chen et al., 2000a; Losada and Mullen, 1994; Switkes et al., 1988). As with bandwidths estimated from adaptation studies, the frequency selectivity of spatial masking is similar for both color and luminance. These results, like those from the adaptation studies described earlier, demonstrate that the color vision system alone can support spatial analysis and that it is functionally similar to the luminance vision system. Another indication of how well a particular mechanism can support spatial vision is the accuracy with which it can discriminate between two patterns that differ slightly in spatial frequency or orientation. When sinusoidal gratings vary in luminance, both spatial frequency and orientation discrimination are exquisitely fine (see De Valois and De Valois, 1988, or Webster et al., 1990, for relevant references). When the stimuli vary in chromaticity with no associated luminance variation, observers can still make surprisingly fine distinctions. A practiced subject can discriminate between two high-contrast isoluminant color-varying gratings that differ in spatial frequency by only 4% or in orientation by as little as 1 degree (Webster et al., 1990). When the grating spatial frequency is low and the contrast is high, the thresholds for discrimination of the spatial frequency or orientation of gratings that vary along an S-cone (tritanopic) axis do not differ significantly from those that vary along an L-M axis. Discrimination thresholds for both chromatic cardinal axes are slightly but reliably higher than comparable measures for gratings that vary in luminance. Two primary conclusions can be drawn from these data. First, the mechanisms that process information about color variations in the absence of luminance are similar in organization to the luminance system, at least at intermediate levels in the visual system. They comprise a set of channels that are bandpass for both spatial frequency and orientation. Second, their selectivity along both spatial frequency and orientation dimensions is only slightly coarser than that of the comparable luminance mechanisms. This suggests that the color vision system is capable of significant spatial pattern analysis in the absence of luminance contrast. The requisite early-level processing is clearly available. These data do not, of course, provide any indication of the frequency with which naturally occurring spatial patterns differ in color with no associated luminance variations, or vice versa. It is unlikely that isoluminant patterns occur often in nature or extend greatly in either space or time.

928

  

D One important task for spatial vision is that of determining how far an object of interest is from the observer. Distance (depth) is critical in navigation, in determining whether or how to escape from a predator, capture prey, grasp food, and many other actions. The twodimensional retinal image, however, reflects visual direction but does not directly represent depth. That must be derived. There are both binocular and monocular cues to depth. The images of a scene in the two eyes are similar but not identical, and binocular disparity, the small differences between these two images, is the stimulus for stereopsis. Stereopsis has been widely studied by the use of random-dot stereograms ( Julesz, 1971), patterns of randomly positioned dots in which some subset of target dots is slightly displaced horizontally in the pattern presented to one eye with respect to the same subset in the stimulus to the fellow eye. To an observer with normal stereopsis, the target dots will appear at a different depth (or distance) than those in the background, even though the image contains no monocular cues to depth. The question of whether stereopsis occurs at isoluminance with random-dot stereograms has occasioned some dispute. Some studies (e.g., Livingstone and Hubel, 1987; Lu and Fender, 1972) have failed to find evidence for the perception of depth in random-dot stereograms at isoluminance. Others (e.g., de Weert and Sadza, 1983; Kingdom and Simmons, 1996; Scharff and Geisler, 1992; Simmons and Kingdom, 1995) have found that depth can be perceived in isoluminant random-dot stereograms. Kingdom et al. (1999) suggest that the specific impairment found with random-dot stereograms at isoluminance is more related to the perception of three-dimensional form than of depth per se. Although stereopsis is perhaps the most widely studied cue to depth, it is often less important than the monocular depth cues in a scene. Many kinds of information in a visual image give information about the distance of objects, either absolute or relative. For example, if one object occludes the image of another, the occluding object must be closer to the observer than the partially hidden object. Although there have been reports that depth from monocular cues disappears at isoluminance (Livingstone and Hubel, 1987), other reports suggest that depth can be signaled at isoluminance by such monocular cues as texture gradients (Troscianko et al., 1991) and motion parallax (Cavanagh et al., 1995). The preponderance of the evidence suggests that the color vision system does carry some information about depth. However, the perception of depth in the absence of luminance contrast appears to be significantly compromised. G I Under many circumstances, the visual world as we perceive it differs significantly from the world as we measure it. One class of such anomalous per-

ceptions, known as geometrical illusions, has often been studied for the insights it provides into the visual processing mechanisms. Livingstone and Hubel (1988) examined several stimuli associated with geometrical illusions and concluded that the illusions either disappeared or were greatly reduced in magnitude when the stimuli were defined by color contrast in the absence of luminance contrast. Li and Guo (1995), on the other hand, measured illusion magnitudes for illusions of orientation (Zöllner), length (Müller-Lyer), and size (Delboeuf ) and found no difference between patterns defined by luminance versus color contrast. However, they found that luminance contrast was necessary for the occurrence of various border and contour illusions such as the Kanisza triangle. The illusory figures disappeared completely at isoluminance. It is tempting to suggest that the resistance of the color vision system to such perceptual errors provides the observer with a useful check on the accuracy of perceptions of spatial patterns, but evidence for this argument is lacking. R P One important task in spatial vision is the determination of the relative positions of different objects in the visual field. The precision with which this can be accomplished with luminance-varying targets is astonishing (e.g., Westheimer and McKee, 1977). Positional misalignments smaller than the diameter of a single cone can be reliably discriminated in the vernier acuity task, for example. Whether the color vision system can support fine judgments of relative position is an interesting question. The small, localized stimuli with which fine alignment hyperacuity is typically demonstrated are not appropriate for use at isoluminance, both because of the eye’s chromatic aberration (which introduces unintended artifacts in such patterns) and because of the relative insensitivity of the chromatic system to high spatial frequencies. It is possible, however, to measure positional alignment using other kinds of targets that are more suitable for the color vision mechanisms, such as two-dimensional Gaussian blobs or Gabor patterns (a sine wave windowed by a Gaussian function) that vary only in color. When these stimuli are used and equated for detectability, comparable levels of performance are found whether the stimuli vary in luminance or in color (Kooi et al., 1991; Krauskopf and Farell, 1991). Within the range of targets to which it is sensitive, then, color vision appears to be able to support judgment of spatial alignment about as well as can luminance vision. H-O T One of the most complex but important tasks the visual system faces is that of integrating contours across space. The detection of an isolated object against a homogeneous background can be accomplished by a simple system with minimal processing capability, but linking multiple spatially separated elements correctly and

determining the contour along which they lie are very difficult indeed. If the color vision system is to play a significant role in spatial vision, it should be capable of contour integration. The failure to see illusory figures such as the Kanisza triangle at isoluminance (Li and Guo, 1995), however, suggests that color vision might lack the ability to integrate across separated sections of a contour. Mullen and her colleagues (McIlhagga and Mullen, 1996; Mullen et al., 2000) have examined contour integration using stimuli that varied along either isoluminant red-green or blue-yellow chromatic axes or in luminance. The task required the perceptual linking of a set of oriented Gabor patterns in order to extract the curving path along which they were placed. The target elements, which had orientations that varied by specified amounts, were presented in a background of similar Gabor patterns of randomly chosen orientations. Subjects performed similarly on contour integration over a range of curvatures, whether stimuli were defined by luminance variations or by variations along either of the two chromatic axes used. The effects of contrast and external noise were also comparable whether the stimuli varied in luminance or in color. However, perceptual linking was compromised by changing either the color (or luminance) axis or the phase of the pattern from element to element. The authors suggested that the three mechanisms use a common contour integration process but that this process is not blind to either color or phase. Another task closely related to contour integration is pattern segregation—in effect, determining which parts of a complex scene belong together. There are many methods of studying pattern segregation. One is to determine how rapidly a target can be detected in a noisy background; another is to ask whether subjects can judge some aspect (e.g., position or orientation) of a region that is defined by a limited number of dimensions, again in a noisy background. When the target differs significantly from its background in color, detection and segregation can be accomplished quickly and efficiently (e.g., D’Zmura, 1991; Gegenfurtner and Kiper, 1992; Li and Lennie, 1997; Nothdurft, 1993; Webster et al., 1998). Webster et al. (1998) used cluttered backgrounds that varied in color in a manner similar to the variation found in natural scenes. They found that prior adaptation to the color distribution of the background significantly affected the speed with which a color-defined target could be detected, depending upon the similarity of the target color to the background distribution. They suggested that adaptation to natural scenes can be an important factor in determining how efficiently visual targets are detected. Another way to examine pattern segregation is to determine the conditions under which the superimposition of two gratings of different orientations is seen as forming a single, coherent plaid as opposed to two separate surfaces. If the

 :    

929

two gratings are similar in spatial frequency, contrast, and color, they will most often appear to form a single plaid pattern. If, however, they differ in color—one red and one green, for example—they will appear to be two different objects (Rauschecker et al., 1973) and will alternate in perceptual rivalry. If a similar plaid pattern is set in motion, it will likely appear to be a single, coherent surface moving in one direction if the two gratings have the same color. If they differ in color, however, they will appear to be two transparent surfaces moving in different directions (Kooi et al., 1992). These and many other studies have demonstrated that the color vision system has significant spatial abilities. It is organized in a manner similar to that of the luminance system, parsing a complex visual scene by multiple spatial frequency and orientation channels that operate in parallel. It can support many of the higher-level tasks important for a versatile visual system. However, in almost every case, performance is somewhat poorer when the stimuli are isoluminant than when they are defined by luminance contrast. These demonstrations of the capabilities of the color vision system at isoluminance are useful because they reveal the extent to which color variations alone can be used to determine the spatial characteristics of the visual world. They reflect both the strengths and the limitations of color vision, and they reveal much about the manner in which the color system carries out spatial analyses. However, they are not necessarily relevant to vision as it normally operates. In nature, virtually all scenes contain both color contrast and luminance contrast. It is rare indeed to find any extended natural scene in which color varies but luminance is constant. So understanding the role that color actually plays in spatial vision requires a different approach. Consider images drawn from nature. Objects typically differ from their backgrounds in both chromaticity and luminance, and a chromatic border will most often also contain luminance contrast. A joint color-luminance border is likely to indicate a change in surface reflectance and thus an object property of interest. The presence of a luminance border with no associated color change, however, is less often indicative of a reflectance border. Because we live in a threedimensional world that is primarily illuminated by a directional light source (the sun), objects cast shadows. The luminance contrast across a shadow border can be quite high, often as high as the luminance contrast across reflectance borders. The presence of shading can be used as a cue to the three-dimensional structure of a scene (see, e.g., Cavanagh and Leclerc, 1989), but it can also be misleading with respect to object borders. The segregation of a complex scene into discrete objects and surfaces is among the first and most critical tasks the visual system faces at higher processing levels. If it is accomplished on the basis of contours regardless of how they are specified, then the presence of

930

  

shadow borders can lead to errors. Chromaticity is not strictly invariant across a shadow border, due to both Rayleigh scattering and reflection from nearby surfaces, but variations in chromaticity are usually small compared to the variations in luminance across shadow borders, and they are rarely apparent. Salient color borders thus may provide an alternative and more reliable indicator of how a visual scene should be segregated when both color and luminance vary.

Spatial vision with both color and luminance variations It is useful to consider both what is known about the characteristics of spatial vision when both color and luminance variations are present and what advantages might accrue from the ability to discriminate color differences in the presence of luminance differences. Gur and Akri (1992), among others, have suggested that studies of isoluminant stimuli in isolation may fail to reveal the full contribution of color to spatial vision. Even if one assumes that the visual system treats color and luminance as separate dimensions, the question of how they interact, if at all, is important. It is well known, for example, that making brightness judgments between two stimuli that differ in hue is difficult (though subjects can do it reliably). This suggests that color differences might interfere with the coding of luminance information. Several studies have raised the question of how color- and luminance-coding systems might interact. Some of these will be reviewed briefly below. A One way of addressing the question directly is to use the same adaptation and masking techniques that have been used to characterize mechanisms within a single dimension. For example, subjects can adapt to a highcontrast luminance-varying grating and then measure the detectability of a color-varying grating of the same spatial frequency, or vice versa. Bradley et al. (1988) compared the effects of cross-adaptation between color and luminance to adaptation and testing with the same stimuli. They found little transfer in either direction from one dimension to the other, even though the adaptation stimuli produced significant losses in sensitivity to test stimuli of the same contrast type (i.e., color or luminance). This result is compatible with a model of separate and largely independent dimensions for color and luminance, but other results (see below) suggest that the relationship is more complex. S M When a suprathreshold grating masking paradigm is used, both color- and luminancevarying masking gratings reduce the detectability of superimposed test gratings of the other dimension (Chen et al., 2000a; De Valois and Switkes, 1983; Mullen and Losada, 1994; Switkes et al., 1988). In the case of a luminance mask and a color-varying test, however, the mask must be of

L-C S  F One way of examining these two models is to determine whether subthreshold summation occurs. If two stimuli are processed by the same underlying mechanism, then a half-threshold amount of one added to a half-threshold amount of the other should produce a threshold response. Consider, for example, a single receptor containing a single photopigment. The only information the receptor can report is the total number of photons it has absorbed. Suppose that it produces a criterion response when it absorbs 100 photons. If that receptor absorbs 10% of the incident photons at 600 nm and only 5% of the incident photons at 450 nm, then it should be possible to produce the same criterion response by either exposing it to 1000 photons of 600 nm light, or 2000 photons of 450 nm light, or a combination of 500 photons of 600 nm light and 1000 photons of 450 nm light. If the responses to the two wavelengths add linearly, then which combination of 600 nm and 450 nm light is used to produce a 100-photon absorption will be irrelevant. A similar argument can be applied to the summation of subthreshold color-varying and luminance-varying stimuli. If they are processed by the same mechanism, the responses they produce should add and subthreshold summation should appear. The detection of a test stimulus should be facilitated by the presence of a low-contrast mask (or pedestal) irrespective of whether it varies in color or in luminance. Thus, it is instructive to determine what test contrast is required for detection when the mask pattern is below or near its own detection threshold. This question was first addressed by Switkes et al. (1988), using red-green isoluminant gratings and luminance-varying gratings matched in space-averaged chromaticity, luminance, and spatial frequency. When the mask varied in luminance and the test was isoluminant red-green, the contrast masking function was dipper-shaped—that is, it showed both facilitation with lowcontrast pedestals and masking at high-mask contrasts. The contrast masking function, however, was not identical to that

Log (Masked Threshold/Unmasked Threshold)

high contrast, significantly above its own detection threshold, before it begins to produce masking. An isoluminant color-varying mask, on the other hand, reduces the detectability of a superimposed luminance-varying test grating as soon as the mask is visible. Suprathreshold masking is consistent with either of two classes of models. If the mask and test gratings are detected by the same underlying mechanism, then cross-dimensional suprathreshold masking should occur in the same manner as masking when mask and test are drawn from the same contrast dimension (both luminance-varying, for example). However, if stimuli drawn from the two dimensions are processed by separate mechanisms, then cross-dimensional suprathreshold masking might reflect inhibitory interactions between the two mechanisms.

Simultaneous Masking 0.8

0.4 YB mask/RG test RG mask/YB test Same-on-same

0

-0.4

-0.8 0.1

1

10

100

Mask Contrast/Threshold Contrast

F 59.4. Simultaneous spatial masking. Mask and test gratings were either isoluminant red-green or luminance-varying yellow-black. When mask and test gratings were the same (i.e., both red-green or both yellow-black), the usual dipper-shaped function was seen ( filled triangles). When the mask was yellow-black and the test was red-green, facilitation (increase in sensitivity) occurred over a broad range of contrasts (gray diamonds). When the mask was isoluminant red-green and the test was yellow-black, there was no facilitation (open squares). Masking increased with mask contrast once the mask exceeded its own detection threshold.

found when test and mask both varied in color or in luminance. Facilitation in the detection of the red-green isoluminant grating occurred over a broad range of low to medium pedestal contrasts but not when the pedestal was actually subthreshold. Only when the mask contrast was quite high did its presence impede the detection of the colorvarying test grating. When the mask was isoluminant red-green and the test was luminance-varying, however, a very different masking pattern appeared. A low-contrast red-green pedestal never reduced the contrast threshold for detecting a superimposed luminance-varying test grating, but once the color-varying mask exceeded its own threshold, it began to reduce the detectability of the luminance-varying test. This rather surprising result has been confirmed and extended to other chromatic axes (Chen et al., 2000a), though Mullen and Losada (1994), using different stimuli and presentation modes, found facilitation for detecting a luminance test at somewhat higher contrasts of a red-green mask. Figure 59.4 illustrates simultaneous spatial masking functions for color and luminance gratings. The filled triangles show the dipper-shaped function obtained with either redgreen masks and red-green tests or yellow-black masks and yellow-black test patterns. Both conditions produce essentially identical results, with facilitation at very low mask contrasts and reduction in sensitivity at higher mask contrasts. The gray diamonds show results from tests with yellow-black masks and red-green test gratings. There is facilitation over a broad range of mask contrasts, with a reduction in sensitivity to the test appearing only at quite high mask contrasts.

 :    

931

The open squares show the results of masking detection of a yellow-black test grating by a red-green mask. There is no increase in test detection sensitivity at any mask contrast. Once the mask exceeds its own detection threshold, it begins to make the test more difficult to detect, and the masking effect increases with mask contrast. Chen et al. (2000b) have modeled the mechanisms underlying the full contrast range of cross-masking results and conclude that they are well fit by a model incorporating cross-mechanism divisive inhibition and a weak excitatory input from a luminance mechanism into the chromatic channels but no excitation from chromatic mechanisms into the luminance channel. Mullen et al. (1997), on the other hand, looked carefully for subthreshold summation between red-green and luminance mechanisms at detection thresholds, using a summation square analysis. They account for their results by a model positing chromatic and luminance mechanisms that are independent at threshold but that demonstrate probability summation. H-S I-C M Our discussion to this point has been cast in terms of two separate systems, one that encodes luminance variations and one that encodes chromatic variations. Another possibility is that there are mechanisms that respond to intensity variations, but that do so in a hue-selective manner. Hue selectivity implies that these are not traditional luminance mechanisms, hence the term intensity coding. There is a clear substrate of hue-selective neurons that respond vigorously to either color variation or luminance variation, from the midget retinal ganglion cells to the parvocellular layers of the LGN and through various cortical regions. It would be surprising if the information they carry about intensity variations were simply discarded, but these color-selective neurons do not respond to luminance contrast in a hue-insensitive manner. For example, an LGN neuron that receives excitatory input from an L cone to its center mechanism and inhibitory input from M cones to its surround will respond with excitation to a full-field intensity increment if the stimulus is shifted toward longer wavelengths. If the intensity increment is shifted toward shorter wavelengths, however, the neuron will be inhibited, not excited (De Valois and De Valois, 1975). These neurons cannot be described as coding luminance per se, since their spectral sensitivity functions differ substantially from the photopic spectral luminous efficiency function, Vl. They may nonetheless be involved in coding information about patterns that vary in intensity. If hue-selective intensity coding mechanisms are important in spatial vision, their presence might be revealed in detection experiments. In macaque monkeys, the selective loss of the parvocellular LGN layers that contain colorselective neurons produces a reduction in contrast sensitivity for luminance-varying patterns (Merigan, 1989; Schiller

932

  

et al., 1990). If a human observer is asked to detect a small spot defined by an intensity increment upon a background, the hue of the stimulus can sometimes be reported at threshold luminance contrasts (Hood and Finkelstein, 1983). Guth and his colleagues (Guth, 1967; Guth et al., 1969) found significant failures of heterochromatic luminance additivity at threshold when the test was a luminance increment against a dark background. These observations all demonstrate that the detection of luminance-varying stimuli is mediated by hue-selective mechanisms under some conditions. On the other hand, selective masking experiments have argued against the existence of mechanisms selective for, say, bright red or dark green (Stromeyer et al., 1999). A compelling clinical case study (Rovamo et al., 1982) also provides evidence for the existence of hue-selective channels that encode information concerning intensity differences. The individual concerned temporarily lost the ability to see achromatic patterns that varied in luminance, but she retained essentially normal contrast sensitivity for luminance-varying gratings viewed through either rose- or green-colored filters. Visually evoked potential measures confirmed the selective nature of the loss. Several kinds of higher-order psychophysical adaptation studies (Hardy and De Valois, 2002; Mayhew and Anstis, 1972; McCullough, 1965; Virsu and Haapasalo, 1973) demonstrate color-selective encoding of intensity variations. For example, alternating adaptation to, say, a bright red–dark red grating of 4 c/deg and a bright green–dark green grating of 1 c/deg will shift the apparent spatial frequency of subsequently viewed 2 c/deg gratings in opposite directions, depending upon the hue (red or green) of the test pattern (Hardy and De Valois, 2002). This suggests that there may be a partial separation of mechanisms that encode information about intensity-varying patterns that differ in hue. A similar dissociation of adaptation aftereffects has also been used to argue for the separation of color and luminance coding mechanisms (e.g., Favreau and Cavanagh, 1981). Figure 59.5 demonstrates encoding by color-selective luminance mechanisms. The subject’s task was to determine which of two spatially separated Gabor patterns appeared to be higher in spatial frequency. In an unadapted state, subjects can make such judgments both accurately and reliably. In this experiment, however, the subject had first adapted to alternating red and green luminance-varying Gabor patterns. When the screen was red, the pattern on the left was 1 c/deg and the one on the right was 4 c/deg; when the screen was green, the pattern on the left was 4 c/deg and that on the right was 1 c/deg. During the test phase, a 2 c/deg pattern was always presented on the left, while the spatial frequency of the test pattern on the right varied. The data show that the apparent spatial frequency of the test patterns was a function of their color. Red 2 c/deg patterns

Proportion Test Judgement Higher

Adapt 4 c/deg Green and 1 c/deg Red 1 0.75 0.5

green red

0.25 0

1.6

1.8 2 2.2 Test Spatial Frequency

2.4

F 59.5. Color-selective change in apparent spatial frequency following differential adaptation to colored luminancevarying patterns. After adapting to a 1 c/deg bright red–dark red pattern alternating with a 4 c/deg bright green–dark green pattern, subsequently viewed 2 c/deg red test patterns (open triangles) appear shifted to higher spatial frequencies. Subsequently viewed 2 c/deg green test patterns (filled squares) appear shifted to lower spatial frequencies. Adaptation and test patterns were presented in the same retinal regions. See text for more details.

appeared to be shifted to still higher spatial frequencies, but green 2 c/deg patterns appeared to be shifted to lower spatial frequencies. Recall that all patterns, whether adaptation or test, varied only in luminance. They differed from each other (i.e., over time) in color, but at any given time, only one color was present on the screen. These data support the idea of color-selective luminance-encoding mechanisms.

The advantages of color for spatial vision We have considered the extent to which the color vision system can support the tasks of spatial vision, the characteristics and limitations of spatial vision based solely upon color differences, the interactions between the traditionally conceived luminance system and the color vision mechanisms, and the possibility of hue-selective intensity encoding. However, a major puzzle remains. The primate visual system has made an extraordinarily large neural investment in color vision, which is among our finest visual abilities. Yet there is little agreement as to the basic functions that color vision serves. Color certainly adds beauty and interest to the visual world, yet it seems unlikely that we evolved such an elaborate mechanism primarily for aesthetic purposes. What then could be the advantages it confers upon those who possess it? Determining the presence, position, and nature of objects in the visual world—the essence of spatial vision—is surely among the most critical tasks of the visual system, and it seems highly likely that color vision plays a role in this most basic function. There are several ways in which adding color information could make spatial vision more effective.

Color vision can make foraging for food easier (Mollon, 1989). Frugivorous species, for example, must both detect their food and judge its state of ripeness. Since many fruits change color as they ripen, detecting them among foliage of a different color and accurately judging their state of ripeness is easier when color can be used as a cue. Any ability that increases one’s success in foraging for food clearly provides an evolutionary advantage. Another use for color vision is conspecific communication. Many species signal either gender or sexual receptivity by color. Finding a mate would in that case be simpler if the animal possesses color vision. Other species—notably humans—communicate emotion by color changes. Anger and embarrassment, for example, can be detected by the accompanying facial flush. Color vision can be of significant assistance in the critical task of segmenting the visual world because color borders are more reliable indicators of reflectance edges than are luminance borders. Color differences profoundly affect the way in which we segregate moving patterns and thus how we perceive the motion direction (Kooi et al., 1992). Segregating objects on the basis of color differences reduces the ambiguities resulting from shadows in our three-dimensional world. Similarly, an object that differs significantly from its surround in color can be preattentively detected. Color then can be used to draw the attention of an observer. The effectiveness of color in drawing attention can be inferred from the existence of species that use color changes for purposes of camouflage. Gegenfurtner and Rieger (2000) argue that color contributes both to the encoding of pattern information and to its retrieval in pattern recognition. They demonstrated that colored images of natural scenes are more accurately recognized than luminance-matched black-white images of the same scenes. They found that color provided an advantage in both the initial encoding stage and the later recognition of the same scenes. They suggest that the addition of color contributes to an enhancement of the memory representation of a complex scene. This interesting suggestion emphasizes the role of color in higher cognitive functions as well as in early pattern segmentation. Color vision differs from luminance vision in certain respects that may be particularly important with respect to higher cognitive functions. Luminance is a one-dimensional continuum, and its perceptual aspect (brightness or lightness) is not easily categorized. Beyond the classification into bright, dark, and medium brightness, it is difficult to describe or to commit to memory the perceptual qualities associated with variations in luminance. Color is three-dimensional, comprising the perceptual aspects of hue, saturation, and brightness. Although two of these dimensions, saturation and brightness, are similar to luminance in being resistant to categorization in themselves, hue is different. A minimum of

 :    

933

four categories (red, green, blue, and yellow, the unique hues of Ewald Hering) are required to describe hues, and we possess other color names in abundance. Without prompting, normal observers typically use about 11 names to classify the colors of reflective objects (Berlin and Kay, 1969; Boynton and Olson, 1990). Some color names (orange, for example) represent a combination of basic hues; others reflect some combination of the three color dimensions (e.g., pink, a desaturated red). If information about a spatial pattern must be stored in memory for later retrieval, its color can be easily coded by reference to a common color category. This is surely easier than trying to encode its position along a continuum such as brightness. In addition, in a species possessing complex language, the communication involved in identifying information about objects is easier when spatial attributes can be joined with color attributes in the description. It sets the conditions necessary for identifying—in effect, recognizing—a previously unseen object. Color vision thus serves many functions in the visual perception of spatial patterns. It supports the spatial analysis of patterns that contain only color contrast, though these are uncommon in nature. Color vision aids in the segregation of complex visual scenes, in the detection and identification of objects of interest, and in conspecific communication. It simplifies encoding and retrieving from memory information about objects and scenes, and it provides a means of categorizing and thus efficiently describing objects. Although we can analyze patterns without color vision, the addition of color to spatial vision enriches us.

Acknowledgments The preparation of this chapter and much of the research on which it was based were supported by grants from the National Science Foundation, the National Eye Institute, the Nippon Telegraph and Telephone Corporation, and the University of California Committee on Research. I thank Wendy Davis and Michael Disch for assisting in the background research for the preparation of this chapter. REFERENCES Berlin, B., and P. Kay, 1969. Basic Color Terms: Their Universality and Evolution, Berkeley: University of California Press. Blakemore, C., and F. Campbell, 1969. On the existence of neurones in the human visual system selectively sensitive to the orientation and size of retinal images, J. Physiol. (Lond.), 203:237–260. Boynton, R. M., and C. X. Olson, 1990. Salience of chromatic basic color terms confirmed by three measures, Vis. Res., 30:1311–1317. Bradley, A., E. Switkes, and K. K. De Valois, 1988. Orientation and spatial frequency selectivity of adaptation to colour and luminance gratings, Vis. Res., 28:841–856.

934

  

Cavanagh, P., and Y. Leclerc, 1989. Shape from shadows, J. Exp. Psychol: Hum. Percep. Perform., 15:3–27. Cavanagh, P., S. Saida, and J. Rivest, 1995. The contribution of color to depth perceived from motion parallax, Vis. Res., 35:1871–1878. Chaparro, A., C. F. Stromeyer, E. P. Huang, R. E. Kronauer, and R. T. Eskew, 1993. Colour is what the eye sees best, Nature, 361:348–350. Chen, C.-C., J. M. Foley, and D. H. Brainard, 2000a. Detection of chromoluminance patterns on chromoluminance pedestals I: threshold measurements, Vis. Res., 40:773–788. Chen, C.-C., J. M. Foley, and D. H. Brainard, 2000b. Detection of chromoluminance patterns on chromoluminance pedestals II: model, Vis. Res., 40:789–803. Curcio, C. A., K. A. Allen, K. L. Sloan, C. L. Lerea, J. B. Hurley, I. B. Klock, and A. N. Milam, 1991. Distribution and morphology of human cone photoreceptors stained with anti-blue opsin, J. Comp. Neurol., 312:610–624. De Valois, K. K., and E. Switkes, 1983. Simultaneous masking interactions between chromatic and luminance gratings, J. Opt. Soc. Am., 73:11–18. De Valois, R. L., and K. K. De Valois, 1975. Neural coding of color, in Handbook of Perception: Seeing, vol. 5 (E. C. Carterette and M. P. Friedman, eds.), New York: Academic Press, pp. 117– 166. De Valois, R. L., and K. K. De Valois, 1988. Spatial Vision, New York: Oxford University Press. de Weert, C. M. M., and K. J. Sadza, 1983. New data concerning the contribution of colour differences to stereopsis, in Colour Vision ( J. D. Mollon, and L. T. Sharpe, eds.), London: Academic Press, pp. 553–562. D’Zmura, M., 1991. Color in visual search, Vis. Res., 31:951–966. Favreau, O., and P. Cavanagh, 1981. Color and luminance: independent frequency shifts, Science, 212:831–832. Gegenfurtner, K. R., and D. C. Kiper, 1992. Contrast detection in luminance and chromatic noise, J. Opt. Soc. Am. A, 9:1880–1888. Gegenfurtner, K. R., and J. Rieger, 2000. Sensory and cognitive contributions of color to the recognition of natural scenes, Curr. Biol., 10:805–808. Granger, E. M., and J. C. Heurtley, 1973. Visual chromaticitymodulation transfer function, J. Opt. Soc. Am., 63:1173– 1174. Gur, M., and V. Akri, 1992. Isoluminant stimuli may not expose the full contribution of color to visual functioning: spatial contrast sensitivity measurements indicate interaction between color and luminance processing, Vis. Res., 32:1253–1262. Guth, S. L., 1967. Nonadditivity and inhibition among chromatic luminances at threshold, Vis. Res., 7:319–327. Guth, S. L., N. J. Donley, and R. T. Marrocco, 1969. On luminance additivity and related topics, Vis. Res., 9:537–575. Hardy, J. L., and K. K. De Valois, 2002. Color-selective analysis of luminance-varying stimuli, Vis. Res., 42:1941–1951. Hood, D. C., and M. A. Finkelstein, 1983. A case for the revision of textbook models of color vision: the detection and appearance of small brief lights, in Colour Vision: Physiology and Psychophysics ( J. D. Mollon and L. T. Sharpe, eds.), London: Academic Press, pp. 385–398. Julesz, B., 1971. Foundation of Cyclopean Perception, Chicago: University of Chicago Press. Kelly, D. H., 1983. Spatiotemporal variation of chromatic and achromatic contrast threshold, J. Opt. Soc. Am., 73:742–750. Kingdom, F. A. A., and D. R. Simmons, 1996. Stereoacuity and colour contrast, Vis. Res., 36:1311–1319.

Kingdom, F. A. A., D. R. Simmons, and S. Rainville, 1999. On the apparent collapse of stereopsis in random-dot-stereograms at isoluminance, Vis. Res., 39:2127–2141. Kooi, F. L., R. L. De Valois, and E. Switkes, 1991. Spatial localization across channels, Vis. Res., 31:1627–1631. Kooi, F. L., K. K. De Valois, E. Switkes, and D. G. Grosof, 1992. High-order factors influencing the perception of sliding and coherence of a plaid, Perception, 21:583–598. Krauskopf, J., and B. Farell, 1991. Vernier acuity: effects of chromatic content, blur and contrast, Vis. Res., 31:735–749. Legge, G. E., and J. M. Foley, 1980. Contrast masking of human vision, J. Opt. Soc. Am., 70:1458–1471. Lennie, P., J. Pokorny, and V. Smith, 1993. Luminance, J. Opt. Soc. Am. A, 10:1283–1293. Li, A., and P. Lennie, 1997. Mechanisms underlying segregation of colored textures, Vis. Res., 37:83–97. Li, C.-Y., and K. Guo, 1995. Measurements of geometric illusions, illusory contours and stereo-depth at luminance and colour contrast, Vis. Res., 35:1713–1720. Livingstone, M., and D. H. Hubel, 1987. Psychophysical evidence for separate channels for the perception of form, color, movement, and depth, J. Neurosci., 7:3416–3468. Livingstone, M., and D. H. Hubel, 1988. Segregation of form, color, movement, and depth: anatomy, physiology, and perception, Science, 240:740–749. Losada, M. A., and K. T. Mullen, 1994. The spatial tuning of chromatic mechanisms identified by simultaneous masking, Vis. Res., 34:331–341. Lu, C., and D. H. Fender, 1972. The interaction of color and luminance in stereoscopic vision, Invest. Ophthalmol. Vis. Sci., 11:482–490. Marron, J. A., and I. L. Bailey, 1982. Visual factors and orientation-mobility performance, Am J. Optom. Physiol. Opt., 59:413–426. Mayhew, J. E., and S. M. Anstis, 1972. Movement aftereffects contingent on color, intensity, and pattern, Percept. Psychophy., 12:77–85. McCullough, C., 1965. Color adaptation of edge-detectors in the human visual system, Science, 149:1115–1116. McIlhagga, W. H., and K. T. Mullen, 1996. Contour integration with color and luminance contrast, Vis. Res., 36:1265–1279. McKeefry, D. J., I. J. Murray, and J. J. Kulikowski, 2001. Red-green and blue-yellow mechanisms are matched in sensitivity for temporal and spatial modulation, Vis. Res., 41:245–255. Merigan, W. H., 1989. Chromatic and achromatic vision of macaques: role of the P pathway, J. Neurosci., 9:776–783. Mollon, J. D., 1989. “Tho’ she kneel’d in that place where they grew . . . ,” J. Exper. Biol., 146:21–38. Mullen, K. T., 1985. The contrast sensitivity of human colour vision to red-green and blue-yellow chromatic gratings, J. Physiol. (Lond.), 359:382–400. Mullen, K. T., W. H. A. Beaudot, and W. H. McIlhagga, 2000. Contour integration in color vision: a common process for the blue-yellow, red-green and luminance mechanisms? Vis. Res., 40:639–655.

Mullen, K. T., S. J. Cropper, and M. A. Losada, 1997. Absence of linear subthreshold summation between red-green and luminance mechanisms over a wide range of spatio-temporal conditions, Vis. Res., 37:1157–1165. Mullen, K. T., and M. A. Losada, 1994. Evidence for separate pathways for color and luminance detection mechanisms, J. Opt. Soc. Am. A, 11:3136–3151. Nothdurft, H.-C., 1993. The role of features in preattentive vision: comparison of orientation, motion and color cues, Vis. Res., 33:1937–1958. Rauschecker, J. P., F. W. Campbell, and J. Atkinson, 1973. Colour opponent neurones in the human visual system, Nature, 245:42–43. Rovamo, J., L. Hyvaerinen, and R. Hari, 1982. Human vision without luminance-contrast system: selective recovery of the redgreen colour-contrast system from acquired blindness, Doc. Ophthalmol. Proc. Seri., 33:457–466. Scharff, L. V., and W. S. Geisler, 1992. Stereopsis at isoluminance in the absence of chromatic aberrations, J. Opt. Soc. Am. A, 9:868–876. Schiller, P. H., N. K. Logothetis, and E. R. Charles, 1990. Role of the color-opponent and broad-band channels in vision, Vis. Neurosci., 5:321–346. Simmons, D. R., and F. A. A. Kingdom, 1995. Differences between stereopsis with isoluminant and isochromatic stimuli, J. Opt. Soc. Am. A, 12:2094–2104. Stromeyer, C. F. III, R. Thabet, A. Chaparro, and R. E. Kronauer, 1999. Spatial masking does not reveal mechanisms selective to combined luminance and red-green color, Vis. Res., 39:2099– 2112. Switkes, E., A. Bradley, and K. K. De Valois, 1988. Contrast dependence and mechanisms of masking interactions among chromatic and luminance gratings, J. Opt. Soc. Am. A, 5:1149– 1162. Troscianko, T., R. Montagnon, J. Le Clerc, E. Malbert, and P. L. Chanteau, 1991. The role of colour as a monocular depth cue, Vis. Res., 31:1923–1929. Van der Horst, G. J., and M. A. Bouman, 1969. Spatiotemporal chromaticity discrimination, J. Opt. Soc. Am., 59:1482– 1488. Van der Horst, G. J., C. M. De Weert, and M. A. Bouman, 1967. Transfer of spatial chromaticity-contrast at threshold in the human eye, J. Opt. Soc. Am., 57:1260–1266. Virsu, V., and S. Haapasalo, 1973. Relationships between channels for colour and spatial frequency in human vision, Perception, 2:31–40. Webster, M. A., K. K. De Valois, and E. Switkes, 1990. Orientation and spatial-frequency discrimination for luminance and chromatic gratings, J. Opt. Soc. Am. A, 7:1034–1049. Webster, M. A., V. E. Raker, and G. Malkoc, 1998. Visual search and natural color distributions, in Human Vision and Electronic Imaging III (B. Rogowitz and T. Pappas, eds.), Bellingham, WA: SPIE, pp. 498–509. Westheimer, G., and S. P. McKee, 1977. Spatial configurations for hyperacuity, Vis. Res., 17:941–947.

 :    

935

60

Pattern-Selective Adaptation in Color and Form Perception MICHAEL A. WEBSTER

. . . I used to think that the aftereffects of persisting stimulation of the retina obtained by prolonged fixation of a display could be very revealing. Besides ordinary afterimages there are all sorts of perceptual aftereffects, some of which I discovered. But I no longer believe that experiments on so-called perceptual adaptation are revealing, and I have given up theorizing about them. . . . J. J. Gibson, The Ecological Approach to Visual Perception (1979, p. 248)

J. J. Gibson was among the most influential perceptual psychologists of the twentieth century. Early in his career he reported a striking visual illusion—the tilt aftereffect (Gibson and Radner, 1937). After tilted lines are viewed for a brief period, a vertical line appears tilted in a direction opposite to the adapting orientation (Fig. 60.1). There are many similar examples of visual aftereffects. For example, to experience the motion aftereffect or waterfall illusion, stare at the water pouring down a fall for a few moments and then shift your gaze to the side. The static rocks will briefly appear to ooze upward. Such aftereffects are a consequence of perceptual adaptation. The visual system adapts or reduces its sensitivity in response to the currently viewed stimulus. These sensitivity changes are normally selective—they adjust to specific properties or patterns of the image—and thus the aftereffects are usually experienced as a bias toward the opposite or more novel image properties. The resulting illusions attest to the malleability of perception and have provided one of the most commonly used tools for probing visual coding. Indeed, adaptation is often referred to as the psychologist’s electrode, for it is routinely used to try to detect and characterize visual mechanisms by measuring how their sensitivities change following adaptation. However, while pattern adaptation has been central to the study of vision, it is less often thought to be important to the actual act of seeing and has even been regarded as an anomaly of perception, arising when the visual system is fatigued by exposure to situations it was never designed to handle. In later years, Gibson went on to found an entire school of perception that emphasized the importance of understanding vision within the context of the rich patterns of information provided by actively exploring the natural visual environment (Gibson, 1979). From this perspective, adaptation was no longer useful as a tool, because the very notion of intervening processes became irrelevant. But as the

936

quote above implies, he also felt that maintaining fixation on a tilted line (a typical procedure for inducing aftereffects) was itself an unnatural task, and thus was irrelevant to understanding the normal dynamics of perception. And in his final major treatise on vision, the aftereffects he helped reveal were relegated to a footnote. The aim of this chapter is instead to emphasize the importance of adaptation, both as a method for understanding the processes mediating perception and as a principle for understanding why things look the way they do. Even brief exposures to a pattern can dramatically alter perception, and this is one reason adaptation remains such a popular paradigm. The following sections review the nature of these perceptual aftereffects and illustrate how they have been used to uncover the visual mechanisms encoding color and form. But if we can recast vision so easily in the lab, how is it being molded by the patterns we are routinely exposed to on the outside, as you walk through a forest or sit reading this page? The final sections take up this question by considering how visual perception is influenced by adaptation to the natural visual environment. The visual world is not random. Natural images have characteristic properties, and exposure to these persisting patterns of stimulation may therefore hold the visual system in specific states of adaptation. These states provide the relevant contexts for understanding natural vision.

Pattern adaptation and visual channels What can an orientation-selective aftereffect tell us about the visual processes underlying form perception? Figure 60.2A shows the kinds of measurements one might record in a study of orientation adaptation. In this plot the angle corresponds to the pattern orientation, while the distance from the origin corresponds to the pattern contrast. Note that we could represent any stimulus within the plane by taking only two “measurements” (e.g., of the component contrasts along the horizontal and vertical axes) and that these could sample contrasts along any pair of axes within the plane. But how many measurements are actually used, and along which axes do they lie? Adaptation experiments address this question by exploring how responses to stimuli are altered after observers are

F 60.1. The tilt aftereffect. Adapting to a counterclockwise tilted line causes a vertical line to appear tilted clockwise, and vice versa.

exposed to and thus adapted by different stimuli. To induce a new state of adaptation, subjects typically view the adapting stimulus for a few minutes and then make judgments about a set of briefly presented test stimuli. Two common types of judgments are used. In one case, sensitivity is probed by finding the threshold for detecting or discriminating the test stimulus. In the second, the subjective appearance of the test is assessed. One way to do this would be to match the apparent orientation of the test by physically adjusting the orientation of a nearby comparison stimulus presented to a part of the retina maintained under neutral adaptation. This asymmetric matching task assumes that the effects of adaptation are confined to the regions of the retina (or their associated pathways) that were exposed to the stimuli. A second approach is to vary the test stimulus physically in order to cancel out a perceptual change. For example, the orientation of a test could be adjusted so that it always appears vertical. This nulling method assumes that any response changes

A

induced by adaptation are equivalent to the responses induced by a physical stimulus. Still other common measures include rating the perceived magnitude of an aftereffect or its perceived duration. Figure 60.2A plots an idealized set of results after adapting to a bar tilted at a clockwise angle. Measures of sensitivity to different orientations would show that adaptation increases the threshold for detecting patterns that have orientations similar to the adapting pattern (Gilinsky, 1968). Measures of appearance would show that after adaptation a vertical line appears tilted counterclockwise. Both aftereffects are consistent with a selective loss in sensitivity to the adapting orientation, and thus imply that adaptation is altering the responses in something that can be selectively tuned for orientation. Results of this kind are usually explained in terms of visual channels—the notion that the visual system encodes information within a bank of filters that respond to different but overlapping ranges along the stimulus continuum (e.g., to different orientations, hues, or directions of motion). Any stimulus is thus represented by the distribution of activity across the set of channels. A further common assumption is that these channels are labeled for particular sensations, so that which stimulus is perceived (e.g., vertical or red) depends on which channels respond, while the magnitude of the stimulus (e.g., contrast or saturation) is encoded by the size of the response (Braddick et al., 1978). Figure 60.2B shows one possible account of the tilt aftereffect based on changes in the distribution of activity across multiple channels. Suppose that adaptation reduces a channel’s sensitivity according to how strongly the channel responded to the adapting stimulus. This would reduce the channel’s responses to a subsequent test stimulus. The test orientations to which it is tuned would become harder to detect, and patterns that are above threshold would appear to have lower contrast. Moreover, the diminished signals

B

F 60.2. Multichannel accounts of the tilt aftereffect. a, Measurements of detection thresholds (elliptical contour) or perceived tilt of a vertical test after adapting to a clockwise bar. b, Both effects can be accounted for by adaptation in orientation-selective chan-

nels that reduces sensitivity in channels tuned to the adapter and thus skews the distribution of responses to the test away from the distribution of responses to the adapter.

:      

937

would reduce its contribution to the collection of channel responses, and thus for nearby test orientations would skew the mean of the distribution away from the mean for the adapting orientation, inducing the perceived aftereffect. (However, this leaves the problem of how distortions in the pattern’s features can be reconciled with their perceived retinal location; Meese and Georgeson, 1996.) Often the studies using adaptation have not been interested in the processes of adaptation itself, but rather in the properties of the channels implied by the adaptation. One question of interest is the bandwidths or profiles of the channels. For example, an adaptation effect that influenced only a narrow range of orientations would imply that the channels are highly selective for orientation. A second commonly asked question concerns the number of channels. If the response changes are selective for the adapting axis, that implies a channel tuned to that axis. If selective aftereffects can be found for many axes, then that might imply many channels. We could thus repeat the measurements of Figure 60.2A for many adapting and test orientations in order to characterize how orientation is represented at the level at which the adaptation alters sensitivity. The results of such studies have shown that sensitivity changes appear selective for any orientation, suggesting that orientation is encoded effectively by a continuum of channels, with bandwidths (orientation range at which sensitivity falls to half the peak) on the order of roughly ±10 degrees (Blakemore and Nachmias, 1971). However, the interpretation of these results is complicated, precisely because any inferences about the underlying channels depends on assumptions about the nature of the adaptation. For instance, the model in Figure 60.2B assumes that each channel adapts independently. Yet suppose that adaptation instead reflects an interaction between channels (Barlow, 1990; Wilson, 1975). For example, Barlow suggested that adaptation involves reciprocal inhibition between two channels that builds up whenever their outputs are correlated. The effect of this mutual repulsion is to bias the channels’ responses until they are statistically independent. An account of the tilt aftereffect based on this principle is shown in Figure 60.3. (For a comprehensive model, see Clifford et al., 2000.) In this example, orientation is encoded by pair of channels that, under neutral adaptation, are tuned to horizontal and vertical. Exposure to the clockwise adapter would produce covarying responses in both channels, leading to inhibition between them. This alters the response within each channel by subtracting a fraction of the response in the second channel. In turn, this reduces the responses to the adapting axis and tilts the tuning function for each channel away from the adapting axis, spherizing the response distribution. Thus, an important feature of this model is that adaptation could induce a selective change in sensitivity even to stimulus directions to which neither channel is tuned.

938

  

A

B

F 60.3. An alternative account of tilt aftereffects based on mutual inhibition between channels. a, Signals along an oblique axis produce correlated responses in channels tuned to horizontal and vertical. b, Inhibition between the channels leads to an oblique rotation of their response axes, decorrelating their outputs.

Consequently, adaptation effects alone do not conclusively reveal the specific channel structure. The actual neural mechanisms underlying patternselective adaptation remain unresolved, though it is clear that the channels defined psychophysically do not reflect passive habituation in a neuron’s responses. Physiological measurements of contrast adaptation in the cortex suggest that the response changes result from a tonic hyperpolarization imposed on a separate stimulus-driven response that is unaffected by adaptation (Carandini and Ferster, 1997). At least some components of the adaptation are extremely rapid (Muller et al., 1999), and can selectively adjust to the co-occurrence or contingencies between pairs of stimuli (Carandini et al., 1997) and alter the shape of an individual neuron’s tuning curve (Movshon and Lennie, 1979; Muller et al., 1999). The fact that very different models can lead to very similar explanations of visual aftereffects shows that the implications of contrast adaptation must be interpreted with caution. On the other hand, the models illustrated share important features. Both assume that stimuli are encoded by a set of channels that are (or can be) selectively tuned to many different directions, and that adaptation alters perception by altering the distribution of responses within these channels. Thus, the presence of a pattern-selective aftereffect remains a powerful source of evidence about the nature of visual representations.

The sites of adaptation As Gibson noted, there are all sorts of perceptual aftereffects. Indeed, we can see the signs of adaptation literally everywhere we look. Neural adjustments begin at the earliest stages in the retina, where processes of light adaptation adjust sensitivity in order to match the ambient light level (Hood, 1998). At the other extreme, some perceptual adap-

tations are actually perceptual-motor adjustments, because they involve recalibration of sensorimotor signals. For example, many studies have examined the visual and behavioral changes that result when observers wear prisms that distort or even invert the visual field (Welch, 1986). Subjects show a remarkable capacity to adjust to these distortions so that they can move about and reach for objects appropriately. This relearning is distinct from a purely visual change because it requires active exploration of the world and primarily reflects changes in perceived body position. At intermediate stages, the visual system adjusts not merely to the average light level, but also to the patterns of light or contrasts in the image (Webster, 1996). These patterns may be stimulus variations in space, time, or color. Classic examples include not only the tilt and motion aftereffects (Mather et al., 1998), but also numerous figural or size-selective aftereffects, in which adaptation to a particular shape or size biases the apparent shape or size of other images (Kohler and Wallach, 1944). Pattern adaptation can also selectively adjust to specific combinations or conjunctions of visual attributes, and in this case is known as contingent adaptation (Stromeyer, 1978). For example, color aftereffects can be induced that are contingent on the spatial orientation or direction of motion of a pattern, or vice versa. Contingent aftereffects are sometimes distinguished from simple pattern aftereffects by their long persistence and by the possibility that they must be actively extinguished rather than passively decaying, characteristics that have blurred the distinction between adaptation and learning. The aftereffects of pattern adaptation primarily reflect sensitivity changes originating in visual cortex. Three lines of evidence support this. First, what is being affected is sensitivity to patterns—to tilted lines, tinted bars, or drifting gratings—and neurons in the primate visual system do not appear to have the requisite selectivity until striate cortex. For example, tuning for orientation, direction of motion, and spatial frequency are properties that first clearly emerge in striate cortex (De Valois and De Valois, 1988). The second source of evidence is that most visual aftereffects show substantial interocular transfer (Blake et al., 1981). That is, an adapting pattern that is viewed only by the right eye can influence a test pattern that is presented only to the left eye. Because signals from the two eyes first converge in the cortex, this is the earliest plausible site at which a sensitivity change could lead to binocular interactions. Finally, direct recordings from neurons along the visual pathway have shown that cortical cells are strongly adapted by patterns, while response changes in geniculate and retinal cells are weaker (Maffei et al., 1973; Ohzawa et al., 1982) though still substantial (Brown and Masland, 2001; Chander and Chichilnisky, 2001; Smirnakis et al., 1997). While striate cortex may therefore be an important site of pattern adaptation, this does not preclude sensitivity changes

at higher levels. A number of aftereffects point to multiple cortical sites in pattern adaptation. For example, distinct motion aftereffects have been found for static versus dynamic test patterns and for simple gratings versus two-dimensional plaids, and these have been attributed to sensitivity changes at different sites or pathways (Mather et al., 1998). Moreover, functional magnetic resonance imaging (fMRI) studies have demonstrated response changes correlated with the motion aftereffect that are strongest in area MT, an extrastriate area specialized for motion (Tootell et al., 1995). Studies of orientation adaptation have provided intriguing clues about the sites of the sensitivity changes controlling the tilt aftereffect. Asymmetrical tilt aftereffects occur between real and illusory contours, and these may reflect differences between striate cortex and area V2 in the representation of subjective contours (Paradiso et al., 1989). Distinct tilt aftereffects can also be demonstrated for oriented contours versus oriented textures, with the former affecting sensitivity changes at relatively high levels of shape coding (Suzuki, 2001). Surprisingly, tilt aftereffects can also be induced by patterns that cannot be consciously perceived because they are too fine to be resolved (He and MacLeod, 2001). This suggests that at least some of the aftereffects arise at relatively early cortical levels before visual awareness.

Adaptation and color vision We can readily distinguish distinct and qualitatively different stages of light adaptation and pattern adaptation in color vision, and can use these effects to characterize how information about color is transformed and represented at successive visual levels. Figure 60.4 shows a standard model of human color vision. At the first stage, light is encoded by the responses in three types of cone that have peak sensitivities at short, medium, or long wavelengths (S, M, or L). Subsequently the signals from the cones are combined to form postreceptoral channels. These channels may draw on receptor signals of the same sign to form luminancesensitive or nonopponent channels, or may receive antagonistic inputs from different cones to form color-sensitive or opponent channels. The two color channels shown receive opposing signals from the L and M cones (L-M) or S opposed by both L and M cones (S-LM). These combinations represent the preferred color directions of cells in the retina and geniculate, and thus are thought to characterize postreceptoral color coding at precortical stages in the visual system (Derrington et al., 1984). One could evaluate a model like Figure 60.4 by asking how a response change in the different channels might alter color perception. Alternatively, we could approach the question from the opposite direction, by measuring the effects of adaptation to a stimulus and then asking what set of channels is consistent with the observed aftereffects. For example,

:      

939

F 60.4. A standard two-stage model of color vision based on three classes of cones transformed into a luminance channel and two cone-opponent color channels.

Figure 60.5A shows a distribution of colors plotted in a space defined by the signals within the L-M and S-LM channels. In this plot different angles correspond to different hues, and saturation or contrast increases with the distance from the white origin. Note that this is very similar to the representation of contrast and orientation in Figure 60.2A. The stimulus distribution in Figure 60.5 is biased in two ways: the average color is not centered on white, and there is greater variance or contrast along the diagonal axis than along other axes of the space. Processes of light adaptation and contrast (pattern) adaptation selectively adjust to each of these properties (Webster and Mollon, 1995). L A Light or chromatic adaptation induces dramatic changes in color vision that are easily demonstrated by the afterimages that are experienced when we fixate a pattern and then switch our gaze to a uniform field. The afterimages arise from lingering sensitivity changes that adjust each location of the retina according to the average light and color it is exposed to. In the example of Figure 60.5A the mean color would look purplish under neutral adaptation. However, adaptation readjusts sensitivity so that the average color appears more achromatic, producing corresponding shifts in the appearance of all colors in the distribution (Fig. 60.5B). To a large extent, the color appearance changes reflect multiplicative gain changes that occur independently within the cones or cone-specific pathways, a process known as von Kries adaptation (Chichilnisky and Wandell, 1995; Webster and Mollon, 1995; Wuerger, 1996). Thus, they represent adjustments at the first stage of color processing and, indeed, at the very beginning of vision. In a classic series of experiments, Stiles (1959) examined the number and color selectivities of the mechanisms underlying chromatic adaptation. Thresholds for detecting a test light were measured in the presence of a uniform adapting background. As the background intensity increases, the processes that respond to the background become less sensitive or light adapt, so that performance follows a characteristic threshold versus intensity curve (Fig. 60.6). The test is detected by the mechanism(s) that are most sensitive on a given adapting background, with a switch between mechanisms revealed by separate branches in the curve. By varying

940

  

F 60.5. The two-color threshold method of Stiles. Thresholds for detecting a blue (475 nm) test on a yellow-green (550 nm) background follow two branches reflecting light adaptation in two different color mechanisms.

the wavelength of the test and adapting lights, Stiles showed that sensitivity was limited by a small number of discrete pi mechanisms with different sensitivities to wavelength, each adjusting independently to the adapting background. The spectral sensitivities of the pi mechanisms are similar but not equivalent to those of the cones. Moreover, Stiles’ work revealed more than three distinct mechanisms under different adapting conditions. These discrepancies have been resolved by showing that chromatic adaptation also depends on second-site adjustments in postreceptoral channels (Pugh and Mollon, 1979). One example of these is transient tritanopia, a loss in sensitivity to a short-wavelength test after turning off a long-wavelength background that is invisible to S cones (Mollon and Polden, 1977). Extinguishing the background should dark-adapt all of the cones and make them more sensitive, yet thresholds for an S cone detected test are temporarily elevated because the rebound from the yellow background saturates responses at a cone-opponent site. Such aftereffects provided an important source of evidence for color opponency. Second-site adjustments in light adaptation can also be seen in the ways that backgrounds influence sensitivity and appearance (Shevell, 1978; Walraven et al., 1990). A background light sets the gain of the visual mechanisms detecting a superimposed test but also physi-

A

B

C

F 60.6. Adaptation in color vision. a, An elliptical distribution of colors plotted in terms of the two cone-opponent axes. Light adaptation adjusts to the mean of the distribution so that the average color appears white (b). Contrast adaptation instead adjusts to the variations in color around the average (c).

cally adds light to the test. However, this added light often has little effect on the test’s appearance, because spatial and temporal filtering subtract the signals from the background, leaving the visual system to respond primarily to spatial and temporal transients. Differences in the time course of sensitivity changes suggest that chromatic adaptation may in fact depend on several sites of both multiplicative and subtractive adjustments (Fairchild and Reniff, 1995; Hayhoe et al., 1987). C A As Figure 60.5B shows, even if we renormalize for the average color in an image, there may often remain a bias in the variance or contrasts in the distribution. Visual mechanisms also adjust to these “patterns” of color through contrast adaptation. This produces changes in color vision that are very different from the effects of light adaptation. Chromatic adaptation in the cones is largely a process that readjusts the white point. Contrast adaptation instead alters the perceived contrasts relative to the average color and thus has very little effect on the mean color itself (Webster and Wilson, 2000). Many studies have examined color adjustments by measuring only the stimulus that looks achromatic. This is perhaps the best setting for detecting chromatic adaptation, but it is the least likely to reveal the presence of contrast adaptation. The first investigators to explicitly study adaptation to color contrast were Guth (1982) and Krauskopf et al. (1982), both by measuring how color vision is affected by adapting to a background that flickered between two colors. An advantage of this approach is that the flickering field did not change the time-averaged luminance or chromaticity of the adapting field, and thus bypassed the early stages of chromatic adaptation to alter sensitivity at more central sites. In fact, like other forms of pattern adaptation, adaptation to color contrast primarily reflects sensitivity changes in the cortex (e.g., Engel and Furmanski, 2001; Lennie et al., 1990). Studies of contrast adaptation thus allow direct measurements of color coding at cortical levels. Krauskopf et al. (1982) used contrast adaptation to explore the spectral sensitivities of cortical color channels and, in particular, to ask which directions in color space they are tuned for. To do this, they measured thresholds for detecting a color change from white after adapting to fields that were sinusoidally modulated in color along different axes within the plane of Figure 60.5. These threshold changes revealed two important properties of the adapted color channels. First, aftereffects were primarily selective for three cardinal directions: an achromatic axis and the L-M and S-LM chromatic axes. For example, after adaptation to an L-M modulation, an L-M test was much harder to see, while sensitivity to an S-LM or luminance-varying test remained largely unaffected. This suggested that the adapted channels are organized in terms of these dimensions, and not in terms

F 60.7. Contrast adaptation in color vision. Adaptation to flicker along the L-M axis reduces responses along this axis, compressing the circle of test stimuli into an ellipse. This biases the perceived hue of tests away from the adapting axis and toward the orthogonal axis.

of the red-green and blue-yellow dimensions predicted by subjective color experience. However, weak selectivity was also observed for adapting directions intermediate to the LM and S-LM axes, suggesting the presence of additional channels tuned to these directions (Krauskopf et al., 1986). This raised the possibility that—like the representation of spatial patterns—the representation of color is elaborated in the cortex. Consistent with this, cells in striate cortex show a much wider range of preferred color directions than geniculate cells (Lennie et al., 1990). Webster and Mollon (1994) extended this paradigm to examine how contrast adaptation alters color appearance. In their studies, subjects viewed adapting and test lights in a field above fixation and then matched the perceived color of the tests by adjusting the physical color in a comparison field presented below fixation. Color changes in the tests were strongly selective for the color axis of the adapting flicker, as illustrated schematically in Figure 60.7 by the matches made after adaptation to the L-M axis. This collapses perceived contrast along the L-M axis, and thus biases the perceived hue of test stimuli away from the L-M axis and toward the orthogonal (S-LM) axis (with a change in contrast but not hue along the adapting and orthogonal axes). Color shifts away from the adapting axis could be induced in any test direction, including the achromatic axis, suggesting that there is no axis that isolates a single type of channel (since the shifts presumably reflect a change in the distribution of responses across channels). These biases in perceived hue are analogous to the biases in perceived orientation in the tilt aftereffect and thus can be thought of as tilt aftereffects in color space. However, compared to orientation the hue shifts are many times larger, exceeding 30 degrees in some cases (compared to 5 degrees or less for typical tilt aftereffects). This difference may result because, when compared in a common metric like Figures 60.2 and 60.5, the color channels are much more broadly tuned than the channels coding orientation, so that the mean of the distribution shifts more with adaptation. The bandwidths of the color aftereffects

:      

941

are in fact well fit by assuming that the channels are defined by different linear combinations of the cones. Like orientation, these selective adjustments to color contrast could be achieved by very different routes. Specifically, to account for the selectivity for multiple directions, we could modify the second-stage mechanisms in Figure 60.4 either by adding many color channels that adapt independently or by allowing for adaptive interactions between them. Models based on each of these assumptions have been developed, and both provide a good fit of the observed color changes (Atick et al., 1993; Clifford et al., 2000; Lennie, 1999; Webster and Mollon, 1994; Zaidi and Shapiro, 1993). Thus, again, adaptation alone does not define the intrinsic number of color channels. However, other experimental approaches point to the presence of many chromatic mechanisms even in a single state of adaptation (Webster, 1996). This supports the idea that the cortex does encode color—like orientation—in a large number of channels, though it does not imply that these adapt independently. Further adaptation effects have revealed a number of additional properties of cortical color channels. For example, adaptation to sawtooth modulations suggests that different populations of channels encode the opposite poles of luminance or chromatic axes (Krauskopf et al., 1982), and the axes may be even further subdivided into channels that code different ranges of contrast (Webster and Wilson, 2000). A number of studies have also explored a simultaneous analog of contrast aftereffects. The perceived contrast of a pattern can be strongly attenuated by embedding the pattern in a high-contrast surround (Chubb et al., 1989). This contrast induction again adjusts selectively to different chromatic axes (Brown and MacLeod, 1997; Singer and D’Zmura, 1994; Webster et al., 2002). Adaptation has also proven useful for probing the spatial selectivities of color mechanisms. For example, adaptation to color spatial patterns (e.g., a red-green edge) makes color patterns with similar orientation and spatial frequency harder to detect (Bradley et al., 1988) and can induce tilt aftereffects in color patterns (Elsner, 1978). Moreover, the tilt aftereffects show selectivity for multiple color directions (Flanagan et al., 1990). Such results show that the affected channels can be tuned to both the color and the spatial properties of stimuli. A classic demonstration of this is the McCollough effect (McCollough, 1965). After viewing a redvertical grating alternated in time with a green-horizontal grating, an achromatic-vertical grating looks greenish, while an achromatic-horizontal grating looks reddish. Thus, the color changes are contingent on the orientation of the adapting patterns. If we think of the bright-red adapting grating as an oblique direction within the color-luminance plane and the achromatic test as vertical, then this aftereffect can again be accounted for by a bias away from the adapting axis. That is, the color change in the test is a tilt

942

  

aftereffect in color-luminance space that is selective for spatial orientation (Webster and Malkoc, 2000).

The functions of adaptation The preceding sections show that adaptation can exert a powerful hold over our perception. But if this influence reflects function rather than fatigue, then we should see tangible signs that it is helping us to see better in everyday contexts. The benefits of light adaptation seem clear. The response range of neurons is very limited, but must be used to encode visual signals over a staggering range of light intensities. Adjusting to the average light level allows the retina to use its full dynamic range to encode the information carried by stimulus contrasts (Craik, 1940; Walraven et al., 1990). Multiplicative adjustments within the cones further allow the visual system to maintain lightness and color constancy by factoring out changes in the mean illumination. Compared to light adaptation, the effects of pattern adaptation appear more subtle, and it has proven much more difficult to demonstrate improvements in visual performance. However, there are several potential benefits. G C One possibility is that adaptation protects against saturation in cortical responses in the same way that light adaptation protects retinal responses. Recordings in striate cells show that prior adaptation tends to center the cells’ contrast response functions around the adapting contrast, and this can allow cells to respond differentially to stimuli that before adaptation led to saturated and thus indistinguishable responses (Albrecht et al., 1984; Sclar et al., 1989). However, psychophysically, only a minority of studies have found that contrast adaptation can improve contrast discrimination (Greenlee and Heitger, 1988; Wilson and Humanski, 1993). G M A related possibility is that contrast adaptation functions to match visual responses to the contrast gamut of the ambient environment to provide contrast constancy (Brown and MacLeod, 1997; Zaidi et al., 1998). In the case of color vision, an interesting example is provided by anomalous trichromats, who have M and L pigments with very similar spectral sensitivities. Because of this, the chromatic signal defined by the L-M difference is very weak, but adaptation might adjust the gain of postreceptoral channels to fill the available range (though whether this occurs in the retina or cortex is uncertain; MacLeod, 2002; Regan and Mollon, 1997). N The idea of gamut matching suggests that the visual system tends to settle around special states that may reflect expectations about the properties of the visual environment. Examples of these states include white for color

or static for motion. Many aftereffects can be seen as a consequence of renormalizations for these states, especially when the adapting stimulus itself appears more “neutral” over time. For example, with adaptation a background color appears less saturated, drifting patterns seem to slow down, and tilted bars may appear more vertical. These adjustments might compensate for mean biases in the world or correct for errors or distortions in the visual system of the observer (Andrews, 1964), thus providing a form of perceptual constancy. For example, we will see below that adaptive renormalizations could maintain constancy for image structure despite variations in retinal image blur. D Pattern adaptation could plausibly improve not only coding within a channel but also coding across channels by removing redundancies between channel responses to provide more efficient representations (Barlow, 1990). Stimuli will often lead to correlated responses within a set of channels. By removing these correlations, adaptation could increase efficiency by allowing each mechanism to code independent information. Recent analyses of adaptation in cortical cells in fact support the role of adaptation in reducing redundancy (Carandini et al., 1997; Muller et al., 1999). L Finally, by adjusting to the correlations between image properties, adaptation provides a mechanism for representing the stimulus associations in the environment and for learning about new ones (Barlow, 1990). This aspect of adaptation is considered further in the concluding section.

Adaptation and the natural visual environment Whether adaptation is important to visual function hinges on whether it actually occurs in natural contexts. Again, there is no question of this for light adaptation. It is universally recognized that these adjustments are both a critical and an intrinsic part of the visual response to any stimulus. In fact, it would be meaningless to try to describe visual responses without assuming a particular state of light adaptation. But what of adaptation to tilted lines and waterfalls? Do the adjustments they reflect hold a similar status in perception? To assess this, it is important to ask how patterns of luminance and color vary within the kinds of natural images we normally encounter, and whether adaptation to these patterns can influence natural visual judgments. E A  V Before exploring the rapid visual adjustments implied by visual aftereffects, it is worth remembering that the very structure of the visual system evolved as a long-term adaptation to the animal’s visual environment. Recent studies have provided powerful insights into visual coding by characterizing statistical prop-

erties of natural images and then asking how these could best be represented by the visual system. For example, color opponency can be seen as a means of removing redundancies across the different cones (Buchsbaum and Gottschalk, 1983), while the spatial structure of receptive fields removes redundancies across space (Srinivasan et al., 1982). In visual cortex, individual cells respond to different scales or spatial frequencies in the image. The response bandwidths increase roughly in proportion to the preferred frequency (e.g., as f ), while the amplitude spectra of natural images instead characteristically vary as the inverse of frequency (e.g., as 1/f ) (Field, 1987). The tuning of cortical cells therefore compensates for the low-frequency bias in natural scenes so that response levels in the cortex are independent of spatial scale (see Chapter 70). S-T A  V E However, the visual environment is not fixed and thus cannot be represented optimally by a visual system with fixed properties. Moreover, the visual system itself undergoes pronounced anatomical and physiological changes during development and aging. Some adjustments in the tuning are therefore important in order to match the system to the ambient environment and to provide stable perceptions despite variations in the observer. Exactly what kinds of natural stimulus patterns the visual system might adjust to is an intriguing but still largely unexplored question. Yet it is clear that natural images provide a powerful stimulus for pattern adaptation. Webster and Mollon (1997) examined changes in color appearance induced by adaptation to the color contrasts characteristic of natural outdoor scenes. Color in natural images is highly constrained and tends to vary along a limited range of axes in color space, from blue-yellow for arid scenes to a greenish S-LM axis for lush scenes. In the former case there is often a very high correlation between the signals along the cardinal chromatic axes, so that color in these scenes is not efficiently represented in the geniculate. However, contrast adaptation might adjust to this bias in the cortex, and observers in different environments should then be adapted by the prevailing colors in different ways. Figure 60.8 shows tests of this by measuring color appearance after adaptation to a succession of colors drawn at random from a natural scene. Light adaptation adjusts to the average color, while contrast adaptation induces large, selective changes in sensitivity to the blue-yellow axis of the adapting distribution. As noted above, natural images have characteristic amplitude spectra to which the visual system might normalize spatial sensitivity. However, image spectra are not entirely constant, but vary because of differences in both scenes and the observer. For example, optical blur steepens the spectrum by reducing finer details in the retinal image, and thus there

:      

943

A

B

F 60.8. Adaptation to natural color distributions. Adaptation to the blue-yellow color variations in an arid scene (a) produces changes in color appearance that are strongly selective for the adapting axis (b). Test colors (unfilled circles) centered on the biased mean of the distribution are matched by colors centered around

white and compressed along the blue-yellow axis of the distribution (filled circles), showing the imprint of both light adaptation to the mean and contrast adaptation to the variations in the distribution. (Unfilled triangles plot the matches predicted by light adaptation alone.)

may be characteristic states of pattern adaptation associated with refractive errors. Visual acuity in fact increases after observers adapt for a period to optical defocus (MonWilliams et al., 1998). Adaptation can also strongly affect the actual appearance of blur (Webster et al., 2002). Exposure to a blurred (sharpened) image causes a focused image to appear too sharp (blurred) (Fig. 60.9). Moreover, the blurred or sharpened adapting images themselves looked better focused the longer they are viewed, suggesting that adaptation is renormalizing perception of image focus. Figure 60.9 also shows a spatial analog of these effects induced by blurred or sharpened surrounds. Similar adaptation and induction effects also influence other judgments of the spatial statistics of images, such as the perception of texture density (Durgin and Huk, 1997). While these effects can be

very rapid, there may also be adjustments at much longer time scales. Fine et al. (2002) recently examined a subject who had had cataracts for decades. Even months after surgery, the world through his new lenses appeared overly sharpened. The perceptual changes in blur adaptation are dramatic, possibly because the natural consequences of adaptation are best revealed by probing them with stimuli and tasks that are natural and relevant to the observer. Human face perception provides a clear example of such tasks. Observers are remarkably adept at recognizing and judging faces based on subtle differences in their configural properties and thus should be highly sensitive to any changes in configuration induced by adaptation. In fact, adaptation to a distorted face alters the appearance of subsequent faces (Webster and

F 60.9. Adaptation to blur. A blurry adapting image causes a focused image to appear sharpened, or vice versa. Bars to the right show similar effects for induction. In each block the central

column of bars are all square edges. Yet the bars abutting the blurred edges appear too sharp, while the bars adjoining sharpened edges appear blurred.

944

  

F 60.10. Adaptation to faces. Adaptation to a contracted face causes the original face to appear expanded, while an expanded face induces the opposite aftereffect.

MacLin, 1999) (Fig. 60.10). Large biases in recognition also occur for adapting and test images that are defined by the configurations characterizing real faces (Kaping et al., 2002; Leopold et al., 2001). Given that we are all exposed to a different diet of faces that are naturally “distorted” relative to the average human face, it seems likely that these adaptation effects influence many aspects of face recognition in everyday viewing. Recent studies have shown that face aftereffects are surprisingly unaffected by large changes in size or location between the adapting and test images (Leopold et al., 2001; Zhao and Chubb, 2001). This strongly suggests that the adaptation involves adjustments to the configural properties of the face, rather than to local, low-level features, and thus supports the possibility that the sensitivity changes reflect processes specialized for face coding. It seems probable that many other aspects of higher-level object recognition are similarly shaped by adaptation.

Adaptation and the phenomenology of perception What are the implications of such adaptation effects for the subjective experience of seeing? One possible implication concerns whether we have shared or distinct perceptions (Webster, 2002). A long-standing philosophical question is what the world might look like if we could see it through the eyes of another. The private nature of our conscious experience may preclude a complete answer, but it is tempting to speculate that any answer will be constrained in important ways by the processes of adaptation. As the preceding sections illustrate, adaptation normalizes our perception according to properties of—or expectations about—the physical world. To the extent that two observers are adapted

to different environments, their visual systems will be normalized in different ways and their subjective experience should differ. For example, if you and I are exposed to a different population of faces, our perception (and not just our judgments) of the same physical facial characteristics are unlikely to agree. To the extent that two observers are exposed to a common environment, adaptation will tend to normalize their perception toward a convergent experience. For example, even if you and I have different refractive errors, our perceptions of image focus may converge because these differences are partially discounted by adaptation to the common spatial structure of scenes. Notably, in both cases, it is the similarities or differences in the environment—and not the intrinsic differences between observers—that determine how the visual system is normalized. Thus, at least some aspects of our private internal experience are controlled by public external variables that can be objectively measured. The effects of adaptation also have important implications for the actual contents of visual awareness. The decorrelation model proposed by Barlow was built around the idea that adaptation serves to discount ambient information in order to enhance sensitivity to novel patterns. These novel patterns, or suspicious coincidences, may be the most important information to the observer, and thus processes that highlight them may be highly beneficial (Barlow, 1990). The role of adaptation in regulating visual salience remains largely unexplored, because most studies have focused on understanding how adaptation influences the encoding of the adapting pattern itself. However, the negative aftereffects characteristic of adaptation support this view, since they “draw attention” to the ways in which the test differs from

:      

945

the adapter. In this regard, it is interesting to note that the consequences of adaptation are often much more striking in the test stimulus than in the stimulus to which we are adapted. Thus, the perceptual aftereffects of color and form are vivid imprints of sensitivity changes that often pass unnoticed during adaptation. If adaptation is part of our everyday visual experience, then perhaps most of what we notice about the world is a perceptual aftereffect.

Acknowledgments This work was supported by Grant EY10834 from the National Eye Institute. REFERENCES Albrecht, D. G., S. B. Farrar, and D. B. Hamilton, 1984. Spatial contrast adaptation characteristics of neurones recorded in the cat’s visual cortex, J. Physiol., 347:713–739. Andrews, D. P., 1964. Error-correcting perceptual mechanisms, Q. J. Exp. Psychol., 16:104–115. Atick, J. J., Z. Li, and A. N. Redlich, 1993. What does postadaptation color appearance reveal about cortical color representation? Vis. Res., 33:123–129. Barlow, H. B., 1990. A theory about the functional role and synaptic mechanism of visual after-effects, in Vision: Coding and Efficiency (C. Blakemore ed.), Cambridge: Cambridge University Press, pp. 363–375. Blake, R., R. Overton, and S. Lema-Stern, 1981. Interocular transfer of visual aftereffects, J. Exp. Psychol.: Hum. Percept. Perform., 7:367–381. Blakemore, C., and J. Nachmias, 1971. The orientation specificity of two visual aftereffects, J. Physiol., 213:157–174. Braddick, O., F. W. Campbell, and J. Atkinson, 1978. Channels in vision: basic aspects, in Handbook of Sensory Physiology VIII (R. Held, H. W. Leibowitz, and H. Teuber, eds.), Berlin: SpringerVerlag, pp. 3–38. Bradley, A., E. Switkes, and K. K. De Valois, 1988. Orientation and spatial frequency selectivity of adaptation to color and luminance gratings, Vis. Res., 28:841–856. Brown, R. O., and D. I. A. MacLeod, 1997. Color appearance depends on the variance of surround colors, Curr. Biol., 7:844– 849. Brown, S. P., and R. H. Masland, 2001. Spatial scale and cellular substrate of contrast adaptation by retinal ganglion cells, Nat. Neurosci., 4:44–51. Buchsbaum, G., and A. Gottschalk, 1983. Trichromacy, opponent colours and optimum colour information transmission in the retina, Proc. R. Soc. Lond. B, 220:89–113. Carandini, M., H. B. Barlow, L. P. O’Keefe, A. B. Poirson, and J. A. Movshon, 1997. Adaptation to contingencies in macaque primary visual cortex, Philos. Trans. R. Soc. Lond. B Biol. Sci., 52:1149–1154. Carandini, M., and D. Ferster, 1997. A tonic hyperpolarization underlying adaptation in cat visual cortex, Science, 276:949–952. Chander, D., and E. J. Chichilnisky, 2001. Adaptation to temporal contrast in primate and salamander retina, J. Neurosci., 21:9904–9916. Chichilnisky, E.-J., and B. A. Wandell, 1995. Photoreceptor sensitivity changes explain color appearance shifts induced by large

946

  

uniform backgrounds in dichoptic matching, Vis. Res., 35:239– 254. Chubb, C., G. Sperling, and J. A. Solomon, 1989. Texture interactions determine perceived contrast, Proc. Natl. Acad. Sci. USA, 86:9631–9635. Clifford, C. W. G., P. Wenderoth, and B. Spehar, 2000. A functional angle on some aftereffects in cortical vision, Proc. R. Soc. Lond. B, 267:1705–1710. Craik, K. J. W., 1940. The effect of adaptation on subjective brightness, Proc. R. Soc. Lond. B, 128:232–247. Derrington, A. M., J. Krauskopf, and P. Lennie, 1984. Chromatic mechanisms in lateral geniculate nucleus of macaque, J. Physiol., 357:241–265. De Valois, R. L., and K. K. De Valois, 1988. Spatial Vision, Oxford: Oxford University Press. Durgin, F. H., and A. C. Huk, 1997. Texture density aftereffects in the perception of artificial and natural textures, Vis. Res., 23:3273–3282. Elsner, A., 1978. Hue difference contours can be used in processing orientation information, Percept. Psychophys., 25:451–456. Engel, S. A., and C. S. Furmanski, 2001. Selective adaptation to color contrast in human primary visual cortex, J. Neurosci., 21:3949–3954. Fairchild, M. D., and L. Reniff, 1995. Time course of chromatic adaptation for color-appearance judgments, J. Opt. Soc. Am. A, 12:824–833. Field, D. J., 1987. Relations between the statistics of natural images and the response properties of cortical cells, J. Opt. Soc. Am. A, 4:2379–2394. Fine, I., H. S. Smallman, P. Doyle, and D. I. A. MacLeod, 2002. Visual function before and after the removal of bilateral congenital cataracts in adulthood, Vis. Res., 42:191–210. Flanagan, P., P. Cavanagh, and O. E. Favreau, 1990. Independent orientation-selective mechanisms for the cardinal directions of color space, Vis. Res., 30:769–778. Gibson, J. J., 1979. The Ecological Approach to Visual Perception, Boston: Houghton Mifflin. Gibson, J. J., and M. Radner, 1937. Adaptation, after-effect and contrast in the perception of tilted lines. I. Quantitative studies, J. Exp. Psychol., 20:453–467. Gilinsky, A. S., 1968. Orientation-specific effects of patterns of adapting light on visual acuity, J. Opt. Soc. Am. A., 58:13–17. Greenlee, M. W., and F. Heitger, 1988. The functional role of contrast adaptation, Vis. Res., 28:791–797. Guth, S. L., 1982. Hue shifts following flicker vs. fused adaptation reveal initial opponent mechanisms, Invest. Ophthalmol. Vis. Sci. (Suppl.), 22:78. Hayhoe, M. M., N. I. Benimoff, and D. C. Hood, 1987. The timecourse of multiplicative and subtractive adaptation process, Vis. Res., 27:1981–1996. He, S., and D. I. A. MacLeod, 2001. Orientation-selective adaptation and tilt aftereffect from invisible patterns, Nature, 411:473–476. Hood, D. C., 1998. Lower-level visual processing and models of light adaptation, Annu. Rev. Psychol., 49:503–535. Kaping, D., A. C. Bilson, and M. A. Webster, 2002. Adaptation and categorical judgments of faces [abstract], J. of Vision, 2:564a. Kohler, W., and H. Wallach, 1944. Figural aftereffects: an investigation of visual processes, Proc. Am. Philos. Soc., 88:269–357. Krauskopf, J., D. R. Williams, and D. W. Heeley, 1982. Cardinal directions of color space, Vis. Res., 22:1123–1131. Krauskopf, J., D. R. Williams, M. B. Mandler, and A. M. Brown, 1986. Higher order color mechanisms, Vis. Res., 26:23–32.

Lennie, P., 1999. Color coding in the cortex, in Color Vision: From Genes to Perception (K. R. Gegenfurtner and L. T. Sharpe, eds.), Cambridge: Cambridge University Press, pp. 235–247. Lennie, P., J. Krauskopf, and G. Sclar, 1990. Chromatic mechanisms in striate cortex of macaque, J. Neurosci., 10:649–669. Leopold, D. A., A. J. O’Toole, T. Vetter, and V. Blanz, 2001. Prototype-referenced shape encoding revealed by high-level aftereffects, Nat. Neurosci., 4:89–94. MacLeod, D. I. A., 2003. Colour discrimination, colour constancy and natural scene statistics, (the Verriest Lecture). In Normal and Defective Color Vision, J. D. Mollon, J. Pokorny, and K. Kroblauch (eds.), London: Oxford University Press, pp 189–217. Maffei, L., A. Fiorentini, and S. Bisti, 1973. Neural correlate of perceptual adaptation to gratings, Science, 182:1036–1038. Mather, G., F. Verstraten, and S. Anstis, eds., 1998. The Motion Aftereffect, Cambridge, MA: MIT Press. McCollough, C., 1965. Color adaptation of edge-detectors in the human visual system, Science, 149:1115–1116. Meese, T. S., and M. A. Georgeson, 1996. The tilt aftereffect in plaids and gratings: channel codes, local signs and “patchwise” transforms, Vis. Res., 36:1421–1437. Mollon, J. D., and P. G. Polden, 1977. An anomaly in the response of the eye to light of short wavelengths, Philos. Trans. R. Soc. B, 278:207–240. Mon-Williams, M., J. R. Tresilian, N. C. Strang, P. Kochhar, and J. P. Wann, 1998. Improving vision: neural compensation for optical defocus, Proc. R. Soc. Lond. B, 265:71–77. Movshon, J. A., and P. Lennie, 1979. Pattern-selective adaptation in visual cortical neurones, Nature, 278:850–852. Muller, J. R., A. B. Metha, J. Krauskopf, and P. Lennie, 1999. Rapid adaptation in visual cortex to the structure of images, Science, 285:1405–1408. Ohzawa, I., G. Sclar, and R. D. Freeman, 1982. Contrast gain control in the cat visual cortex, Nature, 298:266–268. Paradiso, M. A., S. Shimojo, and K. Nakayama, 1989. Subjective contours, tilt aftereffects, and visual cortical organization, Vis. Res., 29:1205–1213. Pugh, E. N., and J. D. Mollon, 1979. A theory of the p1 and p3 color mechanisms of Stiles, Vis. Res., 19:293–312. Regan, B. C., and J. D. Mollon, 1997. The relative salience of the cardinal axes of colour space in normal and anomalous trichromats, in Colour Vision Deficiencies VIII (C. R. Cavonius ed.), Dordrecht: Kluwer, pp. 67–76. Sclar, G., P. Lennie, and D. D. DePriest, 1989. Contrast adaptation in striate cortex of macaque, Vis. Res., 29:747–755. Shevell, S. K., 1978. The dual role of chromatic backgrounds in color perception, Vis. Res., 18:1649–1661. Singer, B., and M. D’Zmura, 1994. Color contrast induction, Vis. Res., 34:3111–3126. Smirnakis, S. M., M. J. Berry, D. K. Warland, W. Bialek, and M. Meister, 1997. Adaptation of retinal processing to image contrast and spatial scale, Nature, 386:69–73. Srinivasan, M. V., S. B. Laughlin, and A. Dubs, 1982. Predictive coding: a fresh view of inhibition in the retina, Proc. R. Soc. Lond. B, 216:427–459. Stiles, W. S., 1959. Color vision: the approach through incrementthreshold sensitivity, Proc. Natl. Acad. Sci. USA, 45:100–114.

Stromeyer, C. F. I., 1978. Form-color aftereffects in human vision, in Handbook of Sensory Physiology VIII (R. Held, H. W. Leibowitz, and H. L. Teuber, eds.), New York: Springer-Verlag. Suzuki, S., 2001. Attention-dependent brief adaptation to contour orientation: a high-level aftereffect for convexity? Vis. Res., 28:3883–3902. Tootell, R. B. H., J. B. Reppas, A. M. Dale, R. B. Look, M. I. Sereno, R. Malach, T. J. Brady, and B. R. Rosen, 1995. Visual motion aftereffect in human cortical area MT revealed by functional magnetic resonance imaging, Nature, 375:139– 141. Walraven, J., C. Enroth-Cugell, D. C. Hood, D. I. A. MacLeod, and J. L. Schnapf, 1990. The control of visual sensitivity: receptoral and postreceptoral processes, in Visual Perception: The Neurophysiological Foundations (L. Spillmann and J. S. Werner, eds.), San Diego, CA: Academic Press, pp. 53–101. Webster, M. A., 1996. Human colour perception and its adaptation, Network: Comput. Neural Syst., 7:587–634. Webster, M. A., 2002. Adaptation, high-level vision, and the phenomenology of perception, in Human Vision and Electronic Imaging VII (B. Rogowitz and T. Pappas, eds.), SPIE 4662:1–11. Webster, M. A., M. A. Georgeson, and S. M. Webster, 2002. Neural adjustments to image blur, Nat. Neurosci., 5:839–840. Webster, M. A., and O. H. MacLin, 1999. Figural after-effects in the perception of faces, Psych. Bull. Rev., 6:647–653. Webster, M. A., and G. Malkoc, 2000. Color-luminance relationships and the McCollough Effect, Percept. Psychophys., 62:659–672. Webster, M. A., G. Malkoc, A. C. Bilson, and S. M. Webster, 2002. Color contrast and contextual influences on color appearance, J. of Vision, 2:505–519. Webster, M. A., and J. D. Mollon, 1994. The influence of contrast adaptation on color appearance, Vis. Res., 34:1993–2020. Webster, M. A., and J. D. Mollon, 1995. Colour constancy influenced by contrast adaptation, Nature, 373:694–698. Webster, M. A., and J. D. Mollon, 1997. Adaptation and the color statistics of natural images, Vis. Res., 37:3283–3298. Webster, M. A., and J. A. Wilson, 2000. Interactions between chromatic adaptation and contrast adaptation in color appearance, Vis. Res., 40:3801–3816. Welch, R. B., 1986. Adaptation of space perception, in Handbook of Perception and Human Performance, vol. 1 (K. R. Boff, L. Kaufman, and J. P. Thomas, eds.), New York: Wiley, pp. 24.1–24.45. Wilson, H. R., 1975. A synaptic model for spatial frequency adaptation, J. Theoret. Biol., 50:327–352. Wilson, H. R., and R. Humanski, 1993. Spatial frequency adaptation and contrast gain control, Vis. Res., 33:1133–1149. Wuerger, S. M., 1996. Color appearance changes resulting from iso-luminant chromatic adaptation, Vis. Res., 36:3107– 3118. Zaidi, Q., and A. G. Shapiro, 1993. Adaptive orthogonalization of opponent-color signals, Biol. Cybern., 69:415–428. Zaidi, Q., B. Spehar, and J. DeBonet, 1998. Adaptation to textured chromatic fields, J. Opt. Soc. Am. A, 15:23–32. Zhao, L., and C. F. Chubb, 2001. The size-tuning of the facedistortion aftereffect, Vis. Res., 41:2979–2994.

:      

947

61

Color Constancy DAVID H. BRAINARD

C     and describe objects. When giving directions, we might provide the detail that the destination is a yellow house. When judging the ripeness of a fruit, we might evaluate its color. The ability to perceive objects as having a well-defined color is quite remarkable. To understand why, it is necessary to consider how information about object spectral properties is represented in the retinal image. A scene is a set of illuminated objects. In general, the illumination has a complex spatial distribution, so that the illuminant falling on one object in the scene may differ from that falling on another. Nonetheless, a useful point of departure is to consider the case where the illumination is uniform across the scene, so that it may be characterized by its spectral power distribution, E(l). This function specifies how much power the illuminant contains at each wavelength. The illuminant reflects off objects to the eye, where it is collected and focused to form the retinal image. It is the image that is explicitly available for determining the composition of the scene. Object surfaces differ in how they absorb and reflect light. In general, reflection depends on wavelength, the angle of the incident light (relative to the surface normal), and the angle of the reflected light (Foley et al., 1990). It is again useful to simplify and neglect geometric considerations, so that each object surface is characterized by a spectral reflectance function, S(l). This function specifies what fraction of incident illumination is reflected from the object at each wavelength. The light reflected to the eye from each visible scene location is called the color signal. For the simplified imaging model described above, the spectral power distribution of the color signal C(l) is readily calculated from the illuminant spectral power distribution and the surface reflectance function: C (l) = E (l)S (l)

(1)

The retinal image consists of the color signal incident at each location after blurring by the eye’s optics. In the treatment here, such blurring may be safely ignored. The imaging model expressed by equation 1 assumes that the light source is spatially uniform, that the objects are flat and coplanar, and that the surface reflectances are Lambertian. It is sometimes referred to as the Mondrian World imaging model. The assumptions of the Mondrian World never hold for real scenes. A more realistic formulation would include a description of the spatial distribution of the

948

illuminant, the geometry of the scene, and how each object’s spectral reflectance depends on the direction of the incident and reflected light (Foley et al., 1990; also see Fig. 61.2). Nonetheless, the Mondrian World is rich enough to provide a useful framework for initial analysis. The form of equation 1 makes explicit the fact that two distinct physical factors, the illuminant and the surface reflectance, contribute in a symmetric way to the color signal. One of these factors, the surface reflectance, is intrinsic to the object and carries information about its identity and properties. The other factor, the illuminant, is extrinsic to the object and provides no information about the object. Given that the color signal at an image location confounds illuminant and surface properties, how is it possible to perceive objects as having a well-defined color? Indeed, the form of equation 1 suggests that changes in illuminant can masquerade perfectly as changes in object surface reflectance, so that across conditions where the illuminant varies, one might expect large changes in the appearance of a fixed object. This physical process is illustrated by Figure 61.1. Each of the two patches shown at the top of the figure corresponds to a region of a single object imaged under a different outdoor illuminant. When the two patches are seen in isolation, their color appearance is quite different: there is enough variation in natural daylight that the color signal is substantially ambiguous about object surface properties. When the two patches are seen in the context of the images from which they were taken, the variation in color appearance is reduced. This stabilization of appearance is by no means complete in the figure, where the reader views small printed images that are themselves part of a larger illuminated environment. For an observer standing in front of the home shown, however, the variation in perceived color is minimal and not normally noticed. This suggests that the visual system attempts to resolve the ambiguity inherent in the color signal by analyzing many image regions jointly: the full image context is used to produce a stable perceptual representation of object surface color. This ability is referred to as color constancy. This chapter is about human color constancy. The literature on color constancy is vast, extending back at least to the eighteenth century (Mollon, in press), and this chapter does not attempt a systematic review. Rather, the goal is to provide an overview of how human color constancy can be studied

F 61.1. Same objects imaged under two natural illuminants. Top: The patches show a rectangular region extracted from images of the same object under different outdoor illuminants. Bottom: The images from which the patches were taken. Images were acquired

by the author in Merion Station, Pennsylvania, using a Nikon CoolPix 995 digital camera. The automatic white balancing calculation that is a normal part of the camera’s operation was disabled during image acquisition. (See color plate 36.)

experimentally and how it can be understood. The next section presents an extended example of how constancy is measured in the laboratory. The measurements show both circumstances where constancy is good and those where it is not; a characterization of human color vision as either “approximately color constant” or “not very color constant” is too simple. Rather, we must characterize when constancy will be good and when it will not. The discussion outlines two current approaches.

The visual system’s ability to achieve simultaneous constancy need not be easily related to its ability to achieve successive constancy. Indeed, fundamental to simultaneous constancy is some sort of segmentation of the image into regions of common illumination, while such segmentation is not obviously necessary for successive constancy (Adelson, 1999). Often results and experiments about successive and simultaneous constancy are compared and contrasted without explicit acknowledgment that the two may be quite different; keeping the distinction in mind as one considers constancy can reduce confusion. This chapter will focus on successive constancy, as many of the key conceptual issues can be introduced without the extra richness of simultaneous constancy. The discussion returns briefly to simultaneous constancy. At the beginning of the chapter, constancy was cast in terms of the stability of object color appearance, and this is the sense in which the experiments presented below assess it. Some authors (Brainard and Wandell, 1988; D’Zmura and Mangalick, 1994; Foster and Nascimento, 1994; Khang and Zaidi, 2002) have suggested that constancy might be studied through performance (e.g., object identification) rather than through appearance per se. One might expect appearance to play an important role in identification, but reasoning might also be involved. Although the study of constancy using performance-based methods is an interesting direction, this chapter is restricted to measurements and theories of appearance.

Measuring constancy This section illustrates how constancy may be measured by describing experiments conducted by Kraft and Brainard (1999; see also Brainard, 1998; Kraft et al., 2002). Before treating the specific experimental design, however, some general remarks are in order. Several distinct physical processes can cause the illumination impinging on a surface to vary. The images in Figure 61.1 illustrate one such process. They were taken at different times, and the spectra of the illuminant sources changed. Color constancy across illumination changes that occur over time is called successive color constancy. Geometric factors can also cause the illumination impinging on a surface to change. This is illustrated by Figure 61.2. All of the effects shown occur without any change in the spectra of the light sources but instead are induced by the geometry of the light sources and objects. Color constancy across illumination changes that occur within a single scene is called simultaneous color constancy.

:  

949

F 61.2. Image formation. Each set of square patches around the side of the image illustrates variation in the light reflected to the eye when surface reflectance is held fixed. Gradient: The two patches shown were extracted from the upper left (L) and lower right (R; above table) of the back wall of the scene. Shadow: The two patches were extracted from the tabletop in direct illumination (D) and shadow (S). Shape: The three patches shown were extracted from two regions of the sphere (T and B; center top and right bottom, respectively) and from the colored panel directly above the sphere (P; the panel is the leftmost of the four in the bottom row). Both the sphere and the panel have the same simulated surface reflectance function. Pose and indirect illum: The four patches were extracted from the three visible sides of the cube (R, L, and T; right, left, and top visible sides, respectively) and from the left side of the folded paper located between the cube and the sphere (I). The simulated surface reflectances of all sides of the cube and of the left side of the folded paper are identical. The image was rendered from a synthetic scene description using the RADIANCE computer graphics package (Larson and Shakespeare, 1998). There were two sources of illumination in the simulated scene: a diffuse illumination that would appear bluish if viewed in isolation and a directional illumination (from the upper left) that would appear yellowish if viewed in isolation. All of the effects illustrated by this rendering are easily observed in natural scenes. (See color plate 37.)

A E E Figure 61.3 shows the basic experimental setup used by Kraft and Brainard (1999). Subjects viewed a collection of objects contained in an experimental chamber. The chamber illumination was provided by theater lamps. The light from the lamps passed through a diffuser before entering the chamber, so that the overall effect was of a single diffuse illuminant. Each lamp had either a red, green, or blue filter, and by varying the intensities of the individual lamps, the spectrum of the chamber illumination could be varied. Because the light in the chamber was diffuse, the viewing environment provided a rough approximation to Mondrian World conditions. The far wall of the experimental chamber contained a test patch. Physically, this was a surface of low neutral reflectance so that under typical viewing conditions it would have appeared dark gray. The test patch was illuminated by the

950

  

F 61.3. Schematic diagram of the experimental apparatus used in the experiments of Kraft and Brainard. An experimental chamber was illuminated by computer-controlled theater lamps. Different filters were placed over individual lamps, so that by varying their relative intensity the overall spectral power distribution of the chamber illumination could be varied. The light from the lamps was passed through a diffuser, producing a fairly homogeneous illumination. The observer viewed a test patch on the far wall of the chamber. The test patch was illuminated by the ambient chamber illumination and also by a beam from a projection colorimeter. The beam from the colorimeter was not explicitly visible, so that the perceptual effect of varying it was to change the apparent surface color of the test patch.

ambient chamber illumination and also by a separate projector. The projector beam was precisely aligned with the edges of the test patch and consisted of a mixture of red, green, and blue primaries. By varying the amount of each primary in the mixture, the light reflected to the observer from the test patch could be varied independently of the rest of the image. The effect of changing the projected light was to change the color appearance of the test, as if it had been repainted. The apparatus thus functioned to control the effect described by Gelb (1950; see also Katz, 1935; Koffka, 1935), wherein using a hidden light source to illuminate a paper dramatically changes its color appearance. The observer’s task in Kraft and Brainard’s (1999) experiments was to adjust the test patch until it appeared achromatic. Achromatic judgments have been used extensively in the study of color appearance (e.g., Chichilnisky and Wandell, 1996; Helson and Michels, 1948; Werner and Walraven, 1982). During the adjustment, the observer controlled

the chromaticity of the test patch while its luminance was held constant. In essence, the observer chose the test patch chromaticity, which appeared gray when seen in the context set by the rest of the experimental chamber. Whether the test patch appeared light gray or dark gray depended on its luminance. This was held constant during individual adjustments but varied between adjustments. For conditions where the luminance of the test patch is low relative to its surroundings, Brainard (1998) found no dependence of the chromaticity of the achromatic adjustment on test patch luminance. This independence does not hold when more luminous test patches are used (Chichilnisky and Wandell, 1996; Werner and Walraven, 1982). The data from the experiment are conveniently represented using the standard 1931 CIE chromaticity diagram. Technical explanations of this diagram and its basis in visual performance are widely available (e.g., Brainard, 1995; CIE, 1986; Kaiser and Boynton, 1996), but its key aspects are easily summarized. Human vision is trichromatic, so that a light C(l) may be matched by a mixture of three fixed primaries: C (l) ~ XP1 (l) + YP2 (l) + ZP3 (l)

(2)

In this equation P1(l), P2(l), and P3(l) are the spectra of the three primary lights being mixed, and the scalars X, Y, and Z specify the amount of each primary in the mixture. The symbol ~ indicates visual equivalence. When we are concerned with human vision, standardizing a choice of primary spectra allows us to specify a spectrum compactly by its tristimulus coordinates X, Y, and Z. The CIE chromaticity diagram is based on a set of known primaries together with a standard of performance that allows computation of the tristimulus coordinates of any light from its spectrum. The chromaticity diagram, however, represents lights with only two coordinates, x and y. These chromaticity coordinates are simply normalized versions of the tristimulus coordinates: x=

X , X +Y +Z

y=

Y X +Y +Z

(3)

The normalization removes from the representation all information about the overall intensity of the spectrum while preserving the information about the relative spectrum that is relevant for human vision. Figure 61.4 shows data from two experimental conditions. Each condition is defined by the scene within which the test patch was adjusted. The two scenes, labeled Scene 1 and Scene 2, are shown at the top of the figure. The scenes were sparse but had visible three-dimensional structure. The surface lining the chamber was the same in the two scenes, but the spectrum of the illuminant differed. The data plotted for each condition are the chromaticity of the illuminant (open circles) and the chromaticity of the observers’ achromatic adjustments (closed circles).

The points plotted for the illuminant are the chromaticity of the illuminant, as measured at the test patch location when the projection colorimeter was turned off. These represent the chromaticity of the ambient illumination in the chamber, which was approximately uniform. The fact that the illuminant was changed across the two scenes is revealed in the figure by the shift between the open circles. The plotted achromatic points are the chromaticity of the light reflected to the observer when the test appeared achromatic. This light was physically constructed as the superposition of reflected ambient light and reflected light from the projection colorimeter. Across the two scenes, the chromaticity of the achromatic point shifts in a manner roughly commensurate with the shift in illuminant chromaticity. R   D  C What do the data plotted in Figure 61.4 say about color constancy across the change from Scene 1 to Scene 2? A natural but misleading intuition is that the large shift in the achromatic locus shown in the figure reveals a large failure of constancy. This would be true if the data plotted represented directly the physical properties of the surface that appears achromatic. As noted above, however, the data plotted describe the spectrum of the light reaching the observer. To relate the data to constancy, it is necessary to combine information from the measured achromatic points and the illuminant chromaticities. Suppose that the observer perceives the test patch as a surface illuminated with the same ambient illumination as the rest of the chamber. Introspection and some experimental evidence support this assumption (Brainard et al., 1997). The data from Scene 1 can then be used to infer the spectral reflectance of an equivalent surface. The equivalent surface would have appeared achromatic had it been placed at the test patch location with the projection colorimeter turned off. Let the reflectance function of the equivalent surface be S˜(l). This function must be such that the chromaticity of E1(l) S˜(l) is the same as the chromaticity of the measured achromatic point, where E1(l) is the known spectrum of the ambient illuminant in Scene 1. It is straightforward to find functions S˜(l) that satisfy this constraint. The inset to Figure 61.4 shows one such function. The function S˜(l) is referred to as the equivalent surface reflectance corresponding to the measured achromatic point. The equivalent surface reflectance S˜(l) allows us to predict the performance of a color constant observer for other scenes. To a constant observer, any given surface should appear the same when embedded in any scene. More specifically, a surface that appears achromatic in one scene should remain so in others. Given the data for Scene 1, the chromaticity of the achromatic point for a test patch in Scene 2 should be the chromaticity of E2(l)S˜(l), where E2(l)

:  

951

F 61.4. Basic data from an achromatic adjustment experiment. The images at the top of the figure show the observer’s view of two scenes, labeled 1 and 2. The test patch is visible in each image. The projection colorimeter was turned off at the time the images were acquired, so the images do not show the results of observers’ achromatic adjustments. The chromaticity diagram shows the data from achromatic adjustments of the test patch made in the context of the two scenes. The open circles show the chromaticity of the illuminant for each scene. The illuminant for Scene

1 plots to the lower left of the illuminant for Scene 2. The closed circles show the chromaticity of the mean achromatic adjustments of four observers. Where visible, the error bars indicate ±1 standard error. The surface reflectance function plotted in the inset at the right of the figure shows the equivalent surface reflectance S˜(l) computed from the data obtained in Scene 1. The closed diamond shows the color constant prediction for the achromatic adjustment in Scene 2, given the data obtained for Scene 1. See the explanation in the text. (See color plate 38.)

is the spectrum of the illuminant in Scene 2. This prediction is shown in Figure 61.4 by the closed diamond. Although the measured achromatic point for Scene 2 does not agree precisely with the constancy prediction, the deviation is small compared to the deviation that would be measured for an observer who had no constancy whatsoever. For such an observer, the achromatic point would be invariant across changes of scene. Thus, the data shown in Figure 61.4 indicate that observers are approximately color constant across the two scenes studied in the experiment. Brainard (1998) developed a constancy index that quantifies the degree of constancy revealed by data of the sort

presented in Figure 61.4. The index takes on a value of 0 for no adjustment and 1 for perfect constancy, with intermediate values for intermediate performance. For the data shown in Figure 61.4, the constancy index is 0.83. This high value seems consistent with our everyday experience that the colors of objects remain stable over changes of illuminant but that the stability is not perfect.

952

  

A P  I R The introductory section stated that illuminant and surface information is perfectly confounded in the retinal image. The data shown in Figure 61.4 indicate that human vision can separate

E1 (l) Scene 1˜

Scene 1 Scene 2 Image 1 Image 2

Scene 2˜ E 2 (l )

F 61.5. Schematic illustration of the ambiguity inherent in color constancy. The figure shows schematically the set of all scenes. Each point in the schematic represents a possible scene. Scenes 1 and 2 from the experiment described in text are indicated by closed circles. Each shaded ellipse encloses a subset of scenes that all produce~ the same image. The scenes represented by open ~ circles, Scenes 1 and 2, produce the same images as Scenes 1 and 2, respectively. The open ellipses each enclose a subset of scenes that share the same illuminant.

these confounded physical factors and achieve approximate color constancy. This presents a paradox. If the information is perfectly confounded, constancy is impossible. If constancy is impossible, how can the visual system be achieving it? The resolution to this paradox is found by considering restrictions on the set of scenes over which constancy holds. Figure 61.5 shows a schematic diagram of the set of all scenes, represented by the thick outer boundary. Each point within this boundary represents a possible scene, that is, a particular choice of illuminant and surface reflectances. The closed circles represent the two scenes used in the experiment described above. These are labeled Scene 1 and Scene 2 in the figure. Denote the retinal image produced from Scene 1 as Image 1. Many other scenes could have produced this same image. This subset of scenes is indicated in the figure by the shaded ellipse that encloses Scene 1. This ellipse is labeled Image 1 in the figure. It also contains Scene 1˜, indicated by an open circle in the figure. Scenes 1 and 1˜ produce the same image and cannot be distinguished by the visual system. Similarly, there is a separate subset of scenes that produce the same image (denoted Image 2) as Scene 2. This subset is also indicated by a shaded ellipse. A particular scene consistent with Image 2 is indicated by the open circle labeled Scene 2˜. Like Scenes 1 and 1˜, Scenes 2 and 2˜ cannot be distinguished from each other by the visual system.

The open ellipse enclosing each solid circle shows a different subset of scenes to which it belongs. These are scenes that share a common illuminant. The open ellipse enclosing Scene 1 indicates all scenes illuminated by E1(l), while the open ellipse enclosing Scene 2 indicates all scenes illuminated by E2(l). The figure illustrates why constancy is impossible in general. When viewing Image 1, the visual system cannot tell whether Scene 1 or Scene 1˜ is actually present: achromatic points measured for a test patch embedded in these two scenes must be the same, even though the scene illuminants are as different as they are for Scenes 1 and 2. Recall from the data analysis above that this result (no change of achromatic point across a change of illuminant) indicates the absence of constancy. The figure also illustrates why constancy can be shown across some scene pairs. Scenes 1 and 2 produce distinguishable retinal images, so there is no a priori reason for the measured achromatic points for test patches embedded in these two scenes to bear any relation to each other. In particular, there is no constraint that prevents the change in achromatic points across the two scenes from tracking the corresponding illuminant change. Indeed, one interpretation of the good constancy shown by the data reported above is that the visual system infers approximately the correct illuminants for Scenes 1 and 2. A mystery would occur only if it could also infer the correct illuminants for Scenes 1˜ and 2˜. Figure 61.6 replots the results from achromatic measurements made for Scene 1 together with the results for a new ˜ ˜ scene, 1˜. The illuminant in Scene 1˜ is the same as that in Scene 2, but the objects in the scene have been changed to ˜ make the image reflected to the eye for Scene 1˜ highly similar ˜˜ to that reflected for Scene 1; Scene 1 is an experimental ˜ approximation to the idealized Scene 1˜ described above. It would be surprising indeed if constancy were good ˜ when assessed between Scenes 1 and 1˜, and it is not. The ˜ achromatic points measured for Scenes 1 and 1˜ are very similar, with the constancy index between them being 0.11.

Discussion C D   I E S The analysis and data presented above show that the degree of human color constancy depends on the choice of scenes across which it is assessed. Thus, it is not useful to summarize human performance through blanket statements about the degree of constancy obtained. Rather, questions about constancy must be framed in conjunction with a specification of the scene ensemble. Natural questions are (1) what ensembles of scenes support good constancy? and (2) how does constancy vary within some ensemble of scenes which

:  

953

F 61.6. Achromatic data when both illuminant and scene surfaces are varied. The images at the top of the figure show the ˜ observer’s view of two scenes, labeled 1 and 1˜. The relation between these scenes is described in the text. The test patch is visible in each image. The projection colorimeter was turned off at the time the images were acquired, so the images do not show the results of

observers’ achromatic adjustments. The chromaticity diagram shows the data from achromatic adjustments of the test patch made in the context of the two scenes. The format is the same as that of Figure 61.4. The equivalent surface reflectance S˜(l) computed from the data obtained in Scene 1 is shown in Figure 61.4. (See color plate 39.)

is intrinsically of interest? An example of the latter would be scenes that occur in natural viewing. The choice of experimental scenes is a crucial aspect of the design of any constancy experiment. Without some a priori restriction the number of possible scenes is astronomical, and systematic exploration of the effect of all possible stimulus variables is not feasible. In choosing an ensemble of scenes for study, different experimenters have been guided by different intuitions. Indeed, it is this choice that most differentiates various studies. A common rationale, however, is to test specific hypotheses about how constancy might operate. The goal is to develop principles that allow generalization beyond the scenes studied experimentally. Two broad approaches have been pursued. The mechanistic approach is based on the hope that constancy is mediated

by simple visual mechanisms and that these mechanisms can be studied through experiments with simple stimuli (e.g., uniform test patches presented on uniform background fields). The computational approach is to develop image processing algorithms that can achieve color constancy and to use insight gained from the algorithms to build models of human performance. This approach is often characterized by the use of stimuli closer to those encountered in natural viewing, as the algorithms are generally designed to take advantage of the statistical structure of natural images. The difference between the mechanistic and computational approaches is not always clear-cut: a mechanistic theory that explains human constancy can always be recast as a computational algorithm, while the action of a given algorithm can probably be approximated by the action of a series of

954

  

plausible neural mechanisms (see, e.g., Marr, 1982, chapter 1). Examples of both approaches are outlined below. T M A Constancy is essentially a relative phenomenon; it can be assessed only by measuring appearance across two (or more) scenes. We cannot say from the data above that constancy is good for Scenes 1 and 2 but ˜ bad for Scene 1˜. Rather, constancy is good across the change from Scene 1 to Scene 2 but bad across the change from ˜ Scene 1 to Scene 1˜. Presumably it is possible to construct some other Scene 3 such that good constancy is revealed ˜ across Scenes 1˜ and 3. What is it about the relation between Scenes 1 and 2 that supports the good constancy observed? A critical feature is that all that differs between them is the spectrum of the ambient illuminant. This design is common to most studies of constancy—stability of appearance is assessed under conditions where the surfaces comprising the scene are held fixed while the illuminant is varied (e.g., Arend and Reeves, 1986; Brainard and Wandell, 1992; Breneman, 1987; Burnham et al., 1957; Helson and Jeffers, 1940; McCann et al., 1976). It is probably the ubiquity of this surfaces-held-fixed design that leads to the oft-quoted generalization that human vision is approximately color constant (e.g., Boring, 1942). When the surfaces in the image are held constant, it is easy to postulate mechanisms that could, qualitatively at least, support the high levels of observed constancy. The initial encoding of the color signal by the visual system is the absorption of light quanta by photopigment in three classes of cone photoreceptors, the L, M, and S cones (for a fuller treatment see, e.g., Brainard, 1995; Kaiser and Boynton, 1996; Rodieck, 1998). The three classes are distinguished by how their photopigments absorb light as a function of wavelength. The fact that color vision is based on absorptions in three classes of cones is the biological substrate for trichromacy. An alternative to using tristimulus or chromaticity coordinates to represent spectral properties of the light reaching the eye is to use cone excitation coordinates. These are proportional to the quantal absorption rates for the three classes of cones elicited by the light. The cone excitation coordinates for a light, r, may be specified by using a three-dimensional column vector È rL ˘ r = ÍrM ˙ Í ˙ ÍÎ rS ˙˚

(4)

It is well accepted that the signals initiated by quantal absorption are regulated by adaptation. A first-order model of adaptation postulates that (1) the adapted signals are determined from quantal absorption rates through multiplicative gain control; (2) at each retinal location the gains

are set independently within each cone class, so that (e.g.,) signals from M and S cones do not influence the gain of L cones; and (3) for each cone class, the gains are set in inverse proportion to a spatial average of the quantal absorption rates seen by cones of the same class. This model is generally attributed to von Kries (1905/1970). The three postulates together are sometimes referred to as von Kries adaptation. Von Kries recognized that models where some of the postulates hold and others do not could also be considered. The first postulate of von Kries adaptation asserts that for each cone class, there is an adapted cone signal (aL for the L cones, aM for the M cones, and aS for the S cones) that is obtained from the corresponding cone excitation coordinate through multiplication by a gain (e.g., aL = gLrL). This may be expressed using the vector notation introduced in Eq. (4). Let the vector a represent the magnitude of the adapted cone signals. Then È aL ˘ È g L a = Ía M ˙ = Í 0 Í ˙ Í ÎÍ aS ˙˚ ÍÎ 0

0 gM 0

0 ˘ È rL ˘ 0 ˙ ÍrM ˙ = Dr ˙Í ˙ g S ˚˙ ÎÍ rS ˙˚

(5)

Because the adapted cone signals a are obtained from the cone excitation coordinates r through multiplication by the diagonal matrix D, this postulate is called the diagonal model for adaptation. It should be emphasized that for the diagonal model to have predictive power, all of the effect of context on color processing should be captured by Eq. (5). In this model, two test patches that have the same adapted cone signals should have the same appearance. In general, it is conceptually useful to separate two components of a model of adaptation (Brainard and Wandell, 1992; Krantz, 1968). The first component specifies what parameters of a visual processing model are allowed to vary with adaptation. The diagonal model provides this component of the full von Kries model. In the diagonal model, the only parameters that can vary are the three gains. The second component of a full model specifies how the processing parameters are determined by the image. The diagonal model is silent about this, but the issue is addressed by the second two assumptions of the full von Kries model. Only cone excitation coordinates within a cone class influence the gain for that cone class, and the specific form of the influence is that the gain at a location is set inversely proportional to the mean excitation in a neighborhood of the location. If the visual system implements von Kries adaptation, the adapted cone signals coding the light reflected from a surface are considerably stabilized across illuminant variation, provided that the other surfaces in the scene also remain fixed (Brainard and Wandell, 1986; Foster and Nascimento, 1994; Lennie and D’Zmura, 1988; see also Finlayson et al., 1994). Indeed, von Kries adaptation is the active ingredient

:  

955

in later versions of Land’s popular retinex account of successive color constancy. In the descriptions of the retinex algorithm, the adapted cone signals are called lightness designators, and these are derived from cone excitations through elaborate calculation. Nonetheless, for successive constancy the calculation reduces to a close approximation to classic von Kries adaptation (Land, 1986; see Brainard and Wandell, 1986; for early descriptions of Land’s work see Land, 1959a, 1959b; Land and McCann, 1971). Qualitatively, then, von Kries adaptation can explain the good constancy shown in experiments where the illuminant is changed and the surfaces in the scene are held fixed. Such adaptation also provides a qualitative account for the poor constancy shown by the data in Figure 61.6, where both the illuminant and surfaces in the scene were changed to hold the image approximately constant. On the basis of the data presented so far, one might sensibly entertain the notion that human color constancy is a consequence of early adaptive gain control. Each of the postulates of von Kries adaptation have been subjected to sharply focused empirical test, and it is clear that each fails when examined closely. With respect to the diagonal model, a number of experimental results suggest that there must be additional adaptation that is not described by Eq. (5). These effects include gain control at neural sites located after signals from different cone classes combine and signal regulation characterized by a subtractive process rather than multiplicative gain control (e.g., Hurvich and Jameson, 1958; Jameson and Hurvich, 1964; Poirson and Wandell, 1993; Shevell, 1978; Walraven, 1976; see also Eskew et al., 1999; Webster, 1996). The diagonal model fails when probed with stimuli carefully crafted to test its assumptions. Does it also fail for natural scenes? The illuminant spectral power distributions E(l) and surface spectral reflectance functions S(l) found in natural scenes are not arbitrary. Rather, these functions tend to vary smoothly as a function of wavelength. This constraint restricts the range of color signals likely to occur in natural scenes. Conditions that elicit performance in contradiction to the diagonal model may not occur for natural scenes. If so, the diagonal model would remain a good choice for studies of how adaptive parameters vary within this restricted domain. The regularity of illuminant and surface spectral functions may be captured with the use of small-dimensional linear models (e.g., Cohen, 1964; Jaaskelainen et al., 1990; Judd et al., 1964; Maloney, 1986). The idea of a linear model is simple. The model is defined by N basis functions. These are fixed functions of wavelength, E1(l), . . . , EN(l). Any spectrum E(l) is approximated within the linear model by a weighted sum of the basis functions E˜ (l) = w1E1 (l) + . . . + w N E N (l) (6)

956

  

F 61.7. Linear model for natural daylights. Top: Three basis functions for the CIE linear model for daylights (CIE, 1986). Bottom: Linear model approximation to measured daylight. Solid line, measurement. Dotted line, reconstruction. The measured daylight was obtained from a database of measurements made available by J. Parkkinen and P. Silfsten on the World Wide Web at http://cs.joensuu.fi/~spectral/databases/download/daylight. htm.

where the weights are chosen to minimize the approximation error. Figure 61.7 plots the basis functions of a threedimensional linear model for natural daylight and shows the linear model approximation to a measured daylight. The same approach can be used to express constraints on surface reflectance functions. The use of linear models has been central to computational work on color constancy (Brainard and Freeman, 1997; Brainard et al., in press; Maloney, 1999). Mondrian World scenes where the illuminants and surface spectra are restricted to be typical of naturally occurring spectra can be referred to as Restricted Mondrian World scenes. When the experimental scenes are from the Restricted Mondrian World, the diagonal model seems to provide a good description of performance. Brainard and Wandell (1992) tested the diagonal model using asymmetric matching. In asymmetric matching, the observer adjusts a matching patch, embedded in one scene, so that its appearance matches that of a test patch presented in another. In the context of the diagonal model, the match is taken to indicate two lights that elicit identical adapted cone signals. Let at and am represent the adapted cone signals for the test and matching patches. Equation (5) then yields -1

a t = Dt r t = a m = D m r m fi r m = [D m ] [Dt ]r t = DtÆm r t tÆm

where the diagonal matrix D

is

(7)

t

DtÆm

È gL Ígm Í L =Í 0 Í Í Í0 ÍÎ

0 g tM g Mm 0

˘ 0 ˙ ˙ 0 ˙ ˙ t ˙ gS ˙ g Sm ˙˚

(8)

If the diagonal model is correct, the cone excitation coordinates of the test and match patches are related by a diagonal matrix whose entries are the ratios of cone gains. A single asymmetric match determines the entries of the matrix DtÆm. Since Eq. (7) must hold with the same matrix DtÆm for any choice of test patch cone coordinates, repeating the experiment with different test patches allows evaluation of the diagonal model. In a successive matching experiment that employed simple synthetic scenes, Brainard and Wandell (1992) found that a large set of asymmetric matching data was in good agreement with the diagonal model. Similar results were obtained by Bauml (1995) also for successive matching and by Brainard et al. (1997) for simultaneous matching. To develop a theory of color constancy that applies to natural viewing, violations of the diagonal model may be small enough to neglect. What about the other postulates of von Kries adaptation, which concern how the gains are set by image context? The notion that the gains are set as a function of a spatial average of the image has been tested in a number of ways. One approach is to examine the equivalent background hypothesis. In its general form, this hypothesis asserts that the effect of any image on the color appearance of a test light is the same as that of some uniform background. If the gains are set by a spatial average of the image, then the equivalent background hypothesis must also hold. Precise tests of the equivalent background hypothesis where spatial variation of color is introduced into a uniform background, indicate that it fails (Brown and MacLeod, 1997; Shevell and Wei, 1998). The general logic is first to find a uniform field and a spatially variegated field that have an identical effect on the appearance of a single test light and then show that for some other test light these two contexts have different effects (Stiles and Crawford, 1932). Like the sharpest tests of the diagonal model, however, these studies did not employ stimuli from the Restricted Mondrian World. Kraft and Brainard (1999) examined whether the spatial mean of an image controls the state of adaptation by constructing two illuminated scenes that had the same spatial mean. To equate the mean, both the illuminant and the scene surfaces were varied between the scenes. Kraft and Brainard then measured the achromatic loci in the two scenes. If the spatial mean were the only factor controlling adaptation, then the achromatic point in the two scenes

should have been the same. A more general constancy mechanism, however, might be able to detect the change in the illuminant based on some other aspect of the images. The achromatic points were distinct, with a mean constancy index of 0.39. Even for nearly natural scenes, control of the gains is not a simple function of the spatial mean of the image: the visual system has access to additional cues to the illuminant. Kraft and Brainard examined other simple hypotheses about control of adaptation in nearly natural scenes and found that none accounted for the data. A key feature in Kraft and Brainard’s (1999) design (see also Gilchrist and Jacobson, 1984; Kraft et al., 2002; McCann, 1994) is that both the illuminant and the surfaces in the scene were varied. When only the illuminant is varied, the data are roughly consistent with adaptation to the spatial mean of the image. Such data do not provide a sharp test, however, since essentially all plausible hypotheses predict good constancy when the surfaces in the image are held fixed across an illuminant change. Only by varying the surfaces and illuminants to differentiate predictions can strong tests be made. This point also applies to studies of the neural locus of constancy. The current state of affairs for the mechanistic approach may be summarized roughly as follows. A gain control model provides a reasonable approximation to performance measured in scenes consisting of illuminated surfaces, but lacking is a theory that links the gains to the image. The agenda is to understand what image factors control the state of adaptation. In the lightness literature, this is sometimes referred to as the anchoring problem (e.g., Gilchrist et al., 1999). Within the mechanistic approach, one recent theme is to study the influence of image contrast (Brown and MacLeod, 1997; Golz and MacLeod, 2002; Krauskopf et al., 1982; Shevell and Wei, 1998; Singer and D’Zmura, 1995; Webster and Mollon, 1991) and spatial frequency content (Bauml and Wandell, 1996; Poirson and Wandell, 1993). Another approach (Bauml, 1995; Brainard and Wandell, 1992; Chichilnisky and Wandell, 1995) is to study rules of combination (e.g., linearity) that allow prediction of parameter values for many images on the basis of measurements made for just a few. T C A The mechanistic approach is motivated by consideration of the physiology and anatomy of the visual pathways. The computational approach begins with consideration about how one could, in principle, process the retinal image to produce a stable representation of surface color. The computational approach focuses on the information contained in the image rather than on the specific operation of mechanisms that extract the information. Computational algorithms often operate in two distinct steps (Maloney, 1999). The first step estimates the illuminant

:  

957

at each image location, while the second step uses the estimate to transform the cone coordinates at each location to an illuminant-invariant representation. Given linear model constraints on natural surface reflectance functions, the second step is quite straightforward (Buchsbaum, 1980) and is well approximated by diagonal gain control (Brainard and Wandell, 1986; Foster and Nascimento, 1994). The deep issue is what aspects of the image carry useful information about the illuminant. This issue is completely analogous to the central issue within the mechanistic approach, namely, what aspects of the image control adaptation. Indeed, the idea linking the computational algorithms to measured human performance is that measured adaptation might be governed by the same image statistics that provide information about the illuminant (Brainard et al., in press; Maloney, 1999). Many algorithms have been proposed for estimating the illuminant from the image (Brainard and Freeman, 1997; Buchsbaum, 1980; D’Zmura and Iverson, 1993; D’Zmura et al., 1995; Finlayson et al., 1997; Forsyth, 1990; Funt and Drew, 1988; Lee, 1986; Maloney and Wandell, 1986). A detailed review of the individual algorithms is beyond the scope of this chapter, but excellent reviews are available (Hurlbert, 1998; Maloney, 1999). Common across algorithms is the general approach of specifying assumptions that restrict the class of scenes and then showing how it is possible to estimate the illuminant within the restricted class. With reference to Figure 61.5, each algorithm is based on a rule for choosing one particular scene from within each shaded ellipse. In practice, different proposed algorithms depend on different image statistics. For example, in Buchsbaum’s (1980) classic algorithm, the illuminant estimate was based on the spatial mean of the cone quantal absorption rates. As a model for human performance, this algorithm may be tested by asking whether adaptation is governed only by the spatial mean. As described above, experiments show that this is not the case. The detailed logic connecting this algorithm to human performance is described in a recent review (Brainard et al., in press). Other computational algorithms depend on different aspects of the image. For example, Lee (1986; see also D’Zmura and Lennie, 1986) showed that specular highlights in an image carry information about the illuminant. This has led to tests of whether human vision takes advantage of the information contained in specular highlights (Hurlbert et al., 1989; Yang and Maloney, 2001). In Yang and Maloney’s (2001) work, the stimuli consisted of realistic computer graphics renderings of synthetic scenes. That is, the locations, spectral properties, and geometric properties of the scene illuminants and surfaces were specified in software, and a physics-based rendering algorithm was used to generate the stimuli. In real scenes, the

958

  

information provided by separate cues tends to covary, which makes it difficult to separate their effects. By using synthetic imagery, Yang and Maloney teased apart the effects of independent cues. They were able to show that specular highlights can influence human judgments of surface color appearance and to begin to delineate the circumstances under which this happens. Delahunt (2001) employed similar techniques to study the role of prior information about natural daylights in successive color constancy. (For computational analysis of the use of such prior information, see Brainard and Freeman, 1997; D’Zmura et al., 1995). The methodology promises to allow systematic study of a variety of hypotheses extracted from the computational literature. G  S C This chapter has focused on successive color constancy, and in particular on the case where the illuminant is approximately uniform across the scene. As illustrated by Figure 61.2, this idealized situation does not hold for natural scenes. When an image arises from a scene with multiple illuminants, one can still consider the problem of successive color constancy. That is, one can ask what happens to the color appearance of an object in the scene when the spectral properties of the illuminant are changed without a change in scene geometry. Little, if any, experimental effort has been devoted to this question. The case of spatially rich illumination also raises the question of simultaneous constancy—how similar does the same object appear when located at different places within the scene? One thread of the literature has emphasized the role of scene geometry (Bloj and Hurlbert, 2002; Bloj et al., 1999; Epstein, 1961; Flock and Freedberg, 1970; Gilchrist, 1977, 1980; Hochberg and Beck, 1954; Knill and Kersten, 1991; Pessoa et al., 1996). Under some conditions, the perceived orientation of a surface in a scene can influence its apparent lightness and color in a manner that promotes constancy. The range of conditions under which this happens, however, is not currently well understood. An interesting aspect of simultaneous constancy is that the observer’s performance can depend heavily on experimental instructions. In a study of simultaneous color constancy, Arend and Reeves (1986) had observers adjust the color of one region of a stimulus display until it appeared the same as another. They found that observers’ matches varied with whether they were asked to judge the color of the reflected light or the color of the underlying surface. More constancy was shown when observers were asked to judge the surface (see also Bauml, 1999; Bloj and Hurlbert, 2002). In a study of successive constancy, on the other hand, Delahunt (2001) found only a small instructional effect. It is not yet clear what conditions support the instructional dichotomy, or whether

the dichotomy indicates dual perceptual representations or observers’ ability to reason from appearance to identity. Recent theories of lightness perception have emphasized simultaneous constancy (Adelson, 1999; Gilchrist et al., 1999). At the core of these theories is that idea that perception of lightness (and presumably color) proceeds in two basic stages. First, the visual system segments the scene into separate regions. Second, image data within regions are used to set the state of adaptation for that region. (At a more detailed level, the theories also allow for some interaction between the states of adaptation in different regions.) The two-stage conception provides one way that results for successive constancy might generalize to handle simultaneous constancy: models that explain successive constancy for uniformly illuminated scenes might also describe the processes that set the state of adaptation within separately segmented regions within a single image (Adelson, 1999). To the extent that this hypothesis holds, it suggests that work on simultaneous constancy should focus on the segmentation process. At the same time, it must be recognized that the segment-estimate hypothesis is not the only computational alternative (see, e.g., Adelson and Pentland, 1996; Funt and Drew, 1988; Land and McCann, 1971; Zaidi, 1998) and that empirical tests of the general idea should also be given high priority.

Acknowledgments I thank P. Delahunt, B. Wandell, and J. Werner for discussion and for comments on draft versions of this chapter. This work was supported by National Eye Institute Grant EY 10016. REFERENCES Adelson, E. H., 1999. Lightness perception and lightness illusions, in The New Cognitive Neurosciences, 2nd ed. (M. Gazzaniga ed.), Cambridge, MA: MIT Press, pp. 339–351. Adelson, E. H., and A. P. Pentland, 1996. The perception of shading and reflectance, in Visual Perception: Computation and Psychophysics (D. Knill and W. Richards, eds.), New York: Cambridge University Press, pp. 409–423. Arend, L. E., and A. Reeves, 1986. Simultaneous color constancy, J. Opt. Soc. Am. A, 3:1743–1751. Bauml, K. H., 1995. Illuminant changes under different surface collections: examining some principles of color appearance, J. Opt. Soc. Am. A, 12:261–271. Bauml, K. H., 1999. Simultaneous color constancy: how surface color perception varies with the illuminant, Vis. Res., 39:1531–1550. Bauml, K. H., and B. A. Wandell, 1996. Color appearance of mixture gratings, Vis. Res., 36:2849–2864. Bloj, M. G., and A. C. Hurlbert, 2002. An empirical study of the traditional Mach card effect, Perception, 31:233–246. Bloj, M., D. Kersten, and A. C. Hurlbert, 1999. Perception of three-dimensional shape influences colour perception through mutual illumination, Nature, 402:877–879.

Boring, E. G., 1942. Sensation and Perception in the History of Experimental Psychology, New York: D. Appleton Century. Brainard, D. H., 1995. Colorimetry, in Handbook of Optics, vol. 1, Fundamentals, Techniques, and Design (M. Bass ed.), New York: McGraw-Hill, pp. 26.1–26.54. Brainard, D. H., 1998. Color constancy in the nearly natural image. 2. Achromatic loci, J. Opt. Soc. Am. A, 15:307–325. Brainard, D. H., W. A. Brunt, and J. M. Speigle, 1997. Color constancy in the nearly natural image. 1. Asymmetric matches, J. Opt. Soc. Am. A, 14:2091–2110. Brainard, D. H., and W. T. Freeman, 1997. Bayesian color constancy, J. Opt. Soc. Am. A, 14:1393–1411. Brainard, D. H., J. M. Kraft, and P. Longère (in press). Color constancy: developing empirical tests of computational models, in Colour Perception: From Light to Object (R. Mausfeld and D. Heyer, eds.), Oxford: Oxford University Press. Brainard, D. H., and B. A. Wandell, 1986. Analysis of the retinex theory of color vision, J. Opt. Soc. Am. A, 3:1651–1661. Brainard, D. H., and B. A. Wandell, 1988. Classification measurement of color appearance, Invest. Opthalmol. Vis. Sci., Suppl., 29:162. Brainard, D. H., and B. A. Wandell, 1992. Asymmetric colormatching: how color appearance depends on the illuminant, J. Opt. Soc. Am. A, 9:1433–1448. Breneman, E. J., 1987. Corresponding chromaticities for different states of adaptation to complex visual fields, J. Opt. Soc. Am. A, 4:1115–1129. Brown, R. O., and D. I. A. MacLeod, 1997. Color appearance depends on the variance of surround colors, Curr Biol., 7:844–849. Buchsbaum, G., 1980. A spatial processor model for object colour perception, J. Franklin Inst., 310:1–26. Burnham, R. W., R. M. Evans, and S. M. Newhall, 1957. Prediction of color appearance with different adaptation illuminations, J. Opt. Soc. Am., 47:35–42. Chichilnisky, E. J., and B. A. Wandell, 1995. Photoreceptor sensitivity changes explain color appearance shifts induced by large uniform backgrounds in dichoptic matching, Vis. Res., 35:239–254. Chichilnisky, E. J., and B. A. Wandell, 1996. Seeing gray through the on and off pathways, Vis. Neurosci., 13:591–596. CIE, 1986. Colorimetry, 2nd ed. (CIE Pub. 15.2), Vienna, Bureau Central de la CIE. Cohen, J., 1964. Dependency of the spectral reflectance curves of the Munsell color chips, Psychon. Sci., 1:369–370. Delahunt, P. B., 2001. An evaluation of color constancy across illumination and mutual reflection changes. Unpublished Ph.D. thesis, Uniueisity of California at Santa Barbara, Santa Barbara. D’Zmura, M., and G. Iverson, 1993. Color constancy. I. Basic theory of two-stage linear recovery of spectral descriptions for lights and surfaces, J. Opt. Soc. Am. A, 10:2148–2165. D’Zmura, M., G. Iverson, and B. Singer, 1995. Probabilistic color constancy. in Geometric Representations of Perceptual Phenomena: Papers in Honor of Tarow Indow’s 70th Birthday (R. D. Luce, M. D’Zmura, D. Hoffman, G. Iverson, and A. K. Romney, eds.), Mahwah, NJ: Erlbaum, pp. 187–202. D’Zmura, M., and P. Lennie, 1986. Mechanisms of color constancy, J. Opt. Soc. Am. A, 3:1662–1672. D’Zmura, M., and A. Mangalick, 1994. Detection of contrary chromatic change, J. Opt. Soc. Am. A, 11:543–546. Epstein, W., 1961. Phenomenal orientation and perceived achromatic color, J. Psychol., 52:51–53.

:  

959

Eskew, R. T., J. S. McLellan, and F. Giulianini, 1999. Chromatic detection and discrimination, in Color Vision: From Molecular Genetics to Perception (K. Gegenfurtner and L. T. Sharpe, eds.), Cambridge: Cambridge University Press, pp. 345–368. Finlayson, G. D., M. S. Drew, and B. V. Funt, 1994. Color constancy—generalized diagonal transforms suffice, J. Opt. Soc. Am. A, 11:3011–3019. Finlayson, G. D., P. H. Hubel, and S. Hordley, 1997. Color by correlation. Paper presented at the IS&T/SID Fifth Color Imaging Conference. Scottsdale, AZ: Color Science, Systems, and Applications. Flock, H. R., and E. Freedberg, 1970. Perceived angle of incidence and achromatic surface color, Percept. Psychophys., 8:251–256. Foley, J. D., A. van Dam, S. K. Feiner, and J. F. Hughes, 1990. Computer Graphics: Principles and Practice, 2nd ed. Reading, MA: Addison-Wesley. Forsyth, D. A., 1990. A novel algorithm for color constancy, Int. J. Comput. Vis., 5:5–36. Foster, D. H., and S. M. C. Nascimento, 1994. Relational colour constancy from invariant cone-excitation ratios, Proc. R. Soc. Lond. B, 257:115–121. Funt, B. V., and M. S. Drew, 1988. Color constancy computation in near-Mondrian scenes using a finite dimensional linear model. Paper presented at the IEEE Computer Vision and Pattern Recognition Conference, Ann Arbor, MI. Gelb, A., 1950. Colour constancy, in Source Book of Gestalt Psychology (W. D. Ellis ed.), New York: Humanities Press, pp. 196–209. Gilchrist, A., and A. Jacobsen, 1984. Perception of lightness and illumination in a world of one reflectance, Perception, 13:5– 19. Gilchrist, A. L., 1977. Perceived lightness depends on perceived spatial arrangement, Science, 195:185. Gilchrist, A. L., 1980. When does perceived lightness depend on perceived spatial arrangement? Percept. Psychophys., 28:527–538. Gilchrist, A. L., C. Kossyfidis, F. Bonato, T. Agostini, J. Cataliotti, X. Li, B. Spehar, V. Annan, and E. Economou, 1999. An anchoring theory of lightness perception, Psychol. Rev., 106:795–834. Golz, J., and D. I. A. MacLeod, 2002. Influence of scene statistics on colour constancy, Nature, 415:637–640. Helson, H., and V. B. Jeffers, 1940. Fundamental problems in color vision. II. Hue, lightness, and saturation of selective samples in chromatic illumination, J. Exp. Psychol., 26:1–27. Helson, H., and W. C. Michels, 1948. The effect of chromatic adaptation on achromaticity, J. Opt. Soc. Am., 38:1025–1032. Hochberg, J. E., and J. Beck, 1954. Apparent spatial arrangement and perceived brightness, J. Exp. Psychol., 47:263–266. Hurlbert, A. C., 1998. Computational models of color constancy, in Perceptual Constancy: Why Things Look as They Do (V. Walsh and J. Kulikowrki, eds.), Cambridge: Cambridge University Press, pp. 283–322. Hurlbert, A. C., H. Lee, and H. H. Bulthoff, 1989. Cues to the color of the illuminant, Invest. Ophthalmol. Vis. Sci., Suppl., 30:221. Hurvich, L. M., and D. Jameson, 1958. Further development of a quantified opponent-color theory, in Visual Problems of Colour II, London: HMSO, pp. 693–723. Jaaskelainen, T., J. Parkkinen, and S. Toyooka, 1990. A vectorsubspace model for color representation, J. Opt. Soc. Am. A, 7:725–730. Jameson, D. B., and L. M. Hurvich, 1964. Theory of brightness and color contrast in human vision, Vis. Res., 4:135–154. Judd, D. B., D. L. MacAdam, and G. W. Wyszecki, 1964. Spectral distribution of typical daylight as a function of correlated color temperature, J. Opt. Soc. Am., 54:1031–1040.

960

  

Kaiser, P. K., and R. M. Boynton, 1996. Human Color Vision, 2nd ed., Washington, DC: Optical Society of America. Katz, D., 1935. The World of Colour (R. B. MacLeod and C. W. Fox, trans.), London: Kegan, Paul, Trench Truber and Co. Khang, B. G., and Q. Zaidi, 2002. Cues and strategies for color constancy: perceptual scission, image junctions and transformational color matching, Vis. Res., 42:211–226. Knill, D. C., and D. Kersten, 1991. Apparent surface curvature affects lightness perception, Nature, 351:228–230. Koffka, K., 1935. Principles of Gestalt Psychology, New York: Harcourt, Brace. Kraft, J. M., and D. H. Brainard, 1999. Mechanisms of color constancy under nearly natural viewing, Proc. Natl. Acad. Sci. USA, 96:307–312. Kraft, J. M., S. I. Maloney, and D. H. Brainard, 2002. Surfaceilluminant ambiguity and color constancy: effects of scene complexity and depth cues, Perception, 31:247–263. Krantz, D., 1968. A theory of context effects based on crosscontext matching, J. Math. Psychol., 5:1–48. Krauskopf, J., D. R. Williams, and D. W. Heeley, 1982. Cardinal directions of color space, Vis. Res., 22:1123–1131. Land, E. H., 1959a. Color vision and the natural image, part I, Proc. Natl. Acad. Sci. USA, 45:116–129. Land, E. H., 1959b. Color vision and the natural image, part II, Proc. Natl. Acad. Sci. USA, 45:636–644. Land, E. H., 1986. Recent advances in retinex theory, Vis. Res., 26:7–21. Land, E. H., and J. J. McCann, 1971. Lightness and retinex theory, J. Opt. Soc. Am., 61:1–11. Larson, G. W., and R. Shakespeare, 1998. Rendering with Radiance: The Art and Science of Lighting Visualization, San Francisco: Morgan Kaufman. Lee, H., 1986. Method for computing the scene-illuminant chromaticity from specular highlights, J. Opt. Soc. Am. A, 3:1694–1699. Lennie, P., and M. D’Zmura, 1988. Mechanisms of color vision, CRC Crit. Rev. Neurobiol., 3:333–400. Maloney, L. T., 1986. Evaluation of linear models of surface spectral reflectance with small numbers of parameters, J. Opt. Soc. Am. A, 3:1673–1683. Maloney, L. T., 1999. Physics-based approaches to modeling surface color perception, in Color Vision: From Genes to Perception (K. T. Segenburtned and L. T. Sharpe, eds.), Cambridge: Cambridge University Press, pp. 387–416. Maloney, L. T., and B. A. Wandell, 1986. Color constancy: a method for recovering surface spectral reflectances, J. Opt. Soc. Am. A, 3:29–33. Marr, D., 1982. Vision, San Francisco: W. H. Freeman. McCann, J. J, 1994. Psychophysical experiments in search of adaptation and the gray world. Paper presented at the IS&T’s 47th Annual Conference, Rochester, NY. McCann, J. J., S. P. McKee, and T. H. Taylor, 1976. Quantitative studies in retinex theory: a comparison between theoretical predictions and observer responses to the “Color Mondrian” experiments, Vis. Res., 16:445–458. Mollon, J. D., in press. The origins of modern color science, in The Science of Color, 2nd ed. (S. K. Shevell ed.), Optical Society of America. Pessoa, L., E. Mingolla, and L. E. Arend, 1996. The perception of lightness in 3-D curved objects, Percept. Psychophys., 58(8): 1293–1305. Poirson, A. B., and B. A. Wandell, 1993. Appearance of colored patterns—pattern color separability, J. Opt. Soc. Am. A, 10:2458–2470.

Rodieck, R. W., 1998. The First Steps In Seeing, Sunderland, MA: Sinauer. Shevell, S. K., 1978. The dual role of chromatic backgrounds in color perception, Vis. Res., 18:1649–1661. Shevell, S. K., and J. Wei, 1998. Chromatic induction: border contrast or adaptation to surrounding light? Vis. Res., 38:1561–1566. Singer, B., and M. D’Zmura, 1995. Contrast gain control—a bilinear model for chromatic selectivity, J. Opt. Soc. Am. A, 12:667–685. Stiles, W. S., and B. H. Crawford, 1932. Equivalent adaptation levels in localised retinal areas. Paper presented at the Report of Discussions of the Vision Physiology Society of London. von Kries, J., 1905/1970. Influence of adaptation on the effects produced by luminous stimuli, in Sources of Color Vision (D. L. MacAdam ed.), Cambridge, MA: MIT Press.

Walraven, J., 1976. Discounting the background: the missing link in the explanation of chromatic induction, Vis. Res., 16:289–295. Webster, M. A., 1996. Human colour perception and its adaptation, Network: Comput. Neural Syst., 7:587–634. Webster, M. A., and J. D. Mollon, 1991. Changes in colour appearance following post-receptoral adaptation, Nature, 349:235– 238. Werner, J. S., and J. Walraven, 1982. Effect of chromatic adaptation on the achromatic locus: the role of contrast, luminance and background color, Vis. Res., 22(8):929–944. Yang, J. N., and L. T. Maloney, 2001. Illuminant cues in surface color perception: tests of three candidate cues, Vis. Res., 41:2581–2600. Zaidi, Q., 1998. Identification of illuminant and object colors: heuristic-based algorithms, J. Opt. Soc. Am. A, 15:1767– 1776.

:  

961

62

Comparative Color Vision GERALD H. JACOBS

F    of natural conditions, the overall photon flux and the distribution of spectral energies reaching the retina vary both spatially and temporally. These variations are the raw material that animals use in the conversion of light to sight. Depending on the nature of the recipient visual system, different features of the light signal can be exploited to yield vision. Specific organizations that allow nervous systems to analyze differences in the distribution of spectral energy have evolved in many different taxa. In most cases, the result is an animal that has some color vision. There are significant variations in the acuteness and nature of this capacity across species, and this chapter underlines that fact by considering the nature, distribution, evolution, and utility of color vision in different animals.

Studying animal color vision: an example The defining conditions for formal descriptions of color vision, as well as many of the techniques used in testing, were developed from studies of human color vision. To provide a reminder of some of the basic features of color vision, and to illustrate how they can be evaluated in studies of color vision in nonhuman subjects, we consider first an example of such an investigation—a study of color vision in the domestic dog. Human subjects can simply be asked to say whether lights, surfaces, or objects appear the same or different, or they can be directed to provide ordered descriptions of their perceptions. Establishing a dialog with a nonhuman subject typically requires either a training regimen, the goal of which is to establish a linkage between a visual stimulus and a response or, alternatively, the examination of some naturally occurring behavior in circumstances where the visual stimuli can be specified. With a compliant subject like the dog, effective communication can be easily initiated using operant conditioning procedures (Neitz et al., 1989). Figure 62.1 summarizes results that characterize several features of dog visual performance. A basic feature is spectral sensitivity (Fig. 62.1A), in which sensitivity to lights of different wavelength content is established. In the dog, this was assessed by progressively decreasing the intensity of a monochromatic light until its presence could no longer be discriminated from a spectrally broadband light to which it had been added. This is not strictly a test of color vision, but the character of the function yields inferences about the

962

nature of the visual system that can, in turn, be linked to color vision. Here the appearance of a sharp decline in sensitivity at a point in the spectrum that separates two regions of higher sensitivity indicates the presence of two different kinds of photopigments in the eye, and it implies that signals initiated by these two have been combined in a spectrally opponent manner. The latter is a hallmark of the neural organization for color vision in a wide variety of species, so its characteristic signature in the dog’s spectral sensitivity function is strongly suggestive of the presence of color vision (Chittka et al., 1992; Jacobs, 1981). Color vision means that an animal can independently process wavelength and intensity information, and it is typically established by asking an animal to discriminate between lights that consistently differ only in their spectral energy distributions. A key in color vision tests of this sort is to make it impossible for animals to use any additional perceptual cues. The results of such a color vision test in which three dogs were required to discriminate between various monochromatic lights and spectrally broadband lights are shown in Figure 62.1B. For most such combinations discrimination performance was nearly perfect, so dogs must have color vision. Note, however, that they failed the test when they were asked to discriminate a narrow band of wavelengths centered at about 480 nm. Such failure means that their color vision is of a particular type. About 1% of all humans experience a similar failure and, like the dog, they are defined as having dichromatic color vision. Most people, however, can easily discriminate all of these spectral lights from the broadband light. These individuals are classified as trichromatic, and thus the normative color vision of humans and dogs is discretely different. The failure of dogs to see a difference between a 480 nm light and one that contains all spectral wavelengths reveals a fundamental feature of color vision in all animals—that stimuli having quite different spectral energy distributions may appear the same. Such perceptual identities constitute color matches, and the analysis of such matches has proven central to understanding the nature of color vision. Another color matching experiment was conducted in which dogs were asked to discriminate various additive mixtures of 500 nm and 440 nm lights from a 480 nm light. The result was that, for most proportions, the mixture of 500 nm and 440 nm lights appeared different to dogs than 480 nm lights, but for a particular ratio of 500 nm and 440 nm there was a

F 62.1. A compilation of measurements relevant to understanding dog color vision. A, spectral sensitivity. B, Color discrimination test. C, Cone pigment absorption spectra. D, Wavelength discrimination. The curves in A and C are normalized to have peak values of 1.0. The details of each of these curves are discussed in the text. (Data taken from Neitz et al., 1989.)

failure of discrimination, so that combination is said to match in color the 480 nm light. The relative amounts of the two mixture lights at this point define a color matching equation to the 480 nm light. These perceptual identities are conceptually powerful because from them inferences can be drawn about the spectral properties of the underlying cone photopigments. The nature of the color matches indicated that the two cone photopigments in the retina that support dichromatic color vision have spectral peaks at about 429 nm and 555 nm, respectively (Fig. 62.1C). Later direct measurements of dog cone pigments showed that these peak estimates were quite accurate ( Jacobs et al., 1993). The tests described established that the dog has color vision, identified its dimensionality (dichromatic), and yielded some strong indications of important features of the biology of vision in this species. But how acute is their color vision? The acuteness of color vision can be assayed in a number of different ways. Shown in Figure 62.1D are results from one such assessment, measurements of wavelength discrimination in which a determination was made of the size of the wavelength change (Dl) required for successful discrimination at various locations in the spectrum. The results indi-

cate that at one point, around 480 nm, dogs have quite acute color vision, with differences of 5 nm or less required for successful discrimination. Away from that point discrimination quickly worsens, with the result that dogs are quite blind to wavelength differences over much of the middle and long wavelength portions of the spectrum. Human dichromats who behave in a similar fashion are often characterized as being “red/green color blind.” This example illustrates how a number of basic features of color vision can be assessed in nonhuman species. These particular laboratory tests, which were easily accomplished for a common mammal, may be difficult or even impossible to apply to other species. In such cases, a range of other behavioral indices can be used. For example, insects like honeybees and wasps visit flowers to harvest nectar and pollen, and this natural behavior has often been exploited in controlled tests to examine insect color vision (Menzel and Backhaus, 1991). Whatever the technique, however, the goals of all animal color vision tests are usually quite similar: to assess the presence of color vision, to determine its dimensionality and acuteness, and to yield inferences about its biological basis and functional utility.

:   

963

Distribution of and variations in animal color vision S  I The example just described shows that although nonhuman species may have color vision, it is not necessarily like ours. From a comparative perspective, then, a basic question is: who has color vision and what is it like? Given that there are millions of species of animals on the planet ranging in size from a few cells to a few tons, this is no small question. In a formal sense, color vision must be assayed behaviorally and, whereas it is true that many excellent behavioral studies have been conducted on representatives from some groups, for the vast majority of animals there is no information. This problem is ameliorated somewhat by our considerable understanding of the biological basis of color vision. For instance, there are strong linkages between the photopigments found in the eyes of animals and their potential for color vision. At minimum, an eye must have two or more types of photopigment that operate at common levels of illumination in order to provide the information required to support color vision, so any animal found to have only a single type of photopigment necessarily lacks color vision (strictly speaking, there may be some specialized cases where this is not true; for an example, see Neitz et al., 1999). Beyond the two pigment types required for color vision, there is significant correlation between the number of pigments present and the dimensionality of vision such that the presence of two pigments is associated with dichromacy, three pigments predicts trichromatic color vision, and so on. During the past quarter century, numerous methods have been developed to measure photopigments, so it is now usually much easier to do this than to undertake a direct examination of color vision, a task often arduous at best. The result is that photopigment measurements are now available for many species and, from these, predictions about color vision can be derived. A second tool for inferring color vision comes from molecular biology. Single genes specify photopigment proteins (opsins), and enough has been learned about the structure of these genes to allow predictions about the properties of the photopigment specified by gene expression. Such analyses can be carried out on small tissue samples, and this provides the possibility of learning about photopigments in animals that may be unlikely targets for behavioral study (because of their size, ferocity, rareness, etc.). Information about photopigment genes is being generated very rapidly, and this too can serve as a useful adjunct to infer color vision. Beyond photopigments and their genes, there are a number of other biological markers that can be exploited to yield some predictions about color vision. Even though there are other indices that can give indications of the presence and nature of color vision, it is clear that they cannot yield the same depth of insight obtained from direct studies of color vision. For example, knowledge

964

  

of the number of types of photopigment and their spectral properties may allow predictions about the dimensionality of color vision, but this by itself cannot predict the acuteness of the capacity, an aspect often more closely linked to the number and distribution of receptors containing the different cone types, as well as their nervous system wiring. Information about opsin genes can be even further removed from the realization of color vision because of the need to infer that the gene actually produces functional photopigment. In sum, although it is common to draw inferences about color vision from measurements of the types suggested, as I shall do freely here, it is important to remember that this is just what they are—inferences. Considering both the direct examinations and those that document mechanisms, studies of animal color vision number in the hundreds, far too many to even list in a brief review. Instead, we consider a few topics that are intended to suggest the extent and nature of animal color vision. Further details on particular groups of animal can be found in the following review articles: insects (Briscoe and Chittka, 2001; Menzel and Backhaus, 1991); crustaceans (Marshall et al., 1999); lower vertebrates (Neumeyer, 1998); fishes (Bowmaker, 1995); birds (Hart, 2001); mammals ( Jacobs, 1993). Those interested in extinct vertebrates should see Rowe (2000). D V Color vision results from neural comparisons of signals from receptor classes that have different spectral sensitivities. Increasing the number of receptor classes allows for an increase in the number of independent comparison channels and, correspondingly, in the potential dimensionality of color vision. Species that entirely lack a capacity for color vision stand at one extreme. Who are they? Not surprisingly, some of these are animals that live where photons are a scarce commodity. Deep-water environments certainly qualify in this regard and, indeed, both marine (Partridge, 1990) and freshwater (Bowmaker et al., 1994) fish that live at great depths often have only rod receptors that contain a single type of photopigment. Some species that live in brighter environments may also lack color vision; for example, cephalopods appear to be color blind (Wells, 1978; Messenger, 2001). Color vision is also absent in some mammals, but for a quite different reason. It has long been known that this condition occurs as an infrequent inherited or acquired defect in humans, but it is also a specieswide characteristic of some other mammals. In all of these animals, the retina has only a single class of cone containing middle-(MM) to long-wavelength-sensitive (L) pigment. A number of different mammalian species are like this, including some rodents (Cobb et al., 1999; Szel et al., 1996), nocturnal primates ( Jacobs et al., 1996), and many— perhaps all—marine mammals (Fasick et al., 1998; Peichl et al., 2001; Levenson and Dizon, 2003). These animals have

the opsin genes required to produce a second, short-wavelength (S) cone pigment, but those genes contain fatal mutations. The implication is that ancestors to these species had color vision that was subsequently abandoned. Why this regressive step occurred in such a broad variety of species is a contemporary puzzle. Mimimal color vision requires at least two sets of photopigments, typically yielding dichromatic color vision. Like the dog, many other mammals apparently also have two pigments and dichromatic color vision (Jacobs, 1993). Dichromacy, however, does not seem to be a common arrangement. Beyond mammals, some fishes that live at midlevel depths have the photopigment potential for dichromacy (Bowmaker, 1995), as do some of the crustaceans (Marshall et al., 1999) and reptiles (Sillman et al., 2001), but most animals that have color vision seem to have escaped the confines of dichromacy. Trichromacy is a much more common color vision arrangement. One reason for this is simply that insect species are abundant, and many of them seem to have trichromatic color vision. Although there is a great diversity of eye types among insects, there is nevertheless considerable commonality in their photopigments, with many species having three different types of photopigment, with respective peaks in the ultraviolet (UV), short (S), and middle (M) wavelengths (Briscoe and Chittka, 2001), although a long-wavelengthsensitive pigment may be added to this basic complement in many species of Lepidoptera. In some insect species, the total number of receptor types with differing spectral sensitivities is increased even further by the presence of screening pigments, photostable pigments that can serve to modify the spectral absorption properties of the photopigments. Although direct tests of color vision have not been done for most insects, the honeybee (Apis mellifera) has the best-studied color vision of any species outside of humans, and it is well documented that the three photopigments of the honeybee underlie acute trichromatic color vision (Backhaus, 1992). In addition to insects, some fishes (mostly living at shallower depths) are known to have three types of cone photopigments and putative trichromacy (Bowmaker, 1995), and, of course, many primates, including humans, are also trichromatic ( Jacobs, 1996). The addition of dimensions of color vision greater than trichromacy has occurred many times, or so it may seem, since many teleost fishes are known to have four types of cone pigment, as do a very large number of birds (Bowmaker, 1995; Hart, 2001). All of these animals are potential tetrachromats, as are other species, such as, turtles (Arnold and Neumeyer, 1987). It must be admitted, however, that of all the species that have four types of cone pigment, actual demonstrations that their operation gets translated into tetrachromatic color vision are very infrequent. Among the species in which tetrachromacy seems to have been

established are several common laboratory animals—goldfish, chickens, and pigeons. In the absence of appropriate tests, direct extrapolations from pigment complement to color vision dimensionality in other birds and fishes should be viewed with caution. Considerable variation in color vision exists within each of these dimensional categories. For example, Figure 62.2 illustrates photopigment spectra and an index of color vision for two representative trichromatic species, an Old World monkey (Macaca mulatta) and the honeybee (A. meliferis), and a tetrachromat, the goldfish (Carassius auratus). The three have very different combinations of photopigment; for example, both honeybee and goldfish have UV-sensitive pigments that are not present in the monkey retina. Note too that the goldfish has an unusually long-wavelength-sensitive pigment. A consequence of these variations is that these species have strikingly different spectral sensitivities. Measurements of the capacity of each of these species to discriminate wavelength differences are illustrated in the right column of Figure 62.2. With a UV photopigment, both honeybee and goldfish have acute color vision at around 400 nm, a part of the spectrum to which the monkey is nearly blind to differences in color. The trichromats have in common two regions of most acute color discrimination, but in accord with differences in their photopigment complements, these regions are in different parts of the spectrum. Correspondingly, three regions of most acute discrimination, around 400, 500, and 610 nm, characterize wavelength discrimination in the tetrachromatic goldfish. The size of the wavelength change required for discrimination in the three species shown in Figure 62.2 suggests the possibility that there may also be real differences in the acuteness of color vision in these species. Comparisons like this illustrate the fact that dimensional characterization of color vision alone does not convey a very complete picture of animal color vision. A simple count of the number of types of spectrally distinct photoreceptors shows that the potential for color vision with dimensionality even higher than tetrachromacy exists in a number of species. The reigning champions in this regard are the stomatopod crustaceans. The eyes of these diverse marine invertebrates contain a profusion of photopigments, with individual species having 11 to as many as 16 types of photopigment (Cronin and Marshall, 1989; Cronin et al., 2000)! And if that is not impressive enough, these eyes also contain a variety of spectral filters, so that that the number of spectrally distinct receptors can be even higher than the number of photopigments. The result is a large number of receptors with narrowed spectral bandwidths. Behavioral experiments have shown that these animals in fact have color vision in that they can successfully discriminate a number of spectral stimuli from achromatic ones in the absence of a consistent brightness cue (Marshall et al., 1996). However, the nature of their color vision is unique, since the different

:   

965

F 62.2. Photopigment absorption spectra (left column) and wavelength discrimination (right column) for three species. Honeybee and macaque monkey have trichromatic color vision; the goldfish is believed to be a tetrachromat. The sensitivity curves have been normalized as noted in Figure 62.1.

photoreceptor types found in stomatopods are separated into rows, and it is believed that the nervous system only allows for a series of spatially local dichromatic comparisons, that is, between pairs of receptors having differing spectral sensitivities. It may be that the stomatopod nervous system allows for further comparisons between these multiple dichromatic channels, but that is not established at present. It has been argued that a system of this kind could be used to enhance color constancy (see Chapter 61), a capacity that should be important, since stomatopods are often forced to make accurate absolute discriminations of the colors of objects (Osorio et al., 1997). W-S B Although crustaceans like the stomatopods just described clearly have color vision

966

  

in a definitional sense, it has long been known that many invertebrates demonstrate a range of natural behaviors that somewhat resemble color vision yet do not satisfy the criteria usually associated with that capacity. These have been termed wavelength-specific behaviors, and they include stereotyped actions like egg laying, feeding, and escape, all behavioral sequences that can be reliably induced by stimulation of one or more spectrally distinct photoreceptors (Menzel, 1979). Wavelength-specific behaviors of this sort have been well documented in Lepidoptera and Hymenoptera, as well as in some Crustacea (Goldsmith, 1990; Marshall et al., 1996). Close examination of these cases often reveals the presence of the sorts of biological mechanisms associated with color vision, that is, more than one spectral class of receptor and neural interactions of inputs from these sep-

arate classes. However, these animals are typically unable to generalize this capacity so that it can be used in other situations. This absence of plasticity in making discriminations in novel situations is usually argued to separate wavelengthspecific behaviors from color vision. The fact that animals that display wavelength-specific behaviors often have multiple types of photopigments and the requisite neural wiring for comparisons of their signals indicates that one needs to be particularly cautious in attributing color vision based solely on the presence of certain biological arrangements.

Evolution of color vision An obvious conclusion drawn from the previous section is that most contemporary species have color vision and that the capacity appears in many different guises. How did this happen? Interest in the evolution of visual capacities is not new; in fact, that concern is a central theme in the most famous book on comparative vision, Gordon Walls’ treatise on The Vertebrate Eye and Its Adaptive Radiation (1942). Although Walls and those who followed him offered plenty of speculation about the evolution of color vision, it is only in the past 15 years that our developing understanding of visual mechanisms has begun to allow some tentative answers to questions about color vision evolution. S  P E Two developments have led to a better appreciation of the evolution of color vision: more and better studies of color vision in contemporary species and the addition of information about the genes that specify photopigment opsins. The latter story starts in 1986 with the announcement of the sequence of the human cone opsin genes (Nathans et al., 1986). Since then there has been a steady accumulation of opsin gene sequences for many other species. Ideas about the evolution of photopigments, and by extension that of color vision, can be derived from comparisons of these sequences (Bowmaker, 1998; Nathans, 1999; Yokoyama and Yokoyama, 1989). One possible interpretation is that it may have begun with something similar to the eyespots found in green algae. These receptors contain pigments that are quite similar in structure to the opsins of both invertebrate and vertebrate photopigments, and that commonality suggests that motile microorganisms like the algae may have been the first to develop photopigments (Deininger et al., 2000). It is believed that in vertebrates a single cone opsin gene (and its photopigment product) emerged first. Later, somewhere between 400 and 1000 million years ago, this progenitor gene duplicated and then diverged in structure, yielding as offspring two types of cone pigment with respective peaks in the short and middle to long wavelengths (Bowmaker, 1998; Nathans et al., 1986; Neitz et al., 2001). The immediate utility of this added pigment is, of course, not known. One speculation is that

F 62.3. A simplified phylogeny for vertebrate photopigments. The interrelationships were inferred from comparisons of opsin gene structures. Illustrated are the relationships between four cone opsin gene families and a single rod opsin gene family. The peak sensitivities are ranges given for photopigments measured in contemporary representatives of each of the named groups. (Modified from Bowmaker, 1998.)

two cone pigments and the wiring requisite for extracting color information may actually have arisen as a device not for producing color vision per se, but rather as a means of eliminating the contaminating effects of flicker from the retinal image (Maximov, 2000). Vertebrate rod photopigments arose from further changes in these original two cone pigments, and subsequently, the number of cone opsin gene families increased to four. Figure 62.3 shows a simplified phylogenetic tree for vertebrate photopigments as inferred from opsin gene structures (parallel structures for the phylogeny of invertebrate photopigments are available in Briscoe and Chittka, 2001). It is noteworthy that although vertebrate rod pigments all have very similar spectral properties, the cone pigments have spread their maximum sensitivities over a much larger portion of the spectrum. As we saw above, cone pigments reflective of the presence of all four families of cone opsin genes appear in many modern animals of several groups. Mammals are unusual in having cone pigments from only two of these families, implying that representatives of the other two gene families were lost sometime during mammalian evolution. One possibility is that this loss was associated with the dominant nocturnality and consequent supremacy of rod vision

:   

967

in early mammals (Walls, 1942). Under those low-light conditions, neither cones nor color vision would likely be very useful. Among mammals only the primates have added a third cone pigment to their retinal repertoire, and this has allowed them to become trichromatic. This happened uniformly for the catarrhines (Old World monkeys, apes, and humans) as a result of an X-chromosome gene duplication that occurred some 30 to 40 million years ago, an event that set the stage for the presence of separate M- and L-cone pigments (Nathans et al., 1986). The platyrrhine (New World) monkeys are very different. The vast majority of these species have polymorphic color vision, with individual animals having any of several versions of trichromatic or dichromatic color vision ( Jacobs, 1998). This polymorphism, in turn, reflects individual variations in cone pigments and cone opsin genes. Like the catarrhines, these monkeys have an autosomal (chromosome 7) S-cone opsin gene, but unlike the catarrhines, they have only a single X-chromosome opsin gene. This gene is polymorphic, thus accounting for individual variations in cone pigments and color vision. How this works is illustrated in Figure 62.4 for the most common arrangement found in platyrrhine monkeys, one where there are three M/L opsin gene alleles and corresponding photopigments. An important consequence of this arrangement is that since males have only a single X chromosome, they necessarily have only a single type of M/L pigment and, thus, dichromatic color vision. With two X chromosomes, females can become heterozygous for their M/L opsin genes, and those that do have two types of M/L cone pigment and trichromatic color vision ( Jacobs, 1984). The result of this arrangement is a striking array of different color vision capacities among conspecific animals. One type of New World monkey departs notably from this polymorphic theme. Howler monkeys (Alouatta) have an opsin gene/photopigment arrangement that is very similar to that of the catarrhines ( Jacobs et al., 1996). This would be expected to give them universal trichromatic color vision (although that fact has not yet been experimentally established). It would appear that something very similar to what happened in catarrhine history also occurred in the line to modern howler monkeys, that is an X-chromosome opsin duplication and subsequent divergence to permit separate M and L opsin genes and photopigments. Until recently, it was believed that the third main group of primate, the more primitive strepsirrhines (lemurs, lorises, and their ilk), are unlike either catarrhines or platyrrhines in having only a single M/L photopigment with no polymorphic variations. This would make them similar to many nonprimate mammals. It has now been discovered, however, that some diurnal strepsirrhines have M/L opsin gene and photopigment polymorphism similar to that described for the platyrrhines ( Jacobs et al., 2002; Tan and Li, 1999), and

968

  

F 62.4. Schematic representation of the photopigment basis for the polymorphic color vision of most New World monkeys. Sketched at the top are the spectral sensitivities of four types of cone. The list at the bottom shows how these cone pigments are combined in individual animals to yield six different cone pigment and color vision phenotypes. (Modified from Jacobs, 1998.)

this means that some individuals in these species could also have trichromatic color vision. The strepsirrhines and platyrrhines are only distantly related, and it seems likely that the photopigment polymorphism seen in two groups arose independently. O D Animal eyes contain a variety of filters that serve to condition the light incident on photoreceptors. Many of these filters are spectrally selective, so they have long attracted attention for the possible role(s) they may play in vision (Douglas and Marshall, 1999). Among these filters, perhaps the most intriguing are the oil droplets located in the inner segments of cone photoreceptors of many birds, turtles, and lizards. Many of these oil droplets are densely pigmented and act as long-pass spectral filters, so that, depending on the location of the spectral cutoff, the droplets appear red, orange, or yellow in fresh tissue. Oil droplets influence the sensitivity of the photoreceptors, as illustrated in the curves of Figure 62.5 that compare the spectral sensitivity of the four classes of cone found in the pigeon retina with and without the influence of accompanying oil

F 62.5. Spectral sensitivities of four types of cone found in the pigeon retina. The two sets of curves show sensitivity without (dashed) and with (continuous) filtering by the oil droplets that are found in each of these four receptor types. (Modified from Vorobyev et al., 1998.)

droplets. It can be seen that the effect of these oil droplets is to significantly narrow the cone absorption spectra, shift the locations of peak sensitivity, and decrease absorption efficiency. In principle, a comparison of neural signals from a population of cones containing the same photopigments while having different types of oil droplets could provide a usable color signal, but there is no evidence that any animal has exploited that possibility. From a theoretical analysis, Vorobyev and colleagues have made a compelling argument to show that oil droplets found in bird retinas can potentially serve to enhance the animal’s ability to discriminate important natural stimuli (such as plumage colors), and they may also serve to improve color constancy (Vorobyev et al., 1998). Whatever their specific roles, and these roles undoubtedly vary across different taxa, the presence of oil droplets in the retinas of many lineages implies that they are an ancient invention, possibly dating back to the ancestors of the earliest land vertebrates more than 400 million years ago (Robinson, 1994). If so, it is the absence of colored oil droplets in the retinas of some contemporary species that is notable. Viewed in this way, the absence of oil droplets reflects the loss of an ancestral adaptation, perhaps occasioned by a shift from a diurnal to a more nocturnal lifestyle. I C V I? Color vision is present in the vast majority of all animals, and where it is absent, as in some mammals, its ancestral presence can often be inferred. The ubiquity of color vision raises the question of why this capacity evolved in so many different lineages. Part of the answer may come directly from a fundamental feature of photopigments. Photopigments have absorption bandwidths that are moderately narrow (half-bandwidths of about 50 to 70 nm) and fixed in size (Fig. 62.2). Consequently, any significant expansion of the spectral window through which an

animal senses its visual world requires that retinas contain populations of pigments with different absorption peaks. Discrete changes in the structures of pigment opsins (at the limit, a single amino acid substitution) that result from mutational changes in opsin genes can produce such new pigments (Asenjo et al., 1994; Merbs and Nathans, 1992). These new photopigments could be coexpressed in receptors along with a native pigment, as indeed they are in some contemporary species (Makino and Dodd, 1996; Rohlich et al., 1994), and this step alone will immediately yield a broadened spectral window ( Jacobs et al., 1999). In most cases, however, pigment opsins get selectively transcribed into single receptors, and this allows both an expanded spectral window and a receptor basis for color vision. Of course, multiple receptor types are necessary but not sufficient to yield color vision. What is also required is a nervous system organized to compare the rate of photon absorption in the different receptor types. This is typically accomplished by the presence of cells whose inputs are such that they generate output signals that are inhibitory/excitatory comparisons of photon capture in the different cone classes. These comparisons yield the spectral opponency property referred to above. Qualitatively similar comparisons are at the heart of the neural analysis of spatial information, in which case they are employed to compare the effects of stimulation at neighboring locations on the photoreceptor mosaic. A consequence is that any nervous system that has evolved to analyze spatial information in this fashion, which is to say virtually all visual systems, already has the basic organization required to set up spectral as well as spatial opponency. It is probably not unreasonable to suggest that virtually any visual system that adds a new type of photopigment will also gain some color vision. Multiple types of photopigment and the appropriate neural comparisons set the stage for the presence of some color vision, but for these organizations to be maintained, to evolve, they must provide some adaptive advantage. The advantages of having color vision may seem self-evident, but is it possible to actually demonstrate the occurrence of such an advantage? One instance may be in the color vision of platyrrhine monkeys. We noted above that most of these species are polymorphic for M/L cone photopigments. The opsin genes specifying these alternative versions are only slightly different in structure, suggesting that they arose by discrete mutational changes (Neitz et al., 1991). The gene arrangement in these monkeys is such that a subset of the female monkeys, those that are heterozygous at the M/L opsin gene site, gain a dimension of color vision and become trichromatic (Fig. 62.4). How many females actually achieve trichromacy depends, in turn, on the relative frequencies of the three gene alleles. The results of a survey indicate that the three versions of these opsin genes are about equally frequent in the population, yielding an arrangement that will

:   

969

maximize heterozygosity and, consequently, the incidence of female trichromacy (Jacobs, 1998). This outcome can best be explained as an example of overdominant selection, a circumstance in which a mutational change yields an advantage only to the heterozygous individual. The strong implication to be drawn from this example is that trichromatic color vision must be an adaptive trait for these monkeys. One potential advantage of adding color vision capacity can be deduced from observations of human vision. At a given level of adaptation, a human observer can discriminate about 100 brightness steps. Because achromatic and chromatic dimensions of vision are orthogonal, adding a single dimension of color vision (i.e., becoming dichromatic) geometrically expands visual capacity (Neitz et al., 2001). So too does adding a third dimension, with the result that it is estimated that the trichromatic human can discriminate in excess of 2 million surface colors (Pointer and Attridge, 1998). This is a huge gain, and so in terms of sheer information-handling capacity, there can be considerable advantage to adding new dimensions of color vision. If the acquisition of color vision may be close to inevitable, and if additional dimensions of color vision enhance discriminative capacity, then why isn’t everyone, say, pentachromatic? The answer to this question about limitations on the dimensionality of color vision will no doubt differ for various lineages, reflective of the visual opportunities available in different environments. In general, however, it is true that there are inevitable costs associated with adding the new receptor types required for additional color vision capacity. For one thing, adding a new photopigment will reduce the number of receptors that contain earlier types of photopigment, and this may reduce the signal-to-noise ratio of each of the cone types, thus lowering overall color vision efficiency; this could be an important factor in limiting the number of cone types (Vorobyev and Osorio, 1998). Another potential problem is that in many visual systems the neural circuits for producing spectral opponency are quite specific, so acquiring a new color vision capacity may require elaborate nervous system changes as well as the simple addition of a photopigment. Adding new neural circuits is metabolically quite costly (Laughlin, 2001). Finally, model studies suggest that in terms of ability to discriminate many natural stimuli, there is probably little to be gained by having more than about four photopigments spread across the 300 nm spectral window available to most species (Menzel and Backhaus, 1991).

Utility of color vision In considering why color vision exists at all, Gordon Walls questioned what it might do for animals that, he suggested, “certainly cannot appreciate sunsets and old masters” (Walls,

970

  

1942). It is conceivable that Walls was mistaken about the aesthetic lives of animals, but in any case his answer was similar to that given above: that color vision can considerably enhance the visibility of objects of interest. In recent years, some have tried to go beyond such generalities by asking how a particular type of color vision is suited to the visual tasks that an animal confronts. To give the flavor of these attempts, I briefly consider one of the best-studied examples. H  F C Honeybees have trichromatic color vision. These insects are dependent on the pollen and nectar offered by flowers, and to obtain these, they have to make reliable discriminations among different flowers. On the other side of the coin, flowers achieve effective pollination by selectively attracting foraging bees. Since there is clear mutual gain for both bees and flowers in ensuring that efficient harvesting occurs, the relationship between floral coloring and bee color vision has been a natural focus for examining the utility of color vision (Kevan and Backhaus, 1998). One step toward linking bee color vision to floral coloring can be made by considering the nature of the signals offered by flowers. Measurements of a large sample (>1000) of flowers drawn from several geographic locations reveal that their spectral reflectance patterns are not random— virtually all flowers can be placed in 1 of 10 categories, and only 5 of these are required to accommodate fully 85% of the entire sample (Chittka et al., 1994). Three such spectral reflectance patterns are illustrated in Figure 62.6. The potential visibility of these targets to honeybees can be crudely predicted by simply considering them in the context of the spectral absorption properties of bee photopigments (Fig. 62.2). For example, the flower whose reflectance spectrum is shown in Figure 62.6A should offer little or no color signal to the bee because its spectral reflectance is virtually constant across the entire spectral absorption range of the bee photopigments. At the same time, however, the pattern of reflectance of this flower does change very rapidly across the spectral absorption windows of the M and L pigments of Old World primates (Fig. 62.2), and so it would be predicted to offer a strong color signal to such observers and indeed it does, appearing red to a human viewer. Almost the reverse holds for the flower of Figure 62.6B. Here there is little reflectance change across the span of the primate cone photopigments, and flowers of this type appear white to a human observer. However, the reflectance pattern of this flower changes rapidly across the spectral bandwidths of the UV and S photopigments of the honeybee and thus offers a potentially strong color signal to the insect. The third reflectance pattern, shown in Figure 62.6C, is that measured for “green” foliage. Note that it shows only a small variation in reflectance pattern across the span of the honeybee photopigment absorption, and one would predict that it might

birds was similarly interpreted as implying that UV signals might be in some way unique. As the number of species known to have specific sensitivity to UV has expanded, and as understanding of the relationships between spectral cues and specific behaviors has broadened, it has become clearer that UV is probably not special (Hunt et al., 2001; Kevan et al., 2001). In concluding this discussion, no one is suggesting that UV signals are not important for some animals in some circumstances, but only that they should probably not be accorded any more significance than that given to other spectral signals.

F 62.6. Reflectance spectra measured for three plants found in the visual environment of honeybees. A, A flower (Justicia rizzinii) that appears red to humans but is presumably uncolored to bees. B, A flower (Bereroa incarna) that is white to humans but colored for honeybees. C, Green leaves. See text for further discussion. (Spectra taken from Chittka et al., 1994.)

appear essentially uncolored to the bee (Chittka et al., 1994). That fact is potentially important because it is from such foliage backgrounds that flower colors frequently need to be discriminated. More quantitative predictions of the ability of bees to discriminate floral colors have been made by considering the spectral absorption and adaptational properties of bee photoreceptors in conjunction with some knowledge of the nature of color coding in the bee visual system (Chittka et al., 1994; Kevan and Backhaus, 1998). These efforts have led to the conclusion that bee visual systems and floral coloring are well matched for the efficient exchange of information (Chittka and Menzel, 1992). Whether or not this and other cases of insect color vision can be explained as evolutionary adaptations of photopigments to specific environmental opportunities is an issue of current debate (Briscoe and Chittka, 2001). The discovery that flowers offer salient visual signals to insects in a portion of the spectrum that is invisible to humans led early investigators to conclude that UV might be special from ecological and evolutionary points of view (Goldsmith, 1994). The later discovery of UV receptors in

T I  A C E A fair conclusion from the material reviewed here is that behavioral experiments and biological inferences have taught us much about animal color vision. In many cases, one can now fairly confidently predict which spectral stimuli certain animals can or cannot discern and then go on to explain why this is so. But what has been learned about what animals may experience when they employ color vision? For instance, a foraging monkey may use color vision to quickly detect a fruit nestled in foliage, but in doing so, does it experience “yellow fruit” in “green foliage,” as a human observer might? There turns out to be precious little to indicate how animal color experiences may be similar to or different from our own or, indeed, if they even see colors in the sense that we do. Transforming visual images so that they simulate for normal human trichromats the appearance of the world to human dichromats allows some insight into the color experiences of those with well-established alternative visual biology (Breitel et al., 1997). These predictions are based on a comparison of color discriminations of trichromats and dichromats and on reports of the perceptions of individuals whose color vision differs in the two eyes. One can never know for certain the color experiences of others, but this provides a reasonable approach to that end. It may be tempting to imagine that one could take color discrimination data and/or knowledge of photopigment complements and similarly deduce what color perceptions of other animals might look like to humans. The assumption in doing this is that the transformations between these indices and color experience are the same for different species. That assumption cannot be evaluated, and consequently such simulations may be enlightened fictions, but nevertheless they are fictions. One higher-level aspect of human color vision is the ability to categorize colors by responding similarly to colors that can be discriminated. Such behavior is evidenced by the consistent use of color names, a process that seems nearly universal across cultural and linguistic boundaries (Abramov and Gordon, 1994). Categorical color perception has been demonstrated to occur in pigeons, and a range of observations suggest its occurrence in nonhuman primates ( Jacobs, 1981; Zentall et al., 1986). This implies that at least some

:   

971

animals are capable of deriving a generalized experience from their color vision abilities, but unfortunately, it does not tell us what that experience might be. In sum, although our personal experience of color makes it hard to believe that animals do not have some similar set of perceptual revelations, what these might be and how they differ across species remain a mystery.

Acknowledgments Preparation of this chapter was supported by a grant from the National Eye Institute (EY02052). I thank Mickey Rowe for numerous helpful comments. REFERENCES Abramov, I., and J. Gordon, 1994. Color appearance: on seeing red—or yellow, or green, or blue, Annu. Rev. Psychol., 45:451– 485. Arnold, K., and C. Neumeyer, 1987. Wavelength discrimination in the turtle Pseudemys scripta elegans, Vis. Res., 27:1501–1511. Asenjo, A. B., J. Rim, and D. D. Oprian, 1994. Molecular determinants of human red/green color discrimination, Neuron, 12:1131–1138. Backhaus, W., 1992. Color vision in honeybees, Neurosci. Biobehav. Rev., 16:1–12. Bowmaker, J. K., 1995. The visual pigments of fish, Prog. Retinal Eye Res., 15:1–31. Bowmaker, J. K., 1998. Evolution of colour vision in vertebrates, Eye, 12:541–547. Bowmaker, J. K., V. I. Govardovskii, S. A. Shukolyukov, L. B. Zueva, D. M. Hunt, V. G. Sideleva, and O. G. Smirnova, 1994. Visual pigments and the photic environment: the Cottoid fish of Lake Baikal, Vis. Res., 34:591–605. Breitel, H., F. Vienot, and J. D. Mollon, 1997. Computerized simulation of color appearance for dichromats, J. Opt. Soc. Am. A, 14:2647–2655. Briscoe, A. D., and L. Chittka, 2001. The evolution of color vision in insects, Annu. Rev. Entomol., 46:471–510. Chittka, L., W. Beier, H. Hertel, E. Steinmann, and R. Menzel, 1992. Opponent colour coding is a universal strategy to evaluate the photoreceptor inputs in Hymenoptera, J. Comp. Physiol. A, 170:545–563. Chittka, L., and R. Menzel, 1992. The evolutionary adaptation of flower colours and insect pollinators’ colour vision. J. Comp. Physiol. A, 171:171–181. Chittka, L., A. Shmida, N. Troje, and R. Menzel, 1994. Ultraviolet as a component of flower reflections, and the colour perception of hymenoptera, Vis. Res., 34:1489–1508. Cobb, J. K., C. Bialozynski, J. Neitz, G. H. Jacobs, and M. Neitz, 1999. UV cone pigment genes from Syrian and Siberian hamsters, Invest. Ophthalmol. Vis. Sci., 40:S353. Cronin, T. W., and N. J. Marshall, 1989. A retina with at least ten spectral types of photoreceptors in a mantis shrimp, Nature, 339:137–140. Cronin, T. W., N. J. Marshall, and R. L. Caldwell, 2000. Spectral tuning and the visual ecology of mantis shrimps, Philos. Trans. R. Soc. Lond. B, 355:1263–1268. Deininger, W., M. Fuhrmann, and P. Hegemann, 2000. Opsin evolution: out of the wild green yonder? Trends Genet., 16:158–159.

972

  

Douglas, R. H., and N. J. Marshall, 1999. A review of vertebrate and invertebrate ocular filters, in Adaptive Mechanisms in the Ecology of Vision (S. N. Archer, M. B. A. Djamgoz, E. R. Loew, J. C. Partridge, and S. Vallerga, eds.), Dordrecht: Kluwer Academic, pp. 95–162. Fasick, J. I., T. W. Cronin, D. M. Hunt, and P. R. Robinson, 1998. The visual pigments of the bottlenose dolphin (Tursiops truncatus), Vis. Neurosci., 15:643–651. Goldsmith, T. H., 1990. Optimization, constraint, and history in the evolution of eyes, Q. Rev. Biol., 65:281–322. Goldsmith, T. H., 1994. Ultraviolet receptors and color vision: evolutionary implications and a dissonance of paradigms, Vis. Res., 34:1479–1487. Hart, N. S., 2001. The visual ecology of avian photoreceptors, Prog. Ret. Eye Res., 20:675–703. Hunt, S., I. C. Cuthill, A. T. D. Bennett, S. C. Church, and J. C. Partridge, 2001. Is the ultraviolet waveband a special communication channel in avian mate choice? J. Exp. Biol., 204: 2499–2507. Jacobs, G. H., 1981. Comparative Color Vision, New York: Academic Press. Jacobs, G. H., 1984. Within-species variations in visual capacity among squirrel monkeys (Saimiri sciureus): color vision, Vis. Res., 24:1267–1277. Jacobs, G. H., 1993. The distribution and nature of colour vision among the mammals, Biol. Rev., 68:413–471. Jacobs, G. H., 1996. Primate photopigments and primate color vision, Proc. Natl. Acad. Sci. USA, 93:577–581. Jacobs, G. H., 1998. A perspective on color vision in platyrrhine monkeys, Vis. Res., 38:3307–3313. Jacobs, G. H., J. F. Deegan II, M. A. Crognale, and J. A. Fenwick, 1993. Photopigments of dogs and foxes and their implications for canid vision, Vis. Neurosci., 10:173–180. Jacobs, G. H., J. F. Deegan II, Y. Tan, and W.-H. Li, 2002. Opsin gene and photopigment polymorphism in a prosimian primate, Vis. Res., 42:11–18. Jacobs, G. H., J. C. Fenwick, J. B. Calderone, and S. S. Deeb, 1999. Human cone pigment expressed in transgenic mice yields altered vision, J. Neurosci., 19:3258–3265. Jacobs, G. H., M. Neitz, and J. Neitz, 1996. Mutations in S-cone pigment genes and the absence of colour vision in two species of nocturnal primate, Proc. R. Soc. Lond. B, 263:705–710. Kevan, P. G., and W. G. K. Backhaus, 1998. Color vision: ecology and evolution in making the best of the photic environment, in Color Vision: Perspectives from Different Disciplines (W. G. K. Backhaus, R. Kliegl, and J. S. Werner, eds.), Berlin: Walter de Gruyter, pp. 163–183. Kevan, P. G., L. Chittka, and A. G. Dyer, 2001. Limits to the salience of ultraviolet: lessons from colour vision in bees and birds, J. Exp. Biol., 204:2571–2580. Laughlin, S. B., 2001. Energy as a constraint on the coding and processing of sensory information, Curr. Opin. Neurobiol., 11:475–480. Levenson, D. H., and A. Dizon, 2003. Genetic evidence for the ancestral loss of SWS cone pigments in mysticete and odontocete cetaceans, Proc. R. Soc. Lond. B, 270:673–679. Makino, C. L., and R. L. Dodd, 1996. Multiple visual pigments in a photoreceptor of the salamander retina, J. Gen. Physiol., 108:27–34. Marshall, J., J. Kent, and T. Cronin, 1999. Visual adaptations in crustaceans: spectral sensitivity in diverse habitats, in Adaptative Mechanisms in the Ecology of Vision (S. N. Archer, M. B. A. Djamgoz, E. R. Loew, J. C. Partridge, and S. Vallerga, eds.), Dordrecht: Kluwer Academic, pp. 285–327.

Marshall, N. J., J. P. Jones, and T. W. Cronin, 1996. Behavioural evidence for colour vision in stomatopod crustaceans, J. Comp. Physiol. A, 179:473–481. Maximov, V. V., 2000. Environmental factors which may have led to the appearance of colour vision, Philos. Trans. R. Soc. Lond. B, 355:1239–1242. Menzel, R., 1979. Spectral sensitivity and color vision in invertebrates, in Handbook of Sensory Physiology, vol VII/6A (H. Autrum, ed.), Berlin: Springer-Verlag, pp. 503–580. Menzel, R., and W. Backhaus, 1991. Colour vision in insects, in The Perception of Colour (P. Gouras, ed.), Boca Raton, FL: CRC Press, pp. 262–293. Merbs, S. L., and J. Nathans, 1992. Absorption spectra of human cone pigments, Nature, 356:433–435. Messenger, J. B., 2001. Cephalopod chromatophores: Neurobiology and natural history, Biol. Rev., 76:473–528. Nathans, J., 1999. The evolution and physiology of human color vision: insights from molecular genetic studies of visual pigments, Neuron, 24:299–312. Nathans, J., D. Thomas, and D. S. Hogness, 1986. Molecular genetics of human color vision: the genes encoding blue, green and red pigments, Science, 232:193–202. Neitz, J., J. Carroll, and M. Neitz, 2001. Color vision: almost reason enough for having eyes, Opt. Photon. News, 12:26–33. Neitz, J., T. Geist, and G. H. Jacobs, 1989. Color vision in the dog, Vis. Neurosci., 3:119–125. Neitz, J., M. Neitz, J. C. He, and S. K. Shevell, 1999. Trichromatic color vision with only two spectrally discrete photopigments, Nat. Neurosci., 2:884–888. Neitz, M., J. Neitz, and G. H. Jacobs, 1991. Spectral tuning of pigments underlying red-green color vision, Science, 252:971–974. Neumeyer, C., 1998. Color vision in lower vertebrates, in Color Vision: Perspectives from Different Disciplines (W. G. K. Backhaus, R. Kliegl, and J. S. Werner, eds.), Berlin: Walter de Gruyter, pp. 149–162. Osorio, D., N. J. Marshall, and T. W. Cronin, 1997. Stomatopod photoreceptor spectral tuning as an adaptation for colour constancy in water, Vis. Res., 37:3299–3309.

Partridge, J. C., 1990. The colour sensitivity and vision of fishes, in Light and Life in the Sea (P. J. Herring, A. K. Campbell, M. Whitfield, and L. Maddick, eds.), Cambridge: Cambridge University Press, pp. 167–184. Peichl, L., G. Behrmann, and R. H. H. Kroger, 2001. For whales and seals the ocean is not blue: a visual pigment loss in marine mammals, Eur. J. Neurosci., 13:1520–1528. Pointer, M. R., and G. G. Attridge, 1998. The number of discernible colours, Color Res. Appl., 23:52–54. Robinson, S. R., 1994. Early vertebrate colour vision, Nature, 367:121. Rohlich, P., T. van Veen, and A. Szel, 1994. Two different visual pigments in one retinal cone cell, Neuron, 13:1159–1166. Rowe, M. P., 2000. Inferring the retinal anatomy and visual capacities of extinct vertebrates, Palaeontol. Electron., 3: http://paleo-electronica.org/2000 Sillman, A. J., J. L. Johnson, and E. R. Loew, 2001. Retinal photoreceptors and visual pigments in Boa constrictor imperator J. Exp. Zool., 290:359–365. Szel, A., P., Rohlich, A. R. Caffe, and T. van Veen, 1996. Distribution of cone photoreceptors in the mammalian retina, Microsc. Res. Tech., 35:445–462. Tan, Y., and W.-H. Li, 1999. Trichromatic vision in prosimians, Nature, 402:36. Vorobyev, M., and D. Osorio, 1998. Receptor noise as a determinant of colour thresholds, Proc. R. Soc. Lond. B, 265:351–358. Vorobyev, M., D. Osorio, A. T. D. Bennett, N. J. Marshall, and I. C. Cuthill, 1998. Tetrachromacy, oil droplets and bird plumage colours, J. Comp. Physiol. A, 183:621–633. Walls, G. L., 1942. The Vertebrate Eye and Its Adaptive Radiation, Bloomfield Hills, MI: Cranbrook Institute of Science. Wells, M. J., 1978. Octopus: Physiology and Behavior of an Advanced Invertebrate, London: Chapman and Hall. Yokoyama, S., and R. Yokoyama, 1989. Molecular evolution of human visual pigment genes, Mol. Biol. Evol., 6:186–197. Zentall, R. T., P. Jackson-Smith, J. A. Jagielo, and G. B. Nallan, 1986. Categorical shape and color coding by pigeons, J. Exp. Psychol. Anim. Behav. Proc., 12:153–159.

:   

973

63

Molecular Genetics of Human Color Vision and Color Vision Defects MAUREEN NEITZ AND JAY NEITZ

T    photoreceptor in the human eye, rods and cones, serve different functions. Rods serve vision only under conditions of very low light levels, such as at night when little light is available. In contrast, cones serve vision at relatively higher light levels. Most of our daily activities are performed in daylight and at room light levels above those where rods contribute significantly to vision but where cones are active. Thus, under most normal conditions, our vision is based on cone photoreceptors. The capacity to see color is a prominent component of cone-based vision, which requires multiple classes of cone photoreceptor. Most humans have trichromatic color vision mediated by at least three cone types, one from each of three well-separated spectral classes. Photopigments are light-sensitive molecules that determine the spectral absorption characteristics of the cones. Thus, for each cone class, there is a corresponding class of cone pigment. The three classes are sometimes referred to as blue, green, and red. However, vision scientists usually refer to them according to their relative spectral sensitivities, short-, middle-, and long-wavelength sensitive (abbreviated S, M, and L). As far as we know, all humans with normal color vision have the same S pigment, and its absorption spectrum has a spectral peak near 415 nm (Bowmaker et al., 1980; Dartnall et al., 1983; Fasick et al., 1999). Traditionally it was assumed that human vision is characterized by a stereotyped set of S, M, and L pigments; however, recently it has become apparent that there is widespread variation in the L and M pigments that underlie normal vision (Dartnall et al., 1983; Merbs and Nathans, 1993; Neitz and Jacobs, 1986; Neitz et al., 1991, 1993; Winderickx et al., 1992a). The L pigments have spectral peaks near 560 nm (Dartnall et al., 1983; Schnapf et al., 1987), and the two most common variants are separated in spectral peak by approximately 3.5 nm (Merbs and Nathans, 1993; Neitz and Jacobs, 1986; Sharpe et al., 1998). Less common normal variants have spectral peaks that are shifted more than 8 nm shorter than the longest absorbing L pigment. M pigments peak near 530 nm (Dartnall et al., 1983; Schnapf et al., 1987), and normal variations also occur but with a lower frequency. Normal color vision requires the presence of the S pigment and at least one pigment from each of the L and M classes. Red-green color vision deficiencies are caused by the absence of expres-

974

sion or function of one class of pigment, either L or M. These color vision defects are extremely common, especially among people of Western European ancestry, for whom the frequency is about 8% in males and 0.4% in females. Molecular genetic experiments have been combined with imaging, electrophysiology, and psychophysics to better understand the basic biology underlying color vision capacity, how the mechanisms arise during development, and how they have evolved. One of the most remarkable aspects of human color vision is the amazingly high degree of normal variation and the high frequency of defects. Color vision deficiency is one of the most frequent genetic anomalies in humans, and considerable progress has been made in understanding the molecular basis for it.

Background T  C V D Normal color vision is the term used for the form of color vision shared by most humans. People with normal color vision can perceive four distinct (or unique) hues: red, yellow, green, and blue. Color information is extracted by neural circuits that compare the outputs of the cones. Red-green color vision is mediated by circuits that compare the outputs of the L and M cones. Blue-yellow color vision is mediated by circuits that compare the output of S cones to the summed outputs of the L and M cones. Together, these two neural systems provide the capacity to distinguish more than 100 different gradations of hue, which can be thought of as the sensations of the four unique hues individually or in combinations, such as shades of yellow-green, blue-green, purple (a red-blue color), and orange (a red-yellow color). People with congenital color vision defects see fewer hues than do people with normal color vision. The term protan refers to color vision defects caused by the absence of functional L-cone pigment, and the term deutan refers to the absence of functional M-cone pigment. Together, protan and deutan defects are the most common inherited forms of color blindness, and they preferentially affect males. Their incidence varies with ethnicity (for a recent review see Sharpe et al., 1999). About 2% of Caucasian men suffer from a protan defect and 6% from a

deutan defect. Only about 1 in 230 females is affected by protan or deutan defects. Within the protan and deutan categories of color vision deficiency, there is variation in the degree to which color vision is impaired. The most severe forms are the dichromatic types, protanopia and deuteranopia, in which color vision is based on just two pigments in two types of cones, either S and M (protanopes) or S and L (deuteranopes). The milder forms are the anomalous trichromacies, protanomaly and deuteranomaly, in which the L or M pigment, respectively, is missing but is replaced by a pigment that allows a reduced form of trichromatic color vision. Understanding the nature of the pigments underlying anomalous trichromacy compared to normal color vision is complicated by the fact that there are normal variations in the L and M pigments. The L and M pigments can be thought of as forming two variable but mutually exclusive classes, illustrated in the lower part of Figure 63.1. Individuals with deuteranomaly, a form of anomalous trichromacy, lack an M pigment, but they have two different pigments from the L class. Individuals with protanomaly lack an L pigment, but they have two slightly different pigments from the M class. Within the anomalous trichromacies there is a wide range of phenotypic variation, with some affected individuals having hue perception approaching normal, while others have color vision almost as poor as a dichromat’s. A third class of congenital color vision deficiency, referred to as tritan, is associated with defects in the S-cone pigment. These defects occur in males and females with equal frequency and are extremely rare, affecting fewer than 1 in 10,000 people. Also extremely rare are the monochromatic color vision defects, the achromatopsias. These disorders are associated with normal rod photoreceptor function but reduced (incomplete achromatopsia) or absent (complete achromatopsia) cone function. One form of incomplete achromatopsia is blue cone monochromacy, which is generally characterized by the absence of both normal L- and Mcone function (Nathans et al., 1989). In the human retina, about 7% of the cones are S, and the remainder are L and M. Blue cone monochromats base their vision on S cones and rods and thus have diminished capacity for all aspects of vision mediated by cones, including color vision and acuity. Rod monochromacy is a form of complete achromatopsia. Affected individuals are completely color-blind and have very poor acuity. This disorder affects up to 1 in 30,000 people (Sharpe et al., 1999). I P  C V D Protan and deutan defects are inherited as X-linked traits and are caused by mutation, rearrangement, or deletion of the genes encoding the L- and M-cone photopigments (Nathans et al., 1986a; Nathans et al., 1986b). These genes lie on the X chromosome, accounting for the pronounced gender differences in the frequency of red-green color vision

F 63.1. Spectral tuning in L and M photopigments. Shaded and unshaded arrows represent L- and M-photopigment genes, respectively. Thin black rectangles within the arrows represent the six exons, which are separated by five much larger introns. The relative sizes of introns and exons are drawn to scale. The seven spectral tuning sites encoded by exons 2 to 5 of the genes are indicated in the upper-left diagram along with the codon/amino acid number. The single-letter amino acid code is used to indicate the amino acid identities. When all seven amino acid residues are the ones shown for the schematic L gene (upper left), the pigment has the longest possible spectral peak (approximately 560 nm). When all the amino acids are those shown for the schematic M gene (upper left), the spectral peak is approximately 530 nm. Transposing the spectral tuning amino acids from L into M and vice versa produces pigments with intermediate spectral sensitivities. Amino acids 277 and 285, encoded by exon 5, produce the largest spectral shifts and define two major classes of pigments, M and L (bottom). Substitutions of amino acids encoded by exons 2, 3, and 4 are responsible for smaller spectral shifts that produce spectral subtypes within the L and M classes. The upper-right column shows the spectral variants of the L-class and M-class pigments encoded by genes that have been identified in humans. The shading surrounding individual exons indicates whether the spectral tuning site(s) specified by that exon encode the amino acids that shift the spectrum long (shaded) or short (unshaded). The wavelength of maximal sensitivity for each of variant pigments shown is an estimate derived by extrapolation from numerous studies (Asenjo et al., 1994; Merbs and Nathans, 1992; Neitz et al., 1995). The single-letter amino acid code is as follows: S, serine; Y, tyrosine; A, alanine; T, threonine; F, phenylalanine; I, isoleucine.

  :    

975

defects. Females have two X chromosomes; males have only one. There is a dosage compensation mechanism, termed X inactivation, to ensure that each cell in the female expresses only the required amount of each X-chromosome gene product. Female somatic cells retain one X chromosome as active; the other one is inactivated. In any given L- or Mcone photoreceptor cell, only one pigment gene from the array on the active X chromosome is expressed. The choice of which X chromosome will be active and which will be inactive is random, so on average, 50% of cells retain the paternal X-chromosome pigment genes as active and 50% retain the maternal pigment genes as active. X inactivation also ensures that the visual pigment genes from the maternal and paternal arrays are expressed in separate populations of cones. If a female carries the genes for color-blindness on one X chromosome and the genes for normal color vision on the other, she will have normal color vision. If she carries genes for color-blindness on both X chromosomes, but together her two X’s specify at least one functional L and one functional M pigment, she will also have normal color vision. A female will be color-blind only if her two X chromosomes together do not specify both a functional L and a functional M pigment. In rare instances, females can exhibit skewed X inactivation, whereby the cells of a given tissue all have the same active X chromosome. This is usually seen only when expression of genes from one of the X chromosomes severely diminishes the survival of cells in which it is active. Thus, in rare circumstances, a female with the genes for color blindness on only one X chromosome could be color-blind if she had a severely skewed X inactivation such that all of her L and M cones expressed genes from the “color-blind” X chromosome. Blue cone monochromacy is also inherited as an X-linked trait, and is caused by a variety of mechanisms including a combination of deletion and mutation of the X-linked visual pigment genes or deletion of cis-acting regulatory elements necessary for the expression of the L- and M-pigment genes (Nathans et al., 1989, 1993). In blue cone monochromats, neither L nor M genes are functionally expressed. Tritan defects are caused by mutations in the S-pigment gene on chromosome 7 and display a dominant inheritance pattern, with incomplete penetrance (Weitz et al., 1992a; Weitz et al., 1992b). Dominance refers to the fact that only one mutant copy of the S-pigment gene is required to cause the color vision defect. Incomplete penetrance means that not everyone who carries the mutant S-pigment gene will exhibit a color vision deficiency. Complete achromatopsia and forms of incomplete achromatopsia besides blue cone monochromacy are inherited as autosomal recessive traits. Defects in two different genes have been implicated in these vision disorders. Each gene encodes a subunit of the cone photoreceptor-specific cyclic-

976

  

GMP gated ion channel, the function of which is critical to the ability of cone photoreceptors to signal that light has been absorbed (Sundin et al., 2000; Wissinger et al., 2001).

Cone photopigments and their genes Photopigments are members of the superfamily of Gprotein coupled receptors. They are composed of two parts: an 11-cis-retinal chromophore and a protein component termed opsin. The chromophore is covalently bound to the opsin via a Schiff ’s base linkage to form the photopigment. The first step in vision is the absorption of light by the 11cis-retinal chromophore, which isomerizes it to all-trans retinal, and this, in turn, causes the opsin to undergo a conformational change, converting the pigment to the activated form. The activated visual pigment triggers a biochemical cascade of events, the phototransduction cascade, which ultimately results in hyperpolarization of the photoreceptor cell via closure of cyclic-GMP gated ion channels (see also Chapter 16). All G-protein coupled receptors are presumed to have evolved from a common ancestor. Likewise, all visual pigments are believed to have evolved from a common ancestor through the processes of gene divergence and duplication (Nathans et al., 1986a; Sharpe et al., 1999). The L and M genes each contain six exons, and each gene encodes an opsin of 364 amino acids (Nathans et al.,1986a). The genes encoding the L- and M-cone pigments share about 98% nucleotide sequence identity. Exons 1 and 6 of the L and M genes are identical; exons 2 through 5 contain nucleotide polymorphisms (Nathans et al., 1986a; Neitz et al., 1995; Winderickx et al., 1993). The S-pigment gene shares only about 45% nucleotide sequence identity with either the L or the M gene, and in evolution, it is estimated that the S gene diverged from the ancestor shared by the L and M genes more than 700 million years ago (Neitz et al., 2001). The S-pigment gene has five exons, which encode an opsin of 348 amino acids (Nathans et al., 1986a). A fraction of the amino acid differences among the opsins are responsible for differences in the relative absorption spectra of the pigments which make color vision possible. S T  C P A common feature of G-protein coupled receptors is that they have seven transmembrane alpha helical segments. In visual pigments, the chromophore is nestled in a hydrophobic pocket that is created by the transmembrane domain of the opsin. Specific interactions between the chromophore and the side chains of amino acids that line the hydrophobic pocket tune the absorption spectra of the pigments, determining the wavelengths of light to which each pigment is sensitive. Our understanding of spectral tuning in the human S pigment is

incomplete, and spectral variation in this pigment has not been observed. In contrast, the amino acid substitutions responsible for the variation in spectral sensitivity between and among the human L and M pigments are relatively well understood. Eighteen amino acid dimorphisms have been identified among and between the human L and M pigments. The effects of amino acid substitutions on the absorption spectra of the pigments have been investigated using a wide variety of experimental approaches (Asenjo et al., 1994; Merbs and Nathans, 1992, 1993; Neitz et al., 1989, 1991; Sharpe et al., 1998; Yokoyama and Radlwimmer, 1998). In vitro, substitutions at seven amino acid positions are required to shift the spectrum from that of the longest absorbing L pigment to that of the shortest absorbing M pigment (Asenjo et al., 1994). These seven substitutions are indicated in Figure 63.1. Two substitutions together, at amino acid positions 277 and 285, produce the spectral separation between M and L classes. Substitutions at each of five other positions (116, 180, 230, 233, and 309) produce small shifts (1 to 4 nm). Thus, a gene can be defined as encoding an M-class or Lclass pigment based on the amino acids specified by codons 277 and 285. M genes specify phenylalanine and alanine at positions 277 and 285, respectively, and peak near 530 nm. L genes specify tyrosine and threonine at positions 277 and 285, respectively, and peak near 560 nm. The other amino acid substitutions that produce small shifts in the wavelength of maximal sensitivity (lmax) can be thought of as producing spectral subtypes of M and L pigments (Fig. 63.1). There are fewer spectral subtypes of M than of L pigment, and the total range of variation in lmax is smaller in M than in L pigments. This appears to be the result of context-specific effects. For example, when a substitution is made at amino acid position 116 (encoded by exon 2) of the L pigment, it produces a shift of 2.5 nm. The same substitution made in the context of an M pigment does not produce a significant shift in spectral sensitivity. Similarly, a substitution at amino acid position 180 (encoded by exon 3) seems to produce a slightly smaller shift when introduced into an M pigment than into an L pigment. As will become apparent below, understanding the mechanism of spectral tuning in the human L and M pigments has provided insight into the molecular genetics of phenotypic variation in human color vision. A   L-  M-P G  N C V Among placental mammals, only primates have trichromatic color vision ( Jacobs, 1993). There are two major primate lineages, Old World primates and New World primates. The Old World primates include African and Asian monkeys, apes, and humans. New World primates inhabit South and Central America. Color vision in these two lineages seems to be at different stages of evo-

lution. Most species of New World monkey have a single visual pigment gene on the X chromosome; however, in diurnal species there are multiple alleles of the X chromosome visual pigment gene, providing the basis for trichromacy in females ( Jacobs, 1983). A female who receives two X chromosomes with different alleles encoding spectrally different pigments will express the two alleles in separate populations of cones because of X inactivation. The two cone populations produced by X inactivation in combination with the S cone provide heterozygous females with three spectrally distinct cone types, and they have trichromatic color vision. The males of theses species and the homozygous females have dichromatic color vision. Old World primates, like humans, have both an L- and an M-pigment gene on the X chromosome (Hunt et al., 1998; Onishi et al., 1999). Thus, it appears that trichromacy like that found in humans arose after the split between New and Old World primates, estimated to have occurred some 60 million years ago (Neitz et al., 2001). One exception is the howler monkey, a species of New World monkey that has uniform trichromatic color vision in both females and males ( Jacobs et al., 1996). Apparently, in this species a duplication of the Xchromosome photopigment gene occurred independently of that which occurred in the Old World primates (Dulai et al., 1999). Human red-green color vision is believed to have its origins in an ancestral primate (or protoprimate) that was the predecessor of all modern Old World monkeys and apes. The photopigment genes in macaque monkeys have recently been examined (Onishi et al., 1999). The vast majority of macaques have one L- and one M-pigment gene that are adjacent to one another on the X chromosome, with the M gene downstream of the L gene. We assume that this is the gene arrangement in the ancestors of modern humans and that it came about because of a relatively recent gene duplication event. It is believed that all placental mammals except Old World primates and a select group of New World primates ( Jacobs et al., 1996) have a single gene encoding a cone pigment on the X chromosome (Fig. 63.2A). Prior to the event that placed two photopigment genes on the X chromosome, our ancestors may have already reached a stage of color vision evolution in which they had multiple alleles of the X chromosome visual pigment gene similar to the situation that allows some New World monkey females, as described above, to enjoy trichromatic color vision. This is illustrated in Figure 63.2A, in which two single photopigment genes, L and M, are located on separate X chromosomes. Figure 63.2B illustrates how, in our ancient ancestor, a second photopigment gene could have been added to one X chromosome by an unequal crossover mechanism (Dulai et al., 1999). The length of the gene insertion extends from about 236 base pairs (bp) upstream of the start of the coding sequence of the M gene to about 18 kilobase pairs (kb) downstream of the last exon.

  :    

977

F 63.2. Trichromatic color vision arose in a primate ancestor by gene duplication. A, Ancestral arrays, like those found in most New World primate species, contain a single visual pigment. The human L- and M-photopigment gene array contains a large number of Alu sequences. From analysis of the DNA sequence of the array, it appears that the region between the Alu sequences indicated is responsible for duplicating the ancestral X chromosome visual pigment gene. B, The photopigment gene array responsible for uniform trichromacy arose when a second photopigment gene was added to the X chromosome. This is proposed to have occurred

by a favorable misalignment of two ancestral chromosomes containing a single pigment gene each. Recombination, possibly facilitated by Alu sequences, produced an X chromosome containing two visual pigment genes. In species that have just one pigment gene on the X chromosome, an enhancer element is present upstream of the gene and is required for expression of the gene. The enhancer was not duplicated when the second photopigment gene was added to the X chromosome. The enhancer has been termed the locus control region (LCR), and in humans it is required for expression of both L and M photopigments.

A feature of the region upstream of the human M gene is the presence of several Alu repeat elements. Human chromosomes contain about 1 million Alu repeat elements, and Alus have been implicated in unequal crossing over. It is possible that during meiosis, the two X chromosomes of a heterozygous female misaligned to bring Alu elements downstream of the L-like gene on one X chromosome into alignment with Alu elements just upstream of the M-like photopigment gene

on the other X chromosome, and these elements may have been involved in facilitating a crossover between the misaligned chromosomes. The result is the insertion of a primordial M-like gene downstream of a primordial L-like gene to form the predecessor of the modern human X chromosome photopigment gene array. This original unequal crossover should be considered to represent the occurrence of an extraordinarily improbable event. Normally, crossing

978

  

F 63.3 Intragenic crossovers alter the number of genes per array, intermix L- and M-gene sequences, and produce new arrays that cause color-blindness in males. Arrow diagrams for L and M genes are as indicated in Figure 63.1. Misaligned X chromosomes, each containing one L- and one M-pigment gene, undergo unequal homologous recombination within the L gene in one array and the M gene in the other array to produce two new arrays. The new

array with three genes contains a chimeric gene which contains exon 5 of the parental L gene and so encodes an L-class pigment. This type of array is common in males with deutan color vision defects. The other array has a single pigment gene that contains exon 5 from the parental M pigment gene, and so encodes an M pigment and is found in protanopes.

over requires the alignment of a long span of two homologous DNA sequences. Even though the presence of highly recombinogenic Alu sequences may have facilitated the event, it still must have been extremely unusual. Consider that Old World and New World monkeys have been separated in evolution by perhaps as long as 60 million years. Most New World species have not enjoyed the benefit of an unequal recombination event to provide them with uniform trichromacy despite the millions of meioses that occurred since they reached the stage where heterozygous females became trichromatic. After the first unequal crossover which placed two highly homologous photopigment genes in tandem on the X chromosome, the probability of subsequent unequal crossovers must have increased dramatically. The repeat unit (Fig. 63.2B), including the gene and intragenic sequences, is nearly 40 kb in length. This provides a large region of homology as the substrate for a crossover when the chromosomes misalign (Fig. 63.3). The combination of the tandem arrangement and their high degree of similarity has made the pigment gene array prone to unequal homologous crossovers, both within (intragenic) and between (intergenic) the genes. Both intra- and intergenic crossovers generate two new arrays, one with more and the other with fewer genes than the parental arrays. In addition, intragenic crossovers intermix the parental L- and M-gene sequences to produce chimeras (Fig. 63.3). The chimeric genes can encode pigments that differ in spectral sensitivity from either parental

pigment, with the magnitude of the spectral difference being determined by the number of spectral tuning sites (Fig. 63.1) that differ in the chimeric pigment compared to the parental pigment of the same class. All intragenic crossovers between normal arrays with two pigment genes result in gene arrays that cause color vision defects; either product of the crossover illustrated in Figure 63.3, when inherited by a male, will confer a color-deficient phenotype. Among humans, there is variation in the number of L and M genes per X chromosome array (Fig. 63.4) (DrummondBorg et al., 1989; Hayashi et al., 1999; Nathans et al.,1986a; Neitz and Neitz, 1995). Presumably, modern humans descended from an ancestor with two visual pigment genes on the X chromosome, and unequal homologous recombination produced the variation in color vision genotype and phenotype in the present population. The variety of arrays seen in modern humans with normal color vision is illustrated in Figure 63.4A. The most common array contains one L gene followed by two M genes. This is quite different from Old World monkeys, in which there is usually one M and one L gene (Onishi et al., 1999), an array type that is seen in only one in five humans. Humans can and often do have more than three pigment genes per X chromosome (Neitz and Neitz, 1995). The differences between humans and monkeys are remarkable. Humans have more photopigment genes on average than monkeys, and there is tremendous widespread variation in the human gene sequences that is absent in monkeys (Tickner et al., 2002).

  :    

979

F 63.4 X-chromosome visual pigment gene arrays underlying normal color vision and color vision deficiencies in the modern human population. A, Arrays underlying normal color vision contain variable numbers of genes encoding L and M photopigments. An L gene is always in the first (5¢-most) position. Additional L genes in normal arrays have most often been observed at the 3¢ end of the array. B, Arrays found in males with deutan color vision defects. Dichromats often have a single X chromosome pigment gene encoding an L-class pigment. Deuteranomalous

males have at least two different genes encoding spectral variants of L-class pigments and may or may not also have M-pigment genes. The M genes from deutan arrays are not expressed in the retina. In some cases deleterious mutations, such as one that substitutes the amino acid arginine for cysteine at position 203, causes the M pigment to be nonfunctional. C, Arrays found in males with protan color vision defects usually do not contain genes encoding L-class pigments but contain one or more genes encoding M-class pigments.

These differences are unexpected because both monkeys and humans arose from a common ancestor, and at present no satisfactory explanation has been offered.

is the dichromatic form, and it affects about 1% of Caucasian males and about 1 in 4000 females. Most deuteranopes are missing all genes encoding M pigments. Usually they have just one visual pigment gene on the X chromosome, and it encodes an L pigment (Fig. 63.4B). Presumably, arrays with a single L gene have been produced by an unequal intergenic recombination between two normal arrays. Deuteranopic males with normal-appearing arrays containing one L and one or two M genes have also been iden-

Molecular genetics of common inherited color vision defects D D Deutan defects are characterized by the absence of the M-cone contribution to vision. Deuteranopia

980

  

tified (Bollinger et al., 2001; Jagla et al., 2002). In these cases, when the M genes were sequenced, it was discovered that they contained a mutation that substituted the amino acid arginine for a cysteine at position 203 (C203R) of the pigment (Fig. 63.4B). This cysteine is highly conserved among all members of the superfamily of G-protein coupled receptors, and is involved in forming an essential disulfide bond in the pigment. The C203R mutation renders the pigment nonfunctional (Kazmi et al., 1997). Deuteranomaly is the anomalous trichromatic deutan defect and is the most common inherited red-green color vision defect, affecting about 5% of Caucasian males. Although deuteranomaly is characterized by the absence of an M-cone contribution to vision, in about two-thirds of deuteranomalous males M genes are present in genomic DNA. These males also have genes to encode two pigments of the L class. Figure 63.4B illustrates an array with three genes: a parental L gene, a parental M gene, and between them, a chimeric gene. This gene arrangement is frequently observed in deuteranomalous males. The chimeric gene derives its 5¢ end from a parental M gene and its 3¢ end from a parental L gene, and encodes what has historically been termed the anomalous pigment. The pigment encoded by the chimeric gene will be of the L class because it has retained exon 5 from the parental L gene (Fig. 63.1). It may differ in spectral sensitivity from the parental L pigment, depending on differences in amino acids at the spectral tuning sites encoded by exons 2, 3, and 4. One long-standing hypothesis suggests that the phenotypic variation in deuteranomaly is directly related to variability in the magnitude of the spectral separation in the underlying X-encoded pigments (e.g., Alpern, 1981). The normal pigments from the L and M classes have overlapping absorption spectra, but they are well separated in spectral peak (Fig. 63.1). Two different wavelengths of light produce different rates of photon capture in the L versus M cones, and the relative difference in photon catch is the basis for color discrimination. In the case of deuteranomaly, color vision is based on two pigment variants from the L class, which are not very different in spectral peak. Thus, it takes a much larger difference in the wavelength of two lights to produce the equivalent relative difference in the rate of photon capture in the two L-cone variants, and color discrimination ability is reduced. This predicts that if the chimeric gene and the parental L gene in the same array encode pigments that differ in spectra, then the person will be deuteranomalous, but if the spectra do not differ, then the person will be a deuteranope. Further, the degree of color vision loss among deuteranomalous males should be greater with a smaller spectral separation between the underlying L-class pigments. In recent experiments to test this hypothesis, DNA sequencing of the genes that encode L-class pigment genes in deuteranomalous males was used to deduce the amino acid sequences of the

pigments, and spectral tuning data were used to predict the spectral separation between them. Color vision phenotypes for the deuteranomalous males were determined by performance on standard color vision tests (Neitz et al., 1996; Shevell et al., 1998). There is a very strong correlation between color vision behavior and the spectral separation between the underlying pigments. However, there are some exceptions in that some deuteranomalous males have much worse color discrimination than others with the same spectral separation between their L-class pigments. One explanation for the poorer performance in some deuteranomalous males is that they have a bias in the ratio of cones expressing their two L-pigment variants. A very large range of variation has been observed in the ratio of L to M cones among males with normal color vision (Carroll et al., 2000; Roorda and Williams, 1999; Vimal et al., 1989). One might expect a similar variation in the ratio of two spectral variants of L cones in deuteranomalous retinas. It may be that color discrimination based on two cones that are quite similar in spectral peak is very sensitive to the proportion of the two cone types. If the ratio is very biased, it might adversely affect the already reduced color discrimination of an anomalous trichromat. The most typical gene arrays underlying deuteranomaly are ones with both M and L genes (Fig. 63.4B). The mystery has been why do men who inherit these arrays have a color vision defect? The C203R mutation has been found in the M genes of a few deuteranomalous males (Winderickx et al., 1992b), but many who have been examined have M genes with no identifiable defect. In a large-scale study to examine gene expression in retinas from male eye donors, Sjoberg et al. (1998) screened for the presence of mRNA from both M and L genes in a 6 mm2 foveal sample from each of 150 male donors. For 6% of the retinas (9/150), no M-pigment mRNA was detected, identifying these as putative deutan retinas. Using a very sensitive assay, the relative amount of L- and M-pigment mRNA was quantified in these retinas, and it was found that M pigment mRNA was below the detection limit of the assay (Bollinger et al., 2000), which was 1 molecule of M-pigment mRNA in 50,000 molecules of L-pigment mRNA. The conclusion was that there is a complete absence of M-gene expression in deutan retinas. It appears that this absence of expression occurs because the last gene in an array with three or more genes is not expressed in the retina (Hayashi et al., 1999). Thus, a primary cause of deuteranomaly is the production of gene rearrangements, as illustrated in Figure 63.3, in which a gene encoding an L-class pigment displaces the M gene to the 3¢ end of the array where it is not expressed. About one-third of deuteranomalous males lack M genes. As illustrated in Figure 63.3, an unequal crossover between ancestral two-gene arrays produces a deuteranomalous array that contains an M gene. However, some

  :    

981

deuteranomalous males (fewer than one-third) have two genes to encode L-class pigments but no M genes (Fig. 63.4B). To delete the M gene from a deuteranomalous array requires a second crossover event. The fact that two crossovers are required to produce a deuteranomalous array that lacks M genes might account for the relatively lower frequency of this array structure in deuteranomals. P D Protan defects are characterized by the absence of an L-cone contribution to vision, and most males with a protan defect do not have genes encoding an L pigment (Fig. 63.4C ). Protanopia is the dichromatic form, and protanomaly is the anomalous trichromatic form; each form affects about 1% of Caucasian men. They have Xchromosome arrays with genes for one or more M-class pigments. The recombination shown in Figure 63.3 generates an array with a single gene that is chimeric, with the 5¢ end from the parental L gene and the 3¢ end from the parental M gene. The gene encodes an M-class pigment because it derives exon 5 from the parental M gene (Fig. 63.1). A male with this array will be a protanope. Depending on the location of the crossover, the chimeric gene may encode a pigment that differs in amino acid sequence or in spectral sensitivity from the parental M pigment. Only rarely have protanopes who have an apparently intact L gene been identified, and the reason for the absence of L-cone function in these individuals is not known. Among males with protanomalous color vision there is phenotypic variation, although it seems to be less pronounced than the variation among deuteranomalous males. This is perhaps expected since there are fewer spectral variants of the M-class pigments (Fig. 63.1). Some protanomalous males have multiple genes (Fig. 63.4C) that encode M-class pigments that are expected to differ in lmax because they differ at spectral tuning positions encoded by exons 3 and 4 of the genes. These small spectral differences are presumably the basis for a small amount of red-green color vision in protanomalous trichromats. However, some protanomalous males have genes encoding M-class pigments that do not differ at the M-pigment spectral tuning sites. There is evidence that in some of these subjects, the M-class pigments differ in optical density. The effect of increasing optical density is due to “self-screening.” Two pigments that have the same spectral peak but differ in relative optical density will have spectral sensitivity curves in which the pigment with greater optical density has a broader spectral sensitivity curve, reflecting higher sensitivity to wavelengths on either side of the spectral peak. Near the peak and on the long-wavelength side of the peak, the difference in optical density between the two pigments qualitatively mimics a difference in spectral peak. We and our colleagues have suggested that amino acid polymorphisms encoded by exon 2 that shift the lmax of L but not of M pigments (Fig.

982

  

63.1) may alter the optical density of the M pigment, providing an effective difference in spectral sensitivity as the basis for a small amount of color discrimination (Neitz et al., 1999). However, some protan males who have genes encoding M-class pigments that differ at exon 2–encoded sites behave as dichromats, not anomalous trichromats ( Jagla et al., 2002). There are at least two possible explanations for this observation. First, it seems likely that the optical density of a pigment may depend on the exact amino acid sequence of the pigment. As mentioned above, there are 18 dimorphic amino acid positions. Whether two pigments that differ by exon 2–encoded substitutions differ in optical density may depend on the complete amino acid sequences of the two pigments. If so, then it is expected that not every set of two pigments that differ by exon 2–encoded amino acids will also differ in optical density. Another explanation is that in order to support color discrimination, the ratio of two M-class cones in a protan retina must be within an optimal range, and that for some protanomalous males, the ratio of cones is not within the required range. The evidence that some protanomalous males have two M-class pigments that do not differ in lmax, but instead differ in optical density is clear (Neitz et al., 1999). The optical density differences correlate with the presence of exon 2–encoded amino acid differences; however, it has not been established that exon 2–encoded differences cause the optical density difference. Conventionally, the diagnosis of color-defective individuals has been based on behavior in a color matching task, the Rayleigh color match, in which the subject is asked to discriminate mixtures of monochromatic red and green lights from a monochromatic yellow light. The clinical instruments used in color matching employ a small stimulus field subtending about 2 degrees. Interestingly, some subjects who cannot discriminate either a pure red or pure green light from yellow when the stimulus is small are able to do so if the stimulus is made large enough. One possible hypothesis had been that many dichromats might have had a very small number of anomalous cones, perhaps concentrated outside the macular region that they used for color vision if the stimulus was large enough. Recent results do not support this hypothesis; it has been shown that many of these people have only one pigment gene on the X chromosome and thus do not have the genetic basis for a second pigment absorbing in the middle to long wavelengths (Crognale et al., 1999). These people have exceedingly poor red-green color vision compared to normal individuals and even compared to most anomalous trichromats, discussed above, who can make redgreen color discriminations when the stimulus is small. The basis for their minimal amount of red-green color discrimination is not known. It may be that they are able to use differences in the spectral absorption properties of different cones that occur because of inhomogeneities in the retina,

such as the optical density difference between cones in the fovea and those in the peripheral retina.

S pigment (Weitz et al., 1992a; Weitz et al., 1992b) and are likely to interfere with protein folding.

A W  A P  C Conventionally, it has been said that anomalous trichromacy occurs when either the normal M or L pigment is replaced by an “anomalous” pigment. These anomalous pigments have been thought of as being abnormal—different from the L and M pigments that underlie normal color vision and present only in people with color vision defects and in female carriers of color vision defects. It has long been understood that female carriers usually have normal color vision; thus, having anomalous pigments and normal pigments in the same retina (in separate cones) does not alter the person’s color vision phenotype from normal to abnormal. However, more recently, it has been realized that there is huge variation in the amino acid sequences and spectral sensitivities of the normal L and M pigments. DNA sequence analysis of the genes encoding pigments underlying deuteranomalous, protanomalous, and normal color vision (Neitz et al., 1995, 1996, 1999; Shevell et al., 1998; Winderickx et al., 1993) reveals that there is overlap between the highly variant sequences of normal L pigments and the anomalous pigments of deuteranomaly, and between normal M pigments and the anomalous pigments of protanomaly. As described below, there is growing evidence for a stochastic model in which the identity of the cones as L versus M is defined solely by the spectral sensitivity of the expressed pigment. Since there is overlap in the spectral sensitivities of normal and anomalous pigments, in the context of the stochastic model, rather than being considered to have both a normal and an anomalous pigment, a deuteranomalous individual can be considered to have two spectral variants of L cones and protanomalous individuals can be considered to have two spectral variants of M cones. Thus, while the concept of normal versus anomalous pigments was useful in the context of earlier theories of color vision, it is somewhat inconsistent with what we now know about the biology underlying normal and color-defective vision.

B C M The genetic causes of blue cone monochromacy are heterogeneous, but all lead to a loss of function of both L and M cones. Two general causes of blue cone monochromacy have been identified (Nathans et al., 1989, 1993). One is the deletion of what has been termed the locus control region (LCR), a DNA element required for expression of both L- and M-pigment genes. The second is the presence of an inactivating point mutation, most commonly a C203R substitution, in an array with a single visual pigment gene. There are quite a few blue cone monochromats for whom the genetic cause has not been discovered. Some blue cone monochromats have been reported to have more than one class of functional cone (Smith et al., 1983). One possibility is that in cases in which the LCR has been deleted, there is still very low level expression of functional L or M pigment, enough to provide residual function of a second class of cone.

Molecular genetics of rare inherited color vision defects T Inherited blue-yellow color vision defects are caused by mutations in the S-cone pigment gene. Tritanopes base color vision on L and M cones, and they lack functional S-cones. The S-pigment gene is on chromosome 7, and since humans are diploid, each S-cone photoreceptor cell has and expresses the S-pigment genes from both copies of chromosome 7. A genetic defect in only one S-pigment gene can be sufficient to cause tritanopia. Three different amino acid substitutions have been identified as causes of tritanopia. All three substitutions are nonconserved amino acid substitutions that occur in the membrane-spanning domain of the

C A  F  I A O T B C M Occasionally, populations with an extraordinarily high incidence of color blindness have been identified. This is true of autosomal recessive incomplete achromatopsia, which generally is extremely rare, but among the Pingelapese islanders in Micronesia, the incidence is about 5%. In that population, the genetic cause has been found: an amino acid substitution in the beta subunit of the cyclic-GMP gated ion channel. All three cone types, S, M, and L, use the same cyclic-GMP gated ion channel in phototransduction. The channel has two subunits, alpha and beta, each encoded by a separate gene. The beta subunit is encoded by a gene identified as CNGB3, which resides on chromosome 8. Mutations in the gene encoding the alpha subunit, CNGA3, which resides on chromosome 2, have also been found in families with rod monochromacy (Wissinger et al., 1998). Recently, CNGA3 mutations were also found in patients with incomplete achromatopsia (Wissinger et al., 2001). The difference between complete and incomplete achromatopsia is that incomplete forms show residual cone function, whereas complete forms do not. One hypothesis is that some of the mutations in the ion channel subunits do not completely abolish function and give rise to the incomplete forms.

X-chromosome pigment gene expression and the identity of photoreceptors as M versus L cones. G E   P G A The rules and mechanisms that govern expression of the Xchromosome visual pigment genes remain largely unknown,

  :    

983

F 63.5. Transcription of L- and M-pigment genes. A, The L- and M-pigment gene array contains a single enhancer that is 5¢ (left) of the genes in the array. Each gene has its own promoter. The sequence of the promoter for the first gene in the array differs slightly from the promoter for other genes in the array. An interaction between the upstream enhancer and the promoter of an individual pigment gene is required for transcription of the gene.

B, In cones that express L pigment, the enhancer interacts only with the promoter for the L gene, and the promoters for the other genes in the array are excluded from interactions with the enhancer. C, In cones that express the M pigment, the enhancer interacts only with the promoter for an M gene, and other pigment gene promoters are excluded from interactions with the enhancer.

but what is known has been important for understanding the molecular genetics of inherited red-green color vision defects. Normal color vision requires at least one L and one M gene from the X-chromosome array to be functionally expressed in the retina, and the M and L genes must be expressed in separate populations of cones. Among humans with normal color vision there is variation in the number of both L and M genes. Most people have more than the two genes that are the minimum required for normal color vision (Fig. 63.4). Arrays with multiple M genes are most common; however, arrays with multiple L genes are not unusual (Drummond-Borg et al., 1989; Jørgensen et al., 1990; Nathans et al., 1986a; Neitz et al., 1995; Sjoberg et al., 1998). Each L- or M-cone photoreceptor expresses only one gene from the X-chromosome array (Hagstrom et al., 2000). Two questions that remain unresolved are (1) how does a photoreceptor choose which gene will be expressed and prevent expression of the other genes from the X-

chromosome pigment gene array? and (2) which of the genes from an individual array are expressed and which are not? The approximately 200 bp region immediately upstream of the coding sequence of each visual pigment gene on the X-chromosome contains promoter elements required for transcription (Fig. 63.5). Although highly homologous, the promoter for the gene in the 5¢ position in the array is not quite identical to the promoters for the other genes in the array (Nathans et al., 1986b). As a consequence of the tandem arrangement of the genes, the downstream genes added by unequal homologous recombination have the same promoter sequence as the parental M gene. Transcription of the individual genes within the Xchromosome array also requires an upstream enhancer that lies 3.5 kb upstream of the first gene in the array. An enhancer is a cis-acting DNA element that promotes transcription of a gene in a distance and orientation independent fashion. The X-chromosome visual pigment gene

984

  

enhancer is conserved across mammalian species, both in sequence and in location (Wang et al., 1992). As shown in Figure 63.2, the enhancer was not included when the second gene was added to create a gene array in our primate ancestor. Thus, all genes in the array must share the same enhancer (Fig. 63.5), the LCR (Wang et al., 1992). A major unresolved issue is what determines the identity of a cone as L versus M. Two possibilities have been considered. The first is that the L and M cones each have a unique identity that is determined before the pigment genes are expressed. During development a cell adopts either an L- or an M-cone cell fate, and as a result, each expresses a unique subset of genes, including transcription factors to specifically activate transcription of an L- or M-pigment gene based on the predetermined identity of the cell. If this is indeed the case, then there can be aberrations in gene expression such that an L pigment could be misexpressed in an M cone and vice versa. In this model, the LCR is a powerful enhancer which may play a role in cell- and tissuespecific transcription but not in determining the identity of the cones as L versus M. The second possibility is that L and M cones are identical until, by the action of a stochastic selection mechanism, each cell randomly makes a one-time choice of one Xchromosome visual pigment gene from the array for expression. In this scenario, termed the stochastic model, the identity of the cone is determined by the spectral sensitivity of the pigment that was randomly chosen for expression, and thus there is no possibility for an L cone to express an M pigment or vice versa. The stochastic model can be considered to be very general, independent of the exact mechanism that makes the random choice. However, one specific theory has been postulated in which the LCR acts as the stochastic selector, forming a stable and irreversible complex with only one pigment gene promoter per cell, thereby ensuring that each cell expresses a single X-chromosome visual pigment gene (Hayashi et al., 1999; Wang et al., 1999) (Fig. 63.5). Our evolutionary ancestors had a single X-linked pigment gene. If for them the binding of the enhancer to the promoter (via DNA-protein and protein-protein interactions) was irreversible, then there would be no need to separately evolve a mechanism to direct mutually exclusive expression of multiple genes once the gene duplication occurred (Fig. 63.2). It is possible, as illustrated in Figure 63.2, that when the gene duplication occurred, our ancestors had already reached a stage in evolution in which there were multiple alleles at the X-chromosome photopigment gene locus. If so, the exclusive interaction of the enhancer with only one Xchromosome pigment gene promoter in each cell could have been exploited as the mechanism to direct the pigments into two separate subpopulations of cells. This allows the possibility that males and homozygous females could have

become trichromatic by the stroke of one highly favorable mutational event, which both added a gene and provided a mechanism for mutually exclusive expression. Extremely attractive are both the idea that the identity of M versus L cones is determined by a stochastic process and the idea that the mutually exclusive expression of L and M genes is the result of exploiting a genetic mechanism that fortuitously preexisted in our ancestors with a single Xchromosome pigment gene. However, there is no direct experimental evidence for the formation of an irreversible enhancer-promoter complex, and as far as we know, there is no precedent for such an irreversible enhancer-promoter interaction in any other system. Even if the simple idea of a permanent binding of the enhancer to the promoter via a protein complex ultimately proves to be incorrect, the stochastic model and the idea of a one-step mutation providing both the photopigment basis for trichromacy and mutually exclusive expression for the new gene are important as general theories independent of the exact mechanisms that are envisioned to implement them. As noted above, in the human retina there is tremendous individual variation in the L:M cone ratio, with an average 2:1 ratio of L to M cones (Carroll et al., 2000; Cicerone and Nerger, 1989; Miyahara et al., 1998; Roorda and Williams, 1999; Rushton and Baker, 1964; Wesner et al., 1991). Under the stochastic model, explaining the observed range of variation in the L : M cone ratio in people with normal color vision requires that the stochastic mechanism choose to activate transcription from L and M genes unequally and with different probabilities for different people. Also, in general, there must be a higher probability that, in normal arrays, an L gene will be chosen for expression. The structure of normal arrays, with an L gene in the first position, followed by M gene(s), suggests the possibility of a relationship between the proximity of a pigment gene to the LCR and the probability of being chosen for expression. Deeb and colleagues have proposed that only the first two genes in the array are expressed at levels significant for color vision, the third gene being effectively out of range of the LCR (Hayashi et al., 1999; Winderickx et al., 1992b). There is very clear evidence that the first gene in the array is preferentially expressed in humans (Balding et al., 1998; Sjoberg et al., 1998) and that the last gene in arrays with three or more genes is not expressed (Hayashi et al., 1999; Winderickx et al., 1992b). As introduced above, this absence of expression of the third gene in arrays with three genes is what causes the color vision defect in many deuteranomalous individuals. However, our understanding is incomplete. For example, there are reported cases in which expression of more than two of the genes from the X-chromosome array has been clearly demonstrated (Sjoberg et al., 1998). Finding exceptions implies that the rule that only two genes are expressed is not absolute. Recently, in a series of experiments using an

  :    

985

extremely sensitive, highly quantitative assay, mRNA from the last gene in the array from a series of male eye donors who each had three or more X-chromosome visual pigment genes, it was demonstrated that the 3¢-most gene in each array was not expressed at a detectable level. The detection limit of the assay was sufficient to detect mRNA from the last gene in the array if it was expressed in a ratio of 1 in 50,000 copies of mRNA from other X-chromosome pigment genes (Bollinger et al., 2000). Our conclusion is that the absence of expression of these genes is complete. In the model where the LCR serves as the stochastic selector, it has been envisioned that the probability of a gene’s being expressed in any cone is determined by its proximity to the LCR. In that model, it is conceived that the probability of expression could be markedly lower for the third gene but is probably not zero (Hayashi et al., 1999; Wang et al., 1999). Presently, research related to the expression of the L and M genes is very active. As more becomes known about this very puzzling process, the mechanisms that govern expression of the X-chromosome photopigment genes will be elucidated. This will, in turn, provide insight into the mechanism that produces the huge variation in L:M cone ratio across individuals. Ultimately, we will also know if the stochastic model is correct. If it is, it suggests that the only difference between an L and an M cone is the photopigment it expresses. This raises the question of how the neural circuits for red-green color vision, which compare the outputs of the L and M cones, arise.

Acknowledgments We thank P. M. Summerfelt for technical assistance and J. Carroll, K. L. Gunther, and C. McMahon for valuable comments and suggestions. The writing of this chapter was supported by National Institutes of Health Grants EY09303, EY09620, & EY01931 and by Research to Prevent Blindness.

REFERENCES Alpern, M., 1981. Color blind color vision, Trends Neurosci., 4:131–135. Asenjo, A. B., J. Rim, and D. D. Oprian, 1994. Molecular determinants of human red/green color discrimination, Neuron, 12:1131–1138. Balding, S. D., S. A. Sjoberg, J. Neitz, and M. Neitz, 1998. Pigment gene expression in protan color vision defects, Vis. Res., 38:3359–3364. Bollinger, K., C. Bialozynski, J. Neitz, and M. Neitz, 2001. The importance of deleterious mutations of M pigment genes as a cause of color vision defects, Color Res. Appl., 26:S100–S105. Bollinger, K., M. Neitz, and J. Neitz, 2000. Topographical expression of first vs. downstream L pigment genes in a retina from a male with two L and one M genes, Invest. Ophthalmol. Vis. Sci., 41(Suppl):S807.

986

  

Bowmaker, J. K., H. J. A. Dartnall, and J. D. Mollon, 1980. Microspectrophotometric demonstration of four classes of photoreceptor in an Old World primate, Macaca fascicularis, J. Physiol., 298:131–143. Carroll, J., C. McMahon, M. Neitz, and J. Neitz, 2000. Flicker photometric electroretinogram estimates of L : M cone photoreceptor ratio in men with photopigment spectra derived from genetics, J. Opt. Soc. Am. A, 17:499–509. Cicerone, C. M., and J. L. Nerger, 1989. The relative numbers of long-wavelength-sensitive to middle-wavelength-sensitive cones in the human fovea centralis, Vis. Res., 29:115–128. Crognale M. A., D. Y. Teller, T. Yamaguchi, A. G. Motulsky, and S. S. Deeb, 1999. Analysis of red/green color discrimination in subjects with a single X-linked photopigment gene, Vis. Res., 39:707–719. Dartnall, H. J. A., J. K. Bowmaker, and J. D. Mollon, 1983. Human visual pigments: microspectrophotometric results from the eyes of seven persons, Proc. R. Soc. Lond. B, 220:115–130. Drummond-Borg, M., S. S. Deeb, and A. G. Motulsky, 1989. Molecular patterns of X-chromosome-linked color genes among 134 men of European ancestry, Proc. Natl. Acad. Sci. USA, 86:983–987. Dulai, K. S., M. von Dornum, J. D. Mollon, and D. M. Hunt, 1999. The evolution of trichromatic color vision by opsin gene duplication in New World and Old World primates, Genome Res. 9:629–638. Fasick, J. I., N. Lee, and D. D. Oprian, 1999. Spectral tuning in the human blue cone pigment, Biochemistry, 38:11593–11596. Hagstrom, S. A., M. Neitz, and J. Neitz, 2000. Cone pigment gene expression in individual photoreceptors and the chromatic topography of the retina, J. Opt. Soc. Am. A, 17:527– 537. Hayashi, T., A. G. Motulsky, and S. S. Deeb, 1999. Position of a “green-red” hybrid gene in the visual pigment array determines colour-vision phenotype, Nat. Genet., 22:90–93. Hunt, D. M., K. S. Dulai, J. A. Cowing, C. Julliot, J. D. Mollon, J. K. Bowmaker, W. Li, and D. Hewett-Emmett, 1998. Molecular evolution of trichromacy in primates, Vis. Res., 38:3299– 3306. Jacobs, G. H., 1983. Within-species variations in visual capacity among squirrel monkeys (Saimiri sciureus): sensitivity differences, Vis. Res., 23:239–248. Jacobs, G. H., 1993. The distribution and nature of colour vision among the mammals, Biol. Rev., 68:413–471. Jacobs, G. H., M. Neitz, J. F. Deegan, and J. Neitz, 1996. Emergence of routine trichromatic colour vision in New World monkeys, Nature, 382:156–158. Jagla, W. M., H. Jägle, T. Hayashi, L. T. Sharpe, and S. S. Deeb, 2002. The molecular basis of dichromatic color vision in males with multiple red and green visual pigment genes, Hum. Mol. Genet., 11:23–32. Jørgensen, A. L., S. S. Deeb, and A. G. Motulsky, 1990. Molecular genetics of X-chromosome-linked color vision among populations of African and Japanese ancestry: high frequency of a shortened red pigment gene among Afro-Americans, Proc. Natl. Acad. Sci. USA, 87:6512–6516. Kazmi, M. A., T. P. Sakmar, and H. Ostrer, 1997. Mutation of a conserved cysteine in the X-linked cone opsins causes color vision deficiencies by disrupting protein folding and stability, Invest. Ophthalmol. Vis. Sci., 38:1074–1081. Merbs, S. L., and J. Nathans, 1992. Absorption spectra of the hybrid pigments responsible for anomalous color vision, Science, 258:464–466.

Merbs, S. L., and J. Nathans, 1993. Role of hydroxyl-bearing amino acids in differentially tuning the absorption spectra of the human red and green cone pigments, Photochem. Photobiol., 58: 706–710. Miyahara, E., J. Pokorny, V. C. Smith, R. Baron, and E. Baron, 1998. Color vision in two observers with highly biased LWS/MWS cone ratios, Vis. Res., 38:601–612. Nathans, J., C. M. Davenport, I. H. Maumenee, R. A. Lewis, J. F. Hejtmancik, M. Litt, E. Lovrien, R. Weleber, B. Bachynski, F. Zwas, R. Klingaman, and G. Fishman, 1989. Molecular genetics of blue cone monochromacy, Science, 245:831–838. Nathans, J., I. A. Maumenee, E. Zrenner, B. Sadowski, L. T. Sharpe, R. A. Lewis, E. Hansen, P. Rosenberg, M. Schwartz, J. R. Heckenlively, E. Trabousli, R. Klingaman, N. T. BechHansen, G. R. LaRouche, R. A. Pagon, W. H. Murphy, and R. G. Weleber, 1993. Genetic heterogeneity among blue-cone monochromats, Am. J. Hum. Genet., 53:987–1000. Nathans, J., T. P. Piantanida, R. L. Eddy, T. B. Shows, and D. S. Hogness, 1986a. Molecular genetics of inherited variation in human color vision, Science, 232:203–210. Nathans, J., D. Thomas, and D. S. Hogness, 1986b. Molecular genetics of human color vision: the genes encoding blue, green, and red pigments, Science, 232:193–202. Neitz, J., J. Carroll, and M. Neitz, 2001. Color vision: almost reason enough for having eyes, Opt. Photon. News, 12:26–33 (electronic copy available at http://www.mcw.edu/cellbio/colorvision/). Neitz, J., and G. H. Jacobs, 1986. Polymorphism of the longwavelength cone in normal human color vision, Nature, 323: 623–625. Neitz, M. and J. Neitz, 1995. Numbers and ratios of visual pigment genes for normal red-green color vision, Science 267:1013–1016. Neitz, M., J. Neitz, and A. Grishok, 1995. Polymorphism in the number of genes encoding long-wavelength sensitive cone pigments among males with normal color vision, Vis. Res., 35:2395–2407. Neitz, J., M. Neitz, J. C. He, and S. K. Shevell, 1999. Trichromatic color vision with only two spectrally distinct photopigments, Nat. Neurosci., 2:884–888. Neitz, M., J. Neitz, and G. H. Jacobs, 1989. Analysis of fusion gene and encoded photopigment of colour-blind humans, Nature, 342:679–682. Neitz, M., J. Neitz, and G. H. Jacobs, 1991. Spectral tuning of pigments underlying red-green color vision, Science, 252:971–974. Neitz, J., M. Neitz, and G. H. Jacobs, 1993. More than three different cone pigments among people with normal color vision, Vis. Res., 33:117–122. Neitz, M., J. Neitz, and G. H. Jacobs, 1995. Genetic basis of photopigment variations in human dichromats, Vis. Res., 35:2095–2103. Neitz, J., M. Neitz, and P. M. Kainz, 1996. Visual pigment gene structure and the severity of human color vision defects, Science, 274:801–804. Onishi, A., S. Koike, M. Ida, H. Imai, Y. Shichida, O. Takenaka, A. Hanazawa, H. Komatsu, A. Mikami, S. Goto, B. Suryobroto, K. Kitahara, T. Yamamori, and H. Komatsu, 1999. Dichromatism in macaque monkeys, Nature, 402:139–140. Roorda, A., and D. R. Williams, 1999. The arrangement of the three cone classes in the living human eye, Nature, 397:520– 522. Rushton, W. A. H., and H. D. Baker, 1964. Red/green sensitivity in normal vision, Vis. Res., 4:75–85. Schnapf, J. L., T. W. Kraft, and D. A. Baylor, 1987. Spectral sensitivity of human cone photoreceptors, Nature, 325:439–441.

Sharpe, L. T., A. Stockman, H. Jägle, H. Knau, G. Klausen, A. Reitner, and J. Nathans, 1998. Red, green, and red-green hybrid pigments in the human retina: correlations between deduced protein sequences and psychophysically measured spectral sensitivities, J. Neurosci., 18:10053–10069. Sharpe, L. T., A. Stockman, H. Jägle, and J. Nathans, 1999. Opsin genes, cone photopigments, color vision, and color blindness, in Color Vision: From Genes to Perception (K. R. Gegenfurtner, and L. T. Sharpe, eds.), New York: Cambridge University Press, pp. 3–52. Shevell, S. K., J. C. He, P. M. Kainz, J. Neitz, and M. Neitz, 1998. Relating color discrimination to photopigment genes in deutan observers, Vis. Res., 38:3371–3376. Sjoberg, S. A., M. Neitz, S. D. Balding, and J. Neitz, 1998. L-cone pigment genes expressed in normal colour vision, Vis. Res., 38:3213–3219. Smith, V. C., J. Pokorny, J. W. Delleman, M. Cozinjinsen, W. A. Houtman, and L. N. Went, 1983. X-linked incomplete achromatopsia with more than one class of functional cones, Invest. Ophthalmol. Vis. Sci., 24:451–457. Sundin, O. H., Y. M. Yang, Y. Li, D. Shu, J. N. Hurd, T. N. Mitchell, E. D. Silva, and I. H. Maumenee, 2000. Genetic basis of total colourblindness among the Pingelapese islanders, Nat. Genet., 3:289–293. Tickner, M. K., M. Neitz, and J. Neitz, 2002. Extreme polymorphism in L & M photopigment genes of humans but not monkeys, [Abstract] 2002 Annual Meeting Abstract and Program Planner [on CD-ROM]. Association for Research in vision and Ophthalmology. Abstract 3792. Vimal, R. L. P., J. Pokorny, V. C. Smith, and S. K. Shevell, 1989. Foveal cone thresholds, Vis. Res., 29:61–78. Wang, Y., J. P. Macke, S. L. Merbs, D. J. Zack, B. Klaunberg, J. Bennett, J. Gearhart, and J. Nathans, 1992. A locus control region adjacent to the human red and green visual pigment genes, Neuron, 9:429–440. Wang, Y., P. M. Smallwood, M. Cowan, D. Blesh, A. Lawler, and J. Nathans, 1999. Mutually exclusive expression of human red and green visual pigment-reporter transgenes occurs at high frequency in murine cone photoreceptors, Proc. Natl. Acad. Sci. USA, 96:5251–5256. Weitz, C. J., Y. Miyake, K. Shinzato, E. Montag, E. Zrenner, L. N. Went, and J. Nathans, 1992a. Human tritanopia associated with two amino acid substitutions in the blue sensitive opsin, Am. J. Hum. Genet., 50:498–507. Weitz, C. J., L. N. Went, and J. Nathans, 1992b. Human tritanopia associated with a third amino acid substitution in the blue sensitive visual pigment, Am. J. Hum. Genet., 51:444–446. Wesner, M., J. Pokorny, S. Shevell, and V. Smith, 1991. Foveal cone detection statistics in color-normals and dichromats, Vis. Res., 31:1021–1037. Winderickx, J., L. Battisti, Y. Hibibya, A. G. Motulsky, and S. S. Deeb, 1993. Haplotype diversity in the human red and green opsin genes: evidence for frequent sequence exchange in exon 3, Hum. Mol. Genet., 2:1413–1421. Winderickx, J., D. T. Lindsey, E. Sanocki, D. Y. Teller, A. G. Motulsky, and S. S. Deeb, 1992a. Polymorphism in red photopigment underlies variation in colour matching, Nature, 356: 431–433. Winderickx, J., E. Sanocki, D. T. Lindsey, D. Y. Teller, A. G. Motulsky, and S. S. Deeb, 1992b. Defective colour vision associated with a missense mutation in the human green visual pigment gene, Nat. Genet., 1:251–256. Wissinger, B., D. Gamer, H. Jägle, R. Giorda, T. Marx, S. Mayer, S. Tippmann, M. Broghammer, B. Jurklies, T. Rosenberg, S. G.

  :    

987

Jacobson, E. C. Sener, S. Tatlipinar, U. Kellner, B. Lorenz, G. Wolff, C. Verellen-Dumoulin, M. Schwartz, F. P. M. Cremers, E. Apfelstedt-Sylla, E. Zrenner, R. Salati, L. T. Sharpe, and S. Kohl, 2001. CNGA3 mutations in hereditary cone photoreceptor disorders, Am. J. Hum. Genet., 69:722–732. Wissinger, B., H. Jägle, S. Kohl, M. Broghammer, B. Baumann, D. B. Hanna, C. Hedels, E. Apfelstedt-Sylla, G. Randazzo, S. G.

988

  

Jacobson, E. Zrenner, and L. T. Sharpe, 1998. Human rod monochromacy: linkage analysis and mapping of a cone photoreceptor expressed candidate gene on chromosome 2q11, Genomics, 51:325–331. Yokoyama, S., and F. B. Radlwimmer, 1998. The “five-sites” rule and the evolution of red and green color vision in mammals, Mol. Biol. Evol., 15:560–567.

64

Linking Retinal Circuits to Color Opponency DAVID J. CALKINS

Mapping color perception to a physiological substrate L C O  C A We are able to perceive an amazingly diverse range of hue, or what we call in the vernacular color. The tremendous variability in the spectral composition of light reflected from surfaces lends itself to eliciting a daunting gamut of more than 100,000 discriminable colors, and the variation in the names we assign these colors is limited only by the scope of human experience. Yet, even with this variation, no demographic culture requires more than 11 color names to describe the quality of any hue (reviewed in Boynton, 1975). Of these 11, 5 can be described using either black or white in combination with the four unique hues—blue, green, yellow, and red (Bornstein, 1973). These four hues are themselves irreducible as percepts, and in that sense, each can be mapped at least conceptually to a perceptual channel whose activity correlates with that hue. The combined activity between channels presumably is what produces the rich variety of colors we experience. The precise design of our visual system rigidly constrains how the activity of the color channels is mapped to hue sensation. Our ability to discriminate surfaces based on differences in spectral reflectance alone arises from a neural comparison of the rates of quantal absorption by the S-, M-, and L-cone photoreceptors (see Lennie and D’Zmura, 1988, for review). This neural comparison delimits color activity in the brain and has a characteristic signature that imposes upon hue perception a natural constraint. The four unique hues are organized into mutually exclusive or opponent pairs, blue/yellow (B/Y) and red/green (R/G). The members in each pair are opponent in the sense that we cannot perceive them simultaneously; their perceptive fields are spatially coextensive and cancellatory. We may perceive hue combinations between these pairs—for example, red and blue yielding a percept that is at once both (namely, violet)— but not combinations within a pair. Thus, there are no such hues as red-green or blue-yellow; we do not experience these percepts and therefore do not have names for them. This inherent phenomenology is explained in abstract terms by identifying each opponent pair with an independent color channel, B/Y and R/G, and through these channels all color perception is mediated.

A diverse body of psychophysical data implies that the B/Y and R/G channels each correlate with a neural pathway in which signals from the three cone types converge with one another in different antagonistic combinations (Hurvich and Jameson, 1957). The particular combination of cone antagonism bestows upon each channel a unique spectral sensitivity that correlates strongly with our perception of hue across the visible spectrum. For B/Y opponency, signals from S cones are combined antagonistically with an additive signal from M and L cones. This combination is abbreviated as S/(M + L), where “/” indicates antagonism or subtraction. Very often the denotation S-(M + L) is used instead, and the psychophysical spectral sensitivity of the channel (which by definition cannot be a negative number) is derived from the absolute value of the difference (Fig. 64.1). For R/G opponency, signals from L cones are combined antagonistically with those from M cones (abbreviated as L/M or L-M; Calkins et al., 1992). There is also a strong input from S cones into the R/G channel with the same polarity as the L-cone signal—the S signal can be canceled with appropriate stimulation of the M cones (Hurvich and Jameson, 1957; Stromeyer et al., 1998). Thus, much of the short-wavelength spectrum appears both blue and red, indicating activation of both the B/Y and R/G channels (DeValois et al., 2000a; Krauskopf et al., 1982; Wooten and Werner, 1979). The denotation “L/M” is a generalization that most visual scientists accept as a reasonable representation of the R/G channel but does not incorporate the S contribution. Each particular combination, S/(M + L) or L/M, therefore represents the minimal condition that describes the defining opponency within a neural pathway consistent with the psychophysical properties of the appropriate color channel. There are therefore two primary considerations in assigning an anatomical substrate to the cone-antagonism within the S/(M + L) and L/M pathways: (1) the source of each pure cone signal prior to the site of antagonistic convergence and (2) the mechanism of antagonism itself. The first addresses the mechanism through which excitation from each cone type is collected or pooled and whether this pooling is indeed independent of the other photoreceptors (represented as “Stage 1” in Fig. 64.1). This stage accurately conveys the spectral sensitivity of the cone, including amplitude changes in cone sensitivity due to adaptation, and

989

Stage 1

S

M

-

Stage 2

-

+

L

+

K1S (K2M K3L)

L

+ (K4L

S

M

+ K5S) -K6M

F 64.1. A two-stage cone-antagonistic model of color opponency. For B/Y opponency (left), signals from S cones are combined antagonistically (indicated by “-”) with the sum of signals from M and L cones (indicated by “+”); the net spectral sensitivity of the channel is given by the absolute value of the difference of the cone terms. Similarly, for R/G opponency (right), a combined signal from L and S cones is subtracted from signals from M cones. The K coefficients scale the spectral sensitivity of each cone and represent the combined effects for both stages in the model.

possibly modulates the cone signal further through its own intrinsic filters. In retinal terms, this stage likely corresponds to one or more types of bipolar cell that collect from cones and feed a glutamatergic excitatory signal forward to the ganglion cells. The second consideration addresses the anatomical site and mechanism through which signals from different cone types converge with opposite polarity, the socalled critical locus of opponency (“Stage 2” in Fig. 64.1; Teller and Pugh, 1983). This stage too could modulate or filter the collected cone signal through its own intrinsic properties. However, unlike the first stage, its output depends not on the spectral sensitivity of one cone type, but on the difference in sensitivity between two or more types. This difference signal forms the spectral signature of the color channel itself (for review, see Calkins and Sterling, 1999). In the primate retina, such antagonism between cones could arise through the convergence of an excitatory signal, say from a bipolar cell, with an inhibitory signal through lateral connections with horizontal cells (GABA-ergic) or amacrine cells (glycinergic), or more likely some combination of these. In the case of B/Y opponency, the antagonism is thought to involve the convergence of strictly excitatory signals from bipolar cells that respond to light with opposite polarity, that is, OFF cells versus ON cells. Thus, cone antagonism is not necessarily synonymous with physiological inhibition. C   C P There is great potential for spatial and temporal modulation of each cone’s signal through the two stages described in Figure 64.1 (e.g., Pugh and Mollon, 1979), the nature of which is well beyond the scope of this chapter. Because the sensitivity of each cone depends on ambient conditions, there is also the potential for spectral modulation (represented by the coeffi-

990

  

F 64.2. Data points represent the spectral sensitivity of the R/G (open squares) and the B/Y (filled circles) color channels estimated from chromatic scaling measurements (Romeskie, 1978). Solid curves were calculated from the model in Figure 64.1. The color of each curve represents the chromatic valence at each wavelength. The bottom traces represent the scaled sensitivity of the R/G and B/Y channels under typical psychophysical conditions. The color of the individual peaks represents the hue sensation of a monochromatic light at that wavelength. Threshold detection of monochromatic test lights would be mediated by the most sensitivity mechanism at each wavelength. Lights at and just above threshold elicit unique hues, while lights well above threshold generally fall and are detected through both R/G and B/Y channels. The location of unique yellow corresponds to the neutral point of the R/G channel, where detection is mediated solely by the B/Y channel (arrow). (See color plate 40).

cients K in Fig. 64.1). In effect, this modulation scales the amplitude of the cone sensitivities relative to one another, depending on the spectral composition and intensity of the ambient illumination. This shift greatly influences the resultant spectral sensitivity of the S/(M + L) and L/M pathways and therefore how the activity of the pathways partitions the visible spectrum into regions of dominant hue (Fig. 64.2). Thus, the color appearance of a monochromatic light under typical psychophysical conditions (i.e., a small spot on a larger adapting field) is directly related to the spectral sensitivity and activity of each channel. This relationship between spectral sensitivity and hue perception allows some key predictions about the wiring of the S/(M + L) and L/M pathways. Most obviously, the output of a pathway should result in one hue and one hue alone, whose quality depends solely on the cone term dominating that output (as indicated in Fig. 64.2). Therefore the unique hues—blue, green, yellow, and red—ought to correlate closely with the activity of a pathway functioning in isolation. This is all to say that the S/(M + L) and L/M pathways should, in some measure, demonstrate separability and

T 64.1

Some reviews of primate retinal circuitry and color vision Review Lennie and D’Zmura (1988) Kaplan et al. (1990) Wässle and Boycott (1991) Lee (1996) Martin (1998) Dacey (1999) Calkins and Sterling (1999) Dacey (2000) Calkins (2001)

Special Focus Early visual pathways and color psychophysics P-cell physiology and receptive field characteristics Retinal mosaics and circuitry Ganglion cell receptive field types Circuitry for receptive field formation Circuits for B/Y and R/G opponency Midget ganglion cells and R/G color channel Cone contributions to ganglion and horizontal cells Circuitry from S cones, B/Y color channel

independence. As a corollary to this condition, the neutral points of each pathway (wavelengths where the cone terms cancel) ought to correspond to a locus of unique hue, where the other pathway solely mediates detection. For example, the wavelength at which the L- and M-cone terms in the L/M pathway cancel one another (570 to 580 nm under typical adaptive conditions) corresponds to unique yellow because detection is mediated only by the (M + L) envelope of the S/(M + L) pathway (Fig. 64.2). The chromatic neutral points are therefore closely dependent on the relative shift in cone sensitivity (reflected in the coefficients K in Figs. 64.1 and 64.2) prior to the stage of antagonistic convergence. The neutral points, then, represent the chromatic signature of the S/(M + L) and L/M pathways, and whatever circuitry in the visual system lends itself to establishing the critical locus of cone antagonism should in some measure support this signature. The optics of the eye, the spatial sampling of retinal neurons, the ratio of rods to cones, and the relative numbers of L and M cones all change dramatically with increasing retinal eccentricity. Despite these variations, for normal trichromatic observers the S/(M + L) and L/M neutral points are remarkably invariant (Hibino, 1992; Kuyk, 1982; Nerger et al., 1995). Along these same lines, when appropriate stimuli are delivered to the peripheral retina, even as far out as 90 degrees, color is perceived with the same range of hues and with the same capacity to discriminate hues as in the fovea (Gordon and Abramov, 1977; Noorlander et al., 1983; Stabell and Stabell, 1982; van Esch et al., 1984). It is true that larger stimuli are required in the peripheral retina to produce comparable sensations, but this is not surprising given the decrease in spatial resolution of retinal mosaics and in the sensitivity of the color channels. L C  C A In short, the fundamental features of the color opponent channels are similar between the fovea and the peripheral retina—despite the common and mistaken belief that color discrimination

is a special function of the central retina. This consistency suggests two general possibilities for wiring the S/(M + L) and L/M pathways. The first possibility is that cone antagonism is established in the retina, within the presynaptic circuitry of one or more types of ganglion cell. In this case, the spectral signature of the color pathways begins with the particular circuitry producing the cone antagonism and is then conveyed to the cortex in a manner that is conserved across retinal eccentricity. This would place both stages of the generalized model in Figure 64.1 within the retina. The second possibility is that the cone antagonism inherent in color opponency is established later in the visual pathways, for example, in V1. In this instance, one or more types of ganglion cell could carry cone signals from the retina (Stage 1 in Fig. 64.1), and the antagonism (Stage 2) would be established in a central neuron where those signals converge with opposite polarity (e.g., ON-center vs. OFF-center cells). In this scheme, any antagonistic interactions within the ganglion cell receptive field, such as those between center and surround, would be ancillary to the critical spectral antagonism established at the central neuron. Spectral variations within the ganglion cell receptive field across retinal eccentricity (e.g., as the ratio of L to M cones changes) could then be washed out by cortical wiring. Most visual scientists are willing to accept that color opponency at least begins in the retina for both B/Y and R/G opponency. This inference is supported by the vast physiological literature demonstrating cone antagonism within the receptive fields of many ganglion cells, mainly in those providing input to the parvocellular (P) region of the lateral geniculate nucleus (LGN; for reviews, see references in Table 64.1). This is especially so for the massive subset of these cells serving the central visual field. There, the net spectral sensitivity to full-field stimulation is generally (but not always) cone antagonistic, either S/(M + L) or L/M (Fig. 64.3), although many other combinations also have been found (de Monasterio and Gouras, 1975). The key issue is the circuitry for this antagonism and whether it is sufficient to explain the

:     

991

log sensitivity

ONL

OPL 400

450

500

550

600

650

H

700

wavelength (nm)

F 64.3. Top: spectral sensitivity of a nominal R/G opponent ganglion cell that responds with an increase in firing rate to long wavelengths (open squares) and with a decrease to middle and short wavelengths ( filled squares). The curve was calculated from the R/G model in Figure 64.1. Bottom: spectral sensitivity of a B/Y opponent ganglion cell that responds with an increase in firing rate to short wavelengths (open circles) and with a decrease to middle and long wavelengths ( filled circles). The curve was calculated from the B/Y model in Figure 64.1. (Data replotted from Zrenner, 1983b.)

spectral, spatial, and temporal properties of the opponent channels. Of particular interest for this chapter is whether the retinal circuitry is conserved across eccentricities or whether more central mechanisms are necessary to explain the consistent spectral signature of the opponent channels. This chapter will explore two general schemes for wiring cone antagonism in a ganglion cell receptive field (Fig. 64.4). The first is through the simple convergence of signals from cones of different types via strictly excitatory cells that respond to light with opposite polarity, that is, OFF versus ON bipolar cells. This sort of wiring is likely to underlie the S/(M + L) spectral sensitivity of the small bistratified ganglion cell, and this chapter will discuss how the cell might contribute to B/Y opponency (see also Calkins, 2001). The second scheme involves the convergence of excitatory and inhibitory inputs in such a way to render the ganglion cell color opponent. For example, the excitatory center of a foveal P (or midget) ganglion cell is derived from a single cone via a midget bipolar cell, while its inhibitory surround is derived from a combination of cone inputs via GABA-ergic or glycinergic lateral connections (Calkins and Sterling, 1999). This chapter will discuss the mechanism through which the ganglion cell’s spectral sensitivity reflects the difference between these signals across retinal eccentricities and the implications for R/G color opponency.

A retinal circuit for B/Y opponency T S-ON/(M + L)-OFF G C One of the purposes of color opponency, at least from the standpoint of

992

  

INL A OFF

IPL

ON

GCL

F 64.4. Two schemes for wiring cone antagonism in the receptive field of retinal ganglion cells. Right: antagonism for the S-ON/(M + L)-OFF ganglion cell (blue) established through converging ON and OFF bipolar cells. Left: antagonism for the midget cell (green) via convergence of excitation from a bipolar cell and inhibition via horizontal (H) and amacrine (A) cells. (See color plate 41.)

serving the efficiency of vision, is to optimize color information by reducing the redundancy resulting from the overlapping spectral sensitivities of the S, M, and L cones (Buchsbaum and Gottschalk, 1983; Derrico and Buchsbaum, 1991). To optimally fill the dynamic range of a ganglion cell with a pure cone-antagonistic signal, its receptive field ought to be spectrally but not spatially antagonistic (Calkins and Sterling, 1999). A small population of relay neurons in the LGN (Derrington and Lennie, 1984; Dreher et al., 1976; Marroco and DeValois, 1977; Wiesel and Hubel, 1966) and their corresponding ganglion cells in the retina (Dacey, 1996; de Monasterio, 1978; de Monasterio and Gouras, 1975; Zrenner, 1983a, 1983b) fulfills these criteria in a manner consistent with B/Y opponency (for

relative sensitivity

1.0

S-ON

(M+L)-OFF

2-3

20-25

2-3

3-4

0.5

0.0

-0.5 -3

-2

-1

0

1

2

3

distance from center (deg)

F 64.5. Spatially coextensive S-ON (open symbols) and (M + L)-OFF (filled symbols) responses of a small bistratified ganglion cell. [Data replotted from Dacey, 1996, and fit to a simple Gaussian (solid curves).]

review, see Calkins, 2001; Rodieck, 1991). These neurons have both the appropriate S/(M + L) spectral signature with a neutral point near 500 nm and the appropriate spatial response, in which the S and M + L components are coextensive (Fig. 64.5). Most of the so-called blue/yellow cells in the literature respond with excitation at the onset of S stimulation and at the offset of M + L stimulation, that is, SON/(M + L)-OFF; a small number of cells demonstrate the reverse configuration (this is discussed below). A critical step forward in linking the S-ON/(M + L)-OFF cell with B/Y opponency was to correlate the physiological receptive field with the morphology of a ganglion cell and a particular synaptic circuitry. Intracellular recordings of spectral responses from macaque ganglion cells with subsequent staining of their dendritic trees revealed a small bistratified ganglion cell (Dacey and Lee, 1994). This ganglion cell has one dendritic arbor deep in the ON stratum of the inner retina and another dendritic arbor, cospatial with but slightly smaller than the first, in the OFF stratum (Calkins et al., 1998; Dacey, 1993b; Rodieck, 1991). The cell responds with excitation to the onset of short-wavelength light and to the offset of middle and long wavelengths, suggesting that the bistratified morphology correlates with segregated ON and OFF inputs in the inner retina. B C A   S-ON/(M + L)-OFF R F What is the source of the different spectral contributions to the receptive field of the small bistratified cell (Stage 1 in Fig. 64.1)? In the ON stratum, dendrites of the cell intermingle with the axon terminals of the socalled blue cone or S bipolar cell. Excitation in the retina is conveyed through the release of glutamate, both at the cone Æ bipolar cell synapse and the bipolar cell Æ ganglion cell synapse (Massey, 1990). Other types of ganglion cell express ionotropic glutamate receptors, which open cation channels upon binding glutamate (Cohen and Miller, 1994;

15 33

F 64.6. Summary of the presynaptic circuitry of the small bistratified ganglion cell in macaque fovea. The ganglion cell collects mostly from one S cone via 33 synapses from two or three S bipolar cells (blue) and from 20 to 25 M and L cones via 15 synapses from three or four DB2 and DB3 cells (yellow). Modified from Calkins (2001).

Lukasiewicz et al., 1997; Peng et al., 1995; Qin and Pourcho, 1995; Zhou et al., 1994). Thus, the S bipolar cell with light stimulation is likely to release glutamate that opens cation channels localized to the dendrites of the S-ON/(M + L)OFF ganglion cell. Because increased light decreases the rate of glutamate release from photoreceptors, an ON bipolar cell must express a metabotropic glutamate receptor that uses a second-messenger cascade to invert the sign of polarization at the cone synapse (Shiells and Falk, 1995; Vardi et al., 1993). The S bipolar cell, like other ON bipolar cells in the mammalian retina, apparently expresses the L-AP4 or mGluR6 receptor (Euler et al., 1996; Hartveit, 1997; Nakajima et al., 1993; Vardi et al., 1998, 2000), indicating that the small bistratified cell’s ON response to short wavelengths arises through excitation mediated by mGluR6 at the first synapse in the circuit. The small bistratified cell also responds with excitation at the offset of yellow light that stimulates M and L cones (Chichilnisky and Baylor, 1999; Dacey and Lee, 1994). In the OFF stratum of the inner plexiform layer, the ganglion cell collects synapses from the DB2 and DB3 types of diffuse bipolar cell (Calkins et al., 1998), whose dendrites collect from each cone they span (Boycott and Wässle, 1999). There, at the cone terminal base, the dendrites of the DB2 and DB3 cells likely express ionotropic glutamate receptors (Haverkamp et al., 2001; Morigiwa and Vardi, 1999), which would open cation channels upon binding glutamate with the offset of M and L stimulation.

:     

993

T S-ON/(M + L)-OFF G C  R E The diameter of the dendritic arbor of the

994

  

convergence (cones)

300 OFF ON

200

100

0 8

M/L:S

Thus, the actual cone antagonism in the S-ON/(M + L)OFF circuit (Stage 2 in Fig. 64.1) is established through distinct, and strictly excitatory, bipolar cell circuits (Fig. 64.6). Through these circuits, S cones and M and L cones effect opposing currents in the small bistratified ganglion cell, and their joint stimulation produces concurrent S-ON and (M + L)-OFF responses. In the macaque fovea, the axon terminals of the S-cone bipolar cell provide 30 to 35 synapses to the ganglion cell in the ON stratum (Calkins et al., 1998). In the OFF stratum, the axon terminals of the DB2 and DB3 cells provide only about half as many synapses. Consequently, about 70% of the excitation in the receptive field is carried via the S-cone circuit. This difference in synaptic weight, when convolved with the number of converging cones, can account for the approximately 40% difference in amplitude between the S-ON and (M + L)-OFF components of the receptive field (Fig. 64.5). The difference may also explain the faster time to peak for the S component of the response (Chichilnisky and Baylor, 1999; Sterling, 1999). Models of the S-ON/(M + L)-OFF receptive field generally are based solely on converging excitation from the parallel bipolar cell circuits. However, two levels of inhibition also contribute to the ganglion cell. In the inner plexiform layer, amacrine cells provide numerous synapses to the ganglion cell dendritic tree (Calkins et al., 1998; Dacey, 1993b; Ghosh and Grünert, 1999), while in the outer plexiform layer, horizontal cells provide a feedback signal to cones proportional to their mean activity and also are likely to directly inhibit bipolar cells (Sterling, 1999). Both levels of inhibition could contribute to a surround mechanism for the bipolar cells (Dacey, 1999). In particular, the H1 horizontal cell collects signals almost exclusively from M and L cones and lacks any substantial contact with S cones, while the H2 horizontal cell collects from and can provide feedback to all three cone types (Chan and Grünert, 1998; Dacey et al., 1996, 2000; Goodchild et al., 1996). Thus, both H1 and H2 cells would be able to contribute to the surround of the DB2 and DB3 bipolar cells, while the H2 cell could contribute to the surround for the S bipolar cell (for review, see Martin, 1998). Nevertheless, the ganglion cell apparently lacks a measurable net surround, and changing the size of a stimulus centered on the receptive field modifies very little the response of the cell (de Monasterio, 1978; Wiesel and Hubel, 1966). One simple explanation for this is that the reduction in activity in the OFF and ON bipolar circuits via horizontal cell feedback is about equivalent because of the overwhelming preponderance of input from M and L cones to both H1 and H2 cells (Rodieck, 1998). What is seen then in the ganglion cell response is only the difference between the excitatory S-ON and (M + L)-OFF components.

7 6 5 4 10

20

30

40

50

60

eccentricity (deg)

F 64.7. Top: convergence of M/L cones to the OFF dendritic tree and of S cones to the ON dendritic tree of the small bistratified ganglion cell in human (thick lines) and macaque (thin lines) retina. Convergence calculated as the product of dendritic tree area (polynomials in Fig. 64.9) with average density of S cones [(from Figs. 6A and 7A) or of M/L cones (from Curcio et al., 1990)]. Bottom: ratio of convergence of M/L cones to S cones in human (stippled trace) and macaque (solid trace) retina. The ratios are nearly constant across retinal eccentricity. Modified from Calkins (2001).

small bistratified cell in the OFF stratum is about 75% that of the ON arbor (Dacey, 1993b). The area encompassed by the OFF tree reflects this difference accordingly (see Figure 9 in Calkins, 2001). Nevertheless, because of their greater density, the estimated convergence of M and L cones is much higher than the convergence of S cones. In the macaque fovea, 20 to 25 M and L cones converge on the ganglion cell, compared to 3 to 4 S cones (Fig. 64.6). The convergence of M and L cones is systematically about fivefold higher than convergence of S cones across the human retina and six- or sevenfold higher across most of the macaque retina (Fig. 64.7). On the other hand, the great width of the spatial aperture of the receptive field of the S cone itself (Williams et al., 1983, 1993), combined with synaptic weighting, effectively molds the S-ON component of the S-ON/(M + L)-OFF receptive field into a continuous, smooth profile that is spatially coextensive with the (M + L)OFF component (see Figure 1 in Calkins, 2001). Thus, both the spatial and spectral response profiles of the S-ON/(M + L)-OFF ganglion cell are consistent with cells involved in B/Y opponency. The ratio of the convergence of M/L cones to S cones remains roughly constant across retinal eccentricity (Fig. 64.7). This consistency would contribute to a uniform spectral neutral point in the response of the S-ON/(M + L)-OFF cell to monochromatic stimulation. In the narrow range of eccentricities tested in the macaque retina, this is certainly the case (Zrenner, 1983a, 1983b). As alluded to earlier in the chapter, the neutral point of the B/Y channel in human observers is also remarkably uniform across a wide range of

resolution (cycles deg -1)

15

acuity ganglion cell

10

5

0 0

10

20

30

40

50

60

eccentricity (deg)

F 64.8. The sampling frequency of the small bistratified ganglion cell matches S acuity in the human retina. The sampling frequency of the ganglion cell is calculated as the corresponding Nyquist frequency, assuming triangular packing and based on the density measurements of Dacey (1993b) outside of the fovea and of Calkins et al. (1998) for the fovea. Acuity measurements were pooled from Daw and Enoch (1973), Green (1972), Mullen (1985), Stiles (1949), Stromeyer et al. (1978), and Williams et al. (1983) for the fovea and from Noorlander et al. (1983) and Hess et al. (1989) for the periphery. Modified from Calkins (2001).

eccentricities (see Figure 6 in Hibino, 1992). In the human retina, the relative number of M versus L cones likely varies with eccentricity (Hagstrom et al., 1998). It would be interesting to determine whether the synaptic weights of the S and M/L components of the ganglion cell receptive field also change with eccentricity to accommodate the M : L ratio. It is difficult to extract from the early extracellular recordings the regularity with which either the S-ON/(M + L)-OFF geniculate cell or its ganglion cell counterpart samples the photoreceptor mosaic. This is important, because for a ganglion cell circuit to be associated with the critical locus of opponency, that circuit ought to sample the retinal mosaic with the same spatial resolution as the color channel in question. The mosaic of the S-ON/(M + L)-OFF small bistratified ganglion cell has been mapped by microinjection in the human and macaque retina for the parafovea and beyond (Dacey, 1993b). These injections provide precise measurement of the cell’s dendritic arbor as a function of eccentricity. Assuming that adjacent arbors “tile” the retina, the average diameter of the dendritic tree provides an estimate of the cell’s density and therefore of its spatial (or Nyquist) sampling rate (Dacey, 1993b). Individual differences between retinas are likely to be substantial, and the great variability between the size of dendritic arbors at a particular eccentricity confounds the estimate of density. Nevertheless, the estimated sampling rate of the small bistratified cell in the human retina agrees reasonably well with measurements of visual acuity based on discriminating S-cone isolating spatial

patterns (Fig. 64.8). Such psychophysical tasks are likely to tap the B/Y color channel, based on the color appearance of the pattern. The Nyquist rate for the fovea is based on identification of the ganglion cell in macaque retina using electron microscopy (Calkins et al., 1998). The finding in that retina was one small bistratified cell for every S cone, and there is little reason to doubt that the same holds for the human fovea. Thus, the small bistratified ganglion cell is likely dense enough to support the spatial acuity of the B/Y channel across eccentricity. Reconstructions of the small bistratified cell with electron microscopy indicate that in the macaque fovea, the ganglion cell collects from about three S cones via 30 to 35 synapses from two or three S bipolar cells (Fig. 64.6; Calkins et al., 1998). Outside of the fovea, the dendritic tree of the ganglion cell in the ON stratum encompasses increasing expanses of retina as the cell’s density declines (Dacey, 1993b). Based on anatomical measurements alone, a small bistratified cell at 10 to 20 degrees of eccentricity would collect from 5 to 10 S cones, while cells at 25 degrees of eccentricity and beyond, it would collect from 10 to 20 S cones. These numbers are consistent with highly sensitive multielectrode recordings from SON/(M + L)-OFF cells, which indicate a convergence of 5 to 15 S cones between 20 and 50 degrees of eccentricity (Chichilnisky and Baylor, 1999). These same multielectrode recordings actually illustrate several important points. The signals from individual S cones to the receptive field of the ganglion cell sum linearly: their combined contribution predicts the net response of the cell to S stimulation. Also, the relative strength of the S signals varies greatly, with one S cone providing the dominant input. Conversely, a particular S cone may contribute to the receptive fields of neighboring small bistratified cells; however, that S cone provides the dominant input to only one ganglion cell (Chichilnisky and Baylor, 1999). These physiological results match very well predictions based on circuitry. A single S cone provides about 70% of the synaptic input to an S bipolar cell. Similarly, a single S bipolar cell dominates in providing synapses to the small bistratified ganglion cell. The result of these two levels of synaptic weighting is that a single S cone outweighs its neighbors in providing excitation to a particular ganglion cell (Calkins, 2000; Calkins et al., 1998; Chichilnisky and Baylor, 1999). If every S cone is so represented, the density of the small bistratified cell ought to match the S cone density. This is clearly so in the primate fovea (Calkins et al., 1998). Outside of the fovea, in human retina, the disparity between the sampling of the small bistratified cell and the sampling of the S mosaic is small (compare Figures 6 and 10 in Calkins, 2001), too small to reject the hypothesis of 1:1 sampling given methodological differences and interretina variation. Therefore, it is reasonable to accept the idea that the spatial sampling of the small bistratified, S-ON/(M + L)-

:     

995

OFF cell matches the acuity of the B/Y channel across retinal eccentricity (Fig. 64.8). C  C:  S-OFF C   R The clear demonstration linking the morphology and circuitry of the small bistratified ganglion cell to the S-ON/ (M + L)-OFF receptive field has reinforced the association of S cones with a single physiological pathway and therefore a single line into the B/Y channel. In many reviews, S cones are depicted as “skipping” contact with additional postsynaptic pathways (e.g., Martin, 1998). Along these lines, the lack of any regularity of physiological recordings from OFF cells with substantial S input has supported the view that S cones simply do not contact OFF pathways to the brain, including any that might support B/Y color vision. However, a careful survey of the literature reveals numerous physiological recordings from neurons with a pure S-OFF response (Calkins, 2001), and certain psychophysical experiments support S input to OFF channels (most recently, McLellan and Eskew, 2000; Shinomori et al., 1999). The physiological examples are relatively rare, compared to the S-ON/(M + L)-OFF cell, but nonetheless persist across multiple decades of investigation. These studies converge upon two distinct profiles of the receptive field. The first resembles the S-ON cell in that the S-OFF response is spatially coextensive with an (M + L)-ON response. The second resembles the textbook spatially and spectrally opponent cell in that the S-OFF response is localized to a center spatially concentric with an inhibitory M + L surround. Complementary ON and OFF mosaics for a particular type of ganglion cell effectively partition the dynamic range of the pathway about the mean light level. Thus, each ganglion cell can use the full range of its spiking capacity to signal with excitation either graded increments or decrements from the mean. If such a strategy is used by the color channels, one might expect a ganglion cell with spatially coextensive S-OFF and (M + L)-ON regions (i.e., (M + L)ON/S-OFF) to contribute to and complete the B/Y opponent channel (Zrenner, 1983a, 1983b; see the discussion of this topic in Sankeralli and Mullen, 2001). To make this argument convincing, though, requires identification of its morphological substrate and demonstration that its presynaptic circuitry optimizes a spectral signal at the price of spatial information, much like the small bistratified cell (Calkins and Sterling, 1999).

New views of R/G opponency C R/G  B/Y O Color opponency serves to optimize the information contained in the quantum catches of the S, M, and L cones through perceptive fields that are spectrally but not spatially antagonistic (Fig. 64.9). Cells involved in establishing the opponent channels at some level should eventually demonstrate receptive fields with the same so-called type II configuration (Calkins and Sterling,

996

  

optimizing space

optimizing color

+

+ + +++ + + + +

F 64.9. To optimize its spatial signal, a ganglion cell should remove the signal component shared with neighbors by subtracting a spatially weighted average of all surrounding cones. To optimize its spectral-difference signal, a ganglion cell should subtract all the signals of one cone type from all the signals of the other types. (Modified from Calkins and Sterling, 1999.)

1999). For B/Y opponency, spectral antagonism without spatial antagonism is established early on in the retina, in the receptive field of the small bistratified ganglion cell. For that cell, the difference in amplitude between the S-ON and (M + L)-OFF components is constant across the spatial extent of the receptive field (Rodieck, 1991). Thus, the only information available in the spike train of the cell is spectral and not spatial. Many cells in V1 demonstrate L/M antagonism in spatially coextensive domains of the receptive field—again, spectral without spatial antagonism. These cells are expected, therefore, to play some role in R/G opponency. Similarly, even earlier in the visual pathways, many cells in the parvocellular region of the LGN, and presumably their ganglion cell counterparts in the retina, also demonstrate type II receptive fields with spatially coextensive L-cone versus M-cone regions (de Monasterio, 1978; de Monasterio and Gouras, 1975; Dreher et al., 1976; Lee, 1996; Reid and Shapley, 1992; Wiesel and Hubel, 1966). These cells generally have the correct spectral signature, with the L/M neutral point near 570 to 600 nm (Rodieck, 1991), and the variability that is present can easily be attributed to different adaptive conditions. Based on these recordings, many investigators have proposed that the type II receptive fields may subserve a specialized pathway for R/G opponency (Hubel and Livingstone, 1990), and others have suggested a retinal circuit similar to that of the small bistratified ganglion cell as the critical locus for L/M antagonism (Calkins and Sterling, 1999; Rodieck, 1991). For B/Y opponency, only 3% to 4% of the ganglion cells in the central fovea need be small bistratified cells to account for B/Y acuity (Calkins and Sterling, 1999). If the B/Y channel requires a corresponding S-OFF cell, this fraction increases by another 3% to 4%. Another 5% to 7% of the ganglion cells in the fovea are “parasol” ganglion cells, and 75% to 85% are midget cells (Boycott and Wässle, 1999). Other types of large and sparsely distributed ganglion cells that are likely uninvolved in color vision probably represent

20

acuity ganglion cell fraction

16 14

20

12 10

15

8 10

6

fraction required (%)

-1 resolution (cycles deg )

25

4

5

dendritic field (human) dendritic field (macaque) receptive field center (macaque)

40

18

2

convergence (cones)

30

30 20 10 0 0

10

20 30 eccentricity (deg)

40

0

0 0

5

10

15

20

25

30

eccentricity (deg)

F 64.10. The fraction of all ganglion cells in the human retina necessary to account for psychophysically observed spatial acuity of the R/G channel. The fraction was calculated as the ratio of the Nyquist sampling density required, assuming triangular packing of ganglion cells, to the total ganglion cell density, using measurements of Curcio and Allen (1990) and assuming three or four ganglion cells per cone for the center fovea (Sjöstrand et al., 1994). Acuity measurements outside of the fovea are from Andersen et al. (1991), and those in the fovea are from Sekiguchi et al. (1993).

another 3% to 5%. Thus, at best, about 10% of the foveal ganglion cells remain that could include a dedicated R/G pathway from the retina. Figure 64.10 shows a plot of spatial acuity for R/G isolating patterns and the fraction of all ganglion cells necessary to account for the corresponding sampling rate. While there may be room for a dedicated R/G cell, the relatively large fraction of cells needed would make finding the cell at least as likely as encountering a small bistratified cell. Yet, in both anatomical and physiological studies, a clear mosaic of ganglion cells with L/M (or M/L) type II receptive fields has not been found. Moreover, if R/G opponency also demands complementary L-ON/MOFF and M-ON/L-OFF mosaics, the fractions in Figure 64.10 must double, and there simply may not be enough ganglion cells to provide the necessary sampling rate. In the absence of a morphologically identified substrate in the retina, many now doubt the accuracy of L/M (or M/L) type II receptive fields derived from extracellular recordings in the retina or LGN. T P C H The early Golgi studies of the retina identified a densely populated ganglion cell with a small cell body and a narrow dendritic tree, with a corresponding bipolar cell providing it input (Polyak, 1941). Both cells were called midget because of their small size and morphology, and the ganglion cell is also called P because of its projections to the parvocellular layers of the LGN (for review, see Calkins and Sterling, 1999). Within the central 6 to 7 degrees or so, each midget ganglion cell collects from only a single cone via a single midget bipolar cell. Similarly,

F 64.11. Cone convergence onto the P cell rises steadily with eccentricity for both macaque and human. Calculation for the region denoted by the shaded lines is based on the range of measurements of macaque P cell receptive field centers in retina and LGN (Croner and Kaplan, 1995; de Monasterio and Gouras, 1975; Derrington and Lennie, 1984) and of macaque cone spatial densities (Packer et al., 1989). This brackets the anatomical convergence calculated from the dendritic field areas and cone densities (calculated from Curcio et al., 1990; Dacey, 1993a; Packer et al., 1989; Watanabe and Rodieck, 1989). Modified from Calkins and Sterling (1999).

over most of the retina, the midget bipolar cell also collects from only a single cone. Conversely, within this region, each cone sends signals to both a single ON and a single OFF midget ganglion cell via an ON and an OFF midget bipolar cell. Outside of the fovea, the number of cones converging onto the midget ganglion cell via numerous midget bipolar cells increases as the dendritic tree of the ganglion cell expands. Nevertheless, at all retinal eccentricities, the midget cell is the most numerous and has the smallest anatomical sampling aperture of all ganglion cells, and therefore sets the limit for cone-mediated spatial acuity (see Dacey, 1993a). The S-ON and (M + L)-OFF regions of the small bistratified cell’s receptive field are spatially coextensive, and the cell lacks a measurable net surround. In contrast, the midget or P cell demonstrates a narrow receptive field center and a spatially concentric, inhibitory surround. The physiological center correlates closely in size with the sampling area of the dendritic tree (Fig. 64.11), while the surround is much broader and only about 55% to 60% the strength of the center (Croner and Kaplan, 1995). This type I receptive field structure is therefore optimized for spatial antagonism and the detection of high-frequency edges (Derrington and Lennie, 1984). While the circuitry for the center of the midget ganglion cell is relatively straightforward, the circuitry for the surround is more complex, probably involving feedback inhibition from both horizontal cells in the outer plexiform layer (OPL) and amacrine cells in the inner plexiform layer (IPL) (Fig. 64.12). As with other ganglion cells, much of the surround of midget cells is likely conveyed through its bipolar cell circuitry (Dacey, 1999; Dacey et al., 2000).

:     

997

+

+

+

+

_

H B

B

B

B

A _

+

+

_

+ +

G F 64.12. The horizontal cell (H) collects directly from both M and L cones and inhibits midget bipolar cell dendrites; amacrine cells (A) also collect M and L input via bipolar cells (B) and inhibit the midget bipolar cell axon terminal and the dendrites of the midget/ P ganglion cell (G). Modified from Calkins and Sterling (1999). (See color plate 42).

By definition, the center mechanism for each midget cell within the central 6 to 7 degrees is spectrally pure, either L or M. This is also likely to be the case for many midget cells within the central 10 degrees, where convergence is still limited to a handful of cones and M and L cones distribute into small clusters of like type. Since recordings began in the early 1960s, nearly all of the physiological recordings from either P cells in the retina or their LGN counterparts have been from cells representing the central 10 to 15 degrees of the visual field. Most of these cells (but not all) demonstrated L/M or M/L cone-antagonistic receptive fields, and P cells were equated with color opponent cells (see Calkins and Sterling, 1999, for review). These recordings seemed to imply that the surround, like the center, must also be spectrally pure—but of the opposing cone type. While some data support this idea, careful physiological measurements indicate that horizontal cells are broadly tuned spectrally (Dacey et al., 1996; Dacheux and Raviola, 1990). Moreover, the anatomical connections to midget cells from amacrine cells

998

  

are also spectrally mixed (Calkins and Sterling, 1996). Thus, there is little basis to support the idea that color opponency in the midget cell of the central retina arises from antagonism between a pure center and a pure surround. Simple models of the receptive field clearly show that random connections forming the surround can render the entire receptive field color opponent (Lennie et al., 1991). The reason is equally simple. Because the surround is so much weaker than the center, having about 60% of the center’s strength (Croner and Kaplan, 1995), whatever cone contribution to the surround is common to the center is simply canceled by the dominant contribution of that cone type to the center. The net result is a more or less balanced antagonism between L and M cones, with resultant L-M or M-L spectral sensitivity yielding the appropriate neutral point (Lennie et al., 1991). In this model, the critic locus for R/G opponency is then placed at the point of convergence of excitatory (center) and inhibitory (surround) inputs forming the midget cell receptive field. With both an ON and an OFF midget ganglion cell for each cone, in the fovea only 30% to 40% of all midget cells need be opponent to account for the discrimination of R/G patterns. That is, if the R/G channel requires complementary L-ON/M-OFF and M-ON/L-OFF pairs, only 30% to 40% of the L and M cones need to contact midget circuits in which the cone antagonism between center and surround is more or less balanced to a psychophysically appropriate spectral signature. Because of its center-surround organization, every midget cell in the retina transmits information regarding spatial edges, regardless of its spectral sensitivity (Derrington and Lennie, 1984). However, because of the mixed surround, the amount of information concerning the difference between L- and M-cone activity will vary significantly across the mosaic of midget cells. For cells in which the surround is biased in favor of the opposing cone type, the spectral antagonism will be strong. For cells in which the surround draws mainly from cones of the same type as the center, the antagonism will be weak or nonexistent. Indeed, physiological recordings verify this great variation in spectral sensitivity (Marroco and De Valois, 1977; Schiller and Colby, 1983; Zrenner and Gouras, 1983). The determinant factor is, of course, the distribution of L and M cones in the overlying cone mosaic, which is subject to spatial variation. M C   P R Outside of the fovea, the convergence of cones to the midget ganglion cell increases substantially (Fig. 64.12). Correspondingly, the probability that the receptive field center draws solely from either L or M cones diminishes with increasing retinal eccentricity. The first systematic recordings from morphologically identified midget ganglion cells in the retinal periphery of the Macaca demonstrated that the majority of cells had

spectral sensitivity without any cone antagonism, that is, a net sensitivity best described as L + M (Dacey, 2000). This is to be expected from cells with centers drawing from increasingly expansive patches of the cone mosaic. On the other hand, the dendritic tree of the midget cell becomes increasingly asymmetric (anisotropic) with increasing retinal eccentricity (Dacey, 1993a; Wässle and Boycott, 1991). Since the mosaic of L and M cones is patchy, with irregular clusters of like type (Mollon and Bowmaker, 1992; Roorda and Williams, 1999), it is possible that the asymmetric geometry of the dendritic tree reflects selective connections with midget bipolar cells from only a single type of cone (but see the discussion of this possibility in Calkins and Sterling, 1999). Recent extracellular recordings from peripheral ganglion cells with P-cell-like receptive fields demonstrated a large contingency of L/M (or M/L) antagonistic responses (Martin et al., 2001). Though it does not follow a priori that these cells were all midget cells, it appears likely that the mosaic of midget cells across the retina includes both cells with and without strong spectral antagonism. To put this in psychophysical perspective, at 20 degrees of eccentricity, the R/G channel samples at 2 to 3 cycles/deg (Anderson et al., 1991). To account for this acuity, only 25% of the total midget population (110 cells/deg2) needs to demonstrate strong L/M (or M/L) antagonism. I  R/G O The most widely accepted model for R/G opponency places the critical locus of cone antagonism within the midget cell’s receptive field, embedded in the antagonistic interactions between the center and the surround. Because of the spatially concentric structure of these interactions, every midget cell transmits information about spatial edges (Derrico and Buchsbaum, 1991; Ingling and Martinez-Uriegas, 1983). On the other hand, the degree to which a midget cell also transmits information regarding spectral contrast (L vs. M) depends greatly on the composition of the center, since the surround is apparently mixed (Kingdom and Mullen, 1995; Mullen and Kingdom, 1996). In the central retina, most (but not all) midget cells are L/M or M/L antagonistic, and the strength of this antagonism depends on the composition of the surround (see above, Lennie et al., 1991). With increasing eccentricity the center size expands, and the degree of L/M antagonism is determined by the relative purity of the center. The point is that, across the entire retina the midget mosaic will demonstrate a highly variable spectral signature (L/M neutral point) from cell to cell, depending on the spatial structure of the overlying cone mosaic (Calkins and Sterling, 1999; Martin et al., 2001). As a confounding factor, in the human retina the relative number of L and M cones varies across the retinal location, with L cones becoming more predominant with increasing eccentricity (Hagstrom et al., 1998). This is also consistent

with reports of L-dominant centers in P cells with increasing eccentricity (Shapley and Perry, 1986). Furthermore, there is great variability in the relative number of L and M cones between human observers (Hagstrom et al., 1998). Yet, the neutral point of the R/G channel is remarkably consistent not only from fovea to periphery (see above), but also between observers with known differences in their numbers of L and M cones (Miyahara et al., 1998). Thus, whatever information about R/G color vision is coming out of the retina via the midget ganglion cell is subject to a great deal of variability, not only because of the inherent circuitry of the cell, but also because of the highly diverse nature of the cone mosaic providing it input.

Building pathways for color vision: some remaining questions S C  C C It is a well-documented psychophysical fact that observers perceive not only “blue” with the stimulation of S cones, but also “red” (DeValois et al., 2000a; Krauskopf et al., 1982; Wooten and Werner, 1979). Thus, most short-wavelength lights are described as both, or as “violet,” and S cones are assumed to provide input not only to the B/Y channel, but also to the R/G channel with the same polarity as the L cones. This explains in part why unique blue occurs with about equal stimulation of S and M cones—the “red” from S cancels the “green” from M (but see DeValois et al., 2000b). If the retinal substrate for the R/G channel is the midget system, then one might expect to find cells with L + S spectral sensitivity for their receptive field center, which is highly unlikely for cells within the central 6 to 7 degrees, where only a single cone provides the center. Only rare descriptions of any such cells exist, either in the retina or the LGN (see Calkins, 2001, for review). Our tendency as vision scientists is to try to correlate what cells we do find with a psychophysical channel, so the S input to the R/G channel is typically simplified out of physiological models—which is tantamount to the “tail wagging the dog.” It is possible that whatever ganglion cells do have S input converge with L/M cells further along in the visual streams to produce the S contribution to the R/G channel (DeValois et al., 2000b). T R   C  R/G O Based on numbers alone (see above), it would seem unlikely that a circuit similar to that of the S-ON/(M + L)-OFF ganglion cell underlies R/G opponency. Yet, there are aspects of this idea that are far more parsimonious than the P cell hypothesis (see Calkins and Sterling, 1999). The receptive field of the midget/P cell is highly optimized for spatial contrast, and whatever spectral contrast exists in the receptive field is highly variable across the retina. It is, of course, possible that only a subset of midget cells, with the appropriate spectral

:     

999

signature, carries R/G color information to the cortex, and that the cortex must “learn” during the development of the visual system to discriminate the output of “spatial” P cells from the output of “spectral” P cells. This might explain the highly regular spectral sensitivity of the R/G channel, and there are seemingly more than enough midget ganglion cells in the retina for a subset of those to represent a dedicated color pathway. Alternatively, the cortex could “demultiplex” whatever variable spectral contrast information is present in the spike train of each midget cell from the inherent spatial contrast (Ingling and Martinez-Uriegas, 1983; Kingdom and Mullen, 1995). A third possibility, though seemingly remote, is that because the spectral antagonism in the midget receptive field is simply an ancillary effect of the structure of the cone mosaic, it is completely irrelevant to the R/G color channel. The critical locus could represent a point in the visual streams where the output of ON and OFF midget cells converge, rendering all subsequent processing coneantagonistic. In this case, only the spectral composition of the excitatory center would matter for creating receptive fields in higher cells with the appropriate L/M signature.

REFERENCES Anderson, S. J., K. T. Mullen, and R. F. Hess, 1991. Human peripheral spatial resolution for achromatic and chromatic stimuli: limits imposed by optical and retinal factors, J. Physiol., 442:47– 64. Bornstein, M. H., 1973. Color vision and color naming: a psychophysiological hypothesis of cultural difference, Psych. Bull., 80:257–285. Boycott, B. B., and H. Wässle, 1999. Parallel processing in the mammalian retina. The Proctor Lecture, Invest. Ophthalmol. Vis. Sci., 40:1313–1327. Boynton, R. M., 1975. Color, hue, and wavelength, in E. C. Carterette and M. P. Friedman (eds.), Handbook of Perception, vol. 5, New York: Academic Press. Buchsbaum, G., and A. Gottschalk, 1983. Trichromacy, opponent colours coding and optimum colour information transmission in the retina, Proc. R. Soc. (Lond.) B, 220:89–113. Calkins, D. J., 2000. The representation of cone signals in the primate retina, J. Opt. Soc. Am. A, 17:597–606. Calkins, D. J., 2001. Seeing with S cones, Prog. Retinal Eye Res., 20:255–287. Calkins, D. J., and P. Sterling, 1999. Evidence that circuits for spatial and opponent color vision segregate at the first retinal synapse, Neuron, 24:313–321. Calkins, D. J., J. E. Thornton, and E. N. Pugh, 1992. Monochromatism determined at a long-wavelength/middle-wavelength cone-anatagonistic locus, Vis. Res., 32:2349–2367. Calkins, D., and P. Sterling, 1996. Absence of spectrally specific lateral inputs to midget ganglion cells in primate retina, Nature, 381:613–615. Calkins, D. J., Y. Tsukamoto, and P. Sterling, 1998. Microcircuitry and mosaic of a blue/yellow ganglion cell in the primate retina, J. Neurosci., 18:3373–3385. Chan, T. L., and U. Grünert, 1998. Horizontal cell connections with short wavelength-sensitive cones in the retina: a compari-

1000

  

son between new world and old world primates, J. Comp. Neurol., 393:196–209. Chichilnisky, E. J., and D. A. Baylor, 1999. Receptive-field microstructure of blue-yellow ganglion cells in primate retina, Nat. Neurosci., 2:889–893. Cohen, E. D., and R. F. Miller, 1994. The role of NMDA and non-NMDA excitatory amino acid receptors in the functional organization of primate ganglion cells, Vis. Neurosci., 11:317– 332. Croner, L. J., and E. Kaplan, 1995. Receptive fields of P and M ganglion cells across the primate retina, Vis. Res., 35:7–24. Curcio, C. A., and K. A. Allen, 1990. Topography of ganglion cells in the human retina, J. Comp. Neurol., 300:5–25. Curcio, C. A., K. R. Sloan, R. E. Kalina, and A. E. Hendrickson, 1990. Human photoreceptor topography, J. Comp. Neurol., 292:497–523. Dacey, D. M., 1993a. The mosaic of midget ganglion cells in the human retina, J. Neurosci., 13:5334–5355. Dacey, D. M., 1993b. Morphology of a small field bistratified ganglion cell type in the macaque and human retina, Vis. Neurosci., 10:1081–1098. Dacey, D. M., 1996. Circuitry for color coding in the primate retina, Proc. Natl. Acad. Sci. USA, 93:582–588. Dacey, D. M., 1999. Primate retina: cell types, circuits and color opponency, Prog. Retinal Eye Res., 18:737–763. Dacey, D. M., 2000. Parallel pathways for spectral coding in primate retina, Annu. Rev. Neurosci., 23:743–775. Dacey, D. M., and B. B. Lee, 1994. The “blue-on” opponent pathway in primate retina originates form a distinct bistratified ganglion cell type, Nature, 367:731–735. Dacey, D. M., B. B. Lee, D. K. Stafford, J. Pokorny, and V. C. Smith, 1996. Horizontal cells of the primate retina: cone specificity without spectral opponency, Science, 271:656–659. Dacey, D. M., O. S. Packer, D. Brainard, B. Peterson, and B. Lee, 2000. Center-surround receptive field structure of cone bipolar cells in primate retina, Vis. Res., 40:1801–1811. Dacheux, R. F., and E. Raviola, 1990. Physiology of H1 horizontal cells in the primate retina, Proc. R. Soc. (Lond.) B, 239:213–230. Daw, N. W., and J. M. Enoch, 1973. Contrast sensitivity, Westheimer function and Stiles-Crawford effect in a blue cone monochromat, Vis. Res., 13:1669–1679. de Monasterio, F. M., and P. Gouras, 1975. Functional properties of ganglion cells of rhesus monkey retina, J. Physiol., 251:167–195. de Monasterio, F. M., 1978. Properties of ganglion cells with atypical receptive-field organization in retina of macaques, J. Neurophysiol., 41:1435–1449. Derrico, J. B., and G. Buchsbaum, 1991. A computational model of spatio-chromatic coding in early vision, J. Vis. Commun. Image Represent., 2:31–38. Derrington, A. M., and P. Lennie, 1984. Spatial and temporal contrast sensitivities of neurones in lateral geniculate nucleus of macaque, J. Physiol., 357:219–240. DeValois, R. L., N. P. Cottaris, S. D. Elfar, L. E. Mahon, and J. A. Wilson, 2000b. Some transformations of color information from lateral geniculate nucleus to striate cortex, PNAS, 97:4997–5002. DeValois, R. L., K. K. De Valois, and L. E. Mahon, 2000a. Contribution of S opponent cells to color appearance, PNAS, 97:512–517. Dreher, B., Y. Fukuda, and R. W. Rodieck, 1976. Identification, classification and anatomical segregation of cells with X-like and Y-like properties in the lateral geniculate nucleus of Old-World primates, J. Physiol., 258:433–452.

Euler, T., H. Schneider, and H. Wässle, 1996. Glutamate responses of bipolar cells in a slice preparation of the rat retina, J. Neurosci., 16:2934–2944. Ghosh, K. K., and U. Grünert, 1999. Synaptic input to small bistratified (blue-ON) ganglion cells in the retina of a new world monkey, the marmoset Callithrix jacchus, J. Comp. Neurol., 413:417–428. Goodchild, A. K., T. L. Chan, and U. Grünert, 1996. Horizontal cell connections with short-wavelength-sensitive cones in macaque monkey retina, Vis. Neurosci., 13:833–845. Gordon, J., and I. Abramov, 1977. Color vision in the peripheral retina. II. Hue and saturation, J. Opt. Soc. Am., 67:202–207. Green, D. G., 1972. Visual acuity in the blue cone monochromat, J. Physiol., 196:415–429. Hagstrom, S. A., J. Neitz, and M. Neitz, 1998. Variations in cone populations for red-green color vision examined by analysis of mRNA, NeuroReport, 9:1963–1967. Hartveit, E., 1997. Functional organization of cone bipolar cells in the rat retina, J. Neurophysiol., 77:1726–1730. Haverkamp, S., U. Grünert, and H. Wa¨ssle, 2001. The synaptic architecture of AMPA receptors at the cone pedicle of the primate retira, J. Neurosci., 21:2488–2500. Hess, R. F., K. T. Mullen, and E. Zrenner, 1989. Human photopic vision with only short wavelength cones post-receptoral properties, J. Physiol., 417:151–172. Hibino, H., 1992. Red-green and yellow-blue opponent color responses as a function of retinal eccentricity, Vis. Res., 32:1955–1964. Hubel, D., and M. Livingstone, 1990. Color puzzles, in Cold Spring Harbor Symposia on Quantitative Biology, vol. LV, Cold Spring Harbor, NY: Cold Spring Harbor Laboratory Press, pp. 643–649. Hurvich, L. M., and D. Jameson, 1957. An opponent-process theory of color vision, Psychol. Rev., 64:384–404. Ingling, C. R., Jr. and E. Martinez-Uriegas, 1983. The relationship between spectral sensitivity and spatial sensitivity for the primate r-g X-channel, Vis. Res., 23:1495–1500. Kaplan, E., B. B. Lee, and R. M. Shapley, 1990. New views of primate retinal function, Prog. Retinal Eye Res., 9:273–336. Kingdom, F. A. A., and K. T. Mullen, 1995. Separating colour and luminance information in the visual system, Spatial Vis., 9:191–219. Krauskopf, J. D. R., M. B. Williams, and D. W. Heeley, 1982. Cardinal directions of color space, Vis. Res., 22:1123–1131. Kuyk, T. K., 1982. Spectral sensitivity of the peripheral retina to large and small stimuli, Vis. Res., 22:1293–1297. Lee, B. B., 1996. Receptive field structure in the primate retina, Vis. Res., 36:631–644. Lennie, P., and M. D’Zmura, 1988. Mechanisms of color vision, CRC Crit. Rev. Neurobiol., 3:333–400. Lennie, P., P. W. Haake, and D. R. Williams, 1991. The design of chromatically opponent receptive fields, in Computational Models of Visual Processing (M. S. Landy and J. A., Movshon, eds.), Cambridge, MA: MIT Press, pp. 71–82. Lukasiewicz, P. D., J. A. Wilson, and J. E. Lawrence, 1997. AMPApreferring preceptors mediate excitatory synaptic inputs to retinal ganglion cells, J. Neurophysiol., 77:57–64. Marroco, R. T., and R. L. DeValois, 1977. Locus of spectral neutral point in monkey opponent cells depends on stimulus luminance relative to the background, Brain Res., 119:465– 470. Martin, P. R., 1998. Colour processing in the primate retina: recent progress, J. Physiol., 513:631–638.

Martin, P. R., B. B. Lee, A. J. R. White, S. G. Solomon, and L. Rüttiger, 2001. Chromatic sensitivity of ganglion cells in the peripheral primate retina, Nature, 410:933–936. Massey, S. C., 1990. Cell types using glutamate as a neurotransmitter in the vertebrate retina, Prog. Retinal Eye Res., 9:399–425. McLellan, J. S., and R. T. Eskew, Jr., 2000. ON and OFF S-cone pathways have different long-wave cone inputs, Vis. Res., 40:2449–2465. Miyara, E., J. Pokorny, V. C. Smith, R. Baron, and E. Baron, 1998. Color vision in two observers with highly biased LWS/MWS cone ratios, Vis. Res., 38:601–612. Mollon, J. D., and J. K. Bowmaker, 1992. The spatial arrangement of cones in the primate fovea, Nature, 360:677–679. Morigiwa, K., and N. Vardi, 1999. Differential expression of ionotropic glutamate receptor subunits in the outer retina, J. Comp. Neurol., 405:173–184. Mullen, K. T., 1985. The contrast sensitivity of human color vision to red-green and blue-yellow chromatic gratings, J. Physiol., 359:381–400. Mullen, K. T., and F. A. A. Kingdom, 1996. Losses in peripheral colour sensitivity predicted from “hit and miss” post-receptoral cone connections, Vis. Res., 36:1995–2000. Nakajima, Y., H. Iwakabe, C. Akazawa, H. Nawa, R. Shigemoto, N. Mizuno, and S. Nakanishi, 1993. Molecular characterization of a novel retinal metabotropic glutamate receptor mGluR6 with a high agonist selectivity for -2-amino-4phosphonobutyrate, J. Biol. Chem., 268:11868–11873. Nerger, J. L., V. J. Volbrecht, and C. J. Ayde, 1995. Unique hue judgments as a function of test size in the fovea at 20-deg temporal eccentricity, J. Opt. Soc. Am. A, 12:1225–1232. Noorlander, C., J. J. Koenderink, R. J. Den Ouden, and B. W. Edens, 1983. Sensitivity to spatiotemporal colour contrast in the peripheral visual field, Vis. Res., 23:1–11. Packer, O., A. Hendrickson, and C. Curcio, 1989. Photoreceptor topography of the retina in the adult pigtail Macaque (Macaca nemestrina), J. Comp. Neurol., 288:165–183. Peng, Y.-W., C. D. Blackstone, R. L. Huganir, and K.-W. Yau, 1995. Distribution of glutamate receptor subtypes in the vertebrate retina, Neuroscience, 66:483–497. Polyak, S. L., 1941. The Retina, Chicago: University of Chicago Press. Pugh, E. N., Jr. and J. D. Mollon, 1979. A theory of the p1 and p3 color mechanisms of Stiles, Vis. Res., 19:293–312. Qin, P., and R. G. Pourcho, 1995. Distribution of AMPA-selective glutamate receptor subunits in the cat retina, Brain Res., 710:303–307. Reid, R. C., and R. M. Shapley, 1992. Spatial structure of cone inputs to receptive fields in primate geniculate nucleus, Nature, 356:716–718. Rodieck, R. W., 1991. Which cells code for color? In From Pigments to Perception (A. Valberg and B. Lee, eds.), New York: Plenum Press, pp. 83–93. Rodieck, R. W., 1998. The First Steps in Seeing, Sinauer Associates. Sunderland, MA. Romeski, M., 1978. Chromatic opponent-response functions of anomalous trichromats, Vis. Res., 18:1521–1532. Roorda, A., and D. R. Williams, 1999. The arrangement of the three cone classes in the living human eye, Nature, 397:520–522. Sankeralli, M. J., and K. T. Mullen, 2001. Bipolar and rectified chromatic detection mechanisms, Vis. Neurosci., 18:127–135. Schiller, P. H., and C. L. Colby, 1983. The responses of single cells in the lateral geniculate nucleus of the rhesus monkey to color and luminance contrast, Vis. Res., 23:1631–1641.

:     

1001

Sekiguchi, N., D. R. Williams, and D. H. Brainard, 1993. Efficiency in detection of insoluminant and isochromatic interference fringes, J. Opt. Soc. Am. A, 10:2118–2133. Shapley, R., and V. H. Perry, 1986. Cat and monkey retinal ganglion cells and their visual functional roles, TINS, 9:229–235. Shiells, R., and G. Falk, 1995. Signal transduction in retinal bipolar cells, Prog. Retinal Eye Res., 14:223–247. Shinomori, K., L. Spillmann, and J. S. Werner, 1999. S-cone signals to temporal OFF-channels: asymmetrical connections to postreceptoral chromatic mechanisms, Vis. Res., 39:39–49. Sjöstrand, J., N. Conradi, and L. Klarén, 1994. How many ganglion cells are there to a foveal cone? Graefes Arch. Clin. Exp. Ophthalmol., 232:432–437. Stabell, U., and B. Stabell, 1982. Color vision in the peripheral retina under photopic conditions, Vis. Res., 22:839–844. Sterling, P., 1999. Deciphering the retina’s wiring diagram, Nat. Neurosci., 2:851–853. Stiles, W. S., 1949. Investigation of the Scotopic and tridchromatic mechanisms of vision by the two-color threshold technique, Rev. d’Opt., 139–163. Stromeyer, C. F., K. Kranda, and C. E. Sternheim, 1978. Selective chromatic adaptation at different spatial frequencies, Vis. Res., 8:427–435. Stromeyer, C. F. III, A. Chaparro, C. Rodriguez, D. Chen, E. Hu, and R. E. Kronauer, 1998. Short-wave cone signal in the redgreen detection mechanism, Vis. Res., 38:813–826. Teller, D. Y., and E. N. Pugh, Jr., 1983. Linking propositions in color vision, in Colour Vision: Physiology and Psychophysics ( J. Mollon and L. Sharpe, eds.), New York: Academic Press, pp. 577–589. van Esch, J. A., E. E. Koldenhof, A. J. van Doorn, and J. J. Koenderink, 1984. Spectral sensitivity and wavelength discrimination of the human peripheral visual field, J. Opt. Soc. Am. A, 1:443–450. Vardi, N., R. Duvoisin, G. Wu, and P. Sterling, 2000. Localization of mGluR6 to dendrites of ON bipolar cells in primate retina, J. Comp. Neurol., 423:402–412. Vardi, N., D. F. Matesic, D. R. Manning, P. A. Liebman, and P. Sterling, 1993. Identification of a G-protein in depolarizing rod bipolar cells, Vis. Neurosci., 10:473–478.

1002

  

Vardi, N., K. Morigiwa, T.-L. Wang, Y.-Y. Shi, and P. Sterling, 1998. Neurochemistry of the mammalian cone “synaptic complex,” Vis. Res., 38:1359–1369. Wässle, H., and B. B. Boycott, 1991. Functional architecture of the mammalian retina, Physiol. Rev., 71:447–480. Watanabe, M., and R. W. Rodieck, 1989. Parasol and midget ganglion cells of the primate retina, J. Comp. Neurol. 289: 434–454. Wiesel, T. N., and D. H. Hubel, 1966. Spatial and chromatic interactions in the lateral geniculate body of the rhesus monkey, J. Neurophysiol., 29:1115–1156. Williams, D. R., R. J. Collier, and B. J. Thompson, 1983. Spatial resolution of the short wavelength mechanism, in Colour Vision: Physiology and Psychophysics (J. Mollon and L. Sharpe, eds.), New York: Academic Press, pp. 487–503. Williams, D. R., N. Sekiguchi, and D. Brainard, 1993. Color, contrast sensitivity, and the cone mosaic, Proc. Natl. Acad. Sci. USA, 90:9770–9777. Wooten, B. R., and J. S. Werner, 1979. Short-wave cone input to the red-green opponent channel, Vis. Res., 19:1053–1054. Zhou, Z. J., D. W. Marshak, and G. L. Fain, 1994. Amino acid receptors of midget and parasol ganglion cells in primate retina, Proc. Natl. Acad. Sci. USA, 91:4907–4911. Zrenner, E., 1983a. Neurophysiological aspects of colour vision in primates. Comparative studies on simian retinal ganglion cells and the human visual system. Monograph. Studies of Brain Function, vol. 9 (V. Braitenberg, H. B. Barlow, T. H. Bullock, E. Florey, O.-J. Grüsser, and A. Peters, eds.), Berlin, Heidelberg, New York: Springer. Zrenner, E. 1983b. Neurophysiological aspects of colour vision mechanisms in the primate retina, in Colour Vision: Physiology and Psychophysics (J. Mollon and L. Sharpe, eds.), New York: Academic Press, pp. 195–211. Zrenner, E., and P. Gouras, 1983. Cone opponency in tonic ganglion cells and its variation with eccentricity in rhesus monkey retina, in Colour Vision: Physiology and Psychophysics (J. Mollon and L. Sharpe, eds.), New York: Academic Press, pp. 211–223.

65

Neural Coding of Color RUSSELL L. DE VALOIS

T   aspects of the neural organization underlying our color vision, it is essential to consider certain problems faced by the visual system in building a system to distinguish among objects on the basis of their chromaticity. Some problems are intrinsic to the physical characteristics of light and objects in the world; others are consequences of limitations in the anatomical and physiological properties of the visual nervous system. As a result, what may appear at first sight to be fairly straightforward problems turn out to be anything but. Let us begin by briefly considering some of these problems.

Confounds in the visual stimulus S  R   I Most of the interesting objects in the world do not themselves emit light, but rather reflect some proportion of whatever light falls on them. To characterize such objects, one needs to determine their reflectance properties, but the amount of light reaching the eye from an object is a product of the illuminant and the reflectance of the object. Since one does not have any independent knowledge of the illuminant, this is one equation with two unknowns and thus formally unsolvable. That is to say, any visual stimulus is massively underspecified, there being an infinite number of possible combinations of surfaces and illuminants that could have produced any given number of photons coming from a particular direction. The severity of this problem is increased by the fact that variations due to the reflectance characteristics of objects are tiny compared to the huge variations in the illuminant. The reflectances of objects vary only over a range of about 20 to 1, with a white object reflecting about 90% of the incident light and a black object about 5%. The variation in light level in the course of a day, on the other hand, can be 1 billion to 1. The daunting task for the visual system is to capture the tiny variations due to objects and separate them from the massive variation due to the illuminant. A related aspect of this problem is the need to separate the wavelength characteristics of objects from those of the illuminant, for the wavelength distribution of the light coming from an object is also a product of the wavelength distribution of the illuminant and the spectral reflectance characteristics of the object. The light reaching us from the sun is reddish at dawn and dusk and bluer at midday. Thus,

both the color and the lightness of objects are indeterminate in the absence of knowledge of the characteristics of the illuminant. To solve these problems and others, the visual system must in effect make (educated) a priori assumptions about the nature of the world—for instance, about which variations in the stimulus are most likely due to the illuminant (and thus to be largely ignored) and which are likely due to visual objects (and thus to be further processed). Most of these assumptions have been acquired through evolution, others through experience with the environment in the course of development. Many of the characteristics of the neural processes can be seen as reflecting the particular assumptions the visual system makes about the world. S I  C A second formally unsolvable problem for the visual nervous system is a consequence of the characteristics of the photopigments that constitute the first stage in the visual neural process. The photopigments have very broad spectral sensitivity; the two major cone types [the L (long-wavelength-sensitive) and M (middle-wavelength-sensitive) cones] absorb light of wavelengths across the whole visible spectrum. A given cone is more likely to absorb light of some wavelengths than of others, but an absorbed photon of any wavelength has precisely the same effect as one of any other wavelength. Thus, at the very first stage, wavelength and intensity information are totally confounded. This is known as the principle of univariance: a receptor is able to signal only one number— the number of photons absorbed; thus, it cannot separately report the wavelength and the intensity of the light incident upon it.

Multiple tasks and bottlenecks An easy error to make when considering any single aspect of vision, such as color vision or space or motion detection, is to look at the process as if it were being carried out in isolation, as if that were the only task for which that part of the system was designed. There do appear to be separate processing units or regions for different aspects of vision, at least to some extent, in the later cortical processing centers. With billions of neurons available in the 35% or so of the cortex devoted to vision, it is perhaps possible to have one group processing only color, another only motion, and so on. This

1003

is not a luxury available in the retina, however, where there are two severe bottlenecks that put a premium on multitasking. R B Although it is a rather odd use of the term, the physical dimensions of the receptors themselves can be considered an information bottleneck. Each receptor occupies a finite space in the array that samples the retinal image, thus excluding from that location a receptor with a different property. If we had some receptors just for color vision and other receptors just for high-acuity luminance vision, then our luminance visual acuity would be limited by the luminance receptors having color receptors interposed between them. As we discuss below, the fact that the same receptors support both color vision and high-acuity luminance vision can be seen to play a role in the locations of the spectral peaks of the different cone types, in the distributions of different cone types across the retina, and in the neural processing of the cone outputs. The receptor bottleneck is probably also a factor in our having evolved a color vision system based on only three different cone types. Increasing the number of different cone types would necessarily result in a coarsening of the array of each variety of cone. O N B A second bottleneck is in the optic nerve. Because the retina is inverted, the ganglion cell axons originate inside the eye and must produce a hole in the receptor layer in each nasal retina to exit the eye. The resulting 5 by 7 degree blind spot—large enough to encompass some 50 nonoverlapping images of the moon— is already an impediment to vision and would be even more so if there were more ganglion cells. A much larger optic nerve might also limit ocular mobility. There is thus a considerable advantage for neurons early in the visual path to multiplex information in order to carry out more than one type of analysis. Again, the consequence of this optic nerve bottleneck can be seen in the processing of color and luminance information in the retina.

Photopigments and receptors A crucial initial stage in color processing is the possession by most humans and other Old World primates of three different cone types containing (at least) three different photopigments. The properties of the photopigments are discussed more completely in Chapter 16. P Critical factors for color vision are the spectral sensitivities of the different cone types. In fact, much of the psychophysical research in color vision in the first half of the twentieth century (e.g., Stiles, 1949) consisted of attempts to determine these functions. The reason is that the

1004

  

extraction of color information is based on the differences between the spectral sensitivities of the different cone types, so the peaks and shapes of the spectral sensitivity functions are critical for understanding color vision. The output of each individual cone, regardless of type, carries no color information, for it completely confounds intensity and wavelength information (the principle of univariance discussed above). Thus, an increase in photon capture by any given cone can result from an increment in the intensity of the light or from a shift toward a more favorable wavelength; a decrease in photon capture correspondingly does not allow one to tell whether there was a decrease in light at that location or a shift to a wavelength to which that cone was less sensitive. However, a long-wavelength light will always result in more photon captures by an L than by an M cone, regardless of its intensity. Increments and decrements in intensity with no change in wavelength will produce increases and decreases in photon capture by both L and M cones, but the ratio of their activities will remain constant. Changes in wavelength, on the other hand, will change the ratio of L to M activity. Intensity information thus lies in the sum of the activity of the different cone types, and color information lies in the differences between the activities of the different cone types. It is these two types of information that are extracted in the early retinal processing. The spectral sensitivities of the human cone pigments have been well established from many different types of experiments, including psychophysical studies of normal and dichromatic human observers (Smith and Pokorny, 1975; von Kries, 1905; Vos and Walraven, 1971; and many others), microspectrophotometry of cone outer segments (e.g., Bowmaker and Dartnall, 1980), recordings of cone electroretinograms ( Jacobs and Neitz, 1993), and recordings from individual cone outer segments (Schnapf et al., 1987). There are three things relevant to color vision to be noted in the cone pigment curves shown in Figure 65.1. Broad spectral adsorption. One is that the absorption curves for the L- and M-cone pigments extend across the whole spectrum, making the L and M cones responsive to all wavelengths and intensities of light. It follows from the principle of univariance that the output of an L or M cone does not in any way specify either the color or the luminance of the light impinging on it. L and M cones thus should not be called or thought of as color receptors. Information about local color and luminance must be computed, as it is at multiple neural levels, by comparisons of the relative activities of several receptors in various combinations. Color and luminance information lies not in the outputs of individual receptors but in the differences and sums of the outputs of various receptors. The S (short-wavelength-sensitive) cones, on the other hand, differ from the L and M cones in this as in a number of other respects. They absorb primarily in the

photopigment genes from their father and their mother. That is indeed the case (Neitz et al., 1993). However, such female observers are nonetheless trichromats; they do not have a four- or five-dimensional color system ( Jordan and Mollon, 1993; Nagy et al., 1981). Contrary to the assumption from the time of Thomas Young (1802), the trichromatic limitation to our color vision lies not in the receptors but at some later processing stage.

Wavelength (nm) 400

500

600

700

S

M

L

1

Log Sensitivity

0

-1

-2

-3

-4

-5 2.7

2.5

2.3

2.1

1.9

1.7

1.5

1.3

Wave number

F 65.1. The spectral sensitivities of the three human cone photopigments. (Based on Stockman and Sharpe, 2000.)

short-wavelength range, and they appear to play a major role only in color vision. Multiple photopigment alleles. A second point is that multiple alleles (varieties) of the L- and of M-cone photopigment genes are present in the normal as well as the colordefective human population. This was first postulated on psychophysical grounds by Alpern (Alpern, 1979; Alpern and Wake, 1977) and later verified by genetic studies (Neitz et al., 1993). While the S-cone photopigment gene, lying in isolation from the other pigment genes on an autosome, is quite uniform across the population, the L- and M-cone photopigment genes lie juxtaposed on the X chromosome (Nathans et al., 1986). One consequence of this is that the L and M genes are occasionally subject to exchange of parts during meiosis, leading to the aforementioned variability in the L and M photopigment spectral sensitivities. Other consequences for color vision of the location of the L and M photopigment genes on the X chromosome derive from the fact that human males get only one X chromosome, while females get two. A very aberrant M or L photopigment gene will thus necessarily produce a significant color anomaly in males. As a consequence, some 8% of human males are either anomalous LM trichromats, with sufficiently deviant L or M photopigments to have color vision significantly different from that of normal observers, or they are LM dichromats, possessing either only an L or only an M photopigment and are thus reduced to a two-dimensional color vision system. Females, on the other hand, inheriting two X chromosomes, are far less likely (about 1% of the population) to have anomalous or dichromatic color vision, since that would entail inheriting the anomaly from each parent. An important corollary of this is that a significant number of females must have, in addition to the S photopigment, not just two but three or four L- and M-cone photopigments, having inherited, for example, slightly different L-

Cone photopigment spacing. A third point to be made with respect to cone spectral sensitivities is that the peaks of the L- and M-cone photopigments are quite close together, about 30 nm apart on average, whereas the S-cone photopigment peak is some 90 and 120 nm away from the others. How are we to understand this arrangement, in particular the close spacing of the L- and M-cone peaks? As discussed above, color information lies in the difference between the spectral sensitivities of the cones, so it is clearly advantageous for color vision to have as wide a separation in spectral peaks as possible. One might thus expect a uniform and wide spacing of the three cone peaks. However, for high-acuity luminance vision, it would be advantageous if the peaks were all identical, that is, if there were only a single cone type to sample veridically the luminance distribution. The human arrangement can be seen as a compromise between these two opposing requirements. The L- and M-cone peaks are sufficiently close that the outputs of these cone types can be (and apparently are) treated as identical in the systems extracting information about intensity variations. However, the L and M peaks are sufficiently far apart to allow also for the extraction of considerable, although suboptimal, information about wavelength variations in the middle to long spectral range. Consistent with this supposition is the paucity of S cones in the central foveola (Ahnelt et al., 1987; Curcio et al., 1991), allowing for maximal sampling of the image by L and M cones in this region of highest acuity. R The primary role of the receptors is to transduce faithfully increments and decrements in incoming light, or wavelength changes in one spectral direction or the other, into variations in neural activity. However, a straightforward linear transduction of photon capture into neural activity would not by itself be sufficient. Two problems, both of which demand an additional very nonlinear process, must also be tackled at very early stages in processing. One problem is that the dynamic range of light levels far exceeds the range that can be encoded in the nervous system. In the course of a day, light intensity can vary by more than 1 billion to 1. On the other hand, information is transmitted down ganglion cell axons by bursts of nerve impulses which, within the typical fixation period of about 250 msec, can span a range at most of about 50 to 1. There is thus a gross

 :    

1005

mismatch between the dynamic range of light intensities and the dynamic range of signals that can be carried by cells in the visual nervous system. The other related problem is the need to maintain the color and brightness appearance of objects relatively constant in the presence of changing illumination. A given surface might reflect 80 photons to a receptor at one moment and 80 million at another moment when the illumination is brighter. For the surface to appear perceptually the same under these two conditions requires some nonlinear process. Major contributions to dealing with each of these problems are made by processes within the receptors themselves, as they must be to avoid saturating the later neural cells. Linear transduction cascade. The receptor’s outer segment membrane is held at an intermediate level of polarization by opposing forces of Na+ inflow through ion pores in the membrane and metabolic effort expelling the Na+ ions by means of ion pumps. Variations in the capture of light by the receptor photopigments located in the disks within the outer segment initiate a cascade of chemical reactions (described more fully in Chapter 16) resulting in changes in ion flow and thus changes in polarization of the membrane. Transient decrements and increments of various amounts in photon capture produce corresponding depolarizations and hyperpolarizations of the outer segment membrane, respectively, in a quite linear transduction process (Pugh and Lamb, 1990). The resulting currents are conducted to the synaptic region of the receptors, where they produce proportional variations in the release of the synaptic transmitter. Nonlinear adaptational processes. This chemical cascade within the receptor’s outer segment has two enzymatic sites, which provide for amplification of the small variations in photon capture that must be detected under dim light conditions. The photoisomerization of the retinal component of rhodopsin (or the similar cone opsins) turns the opsin into an enzyme that can potentially cleave, and thereby activate, numerous transducin molecules. Activated transducin, in turn, activates phosphodiesterase each molecule of which can hydrolyze many cGMP molecules. Since cGMP keeps the ion pores in the outer membrane open, their closure blocks Na+ inflow, hyperpolarizing the membrane. A decrement in photon capture will correspondingly lead to a decrease in phosphodiesterase and thus an increase in cGMP, opening pores and depolarizing the membrane. As light levels increase and photon capture increases or decreases by thousands from instant to instant, the limited amount of available chemicals required in the cascade and the limited number of pores in the outer membrane will reduce the effectiveness of each photon capture by some fraction: it might take a change of 1000 photons under these

1006

  

bright light conditions to have the same effect as a change of 1 photon in dim light. A second adaptational nonlinearity at the initial stage of visual processing involves the inflow of Ca2+ as well as Na+ ions through the outer membrane pores when they are opened by cGMP (Pugh and Lamb, 1990). The Ca2+ inflow acts as a negative feedback, since Ca2+ inhibits the production of cGMP. When elevated cGMP levels open more pores, the inflow cuts back on cGMP production; when cGMP levels fall, the decrease in Ca2+ inflow leads to more cGMP production. These divisive gain control mechanisms have the desired effect of cutting down the dynamic range of the system from 1 billion to 1 to at most a few hundred to 1. They also provide a large initial step toward maintaining lightness constancy in the face of large changes in illumination. Since the adaptation occurs within each receptor, it is a major step toward maintaining color constancy as well. As the illumination becomes redder toward sunset, the L cones will be receiving relatively higher light levels than the M and S cones, and will thus adapt relatively more, thus compensating for the change in chromaticity of the illuminant. So, under steady-state adaptation (as is generally the case since illumination changes only very slowly during the day), the system functions in a quite linear fashion. A change in light level, however, triggers divisive nonlinearities within the receptors which allow the system to adjust to the new adaptation level. Distribution across the retina. There are about 100 million rods and 3 to 5 million cones in the human (and macaque monkey) retina. The S cones are relatively few, only some 7% to 10% of the cone population (Curcio et al., 1991). L and M cones are very thin and densely packed in the center of the fovea. With increasing distance from the foveal center, they become thicker and separated increasingly farther from each other as a consequence of the appearance of S cones and then of rods. By a few degrees away from the foveal center, the cones are all surrounded by rods, which reach their highest concentration some 20 degrees away from the fovea. On the basis of receptor distribution alone, this must lead, as it does, to a loss of spatial acuity for chromatic as well as for photopic intensity-varying patterns with increasing retinal eccentricity. The primary changes with retinal eccentricity relevant to color vision, however, are related to how the retinal organization varies with eccentricity rather than with the distribution of receptors themselves.

Retinal and geniculate processing One can see in the history of studies of vision, in particular of color vision, an increasing realization of the complexity of the problems faced by the visual system, and of the

number of processing levels that must be involved. The most influential early theory of color vision, that of Young (1802) and Helmholtz (1867), was essentially a one-stage model in which three receptors, each responsive to a different color region (red, green, and blue), fed up independent paths from the receptors to the brain to give us our color vision. The opponent-color model of Hering (1878), at least as modified by Schrödinger (1920) and by Jameson and Hurvich (1955), was essentially a two-stage model in which the three cone types were combined in various opposing combinations within the retina to form red-green, blueyellow, and black-white opponent mechanisms that correspond to perceptual unique hues. The discovery of spectrally opponent cells in the monkey lateral geniculate nucleus (LGN) (De Valois et al., 1958), and the evidence that these cells fell into two spectrally opponent classes and one nonopponent class (De Valois, 1965; De Valois et al., 1966) seemed to provide evidence for such a two-stage opponent-color model. In fact, it is now clear that not only are there three distinct principal cell types at each level beyond the receptors in the retinogeniculate path to the striate cortex (V1), but that these three cell types project down separate, largely independent paths through the retina and LGN to separate destinations in the visual cortex. We will briefly describe these three separate anatomical paths (discussed more fully in Chapter 30) and then describe the response characteristics and contribution to color vision of the cells in each. We will also present evidence, however, for further crucial color processing beyond the retina and LGN. R-G-C P Parvo ( Pc ) path. About 80% of the input from the retina to the visual cortex comes by way of the pathway originating largely from individual L and M cones and projecting through the four parvocellular (small cell) layers of the LGN to the cortex. Each L and M cone in the central 80 degrees or so of the retina is contacted by two midget bipolar cells, each of which contacts only this one cone (Wässle and Boycott, 1991). These paired midget bipolars respond in opposite ways to the receptor synaptic transmitter: one depolarizes to increments in photon capture, and the other depolarizes to decrements in photon capture (Famiglietti and Kolb, 1976). The midget bipolars also receive antagonistic input from the same and neighboring L and M cones by way of the feedback onto the cones by horizontal cells. These two inputs form the center and antagonistic surround of the receptive fields (RFs) of the bipolar cell, respectively. Each incremental and decremental midget bipolar feeds into a midget ganglion cell of the corresponding response type. The ganglion cells also receive input from surrounding areas by way of the amacrine cells.

In the central retina, input to the RF center of each midget ganglion cell comes mainly from a single midget bipolar and thus from a single L or M cone. With increasing retinal eccentricity, there is increasing bipolar-toganglion-cell convergence, and the midget ganglion cells get input into their RF centers from more than one midget bipolar and thus in most cases from a combination of L and M cones (Wässle et al., 1994). The midget ganglion cells, in turn, project to one of the four parvocellular LGN layers and from there primarily to layer IVcb of the striate cortex (Hubel and Wiesel, 1972). The LGN on each side receives projections from the temporal half of the ipsilateral retina and from the nasal half of the contralateral retina, but the fibers from the two eyes go to different LGN layers. Thus outputs from the two eyes first come together in the striate cortex. Magno (Mc ) path. About another 10% of the input from the retina to the visual cortex also comes mainly from L and M cones, but with a very different sort of organization. Diffuse bipolar cells receive input even in the foveola from not one but a small group of L and M cones, with antagonistic input from a still larger number of cones in the region. There are again two types of bipolar cells, one which depolarizes to increments in photon capture by the small group of L and M cones and one which depolarizes to decrements in photon capture by the same group of cones. With retinal eccentricity, the number of cones feeding into the center and surround increases still further. There is no evidence that the diffuse bipolars show any selectivity among L and M cones, and thus presumably sum together the outputs of all L+M cones in a region. Although the anatomical origin of the input is not clear, there is physiological evidence that most cells in the magno path receive S-cone input as well (Cottaris et al., in prep.; Derrington et al., 1984). The two types of diffuse bipolars project to separate subregions of the inner plexiform layer, where they contact incremental and decremental parasol ganglion cells, respectively (with amacrine input as well). The axons of the parasol ganglion cells project to one of the two large-cell layers of the LGN and from there primarily to layer IVca of V1. Konio (Kc ) path. As is the case with cells in the Pc and Mc paths, the RF centers of some S-opponent cells depolarize to increments in S-cone photon capture (+S) and those of others (-S) depolarize to S-cone decrements (Valberg et al., 1986). However, unlike the other paths, the S-opponent pathway is very unbalanced, there being about five times as many +S as -S cells. The +S path originates in a so-called S-cone bipolar that contacts one to three S cones, bypassing L and M cones in the region (Mariani, 1984). The S-cone bipolar synaptic region also gets input from H2 horizontal cells that make contact almost exclusively with S cones. The

 :    

1007

S-bipolar contacts a bistratified ganglion cell in the plexiform layer in which incremental midget and parasol ganglion cells contact their respective bipolars. The other half of the bistratified ganglion cell’s dendritic arbor contacts diffuse bipolars at a decremental synaptic level (Dacey and Lee, 1994). The bistratified ganglion cell thus receives a +S signal at one branch of its dendritic tree and a -LM signal at the other. With retinal eccentricity, the S cones become more widely spaced and the number of S cones that feed into each bistratified ganglion cell increases. However, the specificity of the inputs (with only S cones in the RF center) appears to be maintained out to the far retinal periphery (Kouyama and Marshak, 1992). In addition to the six LGN layers that have long been recognized, additional thin layers of cells lying ventral to layers 1 to 4 of the LGN, forming the koniocellular path, have recently been described (Casagrande, 1994). A major component of that path consists of the S-opponent cells whose RF centers originate in the S cones (Hendry and Reid, 2000; Martin et al., 1997). The projection to the cortex of cells in the Kc layers is quite different from those in the main body of the LGN. Rather than projecting to layer IV of V1, these cells project to layers I and III, and mainly in the cytochrome-oxidase blob regions of each cortical module rather than uniformly over the whole cortical region. R P  R C Magno cells. Although Mc ganglion and LGN cells have been widely considered to be summing just the outputs of the L and M cones, and thus carrying a luminance signal (the human luminance function Vl being well fit by 2L+M), considerable evidence indicates that they receive significant input from S cones as well (Cottaris et al., in prep.; Derrington et al., 1984). However, since all three cone types feed into the RF centers of Mc cells with the same polarity, the magno pathway does constitute a largely achromatic channel. The RF surrounds of Mc cells also receive an antagonistic input from the sum of all the cone types. We can thus characterize the RFs of the incremental and the decremental Mc cells, respectively, as (+LMSc-LMSs) and (-LMSc+LMSs), where c and s refer to RF center and surround, respectively. Since it is equally important to detect and characterize dark and light objects, it is not surprising that the relative number of +LMSc and -LMSc magno cells is about the same. The cells in the Mc path have a spatially opponent organization, with antagonistic input from RF center versus surround. They thus signal changes in the amount of light absorbed by a small group of cones relative to the amount absorbed by the cones in a larger, spatially overlapping area. With uniform illumination, largely irrespective of chromaticity, center and surround responses will tend to cancel,

1008

  

but differential illumination within the RF will activate the cell. This organization begins to emphasize intensity information that is likely due to objects and minimize that related to the illuminant. The underlying “assumption” being made by the visual system is that if all the cones in a region absorb more light, or if they all absorb less, it is probably due to a change in the illumination (as when the sun goes behind a cloud or emerges from one). On the other hand, if one group of cones is activated more than its neighbors, this is likely the result of reflections from different objects or different parts of an object. Information about such local intensity variations is captured and carried to the cortex by cells in the Mc pathway. Since they sum together the outputs of different cone types, however, cells in this path carry little information about the chromaticity of patterns. If exactly the same proportions of L, M, and S cones fed into the centers and surrounds of Mc cells, they would respond only to achromatic patterns. However, most Mc cells receive more RF surround inputs from L cones than from M cones and thus have some degree of chromatic opponency (Derrington et al., 1984; Wiesel and Hubel, 1966). Because of this, because of their S-cone inputs, and because they give a small frequencydoubled response to isoluminant stimuli (Lee et al., 1989), it is a mistake to think that isoluminant stimuli would completely silence cells in the magno path. The Mc cells respond with a shorter latency than the other cell types, give a biphasic, transient response to stimuli, and are maximally responsive to high temporal frequencies (de Monasterio and Gouras, 1975; Derrington and Lennie, 1984; Lee et al., 1989; Wiesel and Hubel, 1966). Because they sum over a group of cones in their RF centers rather than receiving input from a single cone, they are tuned to lower spatial frequencies than are cells in the other paths. However, they have a much higher contrast sensitivity than cells in the parvo and konio paths (Kaplan and Shapley, 1986). Despite their larger RF centers, they are thus responsive to quite high spatial frequencies. Konio cells. While cells in the Mc path have mainly a spatially opponent organization, cells in the Kc path have primarily a spectrally opponent organization, signaling changes in the amount of light stimulating one or more S cones relative to the amount stimulating the neighboring L and M cones (+Sc-LMs or -Sc+LMs). Since the RF size of the antagonistic center and surround regions are approximately equal (Derrington et al., 1984; Wiesel and Hubel, 1966), these cells have spectral but little spatial opponency. We shall refer to these as S-opponent cells, or S0. Since the cone photopigments have broad and overlapping spectral sensitivity functions, one cannot identify the wavelength of a pattern by determining which cone is activated. However, differences in the relative activation of the different cone types are related to the spectral distribution of the light, and this information

is captured by a spectrally opponent organization. Shortwavelength light is absorbed relatively more by the S than by the L and M cones, regardless of its intensity. A shift in the light from middle toward short wavelengths will therefore excite a +S0 cell in the Kc path, and a shift toward long wavelengths will excite a -S0 cell. Parvo cells. Finally, the ganglion and LGN cells in the Pc path (about 80% of the total) have both a spatially and a spectrally opponent organization. These cells, at least in the central retina, receive almost all their input from a single L or M cone in their RF center and from a small group of L and/or M cones in the surround. There are thus four varieties of these cells: +Lc-Ms (or L0), -L0, +M0, and -M0. The +L0 and -M0 cells have the same spectral response characteristics (both +L-M), as do the +M0 and -L0 cells (both being +ML). Since they encode the difference between the activity of L cones versus M cones in spectrally opponent organizations, they respond to color variations in the long-wavelength half of the spectrum. (It is the absence of these responses, because of a lack of either the L- or the M-cone pigment, that makes a protanope or deuteranope unable to discriminate red from yellow from green.) However, parvo cells also signal changes in activation of a single L or M cone relative to that in the surrounding L and M cones in a spatially opponent organization. The Pc cells thus carry both chromatic and achromatic information. Relative to Mc cells, cells in the parvo path are sensitive to higher spatial but lower temporal frequencies; they also have sustained monophasic responses and lower contrast sensitivity (de Monasterio and Gouras, 1975; Derrington and Lennie, 1984; Kaplan and Shapley, 1986; Lee et al., 1989; and others). The RF organization of Pc cells, and the consequences of this organization for color, intensity, and spatial processing, are shown in Figures 65.2 and 65.3. The cone-input map of an idealized +Lc-Ms opponent ganglion or LGN cell is diagrammed in Figure 65.2A. The cell receives input from an L cone in the RF center and opposing inputs from M cones in the surround (many Pc cells have a mixed L and M surround rather than a cone-specific surround, but the input from the “wrong” cone type in the surround would only somewhat dilute the strength of the center without eliminating the cone opponency). The critical point (De Valois and De Valois, 1975) is that a Pc cell with such a cone-input map has not one but two quite different receptive field maps, depending on the nature of the stimulus. A luminance change drives both L and M cones in the same direction, but they feed into the Pc cell in opposite directions, so a luminance increment would produce an RF with an excitatory center and an inhibitory surround (Fig. 65.2B). On the other hand, since a pure color change drives L and M cones in opposite directions, their inputs to the Pc cell become not antagonistic but synergistic. In the example shown in Figure

B

A

C F 65.2. A cartoon demonstrating how color and intensity information is multiplexed by cells in the Pc pathway. Part a shows how L and M cones feed into a model +L-M opponent cell. Part b shows how such a cell would respond to intensity increments. These drive L and M cones in the same direction, thereby producing an RF with center-surround antagonism. Part c shows how such a cell would respond to a color shift toward red, which drives L and M cones in opposite directions, thereby producing an RF with center-surround synergism.

65.2C, a color shift toward red produces excitation from the L cone in the RF center and a decrease in inhibition (and thus excitation) from the M cones in the surround as well. As one would predict from the differing RFs for intensity and color changes shown in Figure 65.2, Pc cells show quite different spatial frequency tuning for chromatic versus achromatic stimuli. Figure 65.3 shows the average responses of four typical Pc cells (one of each RF center type, +L0, -L0, +M0, -M0) to isoluminant and to 50% contrast achromatic gratings. It can be seen that these cells respond to somewhat higher spatial frequencies, with spatially bandpass tuning, to achromatic patterns, whereas they show spatially lowpass tuning to pure chromatic patterns. They are most responsive to low-spatial-frequency chromatic stimuli and to high-spatial-frequency achromatic stimuli. (In terms of cone contrast, Pc cells are actually more responsive to chromatic

F 65.3. Averaged data from four L/M opponent LGN cells in response to grating patterns varying in intensity (diamonds) and in color (circles). It can be seen that these cells show bandpass spatial-frequency tuning to intensity variations, but lowpass tuning for color.

 :    

1009

E  C S A fundamental characteristic of our color vision, postulated by Thomas Young (1802) and first verified experimentally by Maxwell (1860), is that the multidimensional spectral space of natural images is collapsed to only a three-dimensional perceptual space: normal human observers are trichromats. Since early color processing of spatially uniform patterns is quite linear, color space can be transformed from one set of axes to another. A common system represents perceptual color space with the three axes of hue, saturation, and brightness. Another useful representation of color space, developed by MacLeod and Boynton (1979) and elaborated by Derrington et al. (1984), is based on the outputs of the three cone types as processed by the different LGN cell types (Fig. 65.4). The three axes in the MBDKL space correspond to the response properties of the Pc, Kc, and Mc LGN cells, respectively. The average responses of populations of each of the different varieties of Pc and Kc cells to shifts from white to various isoluminant chromatic stimuli around the circle in this color space are shown in Figure 65.5. It can be seen that the L-M and M-L Pc opponent cells fire maximally to 0 degrees and 180 degrees, respectively, with no response to 90 degree and 270 degree patterns, whereas the S-opponent Kc cells fire maxi-

1010

  

+LM Luminance axis

patterns across almost the whole spatial frequency range, since in the data shown in Figure 65.2, the cone contrasts of the 50% luminance gratings are about four times greater than those of the color gratings.) The chromatic and achromatic tuning functions of the LGN cells shown in Figure 65.3 look very similar to the corresponding contrast sensitivity functions of normal human observers for chromatic and achromatic gratings, as measured in psychophysical tests (e.g., van der Horst et al., 1967). This suggests that our relative sensitivities for chromatic and achromatic patterns of different spatial frequencies are determined by the response properties of Pc cells at the retinogeniculate level. It appears, then, that Pc cells, with their combined spatial and chromatic opponency, carry both chromatic and achromatic information, but over somewhat different spatial frequency bands. The neural processing in the retina thus begins but does not complete the process of separating color and intensity information. The presence of separate bipolar, ganglion, and LGN cells that respond to increments versus decrements of each cone type gives the parvo path the same advantage discussed above for Mc cells. It provides equal sensitivity to increments and decrements of light. In the case of parvo cells, it also provides for equal sensitivity to shifts in color toward red and toward green. There is one further advantage of this arrangement in the case of Pc cells: it leads to a simple cortical mechanism, discussed below, for eventually separating the color and luminance information that is multiplexed in this path to the cortex.

+M-L L/M

opp

one

nt a

xis

-S ta

en

n po

S

xis

op

+L-M

+S

-LM F 65.4. A diagram of the MBDKL color space based on the response properties of cells in the Pc, Kc, and Mc paths.

mally to 90 degrees and 270 degrees, with no response to 0 degree and 180 degree patterns. Pc and Kc cells thus form orthogonal chromatic axes in this color space. The paired +LMS and -LMS Mc cells fire to achromatic increments and decrements, respectively, along the third axis. The chromatic response characteristics of cells, plus information about the time course of the responses, can also be obtained through reverse-correlation RF mapping with a rapidly presented sequence of isoluminant chromatic stimuli. Figure 65.6 shows such chromotemporal RFs of two representative LGN cells, a +M-L and a +S-LM opponent cell, respectively. Time with respect to stimulus onset is represented along the radii from the center out, and the different orientations represent different angles in the MBDKL isoluminant plane. The chromatic RF regions shown in red reflect a positive correlation between the response and the stimulus, and those shown in blue reflect a negative correlation. Thus, the +M-L cell (Fig. 65.6A), for instance, fires to stimuli from 90 degrees to 270 degrees, with maximum excitation at 180 degrees, and inhibits to stimuli from 270 degrees to 90 degrees, with maximum inhibition at 0 degrees. It can be seen that each of these cells has a response latency of about 50 msec and gives a monophasic, sustained response to the stimuli. Both responses to chromatic flicker (Fig. 65.5) and chromotemporal RF mapping studies (Fig. 65.6) show that LGN cells’ responses to isoluminant stimuli along various color angles not only are bimodally distributed, but they segregate into two distinct, nonoverlapping classes. It has long been thought that the three-dimensional limitation on our

+L-M LGN Cells

-M+L LGN Cells

20.0

20.0

15.0 10.0

15.0 10.0

5.0 0.0

5.0 0.0

-5.0 -10.0

-5.0 -10.0

-15.0

-15.0

-20.0

-20.0 0

90

180

270

n=44 360

0

90

Color Axis

+M-L LGN Cells

25.0 20.0 15.0 10.0 5.0 0.0 -5.0 -10.0 -15.0 -20.0 -25.0

180

270

n=30

360

Color Axis

-L+M LGN Cells 20.0 15.0 10.0 5.0 0.0 -5.0 -10.0 -15.0

0

90

180

270

n=37 360

-20.0 0

Color Axis

90

180

270 n=34

360

Color Axis

+S-M LGN Cells

S+LM LGN Cells

60.0

60.0

40.0

40.0

20.0

20.0

0.0

0.0

F 65.5. Averaged data from a large sample of LGN opponent cells of various response types. Shown are the responses to

full-field lights varying in various directions in MBDKL isoluminant color space.

color vision was set by the presence of only three cone pigments and receptors. However, it has been known for some time that we are trichromats even under mesopic conditions when rods as well as three cone types are active. Furthermore, as discussed above, females who inherit two different L-cone pigment alleles and thus have four cone pigments (with their rods making five receptor types) are still trichromats ( Jordan and Mollon, 1993; Nagy et al., 1981). It therefore appears that the trichromatic limitation on our color vision is set not at the receptor level, but by the presence of just three retinogeniculate cell types projecting down the Pc, Kc, and Mc pathways. One of the striking properties of color perception, emphasized by Hering (1878), is that various hues are not independent of each other but rather stand in an opponent relation. This can be seen in simultaneous and successive color contrast, in which, for instance, a red area induces the appearance of green in nearby regions and in the same region at the offset of the red. This opponent perceptual organization is also reflected in the fact that we can see combinations of red and yellow or blue and green in a single patch of color, but we do not see red and green or blue and yellow in the same place at the same time. Although the chromatic axes of LGN cells do not coincide with our per-

ceptual color axes, as we discuss below, the basic opponent nature of our perceptual color space reflects the opponent processing by the Pc and Kc cells in the retinogeniculate path.

Color processing in the striate cortex M  C R C Cells in the three main retinocortical paths (Pc, Mc, and Kc) project separately to layers IVcb, IVca, and I and III of the striate cortex (V1), respectively, but within the cortex these paths no longer remain discrete. Rather, V1 cells combine the outputs of the different LGN cell types in various ways. As a consequence, the responses of V1 cells do not fall into a few discrete chromatic categories, as at the level of the LGN. Rather, their peak chromaticities are spread across the entire spectrum (Lennie et al., 1990; Thorell et al., 1984). Most of the early cortical processing involves building RFs to detect various spatial aspects of the pattern (e.g., spatial frequency, orientation, motion, depth) within a local cortical region. Cells in these circuits may show some chromatic tuning just because almost all the input from the LGN is color-coded. The chromatic proporties of these cells could be useful in circuits involved in identifying contours or characterizing shapes, but may make no contribution to color

 :    

1011

F 65.6. Shown are the chromo-temporal RFs of two LGN opponent cells, based on reverse-correlation mapping with the rapid presentation of full-field stimuli in various directions in MBDKL color space. In this representation, time from stimulus to response goes out from the center, and the orientation shows the color direction. Red shows positive correlation between stimulus and response of the cell; blue shows a negative correlation. a, The

RF of a +M-L cell. After an initial latency, this cell shows excitation to chromatic stimuli in the +M direction in the LM axis, with maximum response to 180 degrees, and inhibition to the +L direction, with maximum inhibition at 0 degrees. The +S-LM opponent cell whose RF is shown in b responds along the orthogonal color axis, with maximum excitation to 90 degrees and maximum inhibition to 270 degrees. (See color plate 43.)

perception per se. That such may be the case is indicated by the fact, discussed below, that individuals with certain cortical lesions may see no color in the world but still may be able to discriminate objects that differ from their background only in color (Mollon et al., 1980; Victor et al., 1989). On the other hand, some cells found in V1 appear to be significant for color perception per se, in that their response properties reflect certain aspects of color processing which psychophysical evidence indicates must take place at some cortical level, and the chromatic response characteristics of these cells correspond to what one would expect from perception. With respect to color vision per se, the primary cortical processing involves separating color and luminance information, combining LGN cell types to produce cells whose color responses coincide with perceptual color categories, and further separating changes due to the illuminant from those due to visual objects by lateral interactions over large regions.

of four different types of Pc cells also allows for other cortical cells to separate luminance and long-wavelength color information by combining the outputs of Pc cells in two different ways (De Valois and De Valois, 1993; Lennie et al., 1991). When their outputs are summed in one way, the luminance components to their responses sum and the color components cancel. Summed in a different combination, the color components sum and the luminance components cancel. Consider a striate cortex cell that combines inputs from a number of +L0 and +M0 neurons in a region. Such a cell would respond to luminance variations but not to color variations, since both of the cell types that provide its inputs give excitatory responses to luminance increments in the RF center and to decrements in the surround. The color organizations of its inputs are opposite to each other (one being L-M and the other M-L), however, so the color information would cancel. Combined with input from a +S0 cell, this would produce a V1 cell that fires to white and inhibits to black but does not respond to pure color variations. (Input from S cones contributes importantly to white; L+M alone gives yellow, not white.) This is represented in the top row of the model shown in Figure 65.7. Correspondingly, a cortical cell that combined inputs from -L0, -M0, and -S0 cells in a region would fire to black, inhibit to white, and be quite

S  C  L As discussed above, some 80% of the cells in the retinogeniculate projection (those in the Pc path) multiplex color and luminance information. This is true also for many if not most cells in the striate cortex (Thorell et al., 1984). However, the presence

1012

  

+

+M O

+S O

Light

+

+ +

+

-LO

+

+

+

+

+S O

-SO

-SO

+S O

HWR HWR Red Yellow

-SO

Dark

Blue

F 65.7. A model of how different types of LGN cells might be combined at the cortex to form the different perceptual color systems. (After De Valois and De Valois, 1993.)

unresponsive to pure chromatic patterns. On the other hand, a V1 cell receiving inputs from both +L0 and -M0 cells, or from both +M0 and -L0 cells (columns in Fig. 65.7), would respond to color changes but not to luminance variations, since their color responses would add, but their luminance RFs, which are opposite to each other, would cancel. Rotation of color axes and sharpening of chromatic responses. The combination discussed above, in which LM opponent cells with common color characteristics but opposite luminance responses are combined, would separate out the chromatic component of the L/M-opponent cells, but it would not produce cells whose response characteristics correspond to perceptual red-green. The L/M-opponent LGN cells are often incorrectly referred to as forming as the red-green color system and S-opponent cells as constituting the blueyellow color system. I first introduced this nomenclature (De Valois, 1965), but it has become increasingly clear that the peak responses of these LGN-opponent cells do not correspond to the location of the unique hues (De Valois et al., 1997, 2000b), see Figure 65.8. Thus, the region seen as blue does not correspond precisely to the response range of +SLM cells but is intermediate between these and the +M-L cells; the MBDKL directions perceived as green do not correspond to that of +M-L cells but are intermediate between that and -S+LM cells; so also for yellow, although less so for red. A second discrepancy between the spectral-response characteristics of LGN cells and color perception is that the responses of LGN cells to different chromatic directions that produce sinusoidally varying cone contrasts are also sinusoidal (Fig. 65.5), since their responses are quite linear. However, perceptual hue-scaling data are much more peaked (Fig. 65.8). For example, while there is a broad region around 270 degrees to which -S+LM cells respond about

B

40

G

Y

R

0 0

90

180

270

360

450

LGN Opponent Cell Responses

100

HWR HWR Green

60

20

Normalized Spikes/sec

-MO

HWR

80 Percent Hue

+

+L O

Hue Scaling

100

HWR

80 S-LM

60

M-L

-S+LM

L-M

40 20 0 0

90

180

270

360

450

Color Angle (MBDKL deg)

F 65.8. Comparison of perceptual hue scaling with responses of LGN cells. The top half of the figure shows judgments made by observers of the percentage of each color in stimuli at each of many different chromatic angles. The bottom half of the figure shows the averaged responses of a large population of LGN cells of different opponent cell types to similar stimuli. Note that the colors in the perceptual data are rotated with respect to the axes of the geniculate cells and that the perceptual functions are narrower, suggesting the existence of a stage of color processing past the geniculate.

equally, there is only a narrow region (around 290 degrees) which is perceived as being very yellow. Both of these discrepancies between LGN response characteristics and perception suggest that our basic color categorization reflects a cortical rather than an LGN level of processing. In fact, the response properties of some V1 cells correspond much more closely to the perceptual primary colors than do those of LGN cells (De Valois et al., 2000a). Figure 65.9 shows the chromatotemporal RF of two such V1 cells, each tuned to color axes intermediate to the LGN axes but closely corresponding to perceptual green (Fig. 65.9A) and perceptual blue (Fig. 65.9B), respectively. Furthermore, since most V1 cells show an expansive response nonlinearity, their chromatic tuning is typically more sharply peaked than is the case with LGN cells (De Valois et al., 2000a). The relation between LGN and cortical cell responses and color perception has been modeled as (at least) a three-stage process (De Valois and De Valois, 1993); see Figure 65.7. L-M opponent cells excite to the so-called warm colors (red and yellow) and inhibit to the so-called cool colors (blue and green) and M-L cells fire to cool colors and inhibit to warm

 :    

1013

F 65.9. A comparison of responses of LGN cells to isoluminant stimuli (left) with the perceptual hue scaling of these same stimuli by human observers (right). Note that the perceptual colors

are shifted with respect to the LGN axes and are narrower, suggesting a further color processing step in V1. (See color plate 44.)

colors. The further addition of inputs from +S0 or from -S0 cells is required to split these classes into separate red and yellow and separate blue and green systems, respectively. The S-opponent retinogeniculate system, with its relative paucity of cells, essentially plays the role of a color modulator, being combined in various ways with the much more populous Pc cells to form three pairs of color systems.

be, and exactly what computations are carried out there. The output from V1 goes down several paths to later processing areas, including a path from V1 to V2 to V4 and from there to the inferotemporal cortex. There were early reports that area V4 seemed to be particularly involved with color. Almost all V4 cells were reported to be color selective and to be more chromatically selective than cells earlier in the path (Zeki, 1973). Area V4 was thus postulated to be the central color center and the likely site of injuries that produce achromatopsia. More recent studies, however, have found far fewer color-selective cells in V4 than were initially reported, and have found that V4 cells do not differ from LGN cells in their degree of color selectivity (e.g., Schein and Desimone, 1990; Schein et al., 1982). Furthermore, V4 lesions in macaques have been found to produce major deficits in form discrimination but to have minimal effect on color vision (Heywood et al., 1992; Walsh et al., 1992). On the other hand, lesions in an inferotemporal lobe region that spared V4 were found to have a devastating effect on the macaque’s color vision (Heywood et al., 1994). While it is difficult to determine homologous regions in monkey and human prestriate cortex, the clinical literature also indicates that a temporal lobe region anterior to the presumed homolog of monkey V4 is the location of those lesions that lead to achromatopsia (Meadows, 1974). Patients with achromatopsia are incapable of sorting chips by color in the Farnsworth-Munsell 100-hue test, and a functional magnetic resonance imaging study (Beauchamp et al., 1999) has shown that what is presumably this same temporal lobe region is specifically activated in normal observers when they

Color processing beyond the striate cortex For more than a century, there have been reports in the clinical literature of individuals who have lost the perception of color after a cortical lesion resulting from stroke or injury (see Meadows, 1974; Zeki, 1990, for reviews). This condition is called achromatopsia. A number of studies have shown that such individuals may still be able to make discriminations based on color alone, but they perceive no hue in the environment and are unable to identify the colors of objects (Mollon et al., 1980; Victor et al., 1989). It is also of interest that there has been one carefully studied case of a patient who, after a medical crisis, lost for some time the ability to see black-white patterns but retained color perception (Rovamo et al., 1982). She could see a color television program perfectly well, but a black-white program was invisible. Unfortunately, the site of the presumed cortical malfunction that led to this loss is unknown. While the existence of at least one cortical region essential for color perception is clear from the clinical evidence, there is some dispute as to the precise anatomical location of this region, what its inputs from V1 and elsewhere might

1014

  

carry out this task. Several lines of evidence, then, appear to show that not V4 but rather some more anterior temporal lobe region is the cortical area which is crucial for color perception, with its destruction producing cerebral achromatopsia. While some agreement is emerging on the anatomical areas principally involved in processing color information beyond V1, exactly what further computations are being carried out is far from clear. REFERENCES Ahnelt, P. K., H. Kolb, and R. Pflug, 1987. Identification of a subtype of cone photoreceptor, likely to be blue-sensitive, in the human retina, J. Comp. Neurol., 255:18–34. Alpern, M., 1979. Lack of uniformity in colour matching, J. Physiol. (Lond.), 288:85–105. Alpern, M., and T. Wake, 1977. Cone pigments in human deutan colour vision defects, J. Physiol. (Lond.), 266:595–612. Beauchamp, M. S., J. V. Haxby, J. E. Jennings, and E. A. De Yoe, 1999. An fMRI version of the Farnsworth-Munsell 100-hue test reveals multiple color-selective areas in human ventral occipitotemporal cortex, Cereb. Cortex, 9:257–263. Bowmaker, J. K., and H. J. A. Dartnall, 1980. Visual pigments of rods and cones in a human retina, J. Physiol. (Lond.), 298:501–511. Boycott, B. B., and H. Wässle, 1991. Morphological classification of bipolar cells of the primate retina, Eur. J. Neurosci., 3:1069– 1088. Casagrande, V. A., 1994. A third parallel visual pathway to primate area V1, Trends Neurosci., 17:305–310. Cottaris, N. P., D. R. Elfar, and R. L. De Valois, Spatio-temporal receptive field profiles of macaque striate cortex simple cells. In preparation. Curcio, C. A., K. A. Allen, K. R. Sloan, C. L. Lerea, J. B. Hurley, I. B. Klock, and A. H. Milam, 1991. Distribution and morphology of human cone photoreceptors stained with anti-blue opsin, J. Comp. Neurol., 312:610–624. Dacey, D. M., and B. B. Lee, 1994. The “blue-on” opponent pathway in primate retina originates from a distinct bistratified ganglion cell type, Nature, 367:731–735. de Monasterio, F. M., and P. Gouras, 1975. Functional properties of ganglion cells of the rhesus monkey retina, J. Physiol. (Lond.), 251:167–195. Derrington, A. M., J. Krauskopf, and P. Lennie, 1984. Chromatic mechanisms in lateral geniculate nucleus of macaque, J. Physiol. (Lond.), 357:241–265. Derrington, A. M., and P. Lennie, 1984. Spatial and temporal contrast sensitivities of neurones in lateral geniculate nucleus of macaque, J. Physiol. (Lond.), 357:219–240. De Valois, R. L., 1965. Analysis and coding of color vision in the primate visual system, Cold Spring Harbor Symp. Quant. Biol., 30:567–579. De Valois, R. L., I. Abramov, and G. H. Jacobs, 1966. Analysis of response patterns of LGN cells, J. Opt. Soc. Am., 56:966–977. De Valois, R. L., N. P. Cottaris, S. D. Elfar, L. E. Mahon, and J. A. Wilson, 2000a. Some transformations of color information from lateral geniculate nucleus to striate cortex, Proc. Natl. Acad. Sci. USA, 97:4997–5002. De Valois, R. L., and K. K. De Valois, 1975. Neural coding of color, in Handbook of Perception V (E. C. Carterette and M. P. Friedman, eds.), New York: Academic Press, pp. 117–166.

De Valois, R. L., and K. K. De Valois, 1993. A multi-stage color model, Vis. Res., 33:1053–1065. De Valois, R. L., K. K. De Valois, and L. E. Mahon, 2000b. Contribution of S opponent cells to color appearance, Proc. Natl. Acad. Sci. USA, 97:512–517. De Valois, R. L., K. K. De Valois, E. Switkes, and L. Mahon, 1997. Hue scaling of isoluminant and cone-specific lights, Vis. Res., 37:885–897. De Valois, R. L., C. J. Smith, S. T. Kitai, and S. J. Karoly, 1958. Responses of single cells in different layers of the primate lateral geniculate nucleus to monochromatic light, Science, 127:238– 239. Famiglietti, E. V., and H. Kolb, 1976. Structural basis for On- and Off-center responses in retinal ganglion cells, Science, 194:193– 195. Helmholtz, H. von, 1867. Handbuch der Physiologischen Optik, 1st ed., Hamburg: Voss. English translation: (1924) Handbook of Physiological Optics, 3 vols. (J. P. C. Southall ed.), Rochester, NY: Optical Society of America. Hendry, S. H. C., and C. M. Reid, 2000. The koniocellular pathway in primate vision, Annu. Rev. Neurosci., 23:127–153. Hering, E., 1878. Zur Lehre vom Lichtsinne, Vienna: Carl Gerolds Sohn. English translation: (1964) Outlines of a Theory of the Light Sense (L. M. Hurvich and D. Jameson, trans.), Cambridge, MA: Harvard University Press. Heywood, C. A., A. Gadotti, and A. Cowey, 1992. Cortical area V4 and its role in the perception of color, J. Neurosci., 12:4056–4065. Heywood, C. A., D. Gaffan, and A. Cowey, 1994. Cerebral achromatopsia in monkeys, Eur. J. Neurosci., 7:1064–1073. Hubel, D. H., and T. N. Wiesel, 1972. Laminar and columnar distribution of geniculocortical fibers in the macaque monkey, J. Comp. Neurol., 146:421–450. Jacobs, G. H., and J. Neitz, 1993. Electrophysiological estimates of individual variation in the L/M cone ratio, in Colour Vision Deficiencies XI (Documenta Ophthalmologica Proceedings Series 56) (B. Drum ed.), Dordrecht, Kluwer, pp. 107–112. Jameson, D., and L. M. Hurvich, 1955. Some quantitative aspects of an opponent-colors theory. I. Chromatic responses and saturation, J. Opt. Soc. Am., 45:546–552. Jordon, G., and J. D. Mollon, 1993. A study of women heterozygous for colour deficiencies, Vis. Res., 33:1495–1508. Kaplan, E., and R. M. Shapley, 1986. The primate retina contains two types of ganglion cells, with high and low contrast sensitivity, Proc. Natl. Acad. Sci. USA, 83:2755–2757. Kouyama, N., and D. W. Marshak, 1992. Bipolar cells specific for blue cones in the macaque retina, J. Neurosci., 12:1233–1252. Lee, B. B., P. R. Martin, and A. Valberg, 1989. Sensitivity of macaque retinal ganglion cells to chromatic and luminance flicker, J. Physiol. (Lond.), 414:223–243. Lennie, P., P. W. Haake, and D. R. Williams, 1991. The design of chromatically opponent receptive fields, in Computational Models of Visual Processing (M. S. Landy and J. A. Movshon, eds.), Cambridge, MA: MIT Press, pp. 71–82. Lennie, P., J. Krauskopf, and G. Sclar, 1990. Chromatic mechanisms in striate cortex of macaque, J. Neurosci, 10:649–669. MacLeod, D. I. A., and R. M. Boynton, 1979. Chromaticity diagram showing cone excitation by stimuli of equal luminance, J. Opt. Soc. Am., 69:1183–1186. Mariani, A. P., 1984. Bipolar cells in monkey retina selective for the cones likely to be blue-sensitive, Nature, 308:184–186. Martin, P. R., A. J. R. White, A. K. Goodchild, H. D. Wilder, and A. E. Sefton, 1997. Evidence that blue-on cells are part of the

 :    

1015

third geniculocortical pathway in primates, Eur. J. Neurosci., 9:1536–1541. Maxwell, J. C., 1860. On the theory of compound colours, and the relations of the colours of the spectrum, Philos. Trans. R. Soc. Lond., 150:57–84. Meadows, J. C., 1974. Disturbed perception of colour associated with localized cerebral lesions, Brain, 97:615–632. Mollon, J. D., F. Newcombe, P. G. Polden, and G. Ratcliff, 1980. On the presence of three cone mechanimsms in a case of total achromatopsia, in Colour Vision Deficiencies V (G. Verriest ed.), Bristol: Hilger, pp. 130–135. Nagy, A. L., D. I. A. MacLeod, N. E. Heyneman, and A. Eisner, 1981. Four cone pigments in women heterozygous for color deficiency, J. Opt. Soc. Am., 71:719–722. Nathans, J., D. Thomas, and D. S. Hogness, 1986. Molecular genetics of human color vision: the genes encoding blue, green and red pigments, Science, 232:193–202. Neitz, M., J. Neitz, and G. H. Jacobs, 1993. More than three different cone pigments among people with normal color vision, Vis. Res., 33:117–122. Pugh, E. N., and T. D. Lamb, 1990. Cyclic GMP and calcium: the internal messengers of excitation and adaptation in vertebrate photoreceptors, Vis. Res., 30:1923–1948. Rovamo, J., L. Hyvarinen, and R. Hari, 1982. Human vision without luminance-contrast system: selective recovery of the red-green colour-contrast system from acquired blindness, Doc. Ophthalmol. Proc. Ser., 33:457–466. Schein, S. J., and R. Desimone, 1990. Spectral properties of V4 neurons in the macaque, J. Neurosci., 10:3369–3389. Schein, S. J., R. T. Marrocco, and F. M. de Monasterio, 1982. Is there a high concentration of color-selective cells in area V4 of monkey visual cortex? J. Neurophysiol., 47:193–213. Schnapf, J. L., T. W. Kraft, and D. A. Baylor, 1987. Spectral sensitivity of human cone photoreceptors, Nature, 325:439–441. Schrödinger, E., 1920. Grundlinien einer Theorie der Farbenmetric im Tagessehen, Ann. Physik Chem., 63:481–520. Smith, V. C., and J. Pokorny, 1975. Spectral sensitivity of the foveal cone photopigments between 400 and 500 nm, Vis. Res., 15:161–171. Stiles, W. S., 1949. Increment thresholds and the mechanisms of colour vision, Doc. Ophthalmol., 3:138–165.

1016

  

Stockman, A., and L. T. Sharpe, 2000. The spectral sensitivities of the middle- and long-wavelength-sensitive cones derived from measurements in observers of known genotype, Vis. Res., 40:1711–1737. Thorell, L. G., R. L. De Valois, and D. G. Albrecht, 1984. Spatial mapping of monkey V1 cells with pure color and luminance stimuli, Vis. Res., 24:751–769. Valberg, A., B. B. Lee, and D. A. Tigwell, 1986. Neurones with strong inhibitory S-cone inputs in the macaque lateral geniculate nucleus, Vis. Res., 26:1061–1064. van der Horst, G. J. C., C. M. M. de Weert, and M. A. Bouman, 1967. Transfer of spatial chromaticity-contrast at threshold in the human eye, J. Opt. Soc. Am., 57:1260–1266. Victor, J. D., K. Maiese, R. M. Shapley, J. Sidtis, and M. S. Gazzaniga, 1989. Acquired central dyschromatopsia: analysis of a case with preservation of color discrimination, Clin. Vis. Sci., 4:183–196. von Kries, J., 1905. Die Gesichtsempfindungen, in Handbuch der Physiologie des Menschen (W. Nagel ed.), Braunschweig: Vieweg: pp. 109–282. Vos, J. J., and P. L. Walraven, 1971. On the derivation of the foveal receptor primaries, Vis. Res., 11:799–818. Walsh, V., J. J. Kulikowski, S. R. Butler, and D. Carden, 1992. The effects of lesions of area V4 on the visual capabilities of macaques: colour categorization, Behav. Brain Res., 52:81– 89. Wässle, H., and B. B. Boycott, 1991. Functional architecture of the mammalian retina, Physiol. Rev., 71:447–480. Wässle, H., U. Grünert, P. R. Martin, and B. B. Boycott, 1994. Immunocytochemical characterization and spatial distribution of midget bipolar cells in the macaque monkey retina, Vis. Res., 34:561–579. Wiesel, T. N., and D. H. Hubel, 1966. Spatial and chromatic interactions in the lateral geniculate body of the rhesus monkey, J. Neurophysiol., 29:1115–1156. Young, T., 1802. On the theory of light and colours, Philos. Trans. R. Soc. Lond., 92:12–48. Zeki, S. M., 1973. Colour coding in rhesus monkey prestriate cortex, Brain Res., 53:422–427. Zeki, S. M., 1990. A century of cerebral achromatopsia, Brain, 113:1721–1777.

66

The Processing of Color in Extrastriate Cortex KARL R. GEGENFURTNER AND DANIEL C. KIPER

W   of the chromatic properties of cells in the early stages of the primate visual pathways has increased considerably in recent years, little is known about the processing of color information in the cerebral cortex. A number of experimenters have investigated the treatment of color information in the striate visual cortex (V1) of primates (Conway, 2001; Cottaris and De Valois, 1998; Dow and Gouras, 1973; Gouras, 1974; Hubel and Wiesel, 1968; Johnson et al., 2001; Lennie et al., 1990; Livingstone and Hubel, 1984; Michael, 1978a, 1978b, 1978c, 1979; Thorell et al., 1984; Ts’o and Gillbert, 1988; Yates, 1974) and in extrastriate area V4, which has been suggested to play an important role in the cortical analysis of color information (Schein and Desimone, 1990; Schein et al., 1982; Yoshioka et al., 1996; Zeki, 1973, 1980, 1983a, 1983b). However, the results of these studies are anything but equivocal, and the neuronal representation of color in the many other cortical areas has been little investigated. Here we review the current state of knowledge about the processing of color signals in the primate extrastriate cortex. The questions raised by most studies can be grouped into three main classes: First, how is color encoded within particular cortical areas? Studies concerned with this question determine the chromatic properties of individual neurons, as well as the number of cells coding for color within a given cortical area. Second, how specific are the response properties of extrastriate neurons to different visual attributes? The aim of these studies is to determine whether or not color information is treated separately from other attributes, such as form or motion. Third, do the neuronal response properties support perceptual phenomena such as color constancy? In this chapter, our attention will center on these three fundamental aspects of color vision. The vast majority of data have been obtained from macaque monkeys, whose color vision is similar to that of humans (De Valois, 1965; Jacobs, 1993). Our review therefore focuses on this species and is complemented by results obtained from human subjects whenever possible. To describe how color information is encoded within different cortical areas, we review what is known about the chromatic properties of cells and estimates of their proportion within these areas. Chromatic properties of cells are

largely captured by a description of their responses to stimuli varying in color. To determine whether color is treated independently of other visual attributes, we discuss how the chromatic properties of cells relate to other spatiotemporal properties of receptive fields, such as their selectivity to the orientation, direction of motion, or size of visual stimuli. Finally, we examine, whenever possible, the capacity of cells to maintain a stable response to a color stimulus in the face of changes in the stimulus illumination, in the chromatic composition of the background, or in different states of adaptation. Before considering in detail these various aspects of color processing in extrastriate cortex, it is necessary to briefly review the role of the primary visual cortex (V1), which provides the main inputs, directly or indirectly, to all extrastriate visual areas.

Primary visual cortex (V1) In early studies of the primate primary visual cortex, the proportion of chromatically responsive cells was estimated to be relatively low (Hubel and Wiesel, 1968). A few years later, it was found that many cells that respond to luminance variations also respond to color variations, bringing the overall proportion of color-selective cells to about 50% in the striate cortex of macaque monkeys (Gouras, 1974; Dow and Gouras, 1973; Johnson et al., 2001; Thorell et al., 1984; Yates, 1974). These results are supported by studies using functional magnetic resonance imaging (fMRI), which showed a strong color-opponent response in the primary visual cortex of human subjects (Engel et al., 1997; Kleinschmidt et al., 1996). The chromatic properties of V1 cells show both differences from and similarities with those at earlier stages of visual processing (retinal ganglion cells or parvo cells of the lateral geniculate nucleus, pLGN). A number of studies showed that in V1, unlike in the pLGN, the distribution of the cells’ preferred colors does not obviously cluster around particular directions in color space (Lennie et al., 1990; Yoshioka et al., 1996). While most color-selective pLGN cells prefer stimuli modulated either along a roughly red-green or blue-yellow direction, those in the primary visual cortex can have preferences for many other directions. However, Lennie

1017

et al. (1990) also found that pLGN and V1 cells have similar color tuning properties. They showed that a model that successfully describes the tuning properties of pLGN cells (Derrington et al., 1984) also fits the responses of most V1 neurons. This model postulates that the receptive field of cells can be summarized as a linear combination of the inputs from the three different cone classes. In other words, Lennie et al. showed that although V1 neurons, as a whole, sample color space more evenly than pLGN cells, individual cells in V1 are not more selective for color than pLGN cells. Other investigators have reported V1 cells with a narrow color selectivity that deviates significantly from the linear model (Cottaris and De Valois, 1998; Gouras, 1974; Yates, 1974), but overall, the proportion of such narrowly tuned cells appears to be small in V1. A number of reports suggested that the color signals within V1 are carried by a special dedicated population of unoriented cells (Livingstone and Hubel, 1984; Roe and Ts’o, 1999; Ts’o and Gilbert, 1988). These results have been challenged more recently. In particular, Leventhal et al. (1995) found that most cells in the superficial layers of V1 are sensitive to both the orientation and color of a visual stimulus. In addition, many of these cells also signal the stimulus’ direction of motion. More recently, Johnson et al. (2001) showed that V1 cells can simultaneously encode both the chromatic and spatial characteristics of a stimulus. These important findings raise serious doubts about the notion that color information is treated separately from other visual attributes within area V1. While the contribution of V1 cells to color constancy has not been studied systematically, several results bear on that issue. In particular, a number of studies reported the existence of a population of double-opponent cells in the primary visual cortex of primates (Livingstone and Hubel, 1984; Michael, 1978a, 1978b, 1978c, 1979). These cells have a spatially and chromatically antagonistic center-surround organization, and their properties could lead them to play an important role in achieving color constancy (Zeki, 1980). Although some studies reported only a very low incidence of such cells in V1 (Lennie et al., 1990; Ts’o & Gilbert, 1988), their existence has been confirmed by more recent reports (Conway, 2001; Johnson et al., 2001). Finally, a possible role of V1 cells in color constancy is supported by their ability to adapt during prolonged exposure to a habituating stimulus (Lennie et al., 1994), while LGN neurons do not (Derrington and Lennie, 1984). Similar results have been reported by Engel and Furmanski (2001), who used fMRI to investigate the adaptability of the human primary visual cortex to chromatic stimuli. While these results suggest that the response properties of V1 cells could contribute to color constancy, they are by no means conclusive proof that color constancy is achieved in V1.

1018

  

Color coding in extrastriate cortex T P  C-S C The proportion of cells capable of encoding color information appears to be remarkably stable across a number of extrastriate visual areas. Although the methods and criteria adopted by different authors render comparisons quite difficult, perusal of the literature suggests that, as in V1, a large proportion of cells in most extrastriate areas respond to color variations (Felleman and Van Essen, 1987). To obtain objective, quantitative estimates of the population of color coding neurons within V2, Gegenfurtner et al. (1996) measured the responses of cells to a set of drifting bars that span the color space around the isoluminant plane. These stimuli have a constant luminance contrast, and vary only in chromaticity (see Fig. 66.1, along with an example of a V2 cell’s responses to these stimuli). For each cell, the authors computed a color responsivity (CR) index, defined as CR = (Rcol - b) (R white - b) where Rcol is the best response of the cell to any of the colored bars, Rwhite is the response to the white bar, and b is the cell’s baseline firing rate. Cells with CR exceeding 1.4 were classified as colorselective. For the cell shown in Figure 66.1B, CR was 6.03, meaning that the response to a green bar was six times as great as the response to a white bar (after subtracting the baseline). Using this criterion, Gegenfurtner et al. (1996) found that approximately 50% of V2 cells code for the color of a stimulus. This proportion is roughly similar across the various subdivisions of V2 defined by anatomical staining techniques (De Yoe and Van Essen, 1985; Hubel and Livingstone, 1987) and is in agreement with other quantitative studies (Baizer et al., 1977; Levitt et al., 1994; Peterhans and von der Heydt, 1993; Yoshioka et al., 1996). Using the same methods and criteria, Gegenfurtner et al. (1997) found a similar proportion (~54%) of color cells in area V3. Earlier results yielded a lower estimate (Felleman and Van Essen, 1987), but the discrepancy disappears when the criteria used by the different authors are made comparable (Gegenfurtner et al., 1997). Area V4 was proposed to be the main color-specialized area. Zeki (1983a, 1983b) presented cells with a Mondrian display and varied the wavelength composition of the individual rectangles making up the Mondrian pattern, as well as the wavelength composition of the illuminating lights. Using this arrangement, he was able to distinguish two categories of color-selective neurons in V4. A first class responded exclusively to the wavelength composition of the stimulus. A second class showed responses that correlated with the color appearance of the individual patches, as seen by a human observer. No estimates are available on the relative frequency of these two classes. The overall proportion

A

White 900 (Yellow) Luminance

Stimulus Stimulus Set Space 180 0 (Green)

elevation 100

00 (Red) Isoluminant Plane

2700 (Blue) Black

Response (imp/s)

B

368l007 inter CR = 6.03

40 30 20

white blank

10 0 0

red

9 0

blue

180

green

270

yellow

360

Azimuth (deg) F 66.1. A, Schematic diagram illustrating the color space used by Gegenfurtner et al. (1996). At the origin of the color space is a neutral white. Along the L-M axis (red-green), the excitation of the L and M cones covaries to keep their sum constant. Along the S-(L + M) (blue-yellow) axis, only the excitation of the S cones varies. Along the luminance (black-white) axis, the excitation of all cone types varies in proportion to their excitation at the white point. The inverted cone represents a set of stimuli, here with an elevation of 10 degrees, and an azimuth varying between 0 and 360 degrees. Grating stimuli had colors modulated around the neutral white, thus lying on the surface of the cone. Examples of cell responses to such gratings are shown in Figure 66.3. The bar stimuli used to compute the color index (CR) had a single color, chosen on the outer rim of a cone like the one depicted here. B, Responses of V2 cells, located in a CO interstripe, to bars of varying color drifting on a dark background (see text). The solid horizontal line shows the response to white bars having the same luminance as the colored ones (37.5 cd/m2). The dashed horizontal line shows the cell’s baseline firing rate.

of color-selective cells in V4 has varied considerably among authors, from less than 20% (Schein et al., 1982) to 100% (Zeki, 1973). Although a final consensual estimate is lacking, it is now widely agreed that the majority of V4 cells exhibit some color selectivity (Schein and Desimone, 1990). In inferotemporal (IT) cortex, which receives direct inputs from V4, the proportion of color-selective cells is believed to be high as well, and has been estimated between 48% (Gross et al., 1972) and 70% (Komatsu and Ideura, 1993; Komatsu et al., 1992).

Much less is known about the proportion of colorselective cells in areas that are known to be important for motion processing, such as MT and MST. Previously, these areas were assumed as lacking any color selectivity (Zeki, 1983c). However, using a combination of physiological and psychophysical techniques, Albright and collaborators have shown that color information does contribute to the responses of MT neurons (Croner and Albright, 1999; Dobkins and Albright, 1994; Thiele et al., 1999). Moreover, Gegenfurtner et al. (1994) showed that most MT neurons do respond to chromatic variations, although much less vigorously than to luminance modulation. In addition, they showed that most MT cells do not show the response properties that would qualify them as truly color-selective cells, and that their responses to color variations are too small to account for the animals’ behavioral thresholds. Thus, although not totally absent, chromatic information seems to be poorly represented in area MT. These results agree quite well with fMRI responses in the human MT/MST complex (Tootell et al., 1995; Wandell et al., 1999). T P C  C In the retina and LGN, the preferred colors of cells cluster in specific directions of color space (Derrington et al., 1984). In V1, as mentioned above, the distribution of preferred colors is much more uniform, although a faint bias for the red-green and blueyellow directions can be seen (Lennie, 1999; Lennie et al., 1990; Yoshioka et al., 1996), particularly in the populations of nonoriented and simple cells. In addition, most V1 cells prefer stimuli that vary in luminance compared to purely chromatic modulations. These characteristics emerging in V1 hold true for V2 as well. Kiper et al. (1997) showed that the preferred colors of V2 cells do not cluster around particular directions in color space, and that most cells respond preferentially to luminance rather than to chromatic variations. The distribution of preferred color within the V2 population is shown in Figure 66.2A. In this graph, the preferred color of a cell is expressed by two values, azimuth and elevation, which are the polar coordinates of the cells’ preferred direction in the color space depicted in Figure 66.1. An elevation of 90 degrees represents the purely luminance direction, and one of 0 degrees represents a preference for stimuli modulated in the isoluminant plane. In this plane, an azimuth of 0 degrees corresponds approximately to red, 90 degrees to blue, 180 degrees to green, and 270 degrees to yellow (see Derrington et al., 1984, for a complete description of this color space). In addition, Kiper et al. (1997) showed that the distribution of preferred colors holds true irrespective of the cells’ location within any one of the three subcompartments that have been described within V2 (De Yoe and Van Essen, 1985) and irrespective of the cells’ tuning in color space (see below).

  :    

1019

90

A

LGN Cells

Elevation (deg)

B

V1 Cells

45

45

045

90

90

C

0

45 thick

thin

90 inter

135

045

V2

90

60

60

30

30

0 -45

0

45

90

0

45

90

D

0 135 -45

135 V3

0

45

90

135

Azimuth (deg) F 66.2. Scatterplots of the preferred azimuths and elevations cells in the pLGN (A), V1 (B), V2 (C), and V3 (D). The V2 cells are subdivided into the different CO compartments. The azimuths and elevations were derived from the linear model proposed by Derrington et al. (1984).

In V3, the distribution of preferred directions is very similar to that of V2, except for an unusual number of cells giving strong responses for modulations in the blue-yellow direction (Gegenfurtner et al., 1997). These can be seen in Figure 66.2D as the cells clustering around an azimuth of 90 degrees. For cells in area V4, Schein and Desimone (1990) determined the optimal wavelength using narrowband filters. They found that peak responses could be found at all wavelengths, as described above for V1, V2, and V3. Moreover, a small population of V4 cells exhibited several peaks in their wavelength tuning curves, with no systematic relationship between the location of the peaks. For cells in the IT cortex, Komatsu et al. (1992) also reported a rather uniform distribution of preferred colors. As in V4, a subpopulation of 20% of color-selective neurons preferred several colors. The functional significance of these multiple peaks of the wavelength tuning curves is presently not clear. In area MT, Gegenfurtner et al. (1994) reported that although virtually all cells clearly prefer luminance variations, the preferred color of cells clustered in the red-green direction of color space. This is consistent with the notion that area MT derives its major inputs from the middle (M)and long (L)-wavelength-sensitive cones via the magnocellular layers of the LGN (Gegenfurtner et al., 1994; Thiele et al., 1999). Note that short (S)-wavelength-sensitive cone inputs are not totally absent in area MT, as demonstrated by Seidemann et al. (1999) in monkeys and Wandell et al. (1999) in human subjects. Recent results, however, suggest that S cones contribute only to the luminance response of MT cells, and not to any color-opponent signal (Barberini et al., 2001).

1020

  

T T  C  C S Most V1 cells, like those in the LGN, have a rather broad selectivity for color, which is consistent with the hypothesis that they sum their cone inputs linearly, performing simple additions or subtractions between the signals originating in different classes of cones (Lennie et al., 1990). Although cells more selective than predicted by the linear model have been found in V1 (Cottaris and De Valois, 1998; Hanazawa et al., 2000; Lennie et al., 1990); their number seems to be small. In V2 they are much more numerous. Kiper et al. (1997) showed that the linear model failed to describe the tuning properties of a large group (~30%) of V2 colorselective cells. The tuning properties were determined by presenting sinusoidal gratings of different colors around the color circle illustrated in Figure 66.1A. A nonlinear function was used to interpolate between the data points, as shown by the dashed curve in Figure 66.3C. For linear cells, the predictions of both models are essentially identical, and they are not shown separately in Figures 66.3A, 66.3B, and 66.3D. At a constant elevation of 10 degrees, bandwidth was defined as the angular difference between the color vector that gave the best response and the color vector where the response had decreased to 50% of the difference between the firing rate at the peak and 90 degrees away from it. This definition has the advantage that it predicts a constant bandwidth of 60 degrees, the angle whose cosine is 0.5 for linear cells, independently of their preferred azimuth and elevation. The resulting histogram of bandwidths for V2 neurons is shown in Figure 66.4A. The distribution is bimodal, showing distinct subpopulations of narrowly tuned cells (like the one in Fig. 66.3C) and of cells with an approximately linear tuning (like that of Fig. 66.3B), which are clustered around a bandwidth of 60 degrees. To the right are cells for which the investigators could not reliably determine any color-specific tuning because they responded only to the luminance component of the stimuli. Using this measure of bandwidth, neurons could be classified into three categories: linear cells (bandwidth around 60 degrees), narrowly tuned cells (bandwidth lower than 45 degrees), and luminance cells (cells whose responses to the chromatic stimuli did not differ from that to the achromatic stimulus). The resulting classification for a sample of V2 cells (Kiper et al., 1997) and of V3 neurons (Gegenfurtner et al., 1997) is shown in Figure 66.4B. In V2 there is a significant subpopulation of cells whose bandwidths are much narrower than that predicted by the linear model. This subpopulation is almost entirely missing from the V3 sample. Only a handful (5 of 90) of V3 cells showed an indication of narrow tuning in color space. Accordingly, the proportions of luminance and linear cells were slightly higher in V3. In area MT, the responses of nearly all cells are determined by the

A 30

Response (imp/s)

B

376r005 V1

30

20

20

10

10

0

367l010 V2

0 0

90

180

270

360

C

0

90

180

270

360

180

270

360

D 378l027 V2

30

30

20

20

10

10

0

0 0

90

180

270

360

118r008 V3 simple cell

0

90

Azimuth (deg) F 66.3. Responses of cells located in V1 (A), V2 (B, C), and V3 (D) to drifting sinusoidal gratings modulated around the white point, with varying azimuths and a constant elevation of 10 degrees as shown in Figure 66.1. The solid curve fitted through the data represents the prediction of the linear model (Derrington et al., 1984). The horizontal solid line shows the cells’ response to a blackand-white grating having the same luminance contrast as the

Number of cells

A

thick

inter

thin

15 linear

10 5 0 15

30 45 60 75 Color bandwidth (deg)

90 undefined

B Proportion of cells

1.0

V2

V3

0.5

0.0 Luminance cells

Linear cells

Narrowly tuned cells

F 66.4. A, Histogram of the bandwidth of tuning in color space for 84 V2 cells. To the right (“undefined”) are cells that responded only to luminance modulations. For each bandwidth bin, the cells located in different CO compartments are shown in various shades of gray. The linear model’s prediction of a bandwidth of 60 degrees is indicated by the arrow. B, Distribution of luminance, chromatically linear, and chromatically narrowly tuned cells in V2 and V3 samples. Narrowly tuned cells were those with a bandwidth of less than 45 degrees.

colored gratings. The horizontal dashed line shows the response to a blank screen, with a luminance equal to the space-averaged luminance of the other stimuli. The dashed curve in C shows the prediction of the non-linear model. For the cells shown in A, B, and D, the predictions of the nonlinear model were almost identical to those of the linear model and are therefore not shown.

luminance component of the stimuli alone (Gegenfurtner et al., 1994). Perhaps even more than for the topics discussed above, the question of color selectivity of V4 cells is muddled by the use of different methods and criteria. Zeki (1980) reported a narrow wavelength selectivity of all V4 cells, a finding that contributed to the classification of V4 as a color center of the primate brain. However, the interpretation of these results has been questioned on methodological grounds (de Monasterio and Schein, 1982; Schein et al., 1982). Schein and Desimone (1990) showed that most V4 neurons are not more narrowly tuned in their wavelength selectivity than retinal ganglion or LGN cells, but that a subpopulation of narrowly tuned neurons also exists in V4. This result is not surprising if one considers that V4 lies between V2, where narrowly tuned neurons are numerous, and IT cortex, where neurons with a narrow wavelength selectivity have also been reported (Komatsu, 1997). To our knowledge, no systematic study of the chromatic tuning of cells exists for any other cortical area. R  I S An important way to characterize the chromatic properties of cells is to study their responses to stimuli that vary only in chromaticity, so-called isoluminant stimuli. In V1, the majority of cells give stronger responses to stimuli that vary in luminance compared to purely chromatic modulations (Lennie et al., 1990), even

  :    

1021

1022

  

A

Preferred Orientation

360

Isoluminance

270

180

90

0 0

90

180

270

360

Luminance

B

Orientation index

1.5

1.0

Isoluminance

when the stimuli have been equated for cone contrast (Johnson et al., 2001). The same is true for areas V2 (Kiper et al., 1997) and V3 (Gegenfurtner et al., 1997). For the cells giving significant responses to isoluminant stimuli, it is of interest to determine the origin of the response and to compare the receptive properties derived from these stimuli to those obtained with luminance stimuli. For 20 V2 cells and 9 V3 cells that gave robust responses to isoluminant chromatic stimuli, Gegenfurtner and his colleagues (Gegenfurtner et al., 1997; Kiper et al., 1997) measured the cells’ tuning for orientation, spatial, and temporal frequency, as well as their contrast response function. Figure 66.5A shows a scatterplot of the optimal orientations determined using luminance and isoluminant chromatic stimuli. The correlation coefficient between these two measures was 0.98, and the mean difference in optimal orientation was 3.5 degrees. Similarly, selectivity for orientation as defined by an orientation index (Gegenfurtner et al., 1996) was not significantly different for the two types of stimuli (t27 = 1.18, p > .1) and was highly correlated (r = 0.68), as seen in Figure 66.5B. For 19 V2 cells the median optimal spatial frequency was 1.16 c/deg at isoluminance versus 1.03 c/deg for luminance stimuli; this small difference was not statistically significant (t18 = 1.47, p > .05), and the correlation between spatial frequency optima was high (r = 0.87). There was only a small difference in the optimal temporal frequency between the two stimulus types (2.25 Hz at isoluminance versus 2.69 Hz to luminance). There was no variation in the spatial or temporal bandwidths with stimulus type. Finally, there were no significant differences between the shape and steepness of the contrast response functions obtained with both types of stimuli. These results show a strong similarity of the tuning for luminance and isoluminant stimuli, but responses at isoluminance do not necessarily have to be due to coloropponent inputs. Previous research (Dobkins and Albright, 1994; Gegenfurtner et al., 1994) had shown that the isoluminant point can vary slightly from cell to cell, and this can lead to a luminance-based response to nominally, that is, photometrically, isoluminant stimuli. To establish the relative magnitude of the response at or near isoluminance, Gegenfurtner and colleagues tested V2 and V3 cells with a range of stimuli at different elevations around zero (isoluminance). They could thus detect response minima or response nulls even in cells that do not strictly adhere to photometric isoluminance. Two sets of stimuli were used: black and white achromatic gratings of increasing contrasts and heterochromatic gratings, which consisted of an isoluminant colored grating to which was added a black and white achromatic grating. The azimuths of the heterochromatic gratings were chosen to be the ones where the cells gave the best response. If a cell simply responds to the luminance component of the stimu-

0.5

0.0 0.0

0.5 Luminance

1.0

1.5

F 66.5. A, Scatterplot of the preferred orientation of 20 V2 cells ( filled symbols) and 9 V3 cells (open symbols) for chromatic versus achromatic gratings. Preferred orientations were derived from fits of a smooth function to the data. B, Scatterplot of the orientation indices for the same cells.

lus, its response will be the same to the achromatic and heterochromatic gratings of the same luminance contrast. Thus, when the heterochromatic grating is isoluminant, the response will be zero. This zero response, or point of isoluminance for each cell, frequently did not correspond to that predicted by the human photopic luminance sensitivity curve V(l). For these cells, a small amount of luminance contrast needs to be added or subtracted to obtain a zero response. Their response curves for chromatic and achromatic stimuli will be identical but shifted horizontally. On the other hand, cells that receive color-opponent inputs should behave in a different way. They should respond well to all color stimuli, regardless of their luminance contrast.

100

thick thin inter V2

100 75 50 V1 V2 V3 MT

25 0 -20

-10

0

10

20

Luminance contrast (%)

F 66.6. Averaged responses to stimuli with a fixed chromatic contrast but different luminance contrasts (see text). Each neuron’s baseline firing rate was subtracted from the responses. The responses are plotted relative to the response to the gratings with a luminance contrast of 20%. Averaged population responses from simple and complex cells in V1 and V2 cells (averaged across CO compartments), V3, and MT are indicated by different symbols.

These cells should not have a null response for any of the chromatic stimuli in this experiment. Since psychophysical experiments using isoluminant stimuli typically use photometrically isoluminant stimuli, it was of great interest to determine the overall response of cells for these particular stimuli. Cells were shown an isoluminant grating of about 10% root-mean-squared (RMS) cone contrast, and then luminance contrast of up to 20% was added to that grating. The population average response is shown for 319 V1 cells (Michael Hawken, personal communication), 33 V2 cells (Kiper et al., 1997), 71 V3 cells (Gegenfurtner et al., 1997), and 51 MT cells (Gegenfurtner et al., 1994). The surprising aspect of these data is the degree of similarity between the different areas. The response at isoluminance drops to about 30% of the maximal response in all areas (Fig. 66.6). The response does not go to zero. Even in magnocellularly dominated MT, there is a clear response. Moreover, the response is just as big as in V1 or V2, which share parvo- and magnocellular signals. Since all areas show a clear dip at isoluminance in the population average response, it is not surprising that many perceptual functions show a similar degradation at isoluminance. I  C  O V A In addition to the description of individual cells’ chromatic properties, several studies focused on the interactions between color signals and those coding other visual attributes. As described above, the notion that color signals in the cortex are strictly segregated from other attributes has been challenged in area V1 ( Johnson et al., 2001; Leventhal et al., 1995). However, the existence of separate functional processing streams (Ungerleider and Mishkin, 1982) and the discovery of anatomically distinct compartments (the so-called thin stripes, thick stripes, and interstripes, revealed by stain-

Proportion of cells

Response (%)

125

75

V3

50 25 0 Direction Orientation Color Stimulus attribute

Size

F 66.7. Proportion of neurons selective for the direction of motion, orientation, color, or size of a stimulus. As in Figure 66.4, different shades of gray indicate different CO compartments within V2. The proportion of cells in V2 taken as a whole is shown in black, that of V3 in white. The criteria used to classify the cells were identical for all samples.

ing the tissue for the metabolic enzyme cytochrome oxidase) within area V2 (De Yoe and Van Essen, 1985; Livingstone and Hubel, 1987) are still often taken as evidence for the segregation hypothesis in extrastriate visual areas. This question was addressed directly by Gegenfurtner et al. (1996, 1997), who studied the selectivity of V2 and V3 cells to stimulus orientation, direction of motion, size, and color and investigated their interactions. After having quantified the selectivity of single cells for each particular attribute, using indices similar to the color index described above, they first reported on the distribution of these indices within the different subcompartments of area V2. If there is a functional segregation of different visual functions, the tuning properties of neurons in the different V2 compartments should reflect that fact. Although there is no anatomical evidence for the segregation of specialized pathways through area V3, it is of similar interest to determine physiologically whether single V3 neurons are selective for particular stimulus attributes in a way that might mirror anatomical specificity. Alternatively, there might be no functional segregation in the area; single V3 neurons might be selective along several stimulus dimensions. Figure 66.7 shows the proportion of neurons selective for different stimulus attributes in each V2 compartment and in V3. There were some tendencies toward functional segregation: for example, color selectivity was most common in the thin stripes, size selectivity was most common in the interstripes, and orientation selectivity was somewhat less common in the thin stripes. However, despite the evidence for some degree of segregation, there was clearly no absolute segregation of selective sensitivity to color, form, or motion information into different pathways. Neurons showing selectivity to any attribute could be found in each of the compartments. Furthermore, these differences did not depend

  :    

1023

on the particular criteria used to classify cells, and they were essentially the same in all cortical layers (Gegenfurtner et al., 1996). Figure 66.7 also compares the overall proportion of neurons in areas V2 and V3 that are selective for each of these stimulus attributes, using identical classification criteria in each area. It is clear that the only notable difference between V2 and V3 is the greater incidence of direction selectivity in V3 (roughly 40% vs. 20% in V2). In both areas, approximately 85% of the population was orientation-selective, 25% size-selective (endstopped), and 50% color-selective. Another issue is the extent of integration and correlation among the response properties themselves; Gegenfurtner et al. (1996, 1997) investigated the relationships among the tuning characteristics of V2 and V3 neurons for different stimulus attributes. If different stimulus attributes were processed independently, one might expect neurons to show selectivity primarily to one stimulus attribute but not to several attributes simultaneously. Alternatively, it could be that each attribute has the whole range of other visual attributes associated with it. Figure 66.8 shows scatterplots of orientation, direction, and endstopping (i.e., size tuning) indices versus color responsivity for all V2 and V3 cells for which the investigators were able to measure these pairs of characteristics. The solid horizontal and vertical lines indicate the criterion values for classifying a cell as selective to that particular attribute. In both V2 and V3, there was no significant correlation between any of these selectivities. For example, in both areas there were cells that were highly selective for both stimulus color and orientation or for stimulus color and size. These are the cells that fall above and to the right of the criterion lines in Figure 66.8. Where V2 and V3 did seem to differ was in the association between color and direction selectivity. In V3, the investigators observed a population that was highly selective for both of these stimulus attributes; this group seemed to be absent from V2. To quantify the degree of interaction between different attributes, Gegenfurtner et al. used Fisher’s exact test for probabilities (see Hays, 1981, pp. 552–555) and confirmed that the probability with which a given neuron was colorselective did not depend on whether the cell was also selective for stimulus orientation, direction of motion, or size. In other words, these data do not support the hypothesis that the different stimulus attributes are processed in parallel in V2 and V3; rather, it seems that there are neurons tuned to any possible combination of attributes. The stimulus space spanned by color, orientation, direction, and size seems to be covered densely by the population of neurons in these areas. Although such a detailed analysis of the interactions between different visual attributes is not available for other extrastriate areas, several results suggest that the situation is quite similar in a number of other visual areas. In V4,

1024

  

Schein and Desimone (1990) reported that the selectivity of cells for orientation appears uncorrelated with their color properties. In that respect, V4 cells appear similar to the V1 and V2 populations, as supported by Yoshioka et al. (1996). In area MT, the few cells that code for color are also able to signal the direction of stimulus motion (Gegenfurtner et al., 1994), once more arguing against the idea that color signals are segregated from the others. C C The ability to perceive stable colors despite considerable changes in the illumination of a visual scene is considered a fundamental property of the human color vision system. The conditions necessary for efficient color constancy have been studied in numerous psychophysical studies and are still under scrutiny in many laboratories. Surprisingly, very little is known of the physiological basis of color constancy. The vast majority of the available neurophysiological data concerns area V4, and virtually nothing is known about the contribution of cells from other visual areas. The pioneering work of Zeki in the early 1980s (1983a, 1983b, 1983c) described two populations of color-selective cells in area V4. One population (wavelength, or WL cells) responded to colored stimuli in a way that could be predicted by the wavelength composition of the stimulus. A second population (color-coded, or CC cells) gave responses that could not be predicted by the wavelength composition of the stimulus but correlated with its color appearance, as defined by human observers. In other words, CC cells exhibit colorconstant responses; they code for the reflectance properties of an object irrespective of the illumination. Cells with this property seem absent in earlier stages of the visual pathways (Zeki, 1983a, 1983c), including area V2 (Moutoussis and Zeki, 2002), which provides a major input to V4. Unfortunately, most of these reports remain qualitative, and no estimate seems to be available about the frequency of WL and CC cells within V4 and the other cortical visual areas. The notion that V4 plays an important role in color constancy received additional support from lesion studies (Walsh et al., 1993; Wild et al., 1985) showing that V4 lesions result in severe color constancy deficits despite well-preserved color discrimination abilities. These and more recent results using fMRI in humans led Zeki and his collaborators (Zeki and Marini, 1998; Zeki, 1993) to propose that V4 is the centerpiece of the second stage of color processing, concerned primarily with color constancy operations, but without regard for memory or more cognitive aspects of perception. Note, however, that both the electrophysiological results of Zeki and the conclusions drawn from the lesion studies have been challenged on methodological grounds (Lennie and D’Zmura, 1988). In particular, when comparing the responses of V4 neurons to those of V1, Zeki did not scale the spatial dimensions of the Mondrian stimuli to the

A

thin

inter

B

thick

10

10

V2

V3

1

1 0.01

0.1

1

0.1

1

Direction index

D

Color responsivity

C 10

10

1

1 0

0.1

1

0

0.1

1

Endstopping index

F

E 10

10

1

1 0.1

1

0.1

1

Orientation index F 66.8. Association between different response selectivities for V2 (A, C, E) and V3 (B, D, F) cells. The scatterplots show the relations between color selectivity and direction selectivity (A, B), color versus size tuning (C, D), and color versus orientation selectivity (E, F). In A, C, and E, different symbols indicate the various CO compartments and the large symbols show the median values for each group of cells. In B, D, and F, the large symbols indicate the population mean.

neurons’ receptive field size. It is well known that at corresponding eccentricities, receptive fields are much larger in V4 than in V1. Thus, the observed differences between V1 and V4 responses may be partly due to the larger sample of the Mondrian available to V4 than to V1 neurons. Moreover, the chromatic adaptation of the receptive fields was not controlled during these experiments, opening the door for another confounding dimension in the data. Finally, it is not clear whether the performance of animals with V4 lesions reflects a deficit in color constancy, in color discrimination, or in their ability to encode the spatial properties of the stimuli.

Despite all the debate about where in the brain color constancy is achieved, the physiological mechanisms of how we achieve color constancy are little understood. It is widely agreed that the light adaptation properties of cone photoreceptors play an important role. The recent rediscovery of double-opponent cells in area V1 (see above) has opened the debate about the possible role of these cells in color constancy. A potential additional mechanism may lie in the inhibitory surround exhibited by V4 cells. Indeed, Schein and Desimone (1990) found a large, spectrally sensitive surround outside the classical receptive field (CRF) of most V4

  :    

1025

neurons. Although stimulation of the surround alone produced no responses, it could sometimes completely suppress the response to a stimulus in the CRF. For most V4 neurons, the surround’s preferred color matched that of the CRF. This led the authors to propose that V4 could compute color contrast, and that the interactions between center and surround may play an important role in color constancy. Because the role of V4 in color constancy is not firmly established, and because so little is known about the contribution of cells from other areas, it is fair to state that the physiological basis of color constancy is still largely unknown and awaits further research. I T  C C   B? The role of V4 as a color center of the primate brain is controversial. Early studies reported a very high incidence of color-selective cells in area V4, but later ones showed that this proportion is similar to that in other cortical areas of the ventral stream (Schein et al., 1982). As described above, the existence of V4 cells with a narrow tuning in color space is still debated, and so is the role of V4 cells in color constancy. For these reasons, many researchers question the qualification of V4 as the color center of the brain. In fact, some have proposed that other cortical areas play a more important role than V4 for color vision: studying monkeys with cortical lesions that spared V4, Heywood et al. (1995) suggested that it is lesions to the IT cortex, and not to V4, that mimic the human condition of cerebral achromatopsia (i.e., a specific loss of color vision). An analogous conclusion was reached by Hadjikhani et al. (1998), who, using fMRI in human subjects, described a color-selective area (which they called V8) anterior to the human equivalent of V4. These results were, however, challenged by Zeki et al. (1998), who argued that the “new” V8 is nothing but the human equivalent of V4, which had been described previously (McKeefry and Zeki, 1997). Other authors still (Gegenfurtner et al., 1996; Lennie, 1999) argue that color perception is not achieved in one particular cortical area, but that chromatic signals are treated, along with other visual attributes, in several cortical areas. The debate about the existence of a color center in the brain is ongoing, and further results are needed to resolve this issue.

Conclusions The perception of colors is a central component of primate vision. Color facilitates object perception and recognition, and plays an important role in scene segmentation and visual memory. Moreover, it provides an aesthetic component to visual experiences that is fundamental to our perception of the world. But despite its enormous importance and the long history of color vision studies, little is known about the physiological basis of color perception. The treatment of color signals in the retina and LGN is relatively well documented,

1026

  

but the exisiting data on cortical chromatic properties are scarce and often controversial. The summary picture that emerges from our review is that different extrastriate areas differ very little from each other or from area V1 with respect to their chromatic properties. There are two possible conclusions one can draw. The first is that most of the processing for color vision is done early, at the stages of the retinal ganglion cells and the geniculate. The other one is that we need to study color in the cortex more closely and in more detail. There are probably multiple reasons for the paucity of our knowledge on cortical processing of color. One is that some of the preferred models for the physiological investigation of the visual system involve animals with poor color vision, such as cats or ferrets. Another is that the color responses of individual neurons are often weak and, in extrastriate areas, might be severely degraded by anesthesia. Progress in investigating color vision physiology has probably also been slowed by the technical difficulty of producing and controlling suitable stimuli, and maybe even more by the impenetrable jargon used by color scientists! With the recent advances in display technology and brain imaging techniques, and increased communication between vision scientists from different disciplines, most of these obstacles should vanish, and new advances in our understanding of color perception should be made in the near future. REFERENCES Baizer, J. S., D. L. Robinson, and B. M. Dow, 1977. Visual responses of area 18 neurons in awake, behaving monkey, J. Neurophysiol., 40:1024–1037. Barberini, C. L., B. A. Wandell, and W. T. Newsome, 2001. S-cone inputs to area MT sum with L- and M-cone inputs, Soc. Neurosci. (Abstr.), vol 27 no 165. 11. Conway, B. R., 2001. Spatial structure of cone inputs to color cells in alert macaque primary visual cortex (V-1), J. Neurosci., 21(8):2768–2783. Cottaris, N. P., and R. L. De Valois, 1998. Temporal dynamics of chromatic tuning in macaque primary visual cortex, Nature, 395:896–900. Croner, L. J., and T. D. Albright, 1999. Segmentation by color influences responses of motion-sensitive neurons in the cortical middle temporal visual area, J. Neurosci., 19(10):3935– 3951. de Monasterio, F. M., and S. J. Schein, 1982. Spectral bandwidths of color-opponent cells of geniculostriate pathway of macaque monkeys, J. Neurophysiol., 47:214–224. Derrington, A. M., J. Krauskopf, and P. Lennie, 1984. Chromatic mechanisms in the lateral geniculate nucleus of macaque, J. Physiol., 357:241–265. Derrington, A. M., and P. Lennie, 1984. Spatial and temporal contrast sensitivities of neurones in the lateral geniculate nucleus of macaque, J. Physiol., 357:219–240. De Valois, R. L., 1965. Analysis and coding of color vision in the primate visual system, Cold Spring Harbor Symp. Quant. Biol., 30:567–579.

De Yoe, E. A., and D. C. Van Essen, 1985. Segregation of efferent connections and receptive field properties in visual area V2 of the macaque, Nature, 317:8–61. Dobkins, K., and T. D. Albright, 1994. What happens if it changes color when it moves? The nature of chromatic input to macaque visual area MT, J. Neurosci., 14(8):4854–4870. Dow, B. M., and P. Gouras, 1973. Color and spatial specificity of single units in rhesus monkey foveal striate cortex, J. Neurophysiol., 36:79–100. Engel, S. A., and C. S. Furmanski, 2001. Selective adaptation to color contrast in human primary visual cortex, J. Neurosci., 21(11):3949–3954. Engel, S. A., X. Zhang, and B. A. Wandell, 1997. Color tuning in human visual cortex measured using functional magnetic resonance imaging, Nature, 388:68–71. Felleman, D. J., and D. C. Van Essen, 1987. Receptive field properties of neurons in area V3 of macaque monkey extrastriate cortex, J. Neurophysiol., 57:889–920. Gegenfurtner, K. R., D. C. Kiper, J. Beusmans, M. Carandini, Q. Zaidi, and J. A. Movshon, 1994. Chromatic properties of neurons in macaque MT, Vis. Neurosci., 11:455–466. Gegenfurtner, K. R., D. C. Kiper, and S. B. Fenstemaker, 1996. Processing of color, form, and motion in macaque area V2, Vis. Neurosci., 13:161–172. Gegenfurtner, K. R., D. C. Kiper, and J. B. Levitt, 1997. Functional properties of neurons in macaque area V3, J. Neurophysiol., 77:1906–1923. Gouras, P., 1974. Opponent-colour cells in different layers of foveal striate cortex, J. Physiol., 199:533–547. Gross, C. G., C. E. Rocha-Miranda, and D. B. Bender, 1972. Visual properties of neurons in inferotemporal cortex of the macaque, J. Neurophysiol., 35:96–111. Hadjikhani, N., A. K. Liu, A. M. Dale, P. Cavanagh, and R. B. Tootell, 1998. Retinotopy and color sensitivity in human visual cortical area V8, Nat. Neurosci., 1(3):235–241. Hanazawa, A., H. Komatsu, and I. Murakami, 2000. Neural selectivity for hue and saturation of color in the primary visual cortex of the monkey, Eur. J. Neurosci., 12:1753–1763. Hays, W. L., 1981. Statistics, 3rd ed., New York: CBS College Publishing. Heywood, C. A., D. Gaffan, and A. Cowey, 1995. Cerebral achromatopsia in monkeys, Eur. J. Neurosci., 7(5):1064–1073. Hubel, D. H., and M. S. Livingstone, 1987. Segregation of form, color, and stereopsis in primate area 18, J. Neurosci., 4:309–356. Hubel, D. H., and T. N. Wiesel, 1968. Receptive fields and functional architecture of monkey striate cortex, J. Physiol., 195:215–243. Jacobs, G. H., 1993. The distribution and nature of color vision among the mammals, Biol. Rev., 68:413–471. Johnson, E. N., M. J. Hawken, and R. Shapley, 2001. The spatial transformation of color in the primary visual cortex of the macaque monkey, Nat. Neurosci., 4:409–416. Kiper, D. C., S. B. Fenstemaker, and K. R. Gegenfurtner, 1997. Chromatic properties of neurons in macaque area V2, Vis. Neurosci., 14:1061–1072. Kleinschmidt, A., B. B. Lee, M. Requart, and J. Frahm, 1996. Functional mapping of color processing by magnetic resonance imaging of responses to selective p- and m-pathway stimulation, Exp. Brain Res., 110(2):279–288. Komatsu, H., 1997. Neural representation of color in the inferior temporal cortex of the macaque monkey, in The Association Cortex—Structure and Function (H. Sakata, A. Mikami, J. M. Fuster, eds.), Amsterdam: Harwood Academic.

Komatsu, H., and Y. Ideura, 1993. Relationships between color, shape and pattern selectivities of neurons in the inferior temporal cortex of the monkey, J. Neurophysiol., 70(2):677–694. Komatsu, H., Y. Ideura, S. Kaji, and S. Yamane, 1992. Color selectivity of neurons in the inferotemporal cortex of the awake macaque monkey, J. Neurosci., 12(2):408–424. Lennie, P., 1999. Color coding in the cortex, in Color Vision: From Genes to Perception (K. R. Gegenfurtner and L. T. Sharpe, eds.), New York: Cambridge University Press. Lennie, P., and M. D’Zmura, 1988. Mechanisms of color vision, Crit. Rev. Neurobiol., 3:333–400. Lennie, P., J. Krauskopf, and G. Sclar, 1990. Chromatic mechanisms in striate cortex of macaque, J. Neurosci., 10:649–669. Lennie, P., M. J. M. Lankheet, and J. Krauskopf, 1994. Chromatically-selective habituation in monkey striate cortex, Invest. Ophthalmol. Vis. Sci. Suppl. 35, 1662. Leventhal, A. G., K. G. Thompson, D. Liu, Y. Zhou, and S. J. Ault, 1995. Concomitant sensitivity to orientation, direction, and color of cells in layers 2, 3, and 4 of monkey striate cortex, J. Neurosci., 15:1808–1818. Levitt, J. B., D. C. Kiper, and J. A. Movshon, 1994. Receptive fields and functional architecture of macaque V2, J. Neurophysiol., 71:2517–2542. Livingstone, M. S., and D. H. Hubel, 1984. Anatomy and physiology of a color system in the primate visual cortex, J. Neurosci., 4:309–356. Livingstone, M. S., and D. H. Hubel, 1987. Psychophysical evidence for separate channels for the perception of form, color, movement, and depth, J. Neurosci., 7:3416–3468. McKeefry, D., and S. Zeki, 1997. The position and topography of the human colour centre as revealed by functional magnetic resonance imaging, Brain, 120:2229–2242. Michael, C. R., 1978a. Color vision mechanisms in monkey striate cortex: dual-opponent cells with concentric receptive fields, J. Neurophysiol., 41:572–588. Michael, C. R., 1978b. Color vision mechanisms in monkey striate cortex: simple cells with dual opponent-color concentric receptive fields, J. Neurophysiol., 41:1233–1249. Michael, C. R., 1978c. Color-sensitive complex cells in monkey striate cortex, J. Neurophysiol., 41:1250–1266. Michael, C. R., 1979. Color-sensitive hypercomplex cells in monkey striate cortex, J. Neurophysiol., 42:726–744. Moutoussis, K., and S. Zeki, 2002. Responses of spectrally selective cells in macaque area V2 to wavelengths and colors, J. Neurophysiol., 87:2104–2112. Peterhans, E., and R. von der Heydt, 1993. Functional organization of area V2 in the alert macaque, Eur. J. Neurosci., 5:509–524. Roe, A. W., and D. Y. Ts’o, 1999. Specificity of color connectivity between primate V1 and V2, J. Neurophysiol., 82:2719–2730. Schein, S. J., and R. Desimone, 1990. Spectral properties of V4 neurons in the macaque, J. Neurosci., 10:3369–3389. Schein, S. J., R. T. Marrocco, and F. M. de Monasterio, 1982. Is there a high concentration of color-selective cells in area V4 of monkey visual cortex? J. Neurophysiol., 47:193–213. Seidemann, E., A. B. Poirson, B. A. Wandell, and W. T. Newsome, 1999. Color signals in area MT of the macaque monkey, Neuron, 24(4):911–917. Thiele, A., K. R. Dobkins, and T. D. Albright, 1999. The contribution of color to motion processing of motion in macaque middle temporal area, J. Neurosci., 19(15):6571–6587. Thorell, L. G., R. L. De Valois, and D. G. Albrecht, 1984. Spatial mapping of monkey V1 cells with pure color and luminance stimuli, Vis. Res., 24:751–769.

  :    

1027

Tootell, R. B., J. B. Reppas, K. K. Kwong, R. Malach, R. T. Born, T. J. Brady, B. R. Rosen, and J. W. Belliveau, 1995. Functional analysis of human MT and related visual cortical areas using magnetic resonance imaging, J. Neurosci., 15(4):3215–3230. Ts’o, D. Y., and C. D. Gilbert, 1988. The organization of chromatic and spatial interactions in the primate striate cortex, J. Neurosci., 8:1712–1727. Ungerleider, L. G., and M. Mishkin, 1982. Two cortical visual systems, in Analysis of Visual Behavior (D. J. Ingle, M. A. Goodale, and R. J. W. Mansfeld, eds.), Cambridge, MA: MIT Press. Walsh, V., D. Carden, S. R. Butler, and J. J. Kulikowski, 1993. The effects of V4 lesions on the visual abilities of macaques: hue discrimination and colour constancy, Behav. Brain. Res., 53:51–62. Wandell, B. A., A. B. Poirson, W. T. Newsome, H. A. Baseler, G. M. Boynton, A. Huk, S. Gandhi, and L. T. Sharpe, 1999. Color signals in human motion-selective cortex, Neuron, 24(4):901–909. Wild, H. M., S. R. Butler, D. Carden, and J. J. Kulikowski, 1985. Primate cortical area V4 important for colour constancy but not wavelength discrimination, Nature, 313:133–135. Yates, J. T., 1974. Chromatic information processing in the foveal projection (area striata) of unanesthetized primate, Vis. Res., 14:163–173.

1028

  

Yoshioka, T., B. M. Dow, and R. G. Vautin, 1996. Neuronal mechanisms of color categorization in areas V1, V2 and V4 of macaque monkey visual cortex, Behav. Brain Res., 76:51–70. Zeki, S., 1973. Color coding in rhesus monkey prestriate cortex, Brain Res., 53:422–427. Zeki, S., 1980. The representation of colours in the cerebral cortex, Nature, 284:412–418. Zeki, S., 1983a. Color coding in the cerebral cortex: the raction of cells in monkey visual cortex to wavelengths and colors, Neuroscience, 9:741–765. Zeki, S., 1983b. Color coding in the cerebral cortex: the responses of wavelength-selective and color-coded cells in monkey visual cortex to changes in wavelength composition, Neuroscience, 9:767–781. Zeki, S., 1983c. The distribution of wavelength and orientation selective cells in different areas of the monkey visual cortex, Proc. R. Soc. (Lond.), 217:449–470. Zeki, S., 1993. A Vision of the Brain. Cambridge, MA: Blackwell Scientific. Zeki, S., and L. Marini, 1998. Three cortical stages of colour processing in the human brain, Brain, 121:1669–1685. Zeki, S., D. J. McKeefry, A. Bartels, and R. S. J. Frackowiak, 1998. Has a new color area been discovered? Nat. Neurosci., 1(5):335.

67

Improbable Areas in Color Vision SEMIR ZEKI

C    exciting subject that lends itself easily to theorizing. It is especially interesting to do so in relation to the human cerebral cortex, given the huge advances made in the past decade in imaging human brain activity in health and disease. It is also interesting to do so because understanding how the brain constructs colors promises to give significant insights into the cerebral processes underlying aesthetics in the ancient Greek sense, that is, the acquisition of knowledge through the senses. It is not surprising to find, therefore, that color has traditionally attracted the attention of all those who have been concerned with perception and knowledge, including philosophers and physicists. But theorizing, to have any value, must be based on reliable facts and interpretations. How reliable are the facts derived from imaging experiments relating to color vision? And how compelling are the interpretations based on these results? This chapter deals with the quality of evidence that has been used to erect three cortical areas in the human brain— “VP,” “V4v,” and “KO”—all of them related directly or indirectly to color vision. How solid is the evidence for the existence of these areas, and how convincing is the interpretation of their functions? Let me emphasize that I am discussing the human brain, not the primate brain in general, though I shall refer, of course, to the evidence from monkeys from which some of the human areas are etymologically derived. Any conclusions reached about human visual areas must be derived from, and consistent with, evidence obtained from the human brain, without recourse to evidence from the monkey brain or from the brain of any other species. It is, of course, always good if the human evidence is supported by monkey evidence, but it should not have to rely on it. Too often in the recent past, there has been an attempt to hide behind a monkey in pleading for an interpretation, because that interpretation is not sustainable through human evidence alone. This chapter amounts, therefore, to an inquiry into the quality of evidence that we have come to accept in human imaging studies, a quality that I believe falls far short of what the instrumentation that we rely on is capable of delivering.

The location of the human cerebral color center and the visual field representation within it The notion that color may be the function of a specialized cortical area was hinted at several times before Louis Verrey

(1888) published his paper entitled Hémiachromatopsie droite absolue (absolute right hemiachromatopsia). Verrey’s advantage over his predecessors was that he had been able to examine the lesion that had led to the syndrome of acquired achromatopsia (cerebral color blindness). Apart from some involvement of the body of the corpus callosum, the lesion was confined to the fusiform and lingual gyri and was thus located in the inferior part of the occipital lobe (Fig. 67.1). Every word of Verrey’s title, together with the figure representing the lesion in the brain of his achromatopsic patient, is worth studying. Together they tell a great deal about the organization of the visual brain even today, though much of this is not mentioned by Verrey and is read by me into his evidence with hindsight. Foremost among the lessons to be learned is something about the topographic organization of the human visual brain. Given that the lesion producing cerebral hemi-achromatopsia was located in the lingual and fusiform gyri, the title implies that a center located in the lower part of the occipital lobe controls color vision in both the upper and lower parts of the entire contralateral visual hemifield. The discovery of a visual area lying outside the striate cortex was an embarrassment at the time (see Zeki, 1990, 1993, for reviews). Henschen and after him Holmes had concluded, correctly, that the lower part of the calcarine cortex (V1) represents the upper visual field and the upper part of the lower visual field. They had concluded, also correctly, that the calcarine cortex, which Henschen referred to as the “cortical retina” and Holmes as the “visuosensory” cortex, and which we now call the primary visual cortex or area V1, was coextensive with the striate cortex. This led both Henschen and Holmes to conclude, incorrectly, that V1 was the only visual center in the brain and therefore had to receive all visual “impressions,” including color impressions (see Zeki, 1993, for a review). From his single case of achromatopsia, Verrey had concluded, correctly, that there is a color center in the brain, but he had also supposed, incorrectly, that this is part of the “visuosensory” cortex. The implication was obvious: that the “visuosensory cortex” or “cortical retina” of the brain was larger than that supposed by Henschen and by Holmes, that is, it was not confined to the striate cortex. Verrey implied that the primary visual receptive cortex had a specialized subdivision dealing with color. Verrey’s conclusions about the cortical site for color processing (le centre du sense

1029

F 67.1. The sites of the lesions in the brain examined by Louis Verrey. (From Verrey, 1888.)

chromatique), together with his observation that both upper and lower contralateral quadrants were compromised in his unilaterally lesioned patient, implied that both quadrants are mapped in the lower part of the occipital lobe. Such an implicit supposition (which Verrey himself did not explicitly make) obviously cast a doubt on the way that Henschen and Holmes had supposed the visual field is mapped in the occipital lobe. Both Henschen and Holmes dealt with this in the same way, by brushing aside Verrey’s evidence or ignoring it altogether, until it vanished from the literature (for reviews see Zeki, 1990, 1993). There was a price to be paid for ignoring the significance of the finding that an area located in the lower occipital cortex controls color vision in both contralateral quadrants. That peril existed until well into the 1990s. Actually, it is probably with us even today. This is surprising because in 1980, Damasio alluded to it explicitly. He wrote: “one single area in each hemisphere controls color processing for the entire hemifield. This is so regardless of the fact that such an area is eccentrically located, in the lower visual association cortex, classically related to upper quadrant processing only. . . . The classic concept of a concentrically organized visual association cortex no longer appears tenable.” No one took much notice of what Damasio said then, and I don’t think that anyone takes much notice of it today either. There is probably a good explanation for this intellectual scotoma. With time, and with the discovery that “areas 18 and 19” (Brodmann, 1905; von Bonin and Bailey, 1951), or areas V2 and V3 (Cragg, 1969; Zeki, 1969), form concentric rings around V1 (Fig. 67.2), it became customary to consider that the upper part of the visual field is represented in the lower occipital lobe and vice versa. A new habit developed of tagging on the letter “v” or “d” to areas to indicate that they are located in the lower occipital lobe (“v”) and therefore represent upper fields or that they are located in the upper occipital lobe (“d”) and therefore represent lower visual fields. This terminology begged for confusion and

1030

  

F 67.2. Brodmann’s cytoarchitectonic map of the brain. Brodmann numbered the areas according to the sequence in which he studied them. Note how areas 18 and 19 form rings around area 17. (From Brodmann, 1909.)

trouble, which was not long in coming. It was based on an implicit assumption that had already been discounted, namely, that the upper contralateral quadrant is represented in the lower occipital lobe and vice versa. Verrey (1888) had shown that an area located in the lower occipital cortex must represent both contralateral quadrants (otherwise, how would one account for the complete contralateral hemiachromatopsia produced from a unilateral lesion in the lower occipital lobe?), and physiological evidence had shown that that an area located in the upper occipital lobe can represent both contralateral quadrants, as does area V3A (Van Essen and Zeki, 1978) or that it may be located somewhere between the two halves and still do the same, as does area MT in the owl monkey (Allman and Kaas, 1971). Indeed, Allman and Kaas had introduced the more neutral designations of + and - to indicate upper and lower field representation, respectively, without making either explicit or implicit assumptions about whether upper contralateral quadrants are represented only in the lower occipital lobe (Baker et al., 1981). It is unfortunate that their more sensi-

ble designations were not more widely used. If they had been, we might have avoided some of the present difficulties, though even that is not certain. At any rate, with implicit assumptions imposing themselves, if an area was found in the lower occipital cortex, “v” was hastily tagged on to it, irrespective of whether a dorsal counterpart could be found for it; if it was located in the upper occipital lobe, “d” was tagged on to it, again irrespective of whether a ventral counterpart could be found for it. It was but one step from this to describing what Jon Kaas (1993) has called “improbable” areas, ones that represent only one quadrant, with no representation for the other corresponding quadrant. But what kind of visual information is restricted to only one quadrant that it alone should have a cortical representation, without a companion area to represent the same information for the “unrepresented” quadrant? If the proponents of this bizarre terminology thought about it at all, they were not telling. A good example related to color vision is provided by area “VP.”

An improbable area: “VP” Area V2 of the primate brain surrounds area V1 and is, in turn, surrounded by area V3. The latter has a dorsal and a ventral subdivision, just like V2 (Fig. 67.3). Throughout its dorsal and ventral extent, the representation of the horizontal meridian forms a common boundary between it and V2, and the representation of the vertical meridian is at its anterior border (Cragg, 1969; Zeki, 1969). In 1986 a number of papers from Van Essen’s group, summarized in Burkhalter et al. (1986), confirmed the manner in which the visual field is mapped in V3 (see also Shipp et al., 1995). They nev-

F 67.3. A schematic diagram of prestriate cortex and its relation to striate cortex. Red lines show the horizontal meridian, green lines the vertical meridian. (Redrawn from Zeki, 1969.) (See color plate 45.)

ertheless reported that the lower part of V3 does not receive a direct input from V1, unlike the upper part of V3. They also reported that the lower part of V3 contains a high concentration of color cells, again unlike upper V3. This led Van Essen to propose that the lower part of V3 is not part of V3 at all, but a distinct area that they called “VP.” This made “VP” one of Kaas’ “improbable areas,” for it implied that something happening in upper quadrants (in this case color, inter alia) is processed there, without a machinery for processing that same attribute when it occurs in lower quadrants. Also unaccounted for was why, among the areas that process color, “VP” alone should process this attribute in upper visual fields only. Such an asymmetric representation is not characteristic of V1 or V2 or V4—all of them involved in color processing. Burkhalter et al. (1986) gave a halfhearted explanation. They wrote unconvincingly: “any asymmetries relating to V3 and VP might well be cancelled by compensatory asymmetries in other areas. Thus, any perceptual consequences might be rather subtle, but nonetheless would be worth testing for.” Yet the term “VP” has survived unquestioningly and has made an appearance in the human imaging literature with even less evidence. In fact, all the evidence from the human literature speaks against such an asymmetry. In the absence of any direct evidence in favor of a dichotomy between V3 and “VP” in the human, and evidence against it instead, there seems little doubt that the habit of naming the ventral part of human V3 “VP” is inherited from labeling in the macaque monkey. But how reliable is the macaque evidence? The negative anatomical evidence regarding the absence of a direct input from V1 to lower V3 in the macaque is not only doubtful but probably wrong. Such a projection has been found in Cebus (Rosa et al., 2000), and Lyon and Kaas (2001, 2002) have recently found the same not only in the marmoset monkey but also in the macaque monkey. It is instructive to compare the approach of Lyon and Kaas with the approach adopted by some in the imaging community. In their 2001 paper, Lyon and Kaas postulated that a V3 with upper and lower subdivisions would be characteristic of all primate visual brains. But they also tested the postulate by studying the macaque monkey (Lyon and Kaas, 2002). In the brisk world of human imaging experiments, this expensive and time-consuming luxury is seemingly not an option for everyone. Instead, what we have witnessed is the identification of a visual area in the human brain based on questionable evidence from the macaque. This is unfortunate. Given the real differences between species, the identification of human visual areas should be based on human measurements. The anatomical evidence in favor of the absence of a direct input from lower V1 to lower V3 in the monkey is therefore far from convincing, and indeed, all the evidence speaks against it. The physiological evidence that, unlike V3,

:     

1031

there is a high concentration of color cells in “VP” is even less convincing; indeed, all human imaging studies speak against it too. These studies have succeeded in showing that there is an area lying anterior to V2, with the map characteristic of V3, but not a single one of them, even those that use the term “VP” for lower V3, has succeeded in showing any specialization in “VP” for color or any specific activation of “VP” with color stimuli (e.g., De Yoe et al., 1996; Sereno et al., 1995; Zeki et al., 1991). Where the question has been specifically addressed, it has been found that lower V3 is not specifically activated by color (Wade et al., 2002). On the other hand, studies have found that “VP” is activated in the same way as V3 (e.g., Smith et al., 1998). In spite of this, many persist in this folie à plusieurs of calling the lower part of V3 “VP,” oblivious to the fact that to have an area representing one quadrant without having a representation for the other quadrant makes it improbable. Improbable, but also possible. But if the latter, it needs a convincing explanation, which no one has yet provided. Instead, the precedent has been used to describe other improbable areas, as we shall see. Thus, the evidence from the human brain does not support a separation into VP and V3, but rather tells of an area V3 with upper and lower subdivisions, representing lower and upper visual fields, the two subdivisions being activated in the same way. It is, of course, right to record that not all have accepted this division into V3 and “VP” unquestioningly (e.g., Wandell, 1999).

Another improbable area: “V4v” Our early imaging experiments showed that the color center in the human brain is located in the fusiform gyrus (Lueck et al., 1989; Zeki et al., 1991). It was called V4 without attaching a “v” or a “d” to it. At the time, this was not unreasonable. We had used full field stimulation and had no means of distinguishing upper from lower visual field representation within it. Unless one wanted to study its topography, which was not our aim, there was little need for us to do otherwise. We had read Verrey’s paper and remembered its title, which indicated that both upper and lower quadrants are mapped in the “color” center, located for him in the ventral part of the occipital lobe. This, we therefore supposed, was an area in which both quadrants are mapped, and there was little reason to add letters to its upper and lower parts. After all, neither we nor anyone else has bothered to add such letters to area V3A, located in the dorsal part of the occipital lobe, and in which upper and lower quadrants are separately mapped (Van Essen and Zeki, 1978; Zeki, 1978). Moreover, there was good reason to suppose that both upper and lower visual fields are separately mapped within the color center, because clinical evidence has shown more than once that the achromatopsia resulting from lesions there can be restricted to a quadrant.

1032

  

Then came the method of visual field mapping in human prestriate cortex using phase encoding (Engel et al., 1994). The early data were interpreted to reveal an area called “V4v” (De Yoe et al., 1996; Sereno et al., 1995), but at the time, no one seemed much bothered by the fact that they did not reveal its dorsal counterpart, “V4d,” or if they did, they did not communicate their worries. This was strange. How could a sophisticated mapping method that stimulates the entire visual field activate the ventral subdivision of an area but not its dorsal counterpart? The studies of De Yoe et al. suggested that “V4v” overlapped, at least in part, our V4. They wrote: “The location of V4v corresponds to some of the locations identified in positron emission tomography studies as having color selective responses. However there is sufficient variability to make it difficult to be certain that such responses could not have come from VP.” The study of Sereno et al., however, did not mention any overlap with the color center that we had defined. Adding “v” to V4 and calling the area thus defined “V4v” nevertheless implied that this was the ventral part of V4, in which the upper contralateral quadrant alone is mapped. No one seemed to worry much about such an asymmetric representation, at least not in print. Indeed, why should anyone have thought about it at all? There was, after all, the precedent of “VP” to go by. This was something of a puzzle. Could it be true that only upper visual fields are mapped in the part of the color center located in the fusiform gyrus, with lower visual fields mapped elsewhere, perhaps on the lateral surface of the occipital lobe? Though this seemed improbable, we decided to undertake another study in which we mapped the representation of the visual field using colored and achromatic Mondrians presented separately in the upper and lower hemifields. Compared to the sophistication of the phase encoding method, the approach that we used was hoary with age. But the results (McKeefry and Zeki, 1997) showed what we had suspected, and what the clinical evidence from Verrey on had implied, to us at least—that both upper and lower quadrants are separately mapped, side by side, within the color center and that therefore a center located in the lower occipital lobe is indeed responsible for the elaboration of color in both the upper and lower visual fields. This accounted for why lesions in the color center of one hemisphere can lead to a hemi-achromatopsia or even to an achromatopsia restricted to one quadrant (see Meadows, 1974, and Zeki, 1990, for reviews).

Old wine in new bottles The repetition of our experiment, using phase encoding, and the confirmation of our results by Hadjikhani et al. (1998), represents perhaps one of the most surprising events in the history of mapping, not for the results obtained, which

were in fact identical to ours, but for the way in which these results were presented and for what has been read into them. They claimed to have found “a new retinotopic area that we call ‘V8,’ which includes a distinct representation of fovea and both upper and lower visual fields.” This previously undifferentiated cortical area “was consistently located just beyond the most anterior retinotopic area defined previously, area V4v” (emphasis of the “v” added). These claims gained added weight from an accompanying article by Heywood and Cowey (1998) which declared uncritically: “it is area V8, not the favorite candidate V4” that, when lesioned, produces cortical color blindness. Heywood and Cowey claimed that these results show that “the human color center is distinct from area V4. The newlydefined [sio] color area contains a complete retinotopic map of the contralateral visual half field, responds more robustly to color and, unlike V4, is activated by induction of color after effects” (emphasis added), which would seem to leave out of account the earlier discoveries of Sakai et al. (1995) on activation of the color center through color aftereffects.

The many uses of “v” Anyone reading these articles casually may be forgiven for supposing that a new color area, distinct from what we had called human V4, had been found. But how “new” was this area? A slightly more careful reading of the results in Hadjikhani et al. (1998) shows that this “new” area is nothing more than a rediscovery of what we had defined, dressed up in a new name. Hadjikhani et al. wrote: “Based on the anatomical location and functional comparison used here, this collateral [sulcus] color selective patch appears equivalent to the previously reported [Lueck et al., 1998; Zeki et al., 1991] area involved in achromatopsia.” In fact, a comparison of their Talairach coordinates with those given by us in previous publications shows that the “new” color-

selective area, “V8,” is identical in position to the location of our human V4 (Fig. 67.4). Moreover, Hadjikhani et al. found that “When we averaged the Talairach coordinates of the color-selective area ‘V4’ described in previous studies . . . we found that it was about twice as close to the location of our retinotopically defined V8, compared to our retinotopically defined V4v” (emphasis on “v” added). But here comes the catch. Hadjikhani et al. write, in the very next sentence, “This supports all the other evidence that the color selective activity is located in area V8, rather than in ‘V4,’ ” which, presumably, is why they say that they had discovered a “previously undifferentiated” area. To any moderately careful reader, the first part of this sentence says that V8 is almost identical to V4, while the second part says that V8 is distinct and lies anterior to it. How is this feat achieved? Very simply, by dropping the v at the end of the sentence! This is no eristic quibble. The statement would have been unexceptionable if they had written “color selective activity is located in area V8, rather than in V4v” but, then, the claim of a “new,” “previously undifferentiated” visual area would have been difficult to sustain. The omission of the “v,” together with claims of a “new,” “previously undifferentiated” cortical area being discovered, have apparently convinced some of the innocent who work, or comment, on color vision that something new has indeed been discovered. Tootell and Hadjikhani (2001) have since written: “The Talairach coordinates of the original ventral color-selective region (“V8” or “V4” or “VO”) were never in dispute, although this has been a matter of apparent confusion.” The source of the confusion is not hard to trace. It lies in the claim that a “new,” “previously undifferentiated” color area located anterior to V4 has been discovered (Hadjikhani et al., 1998). The confused include even the experts, as witness the article by Heywood and Cowey (1998) about the “newly defined” color area and about the color center being “area V8, not the favorite candidate V4.”

F 67.4. A glass projection of the brain showing three areas discussed in the text. The areas were located by using the Talairach coordinates of the three areas given in the paper by Hadjikhani et al. (1998). O corresponds to area V4 defined in Lueck et al. (1989; Zeki et al., 1991; and McKeefry and Zeki, 1997); X corresponds to the “new” area “V8” of Hadjikhani et al. (1998) and the + to area V4v defined by Sereno et al. (1995). (See color plate 46.)

:     

1033

Some of the comments made about the discovery of a “new,” “previously undifferentiated” color center are worth noting, for they speak volumes about the care with which papers are read, at least in the color vision and imaging communities. I restrict myself to one here, which says: “There is currently a heated debate about the location of the color center in the visual cortex” (Gegenfurtner, 2001). Actually, there isn’t, and there never was. The debate is about whether a new area has been discovered, and the admission of Tootell and Hadjikhani (2001) that it has not settles the issue. There is, of course, an additional debate not alluded to in the above quotation, which is about whether there is an area V4v, distinct from V4 (see below).

Cross-species comparisons In trying to make claims for “V8” as being color selective as against V4 (although they are the same area), Tootell and his colleagues have taken refuge in a long debate about whether macaque V4 has any color selectivity. The argument is along two lines. The first one runs like this: “Zeki believed that monkey V4 is color selective. This is not so, and therefore the color-selective region in human prestriate cortex is not V4 but V8.” That debate, to which I shall return elsewhere, is irrelevant to the issue of whether a “new” color center has been discovered in the human brain, distinct from what was previously described by us. But it also helps to deflect attention from another problem, namely, the status of human “V4v.” The line of argument here is: “Macaque V4, as defined by Zeki, is the fourth visual map after V1. But in human there is a color unselective ‘V4v’ lying anterior to V3 and posterior to the color center called V4 by Zeki and his colleagues and recently rediscovered by us. It is therefore inappropriate to call this rediscovered area V4.” This argument is also irrelevant to whether a “new” color center has been discovered. But it raises an interesting question. Does human “V4v” exist, as an entirely separate and improbable area, which represents upper visual field only? There is no compelling evidence in its favor, which is not to say that there may not be some day. Tried though we have, we have found no evidence for an area “V4v” that is separate from area V4 as we have defined it, or from the same area rediscovered by Hadjikhani et al. (1998). The most extensive and detailed retinotopic mapping experiments in the human visual brain to date have been done recently by Wandell and his colleagues at Stanford (Press et al., 2001; Wade et al., 2002). Their results show convincingly that there is no quarter field map corresponding to a putative “V4v” in ventral occipital cortex. There is thus no area “V4v,” which, given its improbability, is just as well. Their results also show that V4 is the fourth visual map and that it abuts lower V3. Thus, through this work, one of the arguments outlined in the above paragraph is emascu-

1034

  

lated and loses its force. In sum, there is no current compelling evidence for the existence of area “V4v,” and the onus is on its proponents to demonstrate it convincingly and unequivocally. If they cannot do so, they should withdraw it. What is surprising, given the flimsy evidence for the existence of “V4v,” is the numerous papers that, in addressing other aspects of human visual brain organization, refer to it, thus leaving one with doubts about the quality of evidence that is deemed acceptable in the imaging community. It is hard to imagine that the more hard-nosed physiologists would ever have accepted the existence of a “V4v” based on the kind of evidence that is currently available. There is a price to be paid for such uncritical acceptance of the evidence. An example is to be found in a paper by one of the coauthors of the Hadjikhani et al. (1998) paper. Cavanagh et al. (1998), in trying to account for why achromatopsic patients can see moving color stimuli, state, “In humans, the most recent fMRI studies [of Hadjikhani et al.] suggest a more anterior site, V8, for color analysis in a location consistent with the damage in the achromatopsic patients. The human homologue to V4 on the ventral surface (V4v) is probably also damaged in these patients but it includes only a representation of the upper visual fields.” But since V8 is nothing more than human V4 renamed, since Wade et al. (2002) have shown convincingly that there is no quarter field representation corresponding to the “V4v” of Hadjikahni et al. (1998), and since, therefore, the existence of “V4v” is in considerable doubt, the speculation is worse than useless. Such is the price that we pay for having, in the human imaging community, standards of evidence and of proof that fall well short of what is expected, and delivered, in other branches of neurobiology.

The human color center The color center turns out to be more complex than was previously thought (Fig. 67.5). In our work, we have frequently referred to the V4 complex in both monkey and human, and in our most recent work we have described this complex in the human as comprising at least two subdivisions, which we refer to as V4 and V4a (Bartels and Zeki, 2000). V4 itself is topographically organized (Hadjikhani et al., 1998; McKeefry and Zeki, 1997; Wade et al., 2002), while V4a is not obviously so, which is not to say that more sophisticated mapping techniques may not reveal some topographic mapping within it in the future (Bartels and Zeki, 2000). It is important to realize that V4a lies anterior to V4 and therefore anterior also to the “newly discovered” color center of Hadjikhani et al. (1998). It is a part of the color center that they missed in their studies. The topographic organization of V4 accounts well for why the achromatopsia induced by lesions in the fusiform gyrus may involve the whole of the contralateral hemifield

F 67.5. The segregation of the color-selective region in the fusiform gyrus (the V4 complex) into two areas, the posterior retinotopically organized area V4 and the anterior area V4a, as revealed by the reanalysis of the V4 mapping study (McKeefry and Zeki, 1997). a, left: Statistical parametric map (SPM) viewed in glass-brain projections of the comparison of all chromatic stimuli versus their achromatic counterparts for both upper and lower visual field stimulation (group of four subjects; threshold: Z > 4.81, p < .05, corrected for multiple comparisons, equivalent to p < .000001 uncorrected). Right: Slices taken through an SPM of a single subject superimposed on its structural image (slices at x: -33, z: -14 mm). b, Projection of the comparison of either upper field (in red) or lower field (in green) stimulation with color versus their achromatic counterparts onto a ventral view of a human brain

(overlapping regions are shown in yellow). For V4 (bottom), the SPM of the following comparison is projected onto the drawing: (superior colored vs. [superior achromatic + inferior colored + inferior achromatic]); (group of four subjects; threshold: Z = 4.81, p < .05 corrected). For V4a (top), SPMs of a comparison of color versus achromatic stimuli within the corresponding hemifield is projected onto the drawing (threshold: Z = 3.09, p < .001 uncorrected). c, An independent component analysis (ICA) separates spatially independent maps of brain activity without a priori knowledge about the stimulus conditions. ICA isolated the complete V4 complex, including the posterior (V4, bottom) and anterior (V4a) subdivisions in both hemispheres, shown here in the glass-brain view of a single subjects’ brain. (From Bartels and Zeki, 2000.) (See color plate 47.)

(as in the patient of Verrey, 1888) or may be limited to a quadrant. Less certainly, the subdivisions of the color center into V4 and V4a may eventually help to account for what is problematic with the syndrome of cerebral achromatopsia. In some patients, the achromatopsia is transient, whereas in others it is more long-lasting and even permanent. Moreover, in some patients the achromatopsia is less severe than in others and can be described as a dyschromatopsia. In such patients, the color loss is not complete, being greater for some colors (usually the blues and the greens) than for others (see Zeki, 1990, for a review). Moreover, the color vision of some dyschromatopsic patients is very much wavelength based, in that they are not able to compensate for the predominance of one wavelength over another when the wavelength composition of the viewing conditions changes (Kennard et al., 1995). It is possible, but yet to be shown conclusively, that these variations depend upon the extent to which the lesions in the fusiform gyrus involve both subdi-

visions of the human color center (see Bartels and Zeki, 2000, for a review). Of course, there are those who believe that the whole notion of a cortical specialization for color vision is a fantasy and the search for it an outmoded folly. How convincing are their arguments? Here is one of the most forceful. Lennie (1999) writes: “The controls used in most functional imaging studies have involved comparing activity evoked by the presentation of an array of colored surfaces with the activity evoked by the presentation of the same elements at the same luminance, but now set to be gray. Visibility and salience can be quite different. Overall, the evidence for a specialized color pathway is not strong.” Consider this statement carefully. It says essentially that the color center may be nothing more than a salience center. But color does make the visual world more salient. And so there is a color center after all, even after this denial, except that in this instance we call it the “salience center”! If there are good arguments against a

:     

1035

color center in the human brain, why resort to such wholly unconvincing arguments?

An improbable area in the making: “KO” We are currently witnessing the making of another improbable area. It is known as the “kinetic occipital” area or area “KO,” for which an alternative and, to me, more acceptable name is area V3B (Press et al., 2001; Smith et al., 1998). Area “KO” has been described as not only being “specialized in the processing of kinetic contours” but “genuinely specialized” in this task (Van Oostend et al., 1997). The acronym given it by Orban and his group reflects this belief (Dupont et al., 1997; Orban et al., 1995; Van Oostend et al., 1997). Its relation to color vision lies in the current attempts to homologize it with macaque V4 or, more accurately, with the dorsal part of macaque V4, dubbed “V4d” (Tootell and Hadjikhani, 2001), to which I return below. This would naturally make “KO” another improbable area, since all the evidence shows that cells in the upper part of V4 (“V4d”) have their receptive fields in the lower contralateral quadrant (Pinon et al., 1998; Van Essen and Zeki, 1978). This, in turn, would imply that V4d processes kinetic contours in the contralateral lower field only, with no equivalent area for processing the same signals when they occur in the upper contralateral visual field, or at least no declared area. Before going into such speculations, it is important to consider the evidence that “KO” is “genuinely . . . specialized in the processing of kinetic contours” (Van Oostend et al., 1997). The idea that there are brain areas specialized for the processing of forms derived from motion was actually first posited by Gulyas et al. (1994), but it was based on a faulty analysis of their imaging data which has attracted criticism (Frackowiak et al., 1996) and is therefore questionable. How reliable is the evidence of Orban’s group? In their work, Orban’s group (Dupont et al., 1997; Orban et al., 1995; Van Oostend et al., 1997) compared the activity produced in “KO” when humans viewed bars produced from kinetic contours, from luminance differences, and from translational motion. They found that several areas are activated in these comparisons, including V3A, V5, and “KO,” though the last was more active. This led them to conclude that the brain devotes a special area to the processing of kinetic contours. But there is a critical omission in these comparisons. What if “KO” is specialized for the extraction of shapes, no matter how derived, rather than the extraction of shapes from kinetic contours specifically? A critical comparison to distinguish between the two possibilities would be to study the activity in “KO” when contours are extracted from other attributes, for example, equiluminant colors. Color is generally agreed to be the most separate in its cortical representation from motion, even if there is much anatomical opportunity for cross-talk between the two systems and even

1036

  

if such cross-talk does occur (Callaway, 1998). If “KO” is inactive when humans view visual stimuli in which simple shapes are extracted from equiluminant colors, then a fairly strong, though perhaps not definitive, case can be made for “KO” ’s putative specialization. But if it is equally active when contours are extracted from equiluminant colors, as it is when the same contours are extracted from kinetic contours, then “KO” can be safely considered not to be specialized for the processing of kinetic contours. Our own studies show that area V3B, a better name for “KO,” is in fact engaged when shapes are extracted, regardless of their provenance. This is not the place to go into this evidence, for what I am emphasizing is that claims for the specialization of “KO” have been made in the absence of critical comparisons which would establish whether it is specialized in the extraction of shapes from kinetic stimuli. At present, the evidence relating to the “genuine specialization” of “KO” for processing kinetic contours is so incomplete that it should raise considerable doubts in the neuroimaging community. Such evidence may, of course, be forthcoming in the future, but it is not available today. On the other hand, there is considerable evidence for “cue invariance” in the visual brain, by which is meant that cells in different areas of the brain respond to their preferred stimuli no matter how these stimuli are derived, that is, whether from luminance differences, kinetic boundaries, or equiluminant colors (Albright, 1992; Geesman and Andersen, 1996; GrillSpector et al., 1998; Sary et al., 1993). To overwrite this evidence, it is not sufficient to keep repeating that “KO” is specialized for the processing of kinetic contours, as if every utterance establishes its role more solidly. What is needed is a solid experiment to establish its specialty. Such evidence is not currently available.

Precedence replaces evidence The doubtful status of “KO” as an area that is specialized in the processing of kinetic contours is rendered more emphatic by the current attempts to make human “KO” the homolog of the dorsal part of macaque V4, or V4d (Tootell and Hadjikhani, 2001; see also the two contradictory abstracts by Orban’s group—Fize et al., 2001, and Vanduffel et al., 2001), based on the argument that “KO” is the “topolog” of V4d. In replicating the findings of Orban’s group, Tootell and Hadjikhani (2001) believe that they have confirmed that “KO” is indeed specialized for kinetic contours. But they used the same stimuli and paradigm as Orban’s group and in essence replicated Orban’s study. So complete was this repetition that the critical omission, the one that casts doubt on the conclusions of Orban’s group, was not remedied at all. The critical comparison, to determine whether “KO” is also activated as well by shapes derived from other attributes, and especially from equilumi-

nant colors, was not done. The evidence of Tootell and Hadjikhani (2001) therefore adds nothing to the unconvincing evidence of Orban’s group. The issue is further compounded, and actually compromises the status of “KO” further, by the suggestion that it is the homolog of V4d. If so, then “KO” must be registering events occurring in the contralateral inferior quadrant only, since cells in V4d have their receptive fields there (Pinon et al., 1998; Van Essen and Zeki, 1978). This would make “KO” an area that is specialized for the processing of kinetic contours when these occur in the lower contralateral quadrant only, leaving the same activity occurring in the upper contralateral quadrant unrepresented in the cortex, or represented in another area, which would make that latter, yet to be discovered area, another improbable one. In fact, the evidence of Press et al. (2001) suggests otherwise, in showing that both upper and lower contralateral quadrants are mapped in V3B, the better name for “KO,” a fact that appears to have been ignored in the headlong rush to homologize “KO” with V4d, since the latter represents lower contralateral quadrants only. Thus, a circle of confusion, consisting of increasing numbers of improbable areas, follows from the initial assumption that there is an improbable area, “VP.” The best way of breaking that confusion is to get rid of the assumption, or at any rate to test it before accepting it. Where the assumption has been tested, it has not found to be true. It is, of course, possible that this is exactly the way that the cortex is specialized and that, in thinking of areas that represent one quadrant only, Jon Kaas, and I after him, are being extremely unsophisticated in our approach to cortical mapping. Yet one cannot help feeling that the evidence in favor of these improbable areas is very weak at present. The danger is that once an improbable area is accepted on the basis of such weak evidence, one does not even have to produce strong evidence in favor of new improbable areas. Nor is this a hypothetical danger. In justifying their homology of “KO” with V4d, Tootell and Hadjikhani (2001) write: “Although such ‘separated’ quarter-field representations are conceptually unsatisfying, they are not unprecedented: the quarter-field representations in macaque ‘V3’ and ‘VP’ have long been considered separate areas by some investigators, based on empirical differences between V3 and VP” (Burkhalter et al., 1986; Felleman and Van Essen, 1991; Felleman et al., 1997; Van Essen et al., 1986). Yes, by some but not by all (see above). And even if it is accepted by all, the present evidence in favor of areas with just a quarter field representation is still not compelling. To date, at least two (“VP” and “V4v”) have already been convincingly knocked out in both the human and the macaque brains (Lyon and Kaas, 2001, 2002; Wade et al., 2002). This does not auger too well for those who want to use the precedent of dubious quarter field representations to hastily erect new ones.

Conclusion I have already reviewed extensively the early literature on the cortical involvement in color vision, and have tried to show to what extent it was dominated by preconceived notions (Zeki, 1990, 1993). Looking at what has happened since then, it seems that the considerable technological sophistication that is at our disposal for studying the human brain has not been matched by an equal sophistication in thinking about the brain. Where something is improbable, the evidence in its favor should be impeccable. The evidence of the past few years shows, instead, that in studies of cortical color vision, the improbable has imperceptibly become the acceptable and set the precedent for other improbables to become acceptable without the intervention of questioning. The story of color vision as it relates to the cortex is thus a very sad one. In neurobiology it is indeed, to quote the title of Ford Maddox Ford’s book, The Saddest Story. There are, of course, also good things that have happened in our understanding of the role of the cortex in color vision. I will not relate these, but leave it to the commentators on discoveries in cortical color vision to do so.

Acknowledgment The work of this laboratory is supported by the Wellcome Trust, London. This review was concluded in January 2002 and has not been revised since. REFERENCES Albright, T. D., 1992. Form-cue invariant motion processing in primate visual cortex, Science, 255:1141–1143. Allman, J. M., and J. H. Kaas, 1971. A representation of the visual field in the caudal third of the middle temporal gyrus of the owl monkey (Aotus trivirgatus), Brain Res., 31:85–105. Baker, J. F., S. E. Petersen, W. T. Newsome, and J. M. Allman, 1981. Visual response properties of neurons in four extrastriate areas of the owl monkey (Aotus trivirgatus): a quantitative comparison of medial, dorsomedial, dorsolateral, and middle temporal areas, J.. Neurophysiol., 45:397–416. Bartels, A., and S. Zeki, 2000. The architecture of the colour centre in the human visual brain: new results and a review, Eur. J. Neurosci., 12:172–193. Bonin, G. von, and P. Bailey, 1951. The Isocortex of Man. Urbana: University of Illonois Press. Brodmann, K., 1905. Beitrage zur histologischen Lokalisation der Grosshirnrinde. Dritte Mitteilung: Die Rindenfelder der niederen Affen, J. Psychol. Neurol. Lpz., 4:177–276. Brodmann, K., 1909. Vergleichende Lokalisationslehre der Grosshirnrinde in ihren Prinzipien dargestellt auf Grund des Zellenbaues, Leipzig: J. A. Barth. Burkhalter, A., D. J. Felleman, W. T. Newsome, and D. C. Van Essen, 1986. Anatomical and physiological asymmetries related to visual areas V3 and VP in macaque extrastriate cortex, Vis. Res., 26:63–80.

:     

1037

Callaway, E. M., 1998. Local circuits in primary visual cortex of the macaque monkey, Annu. Rev. Neurosci., 21:47–74. Cavanagh, P., M. A. Henaff, F. Michel, T. Landis, T. Troscianko, and J. Intriligator, 1998. Complete sparing of high contrast color input to motion perception in cortical color blindess, Nat. Neurosci., 1:242–247. Cragg, B. G., 1969. The topography of the afferent projections in the circumstriate visual cortex of the monkey studied by the Nauta method, Vis. Res., 9:733–747. Damasio, A., T. Yamada, H. Damasio, J. Corbett, and J. McKee, 1980. Central achromatopsia: behavioural, anatomic, and physiologic aspects, Neurology, 30:1064–1071. De Yoe, E. A., G. J. Carman, P. Bandettini, S. Glickman, J. Wieser, R. Cox, D. Miller, and J. Neitz, 1996. Mapping striate and extrastriate visual areas in human cerebral cortex, Proc. Natl. Acad. Sci. USA, 93:2382–2386. Dupont, P., B. DeBruyn, R. Vandenberghe, A. M. Rosier, I. Michiels, G. Marchal, L. Mortelmans, and G. A. Orban, 1997. The kinetic occipital region in human visual cortex, Cereb. Cortex, 7:283–292. Engel, S. A., D. E. Rumelhart, B. A. Wandell, A. T. Lee, G. H. Glover, E. J. Chichilnisky, and M. N. Shaden, 1994. fMRI of human visual cortex, Nature, 369:525. Felleman, D. J., A. Burkhalter, and D. C. Van Essen, 1997. Cortical connections of areas V3 and VP of macaque monkey extrastriate visual cortex, J. Comp. Neurol., 379:21–47. Felleman, D. J., and D. C. Van Essen, 1991. Distributed hierarchical processing in the primate cerebral cortex, Cereb. Cortex, 1:1–47. Fize, D., W. Vanuffel, K. Nelissen, P. Van Hecke, J. B. Mandeville, R. B. H. Tootell, and G. A. Orban, 2001. Distributed processing of kinetic boundaries in monkeys investigated using fMRI, Soc. Neurosci. Abstr., 11.9. Frackowiak, R. S. J., S. Zeki, J. B. Poline, and K. J. Friston, 1996. A critique of a new analysis proposed for functional neuroimaging, Eur. J. Neurosci., 8:2229–2231. Geesaman, B. J., and R. A. Andersen, 1996. The analysis of complex motion patterns by form/cue invariant MSTd neurons, J. Neurosci., 16:4716–4732. Gegenfurtner, K., 2001. Color in the cortex revisited, Nat. Neurosci., 4:339–340. Grill-Spector, K., T. Kushnir, S. Edelman, Y. Itzhak, and R. Malach, 1998. Cue invariant activation in object-related areas of the human occipital lobe, Neuron, 21:191–202. Gulyas, B., C. A. Heywood, D. A. Popplewell, P. E. Roland, and A. Cowey, 1994. Visual form discrimination from color or motion cues—functional—anatomy by positron emission tomography, Proc. Natl. Acad. Sci. USA, 91:9965–9969. Hadjikhani, N., A. K. Liu, A. M. Dale, P. Cavanagh, and R. B. H. Tootell, 1998. Retinotopy and color sensitivity in human visual cortical area V8, Nat. Neurosci., 1:235–241. Heywood, C., and A. Cowey, 1998. With color in mind, Nat. Neurosci., 1:171–173. Kaas, J. H., 1993. The organization of the visual cortex in primates: problems, conclusions and the use of comparative studies in understanding the human brain, in The Functional Organization of the Human Visual Cortex (B. Gulyas, D. Ottoson, and P. E. Roland, eds.), Oxford: Pergamon Press, pp. 1–11. Kennard, C., M. Lawden, A. B. Morland, and K. H. Ruddock, 1995. Colour identification and colour constancy are impaired in a patient with incomplete achromatopsia associated with prestriate cortical lesions, Proc. R. Soc. Lond., 260:169– 175.

1038

  

Lennie, P., 1999. Color coding in the cortex, in Color Vision (K. R. Gegenfurtner and L. T. Sharpe, eds.), Cambridge: Cambridge University Press, pp. 235–247. Lueck, C. J., S. Zeki, K. J. Friston, M. P. Deiber, P. Cope, et al., 1989. The colour centre in the cerebral cortex of man, Nature, 340:386–389. Lyon, D. C., and J. H. Kaas, 2001. Connectional and architectonic evidence for dorsal and ventral V3, and dorsomedial area in marmoset monkeys, J. Neurophysiol., 21:249–261. Lyon, D. C., and J. H. Kaas, 2002. Evidence for a modified V3 with dorsal and ventral halves in macaque monkeys, Neuron, 33:453–461. McKeefry, D., and S. Zeki, 1997. The position and topography of the human colour centre as revealed by functional magnetic resonance imaging, Brain, 120:2229–2242. Meadows, J. C., 1974. Disturbed perception of colours associated with localized cerebral lesions, Brain, 97:615–632. Orban, G. A., P. Dupont, B. DeBruyn, R. Vogels, R. Vandenberghe, and L. Mortelmans, 1995. A motion area in human visual cortex, Proc. Natl. Acad. Sci. USA, 92:993–997. Pinon, M. C., R. Gattass, and A. P. B. Sousa, 1998. Area V4 in Cebus monkey: extent and visuo-topic organization, Cereb. Cortex, 8:685–701. Press, W. A., A. A. Brewer, R. F. Dougherty, A. R. Wade, and B. A. Wandell, 2001. Visual areas and spatial summation in human visual cortex, Vis. Res., 41:1321–1332. Rosa, M. P. G., M. C. Pinon, R. Gattass, and A. P. B. Sousa, 2000. “Third tier” ventral extrastriate cortex in New World monkey, Cebus apella, Exp. Brain Res., 132:287–305. Sakai, K., E. Watanabe, Y. Onodera, I. Uchida, H. Kato, et al., 1995. Functional mapping of the human colour centre with echo-planar magnetic resonance imaging, Proc. R. Soc. Lond. B, 261:89–98. Sary, G., R. Vogels, and G. A. Orban, 1993. Cue-invariant shape selectivity of macaque inferior temporal neurons, Science, 260:995–997. Sereno, M. I., A. M. Dale, J. B. Reppas, K. K. Kwong, J. W. Belliveau, et al., 1995. Borders of multiple visual areas in humans revealed by functional magnetic resonance imaging, Science, 268:889–893. Shipp, S., J. D. G. Watson, R. S. J. Frackowiak, and S. Zeki, 1995. Retintotopic maps in human prestriate visual cortex: the demarcation of areas V2 and V3, Neuroimage, 2:125–132. Smith, A. T., M. W. Greenlee, K. D. Singh, F. M. Kraemer, and J. Hennig, 1998. The processing of first- and second-order motion in human visual cortex assessed by functional magnetic resonance imaging (fMRI), J. Neurosci., 18:3816–3830. Tootell, R. B. H., and N. Hadjikhani, 2001. Where is “dorsal V4” in human visual cortex? Retinotopic, topographic and functional evidence, Cereb. Cortex, 11:298–311. Vanduffel, W., D. Fize, K. Nelissen, P. Van Hecke, J. B. Mandeville, R. B. H. Tootell, and G. A. Orban, 2001. Cue-invariant shape processing in the awake fixating monkey: a contrast-agent enhanced fMRI study, Soc. Neurosci. Abstr., 11.10. Van Essen, D. C., W. T. Newsome, J. H. R. Maunsell, and J. L. Bixby, 1986. The projections from striate cortex to areas V2 and V3 in the macaque monkey: asymmetries, areal boundaries and patchy connections, J. Comp. Neurol., 244:451–480. Van Essen, D. C., and S. M. Zeki, 1978. The topographic organization of rhesus monkey prestriate cortex, J. Physiol., 277:193–226. Van Oostend, S., S. Sunaert, P. Van Hecke, G. Marchal, and G. A. Orban, 1997. The kinetic occipital (KO) region in man, an fMRI study, Cereb. Cortex, 7:690–701.

Verrey, L., 1888. Hémiachromatopsie droite absolue, Arch. Ophthalmol. (Paris), 8:289–301. Wade, A. R., A. A. Brewer, J. W. Rieger, and B. A. Wandell, 2002. Functional measurements of human ventral occipital cortex: retinotopy and color, Philos. Trans. R. Soc. Lond. B., 357:963–973. Wandell, B. A., 1999. Computational neuroimaging of human visual cortex, Annu. Rev. Neurosci., 22:145–173. Zeki, S. M., 1969. Representation of central visual fields in prestriate cortex of monkey, Brain Res., 14:271–291.

Zeki, S. M., 1978. The third visual complex of rhesus monkey prestriate cortex, J. Physiol., 277:245–272. Zeki, S., 1990. A century of cerebral achromatopsia, Brain, 113:1721–1777. Zeki, S., 1993. A Vision of the Brain, Oxford: Blackwell. Zeki, S., J. D. G. Watson, C. J. Lueck, K. J. Friston, C. Kennard, and R. S. J. Frackowiak, 1991. A direct demonstration of functional specialization in human visual cortex, J. Neurosci., 11:641–649.

:     

1039

IX FORM, SHAPE AND OBJECT RECOGNITION

68

Spatial Scale in Visual Processing ROBERT F. HESS

I   1970s, our views about the nature of visual processing began to change. The default idea, that visual processing produced an image description based on a local feature representation, was being challenged. The new suggestion, which had its origins in visual psychophysics (Campbell and Robson, 1968), was that visual information is analyzed in terms of the amplitude of different Fourier components. According to this approach, information about spatial scale was available to the higher stages of perception and was used in a very specific way. Some found this latter view particularly objectionable because, unlike its applicability to physical optics, neural processes possessed a number of characteristics (nonlinearities and changes in spatial grain with eccentricity) that would potentially render such an analysis inappropriate (Petrov et al., 1980; Westheimer, 1973). However, the issue was developing into one of a more general nature. A slow transition occurred in thinking from the original Fourier proposal to a more general one involving the importance of a local, scale-based spatial analysis. This was essentially a debate about whether perception had access to information separately at each of a number of spatial scales or whether, at an early stage, visual information was rigidly combined across scales for a feature-based analysis. The emerging neurophysiology, which showed that neurons in the striate cortex have receptive fields that could be considered spatial or temporal filters (Campbell et al., 1968, 1969; Glezer et al., 1973; Maffei and Fiorentini, 1973), suggested that, at least at the level of the striate cortex, information could be separately accessed at a range of different spatial scales. Thus, the balance tipped toward the independent scale proposal. However, the more we learn about the properties of cells in extrastriate cortex (Kourtzi and Kanwisher, 2001), the more we realize that scale combination is an essential part of vision. Subsequent work reviewed here suggests a role for both types of analysis: an initial scaleindependent process followed by, in some cases, scale combination. The relative roles of each depend on the particular task and stimuli.

History of the multiscale view The emergence of spatial scale as an important aspect of visual processing came at about the same time from neurophysiological studies of animals and psychophysical studies of humans. In humans, following from the early aerial

reconnaissance work of Selwyn (1948) in England and the applied work of Schade (1956) in the United States, Campbell and Green (1965) began to develop and apply the measurement of contrast sensitivity to better specify human vision. This approach emphasized the importance of visual processing for object sizes larger than the resolution limit. In particular, the contrast sensitivity function describes the relationship between the size of a stimulus and the contrast necessary to just detect it (i.e., its contrast threshold). The stimulus of choice is a sinusoidal grating (black and white bars with a sinusoidal luminance profile modulated about a mean light level) specified in units of spatial frequency (in cycles per degree subtended at the eye). Such a stimulus allows contrast to be altered without affecting the mean adaptational state of the eyes; the retinal image will also be sinusoidal in form. The form of this relationship is shown in Figure 68.1. Human sensitivity is best at intermediate object sizes (or spatial frequencies) and is reduced at both higher and lower spatial frequencies. Although the optics contributes to the reduction in contrast sensitivity at high spatial frequencies, the majority of the falloff at high spatial frequencies and all of the falloff at low spatial frequencies is due to the sensitivity of neural processes. The contrast sensitivity curve depicted in Figure 68.1 is for foveal viewing and photopic light levels. If stimuli are imaged on more peripheral parts of the field or under scotopic light levels, there is preferential loss of sensitivity at high spatial frequencies as a consequence of the reduced neural sensitivity under these conditions. The next major advance came when Campbell and Robson (1968), Pantle and Sekular (1968) and Blakemore and Campbell (1969), provided evidence that the contrast sensitivity function was itself composed of a number of more narrowly tuned, independent spatial mechanisms. A series of psychophysical studies followed, outlining the degree to which these spatial channels, as they were called, were independent (Graham et al., 1978) and the ways in which they interacted (Graham and Nachmias, 1971). During this same period, single-cell measurements from different parts of the visual pathway showed that neuronal receptive fields came in various sizes (Hubel and Wiesel, 1959, 1962) and that there was a systematic scaling of receptive field size with eccentricity in the retina and, to some extent, in the cortex (Hubel and Wiesel, 1968). Up to this

1043

Model. In the MIRAGE model, spatial filters were combined at an early stage, and only symbolic descriptors subsequent to this combination were used (Watt and Morgan, 1985). In the Wilson Line Element Model, later stages of processing had independent access to the output of individual spatial filters, and their outputs were flexibly combined to solve different tasks (Wilson and Gelb, 1984; Wilson and Richards, 1989). In this chapter, I will give examples of the advantages of accessing information at different spatial scales; these include foveal specialization and visual stability under different light levels. After detailing the evidence for independent access to scale information, I will discuss the scale selection rules. Finally, I will give examples of situations where information at different scales is not kept separate but is combined in specific ways (the scale combination rules). F 68.1. Contrast sensitivity function for foveal vision under photopic conditions. Contrast sensitivity (the reciprocal of the contrast needed for threshold detection) is plotted against the spatial frequency of a sinusoidal grating stimulus. This overall sensitivity function is itself composed of a set of more spatially restricted mechanisms termed spatial channels.

time, neurons were not really considered filters (in the sense that they only transmitted information of a particular spatial scale, just as a green filter attenuates blue and red light and only allows light of intermediate wavelengths to pass through); the prevailing idea was that they encoded certain stimulus features, and that they needed to do this at a range of different sizes. The idea that neurons could be sufficiently linear to be considered filters emerged from the retinal work of Enroth-Cugell and Robson (1966), which was very controversial at the time, beginning to have its main impact only a decade later. The primary visual cortex contains cells that are grouped together along a number of key processing dimensions: orientation, ocular dominance, and spatial frequency. Cells with similar spatial frequency preference are grouped together into domains whose map is locally continuous across V1. The distance between domains conforms to the hypercolumn description of cortical organization (Issa et al., 2000). The impact of spatial scale on our thinking in vision research is reflected in the different computational approaches (Marr, 1982; Marr and Hildreth, 1980; Marr and Poggio, 1979; Morrone and Burr, 1988; Watt and Morgan, 1985; Wilson and Gelb, 1984; Wilson and Richards, 1989) that have developed subsequent to the work described above. All the models assume an initial spatial decomposition. They differ in the extent of this decomposition and in the level at which the information from these different spatial filters is combined and analyzed. The two extreme versions of this model were captured in Watt and Morgans’ MIRAGE Model and Wilson’s Line Element

1044

, ,   

Scale and the functional specialization of central vision The relationship between spatial scale and eccentricity has important consequences for visual processing, yet it is poorly understood. Our traditional view of this relationship has been dominated by the measurement of visual acuity and its relationship with eccentricity. Acuity is best at the fovea and deteriorates progressively at more peripheral loci. This is similar to how the receptive field size of retinal ganglion cells changes from fovea to periphery (Crook et al., 1988; de Monasterio and Gouras, 1975; Hubel and Wiesel, 1960). Small receptive fields are confined to the center of the visual field, and progressively larger ones are found in the periphery. In the cortex this situation changes. True, there is gradual enlarging of the average receptive field in regions of the cortex representing more peripheral parts of the visual field. However, at any eccentricity there is a range of sizes of receptive fields. Hubel and Wiesel’s (1968) data suggest that this range is about 4/1 in the fovea. A better description is that in more peripheral parts of the visual field, the range of different-sized receptive fields, and by implication the range of spatial scales, is reduced because the number of cells with small receptive fields declines with eccentricity. Our understanding of how spatial scale varies across eccentricity has been greatly extended by the use of contrast sensitivity measurements for targets of different spatial scale. Numerous investigators have contributed to this understanding. Particularly noteworthy was the study of Robson and Graham (1981), who used stimuli of fixed spatiotemporal bandwidth but well localized in space. They showed that for a wide range of spatial scales, the decline in sensitivity with eccentricity (plotted in absolute units, i.e., in degrees) was linear and depended on spatial scale; the finer the spatial scale, the more rapidly sensitivity fell off with absolute eccentricity. This result is seen in Figure 68.2A,

where contrast sensitivity is plotted against eccentricity in degrees for a range of different spatial frequencies. However, if a relative eccentricity metric (Fig. 68.2B) was used (i.e., measured in periods of the particular spatial scale), then all spatial frequencies within the range 1 to 20 c/deg exhibited a similar falloff; for the vertical meridian, this was found to be 60 periods per decade (contrast sensitivity fell by a decade at an eccentricity equal to 60 periods of a particular spatial frequency). In a subsequent study, Pointer and Hess (1989) extended Robson and Grahams’ approach to spatial frequencies below 1 c/deg. Their results showed that there were three different rules for how spatial scale varied with eccentricity, depending on the spatial range involved. At high to middle spatial frequencies (above 1.6 c/deg), they replicated the previous results of Robson and Graham (1981); the sensitivity gradient was 60 periods per decade. At middle to low spatial fre-

F 68.2. The regional variation of contrast sensitivity. In A and B, the results of Robson and Graham (1981) showing how contrast sensitivity for a range of spatial frequencies declines with eccentricity along the vertical meridian. In A, the results are plotted against eccentricity in degrees, whereas in B, eccentricity is in relative units (periods of the particular spatial frequency). Note that the relative slope (B) is the same for all spatial frequencies in the range tested. (Reproduced with permission from Robson and Graham, 1981.)

quencies (0.2 to 0.8 c/deg) the decline of sensitivity with relative eccentricity increases (30 periods per decade), and at very low spatial frequencies (0.1 c/degree and below), the decline in sensitivity is much more gradual (90 periods per decade). From a purely psychophysical viewpoint, the relationship between spatial scale and eccentricity is fairly straightforward. At most spatial scales, sensitivity is best in the fovea, and the extent of this superiority depends on the spatial scale; the finer the spatial scale, the greater the superiority of the fovea. At very coarse scales, sensitivity is the same in the center of the field as it is in the periphery. There is no evidence for either peripheral superiority or foveal inferiority at coarse scales. Thus, at more eccentric loci, the range of spatial filters available to perception is reduced. Most neurophysiologists would not be comfortable with the above view. This is because this is not reflected in how the receptive field properties of neurons vary with eccentricity in primary cortex, where, as I have already said, the size of the average receptive field increases with eccentricity. This is best seen in the population response using functional imaging. Marrett et al. (1997), using an adaptation of the peripheral phase encoding method, showed that in human V1, spatial scale varies inversely with eccentricity; the fovea contains higher spatial scales, the periphery lower ones. However, what one needs to appreciate is that the psychophysical data reflect the potential contribution of all visual areas, not just V1. Area V2 in primate (and area 18 in cat) is known to contain neurons responding to lower spatial scales than V1 (Foster et al., 1985; Issa et al., 2000; Movshon et al., 1978). Extrastriate visual areas generally have larger receptive fields than V1 and sizable foveal representations. What this means is that, by virtue of the additional information provided by these different extrastriate areas, the fovea has access to additional levels of spatial processing at coarser scales, as well as at the extra-fine scales provided by V1. When we think of foveal specialization, we often think in terms of the magnified areal representation of the fovea in the primary visual cortex. Another way in which the fovea is specialized is that it has access, via the contribution of different visual areas, to a larger range of spatial filters than its peripheral counterpart. This provides a great deal of potential flexibility and computational power. It also represents an important economy in terms of processing bandwidth because not all scales need to be represented at all eccentricities.

Scale and light adaptation One of the most impressive aspects of mammalian vision is the range of light levels over which it operates without greatly sacrificing sensitivity. The usual explanation for this is retinal: the duplex function of the photoreceptors together

:     

1045

with gain control in postreceptoral retinal neurons. The fact that information is processed independently across scale, a mainly postretinal phenomenon, also helps extend our visual range and makes a significant contribution to perceptual stability across different light levels. The relationship between spatial scale and mean light level is best seen in the results of Van Nes and Bouman (1967). They showed that at any one spatial scale there is a characteristic relationship between sensitivity and light level. At low light levels, sensitivity varies as the square root of the light level (so-called De Vries-Rose law) (De Vries, 1943; Rose, 1942, 1948), whereas at higher light levels, sensitivity is independent of the average light level (so-called Weber’s law). The light level at which one behavior gives way to another varies with spatial scale. This is shown in Figure 68.3, in which, for a range of spatial frequencies, contrast sensitivity is plotted against the mean retinal illuminance (in trolands). These results suggest that neurons with smaller receptive fields reach this “transition” luminance at higher light levels than those with larger receptive fields. The psychophysics shows that this Rose-De Vries/Weber behavior occurs at all spatial scales. At the level of the retina, the degree of spatial selectivity also varies with the light level. Enroth-Cugell and Robson (1966) showed that the position of peak responsivity of a neuron shifts to lower spatial frequencies at lower light levels. Curiously, the low-frequency limb of the response function can be in its Weber region when the high-frequency limb is in its De Vries-Rose region. In the cortex, this does not happen. Here the spatial tuning does not alter (either position of peak responsivity or bandwidth); only sensitivity varies with light level. Sensitivity varies in just the way Van

F 68.3. The scale-independent way in which contrast sensitivity varies with mean light level. The results have been replotted from Van Nes and Bouman (1967). Contrast sensitivity is plotted against the mean retinal illuminance in trolands for a range of different spatial frequencies (listed in the top inset). Each spatial frequency response exhibits two different rules, depending on the mean illuminance: a Weber rule at high light levels and a DeVries-Rose rule at low light levels. The transition from the former to the latter also depends on the mean light level. (Reproduced with permission from Van Nes and Bouman, 1967.)

1046

, ,   

Nes and Bouman (1967) described in humans. This is true for simple cells and complex cells in area 17 of cat (Hess, 1990; Kaufman and Palmer, 1990). Thus, by the time one gets to the cortex, reducing the light level seems to affect only sensitivity, not the individual spatial and temporal selectivity of neurons. To the extent that we process information independently at different spatial scales, this provides an opportunity for greater stability in our perceptions at reduced light levels. Our perception of spatial and temporal structure, information derived from the output of many filters, is not greatly affected by reducing the light level. For a change in the average light level by three to six orders of magnitude, our perceptions of spatial and temporal frequency are influenced as little as 10% (Hess, 1990). That sort of perceptual stability is remarkable and contributes significantly to our ability to operate effectively over a wide range of light levels.

Evidence for independent scale processing S  S Contrast perception. Many studies have shown that the perceived contrast of a stimulus depends on the contrast of surrounding structures, a phenomenon known as contrast induction (Cannon and Fullenkamp, 1991; Chubb et al., 1989; Ejima and Takahashi, 1985; Ellemberg et al., 1998; Klein et al., 1974; Mackay, 1973). Two recent studies have shown that the magnitude of contrast induction depends on the spatial frequency difference between test and inducing stimuli (Chubb et al., 1989; Ellemberg et al., 1998). Chubb et al. (1989) showed that if the spatial frequency composition of test and inducing isotropic textures was 1 octave apart, the apparent contrast dropped from 40% to 15%. Ellemberg et al. (1998) showed a similar effect for single one-dimensional Gabor elements. Solomon et al. (1993) showed that this contrast induction is also tuned for orientation. Thus, it appears that the perceived contrast of an isolated region is determined in part by the contrast in adjacent regions and that this lateral influence is mediated within spatial scale. This is a curious result because one would intuitively expect contrast normalization to be mediated by a spatially and temporally broadband mechanism. Contrast-defined structure. Objects can be defined in a variety of ways: by modulations in luminance, contrast, chromaticity, motion, and disparity. Human vision is best at detecting luminance modulation (so-called first-order modulation) but can also detect objects defined by contrast modulation (second-order modulation). Indeed, there is evidence that cortical cells can process both types of information at the level of the striate cortex (Baker, 1999; Zhou and Baker, 1993, 1994, 1996). Though there are different ways of mod-

eling the detection of contrast-defined objects, the standard model involves two stages: a linear first stage composed of bandpass spatial filters and a second stage of linear filtering preceded by a rectifying nonlinearity. It has been of interest to know the relationship between the first and second stages of linear filters; for example, do second-stage filters of a particular spatial scale receive the summed input of all firststage filters? If they do, they would act as generic “texture grabbers” by detecting contrast modulation irrespective of the spatial composition of the texture. This would provide an obvious economy since fewer second-order neurons would be needed to cover all the possibilities of the firstorder input. The results, however, suggest the opposite, namely, that spatial information is not collapsed prior to second-stage detection (Dakin and Mareschal, 2000; Graham et al., 1993; Langley et al., 1996). The current neurophysiology also supports this conclusion. Mareschel and Baker (1998) have shown that second-order detectors are tuned for the spatial frequency and, to a lesser extent, for the orientation of their input carrier frequencies. This means that more second-order detectors are required to cover all possibilities for the spatial tuning of their first-order input, but this may not be quite the problem it was once thought to be (Baker, 1999). This is an example of spatial scale being preserved through at least two different stages of visual analysis. Contour. Recent attempts to understand contour processing have used spatially narrowband elements. Such stimuli allow tests of whether integration occurs between cells with different receptive field properties or between cells with similar receptive field properties. In one approach (Field et al., 1993), each presentation comprises an array of spatially and spatial frequency–localized elements. In one of the two presentations, a subset of these elements are arranged, by virtue of their orientation alignment, to define a contour. When subjects are asked to detect which presentation contains the contour, using a standard two alternate forced-choice psychophysical procedure, performance is found to be at ceiling for straight contours but declines as contour curvature increases (Field et al., 1993). Another reason for using spatially narrowband elements and limiting the processing to just one scale is because of the properties of natural images (Field, 1993). Consider the orientation structure of a fractal edge at a number of different spatial scales. Unlike smooth edges, fractal edges do not exhibit the same consistency of local orientation across scale, making a number of previously proposed edge detection strategies (Canny, 1983; Lowe, 1988; Marr and Hildreth, 1980) problematic. A fractal edge is continuous at each scale, but the precise position and orientation of the edge change across scales. At any position, the edge may show a particular orientation at only one scale. This may be one good

reason for having cortical neurons with bandpass properties (Field, 1987, 1993; Hayes, 1989). The above argument, based on the properties of natural images, suggests that it may be important for the visual system to solve the continuity problem separately at each scale, but is there any direct psychophysical support for this conclusion? The stimuli displayed in Figure 68.4B provide such a test. Here we are using the same paradigm as described above, except that the display is composed of equal numbers of Gabors and phase-scrambled edges. Phase-scrambled edges are micropatterns composed of f + 3f + 5f compound Gabors in which the relative phases are randomized. When the components are phase-aligned, they produce local edge stimuli. The results in Figures 68.4C and 68.4D show that when contour elements alternated between Gabors and phase-scrambled edges (as is also the case for the background elements), good performance (triangles) was maintained across different spatial scales for the Gabor, scales that were common to the phase-scrambled elements. This suggests that the visual system solves the continuity problem independently at each of a number of spatial scales. Interestingly, this was not the case when Gabors were alternated with phase-aligned edges (circular symbols), suggesting the importance of phase alignment in broadband stimuli in determining whether scale combination occurs (Dakin and Hess, 1999). Textures composed of orientation flows have also been shown to exhibit spatial frequency selectivity. Kingdom and Keeble (2000) have shown that texture segmentation for such a task depends on the spatial frequency composition of the local elements, suggesting a scale-selective analysis. S  M Historically, there have been two different models of human motion detection, one using information within different spatial scales and the other involving information after it has been combined across spatial scales. The most popular example of the former is the motion energy model (i.e., the oriented energy in the spatiotemporal spectrum) utilizing neurons with receptive fields narrowly tuned for spatial frequency and orientation (Adelson and Bergen, 1985; van Santen and Sperling, 1985; Watson and Ahumada, 1985). A good test of this involves the motion of a spatially filtered broadband stimulus (spatial noise). For two-flash apparent motion, according to the first proposal above, the direction of displacement would be detected independently at a number of different spatial scales. Dmin (the smallest displacement detected) would be signaled by the detectors working at the finest spatial scale and ultimately would be limited by the signal-to-noise ratio within these detectors. Dmax (the largest displacement detected) would be signaled by the detectors working at the largest scale and ultimately would be limited by their half-cycle limit. An altogether different view (Morgan, 1992; Morgan and Mather,

:     

1047

a

b

(c)

(d)

100

100

Noise edge & Gabor Edge & Gabor

Noise edge & Gabor Edge & Gabor 90

90

Noise & Noise

Percent correct

Percent correct

Noise & Noise 80

70

60

80

70

60

50

50 1

2

4

8

Carrier spatial frequency (c.p.d.)

1

2

4

8

Carrier spatial frequency (c.p.d.)

F 68.4. The scale-independent nature of contour integration. In a and b, an example of the stimuli are shown in which only a subset of the elements have their local orientation aligned along a notional curved contour. In a, all the elements are composed of phase-randomized f + 3f + 5f compound Gabor micropatterns. In b, half of the elements are phase-randomized f + 3f + 5f compound Gabor micropatterns and half are Gabor micropatterns having a spatial frequency corresponding to the abscissa in c and d. In c and d, psychophysical detection results are given for several different

stimulus conditions. Of particular note is the fact that performance is good across a range of spatial Gabor frequencies when the linking is between the phase-randomized f + 3f + 5f compound Gabor micropatterns and Gabors of a single frequency, indicating that the task can be performed independently within each of several different spatial scales (triangles). Note that this is not the case when the compound Gabors exhibit phase coherence (i.e., edges and circles). (Results reproduced with permission from Dakin and Hess, 1999.)

1994) is that the visual system works on the displacements of image features that exist only when information across different spatial scales has been collapsed together by a single spatial filter prior to motion detection. Initially, the independent scale model of motion was supported by the finding that, at least over part of the range (not at low spatial scales), Dmax scaled with the center frequency of bandpass-filtered noise images (Bischof and Di Lollo, 1990, 1991; Chang and Julesz, 1983, 1985; Cleary, 1990; Cleary and Braddick, 1990; Morgan, 1992; Morgan and Mather, 1994). However, this result was equally well described within the feature model of motion because the average separation between image features in bandpassfiltered images also scales with center frequency (Morgan, 1992; Morgan and Mather, 1994). The failure at very low spatial scales was due to the use of white rather than fractal

noise for the broadband stimuli. Fractal noise images (a 1/f amplitude spectrum), unlike white noise images (a flat amplitude spectrum) provide comparable energy for octavewide detectors tuned to different spatial scales (Field, 1987). Stimuli composed of white noise do not adequately stimulate visual detectors tuned to low spatial scales compared with those tuned to higher spatial scales. This resulted in the belief that high spatial frequencies interacted with low spatial frequencies for motion (Chang and Julesz, 1983, 1985; Cleary and Braddick, 1990). Two recent findings, both involving differential spatial filtering between the two frames of a two-flash motion sequence, suggest that the detection of image motion occurs within rather than across spatial scale. The first is the finding that motion can be reliably detected between two image frames so long as there is motion energy within a common

1048

, ,   

spatial frequency band, suggesting that the visual system has equal access to information across spatial scale (Brady et al., 1997; Ledgeway, 1996). The second finding is that motion can be detected between motion sequences of fractal noise that are differentially low- or highpass filtered (Bex et al., 1995). One frame can be unfiltered; the other, either highpass or lowpass filtered. This destroys the correlation between the edge structure but maintains information at common scales (Hess et al., 1998). These results are displayed in Figure 68.5 (in A, symbols show results for lowpass filtering of one frame; in B, symbols show results for highpass filtering of one frame). The multichannel prediction was generated by assuming that Dmax is determined by information carried by the lowest scale supported by the stimulus within the constraints of the half-cycle limit. The spectral characteristics of the spatial noise are again important here. Initial attempts (Morgan and Mather, 1994) with white noise failed because low-frequency mechanisms were severely disadvantaged by the use of white noise stimuli. The use of fractal noise stimuli, which affords similar stimulation of spatial detectors tuned to fine and coarse scales, demonstrates that the detection of motion direction can occur independently at each of several spatial scales. This, of course, does not bear upon exactly what is computed within each of these scales. It could be motion energy (Adelson and Bergen, 1985) or some more scale-localized feature (Eagle, 1996). This is still an open question, one that may be resolved by investigating temporal summation aspects of motion detection. S  S The relationship between spatial scale and stereo processing, like that between spatial scale and motion processing, has been controversial. The initial recep-

F 68.5. The effect of spatial scale on the discrimination of motion direction. A, Direction discrimination data (symbols) for fractal noise images are seen in two-flask apparent motion when only one of the two frames is subject to spatial filtering. B, Results are compared with a multichannel model in which motion direc-

tive field positional disparity model advanced by Barlow et al. (1967) and by Pettigrew et al. (1968) had no specific role for receptive fields of different size. A much later model, where disparity was encoded not by positional displacements of receptive fields but by phase disparities within receptive fields driven by the right and left eyes (Ohzawa and Freeman, 1986; Ohzawa et al., 1990, 1996), did have a specific link to the spatial properties of individual cells. It relied on high spatial frequency tuned cells processing only fine disparities and low spatial frequency tuned cells processing only coarse disparities (the so-called size-disparity correlation). Support for such a size-disparity correlation in human stereo processing has not been clear-cut. For example, Schor and Woods (1983) provided the first psychophysical evidence for a size-disparity correlation by measuring the relationship between stereo sensitivity (Dmin and Dmax) and luminance spatial frequency. For Dmin (the lower disparity limit) below a spatial frequency of 2.4 c/deg, stereo thresholds depend directly on the peak luminance spatial frequency of the stimulus, representing a constant phase limit of around 1/36th of a spatial cycle (Fig. 68.6A). For Dmax (the upper disparity limit), Schor and Woods found a square-root relationship over approximately the same spatial frequency range with an asymptote at around 2.4 c/deg (Fig. 68.6C). This suggested that at least for Dmin, disparity may have been computed within each of a number of independent spatial scales (Schor et al., 1984). Later work (Smallman and MacLeod, 1994, 1997) suggested that such a correlation may occur, at least for lowcontrast targets, across the whole spatial frequency range including that above 2.4 c/deg. The interpretation of these results in terms of the role of spatial channels in stereo

tion is processed independently by channels at a number of different spatial scales and the channel with the highest signal-to-noise ratio determines performance. (Reproduced with permission from Hess et al., 1998.)

:     

1049

B

A

2 2

D min (arc secs)

100 100

8

6 4

4 2 2

10

10 8 6 4

4

2

2

1 2

3

4 5 6

0.1

2

3

10

10

10

10

2

10

D 10

3

10

2

10

1

10

0

10 0.1

3

4 5 6

2

3

10

Spatial Frequency (c/deg)

4 5 6

2

3

4 5 6

1

10

4

3

2

1

0

size 0.5˚

c/object 1 2

1˚

3

2˚

4

4˚

R.F.H.

4 5 6

1

3

0.1

4

2

1/36 cycle R.F.H.

4 5 6

1

C 10

0.5˚ disc 1˚ disc 2˚ disc 4˚ disc

8 6

1

D m ax (arc secs)

8

6

2

0.1

3

4 5 6

2

3

4 5 6

1

10

Spatial Frequency (c/deg)

F 68.6. The scale-dependent nature of local stereo processing. In A and C, results are replotted from Schor and Woods (1983) for Dmin and Dmax, respectively, for a one-dimensional bandpass DOG stimulus in which the overall size and peak spatial frequency covary. In B and D, results for Dmin and Dmax, respectively,

for two-dimensional bandpass fractal noise stimuli in which overall size and peak spatial frequency have been decoupled (Hess et al., 2002). Dmin does scale with the peak spatial frequency of the image irrespective of stimulus size; Dmax does not. Dmax scales with object frequency, not retinal frequency (D).

processing has been controversial. The above interpretation, that there are spatial frequency mechanisms processing stereoscopic information only below 2.4 c/deg (Schor et al., 1984), has been challenged by the results of Yang and Blake (1991) and Kontsevich and Tyler (1994). Yang and Blake’s (1991) masking results and Kontsevich and Tyler’s (1994) modeling results argue that only spatial channels above 2.4 c/deg process stereo information. More recently, Glennerster and Parker (1997) questioned the conclusions of Yang and Blake’s results, stating that they had not taken into account the overall visibility of the stimuli, and argued instead for multiple spatial mechanisms processing stereo information below 2 c/deg, a result supported by subsequent work (Prince et al., 1998). In the majority of the above studies (e.g., Fig. 68.6A, C), the stimulus of choice was a one-dimensional spatially and spatial frequency localized stimulus, a difference of Gaussians (DOGs). One property of such a stimulus is that the peak spatial frequency and overall size of the stimulus covary, leaving one to wonder whether the relationship

described in Figure 68.6A is due to stimulus spatial frequency or size. A more general test of the size-disparity correlation would involve the use of broadband fractal stimuli for the reasons outlined above for the comparable case in motion. Such stimuli contain a range of different spatial scales and are not only representative of natural images but also optimal in stimulating different spatial frequency–tuned neurons (Field, 1987). Assuming that the visual system has independent access to an array of spatial frequency–tuned detectors, each responding up to its individual phase disparity limit, there are some clear predictions for how Dmin and Dmax should vary with low- and highpass filtering of such a broadband stimulus. Dmin should be determined by disparity neurons with the highest spatial frequency tuning (assuming that each spatial detector has comparable internal noise) and Dmax by the lowest. For Dmin, lowpass filtering should reduce stereo performance in a manner corresponding to some fixed fraction of a spatial cycle of the highest spatial frequency channel supported by the stimulus. This fraction will depend on factors such as stimulus contrast

1050

, ,   

because it represents not only a spatial limit but also a signalto-noise limit. For the same reason, highpass filtering should have no effect on performance. Dmax, on the other hand, which is thought to be a predominantly spatial limit, should be limited by the half-cycle limit of the lowest spatial frequency channel supported by the stimulus. Therefore, it should display a falloff with filter cutoff frequency corresponding to a fixed phase relationship for highpass filtering but no effect for lowpass filtering. These predictions follow from possibly the simplest view of the relationship between spatial frequency–tuned disparity mechanisms: that the channel with the highest signal-to-noise ratio will determine performance. There are, of course, many other possibilities. For example, disparity mechanisms may receive input from the combined output of many spatial channels, and other factors may limit Dmax and Dmin (e.g., the type of local primitive derived from such a multiscale analysis). The effect of various types of spatial filtering on stereopsis is seen in Figures 68.6B and 68.6D. Figure 68.6B shows how Dmin for a broadband fractal image varies with lowpass filtering (highpass filtering has no effect; data are not presented). Irrespective of the stimulus size (here, spatial frequency and stimulus size have been studied separately), Dmin gets progressively worse with progressive lowpass filtering (slope = 1/36th of the period of the highest frequency), bearing out the conclusions of Schor and Woods (1983). Figure 68.6D shows similar results for Dmax, but this time in terms of highpass filtering (lowpass filtering has no effect; data are not presented). However, the relationship is now much shallower (square root) than the independent scale prediction (slope of unity), and there is an effect of stimulus size. In fact, Dmax seemed to show the scale-dependent prediction (slope of unity) when we compared performance for stimuli not with the same spatial frequency (i.e., cycles/degree) but with the same object frequency (cycles/object), suggesting that it follows the information content of the stimulus rather than a spatial scale limit imposed by the visual system. Although Dmin can be thought of as reflecting the activity of the highest spatial frequency–tuned disparity neurons supported by a particular stimulus, Dmax cannot. It must involve information combined across scale. The Ohzawa-Freeman (1986) model would work for Dmin but not for Dmax. A combination of the models of both Ohzawa and Colleagues (1996) and Barlow et al. (1967) might be able to explain both limits. There is more to stereopsis than the measures Dmax and Dmin. For example, local stereopsis can support the perception of complex three-dimensional surfaces (Tyler, 1974). Does spatial scale maintain its relevance at this higher level? The answer is yes, at least in some cases. Take the example of two identical sinusoidal surfaces defined by stereo added 180 degrees out of phase (Fig. 68.7). Such a stimulus is difficult to disambiguate when each surface is

made up of an array of micropatterns comprising the mixture of two spatial frequencies (Fig. 68.7A) an octave apart. Yet these surfaces can be disambiguated (Fig. 68.7B) when each surface contains its own exclusive spatial scale (Kingdom et al., 2001). This segregation by spatial scale has obvious ecological relevance because similar objects (e.g., foliage) at different depths will be represented at a different spatial scale, and this difference in scale by itself can aid segregation. S  C Any understanding of the relation between chromatic and achromatic processing necessitates an understanding of spatial scale. A comprehensive comparison of spatial processing in chromatic and achromatic vision was made using the contrast sensitivity function (Granger and Heurtley, 1973; Mullen, 1985; Sekiguchi et al., 1993). This showed that the overall spatial sensitivity of mechanisms that process red-green and blue-yellow is very different from that of the achromatic system. Chromatic sensitivity is lowpass, compared with the bandpass response of the achromatic system. Furthermore, although chromatic sensitivity is worse at higher spatial frequencies compared with its achromatic counterpart, its low spatial frequency sensitivity is better. These results are shown in Figure 68.8. Chromatic and achromatic processes work best at complementary spatial scales. Color vision does not provide the sort of detail that achromatic vision is capable of, but it does provide the ability to segment large regions based on subtle color differences. The spatial properties of the individual mechanisms underlying human chromatic sensitivity are similar to those previously described for achromatic vision, namely, a more narrowly tuned bandpass mechanism extending over the entire range (Bradley et al., 1988; Losada and Mullen, 1994, 1995; Mullen and Losada, 1999). The scale-dependent rules for how sensitivity falls off across the visual field described above for the luminance grating are also relevant to chromatic sensitivity, with the proviso that red-green chromatic sensitivity falls off much more rapidly than its achromatic counterpart (Mullen, 1991). Blue-yellow and red-green cone opponency are distributed differently across the visual field, suggesting different underlying neural constraints (Mullen and Kingdom, 2002). However, this difference, like the above chromatic/achromatic difference, does not itself depend on spatial scale (K. T. Mullen, personal communication).

Scale selection The ability to be able to process information independently at different scales has some obvious advantages. These involve situations where scales are differentially affected, for example, with reduced luminance levels, peripheral viewing,

:     

1051

F 68.7. A scale-dependent example of transparency for global stereopsis. a and b, Example of dual-surface disparity gratings. Fusion of the two stereo pairs in a and b results in the percept of an obliquely oriented corrugated structure. This structure is actually composed of two interwoven sinusoidal disparity surfaces added out of phase. In a, each surface is composed of Gabor ele-

ments of a single scale, though there is an octave difference in the scale that represents each surface. In this case, the transparent nature of the two surfaces is perceived. In b, the Gabors of different scale are not segregated with respect to the surfaces. In this case, transparency is not perceived. (Reproduced with permission from Kingdom et al., 2001.)

motion, stereo, and color. Under these situations, common information derived from multiple scales will be less corrupted and, as a consequence, perceptions will be more stable. This benefit comes at a cost. Firstly, some of the more interesting image features occur only after scale combination; secondly, for information that is carried independently within multiple scales, one has to decide what scale to select and what scale to ignore: the problem of scale selection. Intelligent selection of spatial scale is always better than a dumb combination, but the questions is, “What constitutes intelligent selection?” A number of rules have been proposed, each with application to a specific task. These range from selection of the scale with the best signal-to-noise ratio or the least variability to the tracking of filter output features across scale.

Initially, at least for texture tasks, rules were delineated for the selection of features for segregation (Julesz, 1981). Some of these feature rules for textures can be recast as filter selection, especially where the contrast polarity of features is important. A number of suggestions have been made for the selection of the appropriate scale of analysis. For example, within the image processing field, features such as zero crossing from the output of filters of different spatial scale were identified as important markers of image features. In one proposed scale selection rule, these zero crossings were tracked across scale and the highest scale at which they persisted was selected (Witkin and Tennenbaum, 1983). This initial approach, which was successful for onedimensional features, ran into difficulty in its application to two-dimensional features (Yuille and Poggio, 1986). Another

1052

, ,   

suggestion was to use the scale with the best signal-to-noise ratio. A number of studies on motion (Bex et al., 1995; Brady et al., 1997; Eagle, 1996; Hess et al., 1998) and stereo (Hess et al., 2002) have assumed this. Malik and Perona (1990) proposed a “leader-takes-all” rule for a texture discrimination task. Elder and Zucker (1998) proposed the use of the minimum reliable scale for edge detection. Finally, Dakin (1997) proposed a statistical rule, the minimization of local orientation variance for a global orientation task.

Evidence for scale combination

F 68.8. Spatial scale differences between achromatic and chromatic vision. Contrast sensitivity functions for achromatic and chromatic stimuli show that while the former excels at high spatial frequencies, the latter excels at low spatial frequencies. (Reproduced with permission from Mullen, 1985.)

F 68.9. Object recognition does not occur independently within different scales. In the top frame, recognition is impaired when the image is block-quantized (top, middle) and to a much greater extent than expected on the basis of the available low spatial frequency information (top, right). This effect was originally thought to be due to critical band masking (Harmon and Julesz,

Useful information in an image occurs at different scales within the one spatial region as well as at the one scale across different regions. Scale combination may help provide important object-based information in the presence of surface noise (Marr, 1982). Below is a discussion of specific situations where a rigid scale combination has been shown to occur. In some cases, it is not the stimuli but the task that determines whether the visual system combines information across scale or analyzes it separately at each of several scales. An example of the task-dependent nature of scale combi-

1973), but adding high frequencies (bottom, middle) does not impair performance (Morrone and Burr, 1983) to the same extent as block quantization (bottom, left). Flipping the luminance polarity (bottom, right) of the original block-quantized edges does aid recognition (Hayes, 1989). (Reproduced with permission from S. C. Dakin.)

:     

1053

nation is in local versus global discrimination tasks. For both spatial detection (Dakin and Bex, 2001) and motion detection (Bex and Dakin, 2000), it has been shown that at a local level elements are grouped in a scale-dependent way, but at a global level these local groupings are integrated across scale. On the other hand, proposals have been made for a rigid scale combination rule operating from coarse to fine scales. This was originally proposed to solve the correspondence problem for stereo (Marr and Poggio, 1979) but later was transformed into a model of spatiotemporal analysis where progressively finer scales are analyzed at progressively later times (Parker et al., 1997; Watt, 1987). S Harmon and Julesz (1973) were the first to show that images that are block-quantized are difficult to recognize. Compare Figure 68.9 (top row). The low spatial frequency

F 68.10. Spatial frequency tuning of local and global grouping for the recognition of structure in Glass patterns. In the top two frames, local sensitivity (the reciprocal of the signal-to-noise ratio at threshold) is plotted as a function of the spatial frequency of the element paired with a 2 c/deg dipole element. Here sensitivity directly reflects the sensitivity of the underlying mechanism.

1054

, ,   

components shown in the right top figure are also present in the middle top figure but cannot be used to aid recognition. Though at first this was thought to be due to “masking” of the low spatial frequencies by the high spatial frequencies introduced by the quantization, it was later realized that it had more to do with phase continuity across scale (Canny, 1983; Hayes, 1989; Morrone and Burr, 1983). Compare Figure 68.9 (bottom row). In the middle bottom figure, additional high spatial frequencies similar to those introduced by blocking are added, but recognition is not so impaired as it is in the bottom left figure. Image identification is restored when the phase relationships are disrupted between the (blocking) high spatial frequencies and the (original) low spatial frequency components of the image. This is seen in Figure 68.9 (bottom right) where the phase of the

In the bottom two frames, global sensitivity is plotted as a function of the spatial frequency of a masking stimulus. Here sensitivity is related (inversely to the sensitivity of the underlying mechanism. Local grouping occurs within scale; global grouping occurs across scale. (Reproduced with permission from Dakin and Bex, 2001.)

high-frequency components introduced by blocking has been shifted by 180 degrees; now recognition is easier. If the visual system carried out an independent scale analysis without rigid combination, a block-quantized image should be recognizable via its low spatial frequency components. For example, faces that are blurred are not unrecognizable (Fig. 68.9 (top right)). That this is not the case suggests that although processing may initially occur within scale, it is only by combining information across scale that interesting image features can be revealed. T P Glass patterns are visual textures composed of a field of element pairs (dipoles) whose global orientation is determined by a simple geometric transformation (Glass, 1969). These patterns are ideal for looking at scale combination because it has been shown that this task can be performed equally well at a number of independent spatial scales (Dakin, 1997). To detect the global structure in these multielement arrays, observers must perform local grouping at the level of the dipoles in order to detect the local oriented structure and global grouping at the level of the array as a whole in order to combine these local orientation estimates to derive global structure. Dakin and Bex (2001) have shown that the local grouping operation occurs independently at each of a number of spatial scales (i.e., has bandpass tuning), whereas the global grouping combines information across spatial scale (i.e., has lowpass tuning). The spatial tuning results for these two processes are seen in Figure 68.10 for the geometric transformations of rotation, translation, and expansion. B P Our understanding of how we discriminate blur is still unresolved. There have been a number of different proposals. In some cases (Elder and Zucker, 1998; Field and Brady, 1997; Mather, 1997; Watt and Morgan, 1983), the neural mechanism involved would need to collapse information across spatial scale, whereas in others (Georgeson, 1994), multiscale or single-scale (Elder and Zucker, 1998) processing would suffice. The straightforward suggestion that blur is signaled by the degree of activity at the highest spatial scale is not correct because we are better at discriminating a change in blur from a reference edge that is already blurred compared with a reference edge that is perfectly sharp (Watt and Morgan, 1983). This implies either that we cannot independently access the highest spatial frequency mechanisms that we possess to signal blur or that we choose not to use it for other reasons (i.e., if the finest scale is too noisy). Watt and Morgan (1983) proposed that there is a rigid combination of filter outputs (MIRAGE model) prior to blur analysis. They suggested that blur is signaled by the distance between adjacent peaks and troughs in the output of a second-derivative filter. Field and Brady (1997) and Mather (1997) proposed that

images look blurred when there is a change in the relative amplitude of detectable structure at different frequencies. To determine this would require a comparison of the activity across rather than solely within scales. Georgeson (1994) proposed (see also Kayargadde and Martens, 1994) another way of computing blur for aperiodic and periodic sinusoids based on a multiscale template model (Georgeson, 2001) in which blur is encoded by finding which of a set of multiscale Gaussian derivative templates best fits the second derivative profile of the edge; this model is simple, multiscale, fits well and (for one-dimensional edges) has no free parameters, and can be readily implemented by simple cells. An important contribution was also made by Elder and Zucker (1998), who used adaptive scale filtering; at each location, the system uses the smallest reliable scale, defined by signal-tonoise ratio. M V Even though the early stages of visual processing involve the activity of spatiotemporal separable filters, there is evidence that human discrimination of image motion is in terms of velocity rather than temporal frequency. To achieve this, it is necessary to have filters with inseparable spatiotemporal properties (i.e., having sensitivity envelopes oriented in space-time), necessitating the combination of information across spatial and temporal scales prior to the processing of image velocity. Psychophysical studies have provided evidence for the importance of velocity in motion processing. Thompson (1981) showed that a common metric for motion adaptation was in terms of velocity rather than spatial and temporal frequency. McKee et al. (1986) showed the importance of velocity for the discrimination of image motion. More recently, it has been shown that the visual system integrates information across different orientation bands in a nonselective manner, leading to suboptimal performance in some situations (Schrater et al., 2000). Perrone and Thiele (2001) have provided neurophysiological support for the receptive field properties of neurons at the level of area MT in the extrastriate cortex being oriented in space-time. This suggests that information previously contained within individual spatial and temporal scales in area V1 has been combined at the level of MT to encode velocity.

Combination rules At some stage within the visual process, for particular tasks, it is advantageous to combine information across spatial scale to derive interesting image features. The type of combination will depend critically on the nature of the image feature. Many different combination rules have been proposed, ranging from “blind” (i.e., indiscriminate) pooling to optimal pooling strategies. The main determinant is the particular feature involved.

:     

1055

For example, if the feature is the global orientation in an image, then it has been shown that information is first collapsed across scale or spatial frequency. Similarly, if the selected feature is the local spatial frequency structure, combination across orientation occurs (Dakin and Bex, 2001; Olzak and Thomas, 1991, 1992; Thomas and Olzak, 1996). There is also evidence that the visual system contains higherlevel mechanisms that sum across spatial scale to process the orientation of image features but do not sum across scale to process either the spatial frequency or the contrast of image features (Olzak and Wickens, 1997). If the feature is local velocity, Schrater et al. (2000) have shown that the visual system uses a fixed combination rule over orientation rather than one adapted to the spatial properties of a particular stimulus. Their result is consistent with velocity-tuned detectors that measure the energy in fixed (i.e., blind pooling), orientationally broadband, planar regions of spatiotemporal frequency. If the feature is the global structure in either Glass (Dakin and Bex, 2001) or random-dot kinematograms (Bex and Dakin, 2000), information is combined across spatial scale, with greater weight given to lower spatial frequencies. If the feature is the global perception of plaids composed of two oblique sinusoidal gratings, information is collapsed across orientation prior to edge extraction (Georgeson and Meese, 1997). A special case of interscale combination occurs for edge features where the combination process is phase sensitive. A number of computational schemes for edge detection exploit this by assessing the correlated activity between filter outputs across scale (Canny, 1983; Georgeson, 1992, 1994; Lowe, 1988; Marr, 1982; Marr and Hildreth, 1980; Morrone and Burr, 1988; Torre and Poggio, 1986). Georgeson and Meese (1997) report that the addition of a 3f component, if added in square wave phase to one component of their plaid stimuli, breaks down the previous combination across orientation. The perception of the plaid is now in terms of its components. This suggests that phase-sensitive combination of scale to extract edge features may be a special case and may take precedence over interorientation linking. Dakin and Hess (1999), using a contour integration task between elements with broad- and narrowband spatial properties, also argue that edge extraction may represent a special case and may occur concurrently with or prior to contour integration.

Conclusions Some have argued that the visual system processes information only within scale, while others have stated that information is processed only across scale by the visual system. However, one can find examples of within- as well as acrossscale analysis in vision, and this is quite task dependent. For

1056

, ,   

some tasks, scale combination occurs at a later stage in visual processing than for others. One example is efficient coding of edges. At the level of the striate cortex, humans have neurons with the appropriate spatial tuning properties to encode edge structure successfully via its harmonic components. However, if edges are an important feature of everyday images and if we want to achieve as sparse a code as possible (Olshausen and Field, 1996, 1997; see also Chapter 108 in this volume), it might be advantageous to also have some hardwired edge-detecting neurons (combination across scale) that would encode such common natural features more sparsely. Having such detectors would not preclude the use of multiscale processing for images with similar spatial frequency components (as edges) but where the phase relations are random (e.g., noise). This is an example of where within-scale and combined-scale analysis could coexist at the same stage of processing and even be capable of competing to accomplish similar tasks. Initially, the within-scale processing idea was taken to extreme limits by assuming that the visual system did a formal Fourier analysis of the retinal image. This view has now given way to the more moderate proposal that visual information can be processed separately at each of a number of scales within the same region of the field, at least in early parts of the cortical pathway. Understanding this provides predictive power about what limits various types of visual processes. It is, of course, not the full story. In some cases, for example, image velocity encoding, spatial combination is a fundamental part of the extraction of the relevant visual information. Spatial scale is now a fundamental part of our thinking in vision. It is as hazardous to neglect it as it is to be unaware of its limitations.

Acknowledgments I am grateful to all my collaborators throughout the years for their insight, patience, and understanding and to the Canadian Institutes of Health Research and National Science and Engineering Research Council of Canada for their support. I am grateful to Rebecca Achtman for helping with the illustrations. REFERENCES Adelson, E. H., and J. R. Bergen, 1985. Spatio-temporal energy models for the perception of motion, J. Opt. Soc. Am. A, A2:284– 299. Baker, C. L., Jr., 1999. Central neural mechanisms for detecting second-order motion, Cur. Opin. Neurobiol., 9:461–466. Barlow, H. B., C. Blakemore, and J. D. Pettigrew, 1967. The neural mechanism of binocular depth discrimination, J. Physiol. (Lond.), 193:327–342. Bex, P. J., N. Brady, et al., 1995. Energetic motion detection, Nature (Lond.), 378:670–672.

Bex, P. J., and S. C. Dakin, 2000. Narrowband local and broadband global spatial frequency selectivity for motion perception, Invest. Ophthalmol. Vis. Sci. (Suppl.), 41:s545. Bischof, E. H., and V. Di Lollo, 1990. Perception of directional sampled motion in relation to displacement and spatial frequency: evidence for a unitary motion system, Vis. Res., 9:1341–1362. Bischof, W. F., and V. Di Lollo, 1991. On the half-cycle displacement limit of sampled directional motion, Vis. Res., 31:649–660. Blakemore, C., and F. W. Campbell, 1969. On the existence of neurones in the human visual system selectively sensitive to the orientation and size of retinal images, J. Physiol. (Lond.), 203:237–260. Bradley, A., E. Switkes, and K. De Valois, 1988. Orientation and spatial frequency selectivity of adaptation to color and luminance gratings, Vis. Res., 28:841–856. Brady, N., P. J. Bex, and R. E. Fredericksen, 1997. Independent coding across spatial scales in moving fractal images, Vis. Res., 37:1873–1884. Campbell, F. W., and J. G. Robson, 1968. Application of Fourier analysis to the visibility of gratings, J. Physiol. (Lond.), 197:551–566. Campbell, F. W., B. Cleland, and C. Enroth Cugell, 1968. The angular selectivity of visual cortical cells to moving gratings, J. Physiol. (Lond.), 198:237–250. Campbell, F. W., G. F. Cooper, and C. Enroth Cugell, 1969. The spatial selectivity of the visual cells of the cat, J. Physiol. (Lond.), 203:223–235. Campbell, F. W., and D. G. Green, 1965. Optical and retinal factors affecting visual resolution, J. Physiol. (Lond.), 181:576–593. Cannon, W. M., and S. C. Fullenkamp, 1991. Spatial interactions in apparent contrast: inhibitory effects among grating patterns of different spatial frequencies, spatial positions and orientations, Vis. Res., 31:1985–1998. Canny, J. F., 1983. Finding edges and lines in images. Boston: MIT AI Laboratory Technical Report. Chang, J. J., and B. Julesz, 1983. Displacement limits, directional anisotropy and direction versus form discrimination in random dot cinematograms, Vis. Res., 23:639–646. Chang, J. J., and B. Julesz, 1985. Cooperative and non-cooperative processes of apparent motion of random dot cinematograms, Spatial Vis., 1:39–41. Chubb, C., G. Sperling, and J. A. Solomon, 1989. Texture interactions determine perceived contrast., Proc. Natl. Acad. Sci. USA, 86:9631–9635. Cleary, R., 1990. Contrast dependence of short range apparent motion, Vis. Res., 30:463–478. Cleary, R., and O. J. Braddick, 1990. Direction discrimination for bandpass filtered random dot kinematograms, Vis. Res., 30:303–316. Crook, J. M., B. Lange-Malecki, B. B. Lee, and A. Valberg, 1988. Visual resolution of macaque retinal ganglion cells, J. Physiol. (Lond.), 396:205–224. Dakin, S. A., 1997. The detection of structure in Glass patterns: psychophysics and computational models, Vis. Res., 37:2227– 2259. Dakin, S. A., and P. J. Bex, 2001. Local and global visual grouping: tuning for spatial frequency and contrast, J. Vis., 1:99–112. Dakin, S. C., and R. F. Hess, 1999. Contour integration and scale combination processes in visual edge detection, Spatial Vis., 12:309–327. Dakin, S. C., and I. Mareschal, 2000. Sensitivity to contrast modulation depends on carrier spatial frequency and orientation, Vis. Res., 40:311–329.

de Monasterio, F. M., and P. Gouras, 1975. Functional properties of ganglion cells in the rhesus monkey retina, J. Physiol. (Lond.), 251:167–195. De Vries, H., 1943. The quantum nature of light and its bearing upon the threshold of vision, the differential sensitivity and visual acuity of the eye, Physica, 10:553–564. Eagle, R. A., 1996. What determines the maximum displacement limit for spatially broadband kinematograms, J. Opt. Soc. Am. A, 13:408–418. Ejima, Y., and S. Takahashi, 1985. Apparent contrast of a sinusoidal grating in the simultaneous presence of peripheral gratings, Vis. Res., 25:1223–1232. Elder, J. H., and S. W. Zucker, 1998. Local scale control for edge detection and blur estimation, IEEE Trans. Pattern Analysis Machine Intell., 20:699–716. Ellemberg, D., F. E. Wilkinson, H. R. Wilson, and A. S. Arsenault, 1998. Apparent contrast and spatial frequency of local texture elements, J. Opt. Soc. Am. A, 15:1733–1739. Enroth-Cugell, C., and J. G. Robson, 1966. The contrast sensitivity of retinal ganglion cells of the cat, J. Physiol. (Lond.), 187:517–552. Field, D. J., 1987. Relations between the statistics of natural images and the response properties of cortical cells, J. Opt. Soc. Am. A, 4:2379–2394. Field, D. J., 1993. Scale-invariance and self-similar “wavelet” transforms: an analysis of natural scenes and mammalian visual systems, in Wavelets, Fractals and Fourier Transforms (M. Marge, J. C. R. Hunt, and J. C. Vassilicos, eds.), Oxford: Clarendon Press, pp. 151–193. Field, D. J., A. Hayes, and R. F. Hess, 1993. Contour integration by the human visual system: evidence for a local “association field,” Vis. Res., 33:173–193. Field, D. J., and N. Brady, 1997. Visual sensitivity, blur and the sources of variability in the amplitude spectra of natural scenes, Vis. Res., 37:3367–3384. Foster, K. H., J. P. Gaska, M. Nagler, and D. A. Pollen, 1985. Spatial and temporal frequency selectivity of neurones in visual cortical areas V1 and V2 of the macaque monkey, J. Physiol. (Lond.), 365:331–363. Georgeson, M., 1992. Human vision combines oriented filters to compute edges, Proc. R. Soc. Lond. B, 249:235–245. Georgeson, M. A., 1994. From filters to features: location, orientation, contrast and blur, in Higher-Order Processing in the Visual System (CIBA Foundation Symposium 184) (M. J. Morgan ed.), Chichester, UK: Wiley. Georgeson, M. A., 2001. Seeing edge blur: receptive fields as multiscale neural templates, presented at the Vision Sciences Conference, Sarasota, FL. Georgeson, M. A., and T. S. Meese, 1997. Perception of stationary plaids: the role of spatial filters in edge analysis, Vis. Res., 37:3255–3271. Glass, L., 1969. Moiré effects from random dots, Nature, 243:578– 580. Glennerster, A., and A. J. Parker, 1997. Computing stereo channels from masking, Vis. Res., 37:2143–2152. Glezer, V. D., A. M. Cooperman, V. A. Ivanov, and T. A. Tscherbach, 1973. Investigation of complex and hyper-complex receptive fields of visual cortex of the cat as spatial frequency filters, Vis. Res., 13:1875–1904. Graham, N., and J. Nachmias, 1971. Detection of grating patterns containing two spatial frequencies: a comparison of single channel and multi-channel models, Vis. Res., 11:251– 259.

:     

1057

Graham, N., J. G. Robson, and J. Nachmias, 1978. Grating summation in fovea and periphery, Vis. Res., 18:815–825. Graham, N., A. Sutter, and H. Venkatesan, 1993. Spatial frequency and orientation selectivity of simple and complex channels in regional segregation, Vis. Res., 33:1893–1911. Granger, E. M., and J. C. Heurtley, 1973. Visual chromaticitymodulation transfer function, J. Opt. Soc. Am., 63: 1173–1174 Harmon, L. D., and B. Julesz, 1973. Masking in visual recognition: effects of two dimensional filtered noise, Science, 180:1194–1197. Hayes, A., 1989. Representation by images restricted in resolution and intensity range. Ph.D. thesis, Department of Psychology, University of Western Australia. Hess, R. F., 1990. Vision at low light levels: role of spatial, temporal and contrast filters, Ophthalmol. Physiol. Opt., 10:351–359. Hess, R. F., P. J. Bex, E. R. Fredericksen, and N. Brady, 1998. Is human motion detection subserved by a single or multiple channel mechanism? Vis. Res., 38:259–266. Hess, R. F., H.-C. Liu, and Y. Z. Wang, 2002. Luminance spatial scale and local stereo-sensitivity, Vis. Res., 42:331–342. Hubel, D. H., and T. N. Wiesel, 1959. Receptive fields of single neurons in the cat’s striate cortex, J. Physiol. (Lond.), 148:574– 591. Hubel, D. H., and T. N. Wiesel, 1960. Receptive fields of optic nerve fibres in the spider monkey, J. Physiol. (Lond.), 154:572–580. Hubel, D. H., and T. N. Wiesel, 1962. Receptive fields, binocular interaction and functional architecture in the cat’s visual cortex, J. Physiol. (Lond.), 160:106–154. Hubel, D. H., and T. N. Weisel, 1968. Receptive fields and functional architecture of monkey striate cortex, J. Physiol. (Lond.), 195:215–243. Issa, N. P., C. Trepel, and M. P. Stryker, 2000. Spatial frequency maps in cat visual cortex, J. Neurosci., 20(22):8504–8514. Julesz, B., 1981. Textons, the elements of texture perception, and their interactions, Nature, 290:91–97. Kaufman, D. A., and L. A. Palmer, 1990. The luminance dependence of the spatiotemporal response of cat striate cortical cells, Invest. Ophthalmol. Vis. Sci. (Suppl.), 31:398. Kayargadde V. and J. B. Martens, 1994. Estimation of edge parameters and image blur using polynomial transforms. CVGIP: Graphical models and image processing, 56:442–461. Kingdom, F. A. A., and D. R. T. Keeble, 2000. Luminance spatial frequency differences facilitate the segmentation of superimposed textures, Vis. Res., 40:1077–1087. Kingdom, F. A. A., L. R. Ziegler, and R. F. Hess, 2001. Luminance spatial scale facilitates depth segmentation, J. Opt. Soc. Am. A, 18:993–1002. Klein, S. A., C. F. Stromeyer, and L. Ganz, 1974. The simultaneous spatial frequency shift: a dissociation between the detection and perception of gratings, Vis. Res., 14:1421–1432. Kontsevich, L. L., and C. W. Tyler, 1994. Analysis of stereothresholds for stimuli below 2.5 c/deg, Vis. Res., 34:2317–2329. Kourtzi, Z., and N. Kanwisher, 2001. Representation of perceived object shape by the human lateral occipital complex, Science, 293:1506–1509. Langley, K., D. J. Fleet, and P. B. Hibbard, 1996. Linear filtering precedes non-linear processing in early vision, Curr. Biol., 6:891–896. Ledgeway, T., 1996. How similar must the Fourier spectra of the frames of a random dot kinematogram be to support motion perception? Vis. Res., 36:2489–2495. Losada, M. A., and K. T. Mullen, 1994. The spatial tuning of chromatic mechanisms identified by simultaneous masking, Vis. Res., 34:331–341.

1058

, ,   

Losada, M. A., and K. T. Mullen, 1995. Color and luminance spatial tuning estimated by noise masking in the absence of offfrequency looking, J. Opt. Soc. Am. A, 12:250–260. Lowe, D. G., 1988. Organization of smooth image curves at multiple spatial scales, in Proceedings of the Second International Conference on Computer Vision, New York: IEEE Computer Society Press. Mackay, D. M., 1973. Lateral interaction between neural channels sensitive to texture density, Nature (Lond.), 245:159–161. Maffei, L., and A. Fiorentini, 1973. The visual cortex as a spatial frequency analyser, Vis. Res., 13:1255–1267. Malik, J., and P. Perona, 1990. Preattentive texture discrimination with early visual mechanisms, J. Opt. Soc. Am. A, 7:923–932. Mareschal, I., and C. L. Baker, 1998. A cortical locus for the processing of contrast-defined contours, Nat. Neurosci., 1:150–154. Marr, D., 1982. Vision, San Francisco: W. H. Freeman. Marr, D., and E. Hildreth, 1980. Theory of edge detection, Proc. R. Soc. B, 207:187–217. Marr, D., and T. Poggio, 1979. A computational theory of human stereo vision, Proc. R. Soc. B, 204:301–328. Marrett, S., A. M. Dale, J. D. Mendela, M. I. Sereno, A. K. Liu, and R. B. H. Tootell, 1997. Preferred spatial frequency varies with eccentricity in human visual cortex, NeuroImage, 3:s157. Mather, G., 1997. The use of image blur as a depth cue, Perception, 26:1147–1158. McKee, S. P., G. H. Silverman, and K. Nakayama, 1986. Precise velocity discrimination despite random variations in temporal frequency and contrast, Vis. Res., 26:609–619. Morgan, M. J., 1992. Spatial filtering preceeds motion detection, Nature, 335:344–346. Morgan, M. J., and G. Mather, 1994. Motion discrimination in 2-frame sequences with differing spatial frequency content, Vis. Res., 34:197–208. Morrone, M. C., and D. C. Burr, 1983. Added noise restores recognition of coarse quantized images, Nature, 305:226–228. Morrone, M. C., and D. C. Burr, 1988. Feature detection in human vision: a phase-dependent energy model, Proc. R. Soc. B, 235:221–245. Movshon, J. A., I. D. Thompson, and D. J. Tolhurst, 1978. Spatial and temporal contrast sensitivity in areas 17 and 18 of the cat’s visual cortex, J. Physiol. (Lond.), 283:101–130. Mullen, K. T., 1985. The contrast sensitivity of human colour vision to red-green and blue-yellow chromatic gratings, J. Physiol. (Lond.), 359:381–400. Mullen, K. T., 1991. Colour vision as a post-receptoral specialization of the central visual field, Vis. Res., 31:119–130. Mullen, K. T., and F. A. A. Kingdom, 2002. Differential distribution of red-green and blue-yellow cone opponency across the visual field, Vis. Neurosci., 19:1–10. Mullen, K. T., and M. A. Losada, 1999. The spatial tuning of color and luminance peripheral vision measured with notch filtered noise, Vis. Res., 39:721–731. Ohzawa, I., G. C. DeAngelis, and R. D. Freeman, 1990. Stereoscopic depth discrimination in the visual cortex: neurones ideally suited as disparity detectors, Science, 249:1037–1041. Ohzawa, I., G. C. DeAngelis, and R. D. Freeman, 1996. Encoding of binocular disparity by simple cells in cat’s visual cortex, J. Neurophysiol., 75:1779–1805. Ohzawa, I., and R. D. Freeman, 1986. The binocular organization of simple cells in cat striate cortex, J. Neurophysiol., 56:221– 242. Olshausen, B. A., and D. J. Field, 1996. Emergence of simple-cell receptive field properties by learning a sparse code for natural images, Nature, 381:607–609.

Olshausen, B. A., and D. J. Field, 1997. Sparse coding with an overcomplete basis set: a strategy employed by V1? Vis. Res., 37:3311–3325. Olzak, L. A., and J. P. Thomas, 1991. When orthogonal orientations are not processed independently, Vis. Res., 31:51–57. Olzak, L. A., and J. P. Thomas, 1992. Configural effects constrain Fourier models of pattern discrimination, Vis. Res., 32:1885– 1898. Olzak, L. A., and T. D. Wickens, 1997. Discrimination of complex patterns: orientation information is integrated across spatial scale; spatial frequency and contrast information are not, Perception, 26:1101–1120. Pantle, A., and R. Sekular, 1968. Size detecting mechanisms in human vision, Science, 62:1146–1148. Parker, D. M., J. R. Lishman, and J. Hughes, 1997. Evidence for the view that spatiotemporal integration in vision is temporally anisotropic, Perception, 26:1169–1180. Perrone, J. A., and A. Thiele, 2001. Speed skills: measuring the visual speed analysing properties of primate MT neurons, Nat. Neurosci., 4:526–532. Petrov, A. P., I. N. Pigarev, and G. M. Zenkin, 1980. Some evidence against Fourier analysis as a function of the receptive fields in cat’s striate cortex, Vis. Res., 20:1023–1025. Pettigrew, J. D., T. Nikara, and P. O. Beshop, 1968. Binocular interaction on single units in cat striate cortex: simultaneous stimulation by single moving slit with receptive fields in correspondence, Exp. Brain Res., 6:394–410. Pointer, J. S., and R. F. Hess, 1989. The contrast sensitivity gradient across the human visual field: emphasis on the low spatial frequency range, Vis. Res., 29:1133–1151. Prince, S. J. D., R. A. Eagle, and B. J. Rogers, 1998. Contrast masking reveals spatial-frequency channels in stereopsis, Perception, 27:1345–1355. Robson, J. G., and N. Graham, 1981. Probability summation and regional variation in contrast sensitivity across the visual field, Vis. Res., 21:409–418. Rose, A., 1942. Quantum and noise limitations of the visual process, J. Opt. Soc. Am., 43:715–725. Rose, A., 1948. The sensitivity performance of the human eye on an absolute scale, J. Opt. Soc. Am., 38:196–208. Schade, O. H., 1956. Optical and photo-electric analog of the eye, J. Opt. Soc. Am., 46:721–739. Schor, C. M., and I. C. Woods, 1983. Disparity range for local stereopsis as a function of luminance spatial frequency, Vis. Res., 23:1649–1654. Schor, C. M., I. C. Woods, and J. Ogawa, 1984. Binocular sensory fusion is limited by spatial resolution, Vis. Res., 24:661–665. Schrater, P. R., D. C. Knill, and E. P. Simoncelli, 2000. Mechanisms of visual motion detection, Nat. Neurosci., 3:64–68. Sekiguchi, N., D. R. Williams, and D. H. Brainard, 1993. Aberration-free measurements of the visibility of isoluminant gratings, J. Opt. Soc. Am. A., 10:2105–2117. Selwyn, E. W. H., 1948. The photographic and visual resolving power of lenses, Photo. J., 88:6–12, 46–57. Smallman, H. S., and D. I. A. MacLeod, 1994. Size-disparity correlation in stereopsis at contrast threshold, J. Opt. Soc. Am., 11:2169–2183.

Smallman, H. S., and D. I. A. MacLeod, 1997. Spatial scale interactions in stereosensitivity and the neural representation of binocular disparity, Perception, 26:977–994. Solomon, J. A., G. Sperling, and C. Chubb, 1993. The lateral inhibition of perceived contrast is indifferent to on-center/off-center segregation, but specific to orientation, Vis. Res., 33:2671– 2683. Thomas, J. P., and L. A. Olzak, 1996. Uncertainty experiments support the roles of second order mechanisms in spatial frequency and orientation discriminations, J. Opt. Soc. Am. A, 13: 689–696. Thompson, P., 1981. Velocity after-effects: the effects of adapting to moving stimuli on the perception of subsequently seen moving stimuli, Vis. Res., 21:337–345. Torre, V., and T. A. Poggio, 1986. On edge detection, IEEE Trans. Pattern Analysis and Machine Intell., 8:147–163. Tyler, C. W., 1974. Depth perception in disparity gratings, Nature (Lond.), 251:140–142. Van Nes, F. L., and M. A. Bouman, 1967. Spatial modulation transfer in the human eye, J. Opt. Soc. Am., 57:401–406. van Santen, J. P. H., and G. Sperling, 1985. Elaborated Reichardt detectors, J. Opt. Soc. Am. A, 2:300–321. Watson, A. B., and A. J. Ahumada, 1985. Model of human visual motion sensing, J. Opt. Soc. Am. A, 2:322–341. Watt, R. J., 1987. Scanning from coarse to fine spatial-scales in the human visual system after the onset of the stimulus, J. Opt. Soc. Am. A, 4:2006–2021. Watt, R. J., and M. J. Morgan, 1983. The recognition and representation of edge blur: evidence for spatial primitives in human vision, Vis. Res., 23:1465–1477. Watt, R. J., and M. J. Morgan, 1985. A theory of the primitive spatial code in human vision, Vis. Res., 25:1661–1674. Westheimer, G., 1973. Fourier analysis of vision, Invest. Ophthalmol. Vis. Sci., 12:86–87. Wilson, H. R., and D. Gelb, 1984. Modified line-element theory for spatial-frequency and width discrimination, J. Opt. Soc. Am. A, 1:124–131. Wilson, H. R., and W. A. Richards, 1989. Mechanisms of contour curvature discrimination, J. Opt. Soc. Am. A, 6:106–115. Witkin, A., and J. Tennenbaum, 1983. On the role of structure in vision, in Human and Machine Vision ( J. Beck, B. Hope, and A. Rosenfeld, eds.), London: Academic Press. Yang, Y., and R. Blake, 1991. Spatial frequency tuning of human stereopsis, Vis. Res., 31:1177–1189. Yuille, A. L., and T. Poggio, 1986. Scaling theorems for zerocrossings, IEEE Trans. Pattern Analysis Machine Intell., 8:15–25. Zhou, Y. X., and C. L. Baker, Jr., 1993. A processing stream in mammalian visual cortex neurons for non-Fourier responses, Science, 261:98–101. Zhou, Y. X., and C. L. Baker, Jr., 1994. Envelope-responsive neurons in areas 17 and 18 of cat, J. Neurophysiol., 72:2134– 2150. Zhou, Y. X., and C. L. Baker, Jr., 1996. Spatial properties of envelope-responsive cells in area 17 and 18 neurons of the cat, J. Neurophysiol., 75:1038–1050.

:     

1059

69

Spatial Channels in Vision and Spatial Pooling HUGH R. WILSON AND FRANCES WILKINSON

T    channels in vision seems to have arisen from an early analogy to the auditory system. In audition it was known from the work of von Békésy (reviewed 1960) and others that the curve plotting auditory thresholds as a function of temporal frequency was in fact the envelope of many narrower tuning curves, each reflecting the response of hair cells at a particular locus along the basilar membrane. Once cosine gratings were introduced into vision by Schade (1958), it became possible to measure the spatial contrast sensitivity function (CSF; see Chapter 68), which plots the relationship between visual sensitivity (reciprocal of threshold contrast) and spatial frequency. The analogy then became obvious and begged to be tested: if spatial frequency in cycles per degree is analogous to auditory temporal frequency in hertz, then perhaps the CSF might also represent the envelope of many more narrowly tuned visual channels, just as in the case of auditory thresholds. Indeed, Campbell and Robson (1968), who pioneered the exploration of spatial channels in vision, set out to study vision by exploring this analogy to audition. As is now well known, they and others subsequently confirmed that the visual system contains multiple spatial channels, each tuned to a narrower range of spatial frequencies and orientations than the visual system as a whole. Furthermore, these channels can now be rather closely linked to responses of orientation-selective neurons in primary visual cortex (V1). Early studies of spatial channels were based upon two assumptions. First, it was assumed that each channel was linear once the stimulus was strong enough to surpass a threshold value. Second, it was assumed that the channels processed visual information in parallel and independently of one another. Both of these assumptions have crumbled in recent years, and numerous studies have shown that linearity and independence are at best true only at the detection threshold. The demise of channel independence resulted from four major discoveries: contrast gain controls, collinear summation, low-level perceptual learning, and spatial pooling. Accordingly, this chapter will first review the major evidence leading to a characterization of spatial channels in vision. Against this background, modifications necessitated by these four major discoveries will be introduced.

1060

The contemporary channel concept thus becomes one in which visual analyses on multiple spatial scales are integrated through nonlinearity and spatial pooling to begin the extraction of visual information relevant to texture, form, motion, and depth perception. The links to these different forms of perception cannot be developed fully here, but the interested reader will find these connections discussed elsewhere (Wilson and Wilkinson, 1997).

Evidence for spatial channels Following the auditory analogy, Campbell and Robson (1968) measured detection thresholds for cosine gratings as a function of spatial frequency. At low temporal frequencies, the data (plotted as sensitivities which are reciprocals of threshold contrasts) describe a bandpass function, the CSF, with a peak at about 3 to 5 c/deg (Fig. 69.1). A convenient mathematical description of the CSF as a function of spatial frequency w is given by Wilson and Giese (1977): CSF (w) = Mw a exp(-w f )

(1)

where the peak frequency can easily be shown to be w = af. For low temporal frequency presentations, a ~ 1, M = 150, and f = 5 gives a good approximation to typical CSF data, and this function is plotted in Figure 69.1. Data obtained under transient or high temporal frequency conditions produce values of a ~ 0.4. To test the idea that the CSF might represent the envelope of many more narrowly tuned channels, Campbell and Robson (1968) conducted a summation experiment. Their logic was simple: measure the threshold for a complex spatial pattern composed of many widely separated spatial frequencies. If that threshold was simply a weighted sum of the thresholds for the individual components, the CSF must describe the properties of a single spatial channel that summed all spatial frequencies proportionately. If, however, the complex pattern reached threshold only when one of its spatial frequency components reached threshold independently, then the CSF must be the envelope of many spatial channels tuned to narrow ranges of spatial frequencies. The data, obtained using square wave gratings, were unambiguous: the visual system did not add up all spatial frequencies

Adapting Frequency

CSF

100

10

1

1

10

100

Spatial Frequency (cpd)

F 69.1. Typical CSF for sine wave gratings presented at low temporal frequency. The solid curve is from equation 1. The dashed notch depicts the effect of adaptation to the spatial frequency indicated.

equally but rather behaved like a bank of independent spatial channels, each sensitive to a different range of spatial frequencies. In the wake of this discovery, which supported the analogy to audition, the race was on to characterize the spatial frequency bandwidths of individual channels. The earliest approach employed spatial frequency adaptation, and results were published almost simultaneously by Pantle and Sekuler (1968) and Blakemore and Campbell (1969). In this technique, subjects viewed a high-contrast grating of fixed spatial frequency w for several minutes, moving their eyes across the bars to minimize conventional afterimage formation. Measurement of the CSF after this adaptation period revealed a notch of depressed sensitivities centered on w, as illustrated in Figure 69.1. Bandwidth estimates based on the adaptation technique fell in the 1 to 2 octave range. (One octave is a factor of 2; 2 octaves are a factor of 4; so n octaves are a factor of 2n.) Thus, adaptation studies suggested that spatial frequencies differing by a factor of 2 to 4 would be processed by independent spatial channels. Strong support for this range was provided by the classic study of Graham and Nachmias (1971), who showed convincingly that spatial frequencies differing by a factor of 3 (i.e., 1.6 octaves) were processed independently at threshold. The subsequent 14 years saw many attempts to measure spatial channel bandwidths more precisely using a variety of techniques. Several approaches using a technique known as subthreshold summation produced bandwidth estimates as narrow as 0.33 octave (Kulikowski and King-Smith, 1973; Sachs et al., 1971). Subsequent work, however, showed that these figures were artifactually narrowed as a consequence of the spatial beat pattern that occurs when cosine gratings

of very similar spatial frequencies are added together (see Wilson, 1991, and Wilson and Wilkinson, 1997, for further discussion). An oblique masking technique that was not subject to these problems was developed in our laboratory (Wilson et al., 1983). High-contrast masking gratings oriented at an angle of 15 degrees from vertical were superimposed on vertical test patterns with a 1 octave bandwidth (sixth spatial derivatives of Gaussians, D6s). In masking experiments the threshold elevation is defined as the ratio of the test pattern threshold in the presence of the masking grating to the test threshold measured with no mask present, so a threshold elevation of 1.0 would indicate that the mask had no effect on the test. In our experiments, the spatial frequencies of both test D6s and masking gratings were varied in half-octave steps from 0.25 up to 22.6 c/deg so that every test D6 was paired with a wide range of mask spatial frequencies. The resulting 14 threshold elevation curves, each obtained with a different D6 test spatial frequency, were fit quantitatively with a set of just six underlying visual channels tuned to peak frequencies of 0.8, 2.0, 2.8, 4.0, 8.0, and 16.0 c/deg (Wilson et al., 1983). The spatial frequency tuning curves for four of these visual channels are plotted in Figure 69.2. Note that the envelope of these curves describes the CSF fairly well. Although the notion of just six visual channels in the fovea remains controversial, it is nevertheless true that these six suffice to encode all of the spatial frequency information present in the stimulus; more than six channels would be redundant. Furthermore, shifting peak channel sensitivities to lower spatial frequencies in the visual periphery produces an effective continuum of channel tunings across the visual system (Swanson and Wilson, 1985). In a subsequent experiment, Phillips and Wilson (1984) used masking to measure orientation bandwidths, and these were found to vary from ±30 degrees at half amplitude for the lowest spatial frequencies down to ±15 degrees for the highest spatial frequency mechanisms. These estimates of both spatial frequency and orientation bandwidths obtained by masking were in good quantitative agreement with bandwidths of single neurons in macaque area V1 as measured by De Valois et al. (1982). A graph comparing both spatial frequency and orientation bandwidths in macaques and humans can be found elsewhere (Wilson, 1991). The two-dimensional spatial receptive fields, RF(x, y ), of these visual channels could be well described using combinations of Gaussian functions RF (x, y) = A{exp(-x 2 s12 ) - B exp(-x 2 s22 ) + C exp(-x 2 s23 )} exp(- y 2 s2y )

(2)

and a table of the various constants can be found in Wilson (1991). Furthermore, the fact that both spatial frequency and orientation bandwidths decrease with increasing peak frequency indicates that spatial processing by the visual system

  :     

1061

Increment Threshold

0º 22.5º

Sensitivity

100

10

10 1 0.01

2 cpd 4 cpd 8 cpd 16 cpd

1 1

10

100

Spatial Frequency (cpd)

F 69.2. CSF with four underlying spatial frequency selective visual channels. The channels are most sensitive at 2, 4, 8, and 16 c/deg, and their envelope provides a good fit to the CSF above about 0.75 c/deg. (Data from Wilson et al., 1983.)

cannot be accurately described by a wavelet transform. Indeed, visual channels are performing neither a wavelet nor a Fourier transform (Wilson and Wilkinson, 1997). Rather, the data corroborate the early hypothesis of Thomas (1970) that visual channels reflect properties of cortical receptive fields of varying size and preferred orientation. In order to infer the tuning curves in Figure 69.2, it was necessary to determine how masking varied with mask contrast. In a pioneering study, Nachmias and Sansbury (1974) had shown that threshold elevations due to masking were described by the “dipper-shaped” function of mask contrast depicted in Figure 69.3. As mask contrast increased from zero, test pattern thresholds initially decreased by a factor of about 2. At higher mask contrasts, however, test pattern thresholds rose substantially, resulting in large threshold elevations. Nachmias and Sansbury suggested that the masking dipper function reflected properties of a contrast nonlinearity in visual channels that was accelerating at subthreshold contrasts but compressive at suprathreshold contrasts. A suitable form for this function was found to be N +e

F (C ) =

MC aN +C N

(3)

where M and a are constants. Typically, 2 £ N £ 4, and e ~ 0.5 (Wilson, 1980; Wilson et al., 1983). Note that if e = 0, this is just a Naka-Rushton (1966) function of the type that has been successfully used to describe the responses of V1 cortical neurons (Albrecht and Hamilton, 1982; Sclar et al., 1990; see Chapter 47). This contrast nonlinearity generates the dipper function in Figure 69.3 on the assumption that contrast increments at threshold, D, are defined by the relation

1062

, ,   

0.1

1 Mask Contrast (%)

10

F 69.3. Typical contrast increment threshold functions. The solid curve illustrates grating contrast increment thresholds as a function of mask grating contrast when mask and test gratings are at the same orientation, 0 degrees. Note the characteristic dip at the location of the arrow, which was first reported by Nachmias and Sansbury (1974). The dashed line shows the effect of a masking grating at an angle of 22.5 degrees relative to the test grating. As reported by Foley (1994), the dip disappears in this case.

F (C + D) - F (C ) = 1

(4)

Masking data were corrected for this contrast nonlinearity to obtain the tuning curves plotted in Figure 69.2 (Wilson et al., 1983). This completes a brief historic review of visual channels as they were understood around 1990 (Graham, 1989; Wilson, 1991). To predict the response of an array of psychophysically defined cortical neurons, the receptive field description in equation 2 was first convolved with the luminance profile of the stimulus. Following this, the nonlinearity in equation 3 was applied pointwise to the convolution result to produce an estimate of neural responses to the pattern in question (see Wilson, 1991, for details). Thus, visual channels were assumed to process the retinal image entirely independently and in parallel. As we shall see, this formulation was radically altered by research during the subsequent decade.

Contrast gain controls The first challenge to independent channels came from theoretical work on contrast gain controls by Heeger (1992). Heeger’s gain control model was originally designed to describe responses of V1 simple cells, but here we shall adapt it to the discussion of visual channels defined psychophysically. The first novel idea in this model is that the function in equation 3 does not represent a static nonlinearity in each independent channel. Instead, it represents a combination of two processes that evolve over time. The first of these is an accelerating nonlinear response of each channel to its optimal stimulus, which we shall take to be of the form CN (Heeger focused on the special case N = 2). The second process is nonlinear feedback that produces the denomina-

tor in equation 3 when the system reaches the steady state. To see how this works in the simplest case, where only the ith channel is stimulated by a pattern of contrast Ci, consider the following pair of equations which describe the temporal response Ri of the ith channel as well as the response of the gain control signal G: C iN dRi = - Ri + N ( M - G ) dt s dG = -G + Ri dt

(4)

These form a pair of coupled, nonlinear differential equations describing a feedback loop from Ri to G and then back to Ri. The first equation is cast in the form of conduction dynamics similar to those in the Hodgkin-Huxley equations (see Carandini et al., 1999; Wilson, 1999). The dynamics in equation 4 are implicit in Heeger’s (1992) model, which focused on the steady state or equilibrium solution. The steady state is obtained by setting dRi/dt = 0 and dG/dt = 0 and then solving the simultaneous algebraic equations (Wilson, 1999). The solution to the second equation, G = Ri, can be substituted into the first to yield the following expression for Ri: Ri =

MC iN s N + C iN

(5)

This is just a Naka-Rushton (1966) function with a form very similar to equation 3. In equation 5, however, the denominator arises as a result of the nonlinear feedback process described by equation 4, so the nonlinearity is dynamic rather than static. The second, and even more important idea in Heeger’s (1992) model is that the feedback should incorporate not just Ri, but also a sum of responses across orientations and a range of spatial frequencies. To implement this, Heeger postulated that the steady state of such a network would be M C iN +e Ri = N s + Â j C Nj

(6)

where the summation ranges across all orientations and a range of spatial frequencies. In Heeger’s cortical cell model e = 0, but as already noted, e > 0 for psychophysical channels. (It should be mentioned that this expression is easy to derive for the equilibrium of a feedforward network, but it is an approximation for feedback networks.) Equation 6 indicates that the response of each visual channel is normalized by the sum of the stimuli to a wide range of visual channels, hence the concept of a gain control. The implication is that visual channels subject to a gain control are not independent. Rather, they are coupled through the divisive nonlinearity in the denominator of equation 6. A critical psychophysical test of equation 6 was performed by Foley (1994). Foley masked vertical cosine gratings with

cosine gratings at a range of different orientations. When vertical masks were used with vertical test gratings, threshold elevations as a function of mask contrast produced the characteristic dipper function illustrated by the solid line in Figure 69.3. When the masking grating orientation fell outside the bandwidth of the channel responding to the test grating, however, the dipper disappeared and the data assumed the shape of the dashed curve in Figure 69.3. This result is predicted by equation 6, and in fact, the curves in Figure 69.3 were generated from the equation. The static nonlinearity in equation 3, however, predicts that masks outside the orientation bandwidth should have no effect at all. Thus, Foley’s (1994) study provided crucial support for the form of Heeger’s (1992) contrast gain control mechanism embodied in equation 6, which links channels through a divisive pooling process. Physiological support for contrast gain controls in V1 was first reported by Bonds (1989, 1991). He showed that V1 neurons could be actively inhibited by gratings outside their orientation bandwidth. Furthermore, this inhibition served to sharpen the orientation tuning of individual neurons. This implies that contrast gain controls in V1 play an active role in the formation of cortical-oriented receptive fields, a point to be amplified below.

Collinear facilitation In addition to gain controls, another line of research has provided evidence for a different type of functional interaction among visual channels. This research was pioneered by Polat and Sagi (1993, 1994). Instead of masking a target pattern with a superimposed mask, they chose to separate the mask spatially from the target. Based on preliminary experiments, both test and mask patterns were Gabor functions with a space constant l equal to the wavelength of the dominant spatial frequency: G (x, y) = cos(2p x l) exp(- (x 2 + y 2 ) l2 )

(7)

Polat and Sagi (1993, 1994) used spatial configurations comprising two mask Gabors flanking a central target Gabor, as depicted in Figure 69.4. Threshold elevations for the central Gabor were then measured as a function of the separation of the masking Gabors in multiples of l. For a value of l = 0.075 degree (spatial frequency of 13.3 c/deg), Polat and Sagi (1993) found that the mask reduced the threshold of the target Gabor by about 50% when the mask was separated by 3l to 4l. This facilitation by the mask, however, was largely specific to the configuration in Figure 69.4A (separation of 4l), as little or no target facilitation was found when the mask orientations were varied by 30 degrees, as in Figure 69.4B. A subsequent study showed that when both target and mask Gabors were oriented 45 degrees away from the line connecting the centers of the patterns, as in Figure 69.4C, there was again almost no facilitation of the

  :     

1063

F 69.4. Spatial configurations of Gabor patches used in studies by Polat and Sagi (1993, 1994). Thresholds were measured for the central patch as a function of the orientation of the top and bottom flanking patches. Significant facilitation (threshold

decrease) was found in condition a, where the patches form a collinear configuration. Configurations b and c, which lack collinearity, failed to produce significant facilitation.

target Gabor (Polat and Sagi, 1994). Control experiments showed that this pattern of results did not depend on either the spatial frequency of the Gabors or their absolute orientations. These experiments clearly indicate that there is lateral facilitation among orientation-selective visual channels that are spatially aligned along their preferred orientation, as in Figure 69.4A. Neither spatial alignment alone (Fig. 69.4B) nor orientation alignment away from the common spatial alignment (Fig. 69.4C) generates facilitation. This pattern of results provides clear evidence for collinear facilitation: lateral spatial facilitation along a line defined by a channel’s preferred orientation. Several further studies have also provided evidence for collinear facilitation. Field et al. (1993) studied contours defined by chains of Gabor functions embedded in a random field of Gabors. They showed that subjects could detect these contours only when the Gabors defining the contour were either collinear or tangent to smooth curves. From this they inferred the existence of a recurrent association field linking spatially adjacent visual channels with similar orientations (Chapter 70). Similarly, Kovács and Julesz (1993) argued that collinear facilitation could explain the salience of Gabor-defined circular contours in Gabor background noise. Finally, Wilson et al. (2001) demonstrated that dominance waves in binocular rivalry travel faster around an annulus of concentric contour than they do around an annulus of radial contour. This wave speed increase was interpreted as resulting from collinear facilitation. Anatomical and physiological evidence supports the existence of connections mediating collinear facilitation in primary visual

cortex. Studies on both cats and monkeys in several laboratories have shown that long-range excitatory connections preferentially occur between orientation columns having the same preferred orientation and an approximately collinear spatial arrangement (Das and Gilbert, 1995; Malach et al., 1993; Polat et al., 1998). Collinear facilitation provides yet another indication that visual channels do not process the retinal image independently. Rather, they interact via long-range excitatory connections to enhance contour salience. Furthermore, collinear facilitation demonstrates that visual channels cannot be fully described by oriented visual filters such as that in equation 2. Rather, orientation selectivity itself is now thought to result in part from the action of recurrent collinear connections (Vidyasagar et al., 1996), and several recent V1 models have implemented this idea in detail (McLaughlin et al., 2000; Somers et al., 1998). The picture of visual channels that has emerged comprises a network of excitatory and inhibitory neurons like those diagrammed in Figure 69.5. Each gray patch represents a local neural ensemble containing multiple oriented units (only three are shown for clarity). All of these provide excitation to an inhibitory gain control neuron I, which in turn provides divisive feedback inhibition to the oriented units. In addition, oriented units in ensembles at different spatial locations form mutual long-range excitatory connections, provided that their locations are collinear. Several of these collinear connections are depicted by the double-headed arrows between identical orientations in different local networks. This figure captures the salient aspects of visual channel structure as understood today.

1064

, ,   

I

I

I

I

F 69.5. Schematic network for V1 channel interactions. Local modules enclosed by gray rectangles incorporate gain controls that link channels with different preferred orientations (only three are shown for clarity) via inhibitory feedback from neurons I. Facilitation is depicted by the arrows between modules linking channels with similar preferred orientations that are collinear in space. Both collinear facilitation and divisive gain control circuits contribute to channel orientation selectivity.

Perceptual learning The description of visual channels in terms of oriented filters (equation 2) implies that these filters have fixed properties resulting from some combination of genetics and early visual development. As discussed in the previous two sections, however, visual filter characteristics are now known to reflect network interactions including contrast gain controls and collinear facilitation. If this is the case and connections (synapses) in these networks could be modified by experience, then it would be predicted that visual filters could be changed as a result of perceptual learning. A model of early synaptic modification resulting from perceptual experience was first proposed in the context of spatial frequency adaptation (Wilson, 1975), where it was shown that changes in inhibitory connection strengths based on a Hebbian correlation rule would produce appropriate changes in visual filter shapes. Early experimental evidence for visual improvement resulting from perceptual learning was provided by Ball and Sekuler (1982). They trained subjects on discrimination of motion direction and found that subjects’ performance improved over time. Furthermore, the improvements were specific to the direction of motion used during training and lasted for at least 10 weeks. These data are consistent with the hypothesis that bandwidths of direction-selective neurons become narrower as a result of training, thereby reducing direction discrimination thresholds. The hypothesis that visual channel bandwidths are narrowed as a result of perceptual learning was tested directly by Saarinen and Levi (1995). McKee and Westheimer (1978) had already demonstrated that vernier acuity could be improved about twofold with extensive training. To deter-

mine whether such vernier improvements resulted from narrowing of orientation bandwidths, Saarinen and Levi first measured orientation bandwidths using a masking paradigm; then they trained subjects on a vernier task until perceptual learning occurred. Subsequent measurement of orientation bandwidths revealed that they had indeed become narrower by about a factor of 2, and the degree of narrowing correlated well with the extent of individual improvement on the vernier task. Reference to Figure 69.5 suggests two ways in which synaptic modification might cause sharpening of channel bandwidths. One possibility would be an increase in the strength of synapses mediating collinear facilitation between oriented units that were simultaneously activated during training. However, such excitatory synaptic facilitation would have to be tempered by some form of inhibitory gain control increase to prevent runaway facilitation from triggering seizure-like activity such as migraine auras. An alternative model for the alteration of channel orientation bandwidths involves selective increases in the strength of inhibitory synapses in the local gain control circuits (Wilson and Humanski, 1993). Increases in inhibition from neighboring orientations in the model serve to sharpen the orientation tuning of the remaining units (which can also account for the tilt aftereffect). It is most likely, however, that a combination of excitatory and inhibitory synaptic modifications underlies perceptual learning, but further work will be required to elucidate the details.

Spatial pooling While collinear facilitation may occur along nearly abutting contours in densely populated arrays of oriented elements, there is ample evidence that close juxtaposition of visual information may also interfere with the processing of individual elements. This problem is particularly marked in the visual periphery, where the phenomenon is known as lateral masking. Early studies of lateral masking typically employed letters or other complex elements, and the loss of processing ability in the periphery under conditions of element proximity or crowding was viewed as masking or interference. Clearly, lateral interactions can interfere with perceptual processing in situations in which individual visual elements provide different and essential information (as in letter strings making up words). However, in the presence of an intact fovea, such a test situation is highly artificial in that an individual would normally fixate to place such stimuli on the fovea where such lateral interaction does not occur or is greatly reduced. Only in abnormal conditions such as amblyopia is crowding a problem in foveal vision. This raises the possibility that lateral masking in the periphery actually represents the

  :     

1065

pooling or combination of information either as a means of economy of processing or for some other purpose. For example, when the elements are repetitive, lateral masking may reflect a trade-off of detailed local information in exchange for the encoding of global stimulus properties. Thus, a visual texture may, based on its global statistical properties, be appreciated as coarse or sparse, of high or low luminance and contrast, periodic or random without detailed knowledge of the fine structure of each component element. Wilkinson et al. (1997) demonstrated that discrimination thresholds in the periphery are elevated for three properties of a target Gabor element—contrast, spatial frequency, and orientation—when placed in the center of an array of similar elements, and proposed that their data could be explained if complex cells with large receptive fields pooled the input of several subunits. Consistent with the notion that this pooling might be involved in texture analysis were the later findings of this group (Ellemberg et al., 1998) that not only discrimination thresholds, but also the percept of Gabor contrast and spatial frequency were affected by proximity, again suggesting pooling. Altered perceived contrast and contrast discrimination thresholds have also been reported by Snowden and Hammett (1998) using gratings in an annular format; they too reported much stronger effects in the periphery than in the fovea. These authors proposed that this surround masking is just a more general case of pattern masking. However, a more recent study (Parkes et al., 2001) on orientation masking provided evidence that local orientation signals from target and surround stimuli are combined in a way that reflects contributions of the individual elements rather than suppression or masking of the central target information by the surrounding distractors. Thus, some form of weighted pooling is supported. Interestingly, these authors also demonstrated that pooling can occur in the fovea when the subject does not have prior information about the location of a target in an array, suggesting that focal attention may be able to drive a switch from pooling to nonpooling in foveal vision. In this context, it is interesting to note that unpublished work on texture perception from our lab (Wilkinson and Peterson, 1989) provides evidence that the visual system may be organized to process global texture and local feature information in parallel from the parafovea and fovea, respectively. Textural judgments based on full-field stimuli were comparable to those based on parafoveal stimulation alone but not on foveal stimulation alone. This implies that when visually inspecting a textured surface, we may be encoding the contrast, granularity, and other global features through parafoveal pooling at the same time that we use our foveas to discern that nature of the texture’s components. This possibility certainly merits further experimental investigation.

1066

, ,   

Higher-level pooling Work in several laboratories over the past few years has provided strong evidence that as patterned input is channeled up through the visual pathways, there is increasing pooling of information across spatial channels. This is clearly the case in both the motion and form domains; here we restrict our summary to the latter. From the earliest work on the stimulus preferences of neurons in higher cortical areas such as inferotemporal cortex (Gross et al., 1972), it has been evident that information conveyed through the simple spatial channels of this discussion must be pooled in complex combinations at higher levels in order to represent objects of such complexity as faces. However, until recently there was little available evidence as to the stages of this process beyond the recognition of a hierarchical anatomical organization with abundant cross-talk between levels. In a series of recent psychophysical studies (Wilson and Wilkinson, 1998; Wilson et al., 1997), we have demonstrated that information is summed across orientations in highly specific configural ways to extract various global structures from the retinal input (ellipses, radial configurations). Functional imaging work from our laboratories (Wilkinson et al., 2000) provides converging evidence with human event-related potentials (Allison et al., 1999) and primate single unit electrophysiology (Gallant et al., 1993) pointing to cortical area V4 as a crucial intermediate level of processing where such pooling occurs.

Discussion A decade ago, visual channels were viewed as a rigid basis set of oriented, spatial frequency tuned filters that processed the retinal image independently and in parallel. Over the past 10 years, channel independence has completely dissolved, to be replaced by a model in which channel properties emerge from a network of excitatory and inhibitory interactions epitomized by collinear facilitation and divisive contrast gain controls. These networks are also malleable in their connectivity, as manifested by perceptual learning. Furthermore, there is now evidence that spatial pooling of channel outputs at higher cortical levels represents a subsequent processing stage in which visual shapes are extracted from the input and measured. Granted that excitatory and inhibitory interactions mediate channel properties, let us consider their functional significance. Contrast gain controls produced by divisive feedback serve to correct the neural representation of an image for variations in stimulus contrast. The fact that humans can recognize a face in sharp directional lighting as well as in diffuse, almost foggy conditions attests to the effectiveness of contrast gain controls. Collinear facilitation pro-

duced by weak long-range excitatory feedback serves to bind continuous contours together and enhance their salience as a prelude to further processing. Collinear facilitation does not, however, contribute to the measurement of object geometry. In peripheral vision, channel responses are pooled to extract statistical characteristics of visual textures. Finally, configural pooling extracts significant geometric shapes (ellipses, radial structure) from the retinal image and begins their measurement. Current research is driven by the hypothesis that channel interactions evolved to perform important computations for shape analysis in form vision. In closing, we mention two challenging but unresolved issues concerning the role of multiple spatial frequency channels in vision. One likely function of interactions across spatial frequency scales is to maintain size constancy. It seems likely that pooling to extract shape information occurs in parallel on several spatial scales. Inhibition and subsequent pooling across scales could then produce neural units able to respond to a shape independent of size. A second possibility is that lower spatial frequencies provide context for the incorporation of object details encoded at higher spatial frequencies. These and other aspects of visual channel interactions provide a rich source of research hypotheses for the future. REFERENCES Albrecht, D. G., and D. B. Hamilton, 1982. Striate cortex of monkey and cat: contrast response function, J. Neurophysiol., 48:217–237. Allison, T., A. Puce, D. D. Spencer, and G. McCarthy, 1999. Electrophysiological studies of human face perception. I: potentials generated in occipitotemporal cortex by face and non-face stimuli, Cereb. Cortex, 9:415–430. Ball, K., and R. Sekuler, 1982. A specific and enduring improvement in visual motion discrimination, Science, 218:697–698. Blakemore, C., and F. W. Campbell, 1969. On the existence of neurones in the human visual system selectively sensitive to the orientation and size of retinal images, J. Physiol., 203:237– 260. Bonds, A. B., 1989. Role of inhibition in the specification of orientation selectivity of cells in the cat striate cortex, Vis. Neurosci., 2:41–55. Bonds, A. B., 1991. Temporal dynamics of contrast gain in single cells of the cat striate cortex, Vis. Neurosci., 6:239–255. Campbell, F. W., and J. G. Robson, 1968. Application of Fourier analysis to the visibility of gratings, J. Physiol., 197:551–566. Carandini, M., D. J. Heeger, and J. A. Movshon, 1999. Linearity and gain control in V1 simple cells, in Cerebral Cortex, vol. 13, Models of Cortical Circuitry (P. S. Ulinski and E. G. Jones, eds.), New York: Plenum, pp. 401–443. Das, A., and C. D. Gilbert, 1995. Long range cortical connections and their role in cortical reorganization revealed by optical recording of cat primary visual cortex, Nature, 375:780–784. De Valois, R. L., D. G. Albrecht, and L. G. Thorell, 1982. Spatial frequency selectivity of cells in macaque visual cortex, Vis. Res., 22:545–559.

Ellemberg, D., F. Wilkinson, H. R. Wilson, and A. S. Arsenault, 1998. Apparent contrast and spatial frequency of local texture elements, J. Opt. Soc. Am. A, 15:1733–1739. Field, D. J., A. Hayes, and R. F. Hess, 1993. Contour integration by the human visual system: evidence for a local “association field,” Vis. Res., 33:173–193. Foley, J. M., 1994. Human luminance pattern vision mechanisms: masking experiments require a new model, J. Opt. Soc. Am. A, 1710–1719. Gallant, J. L., J. Braun, and D. C. Van Essen, 1993. Selectivity for polar, hyperbolic, and Cartesian gratings in macaque visual cortex, Science, 259:100–103. Graham, N., 1989. Visual Pattern Analyzers, New York: Oxford University Press. Graham, N., and J. Nachmias, 1971. Detection of grating patterns containing two spatial frequencies: a comparison of singlechannel and multiple-channel models, Vis. Res., 11:251–259. Gross, C. G., C. E. Rocha-Miranda, and D. B. Bender, 1972. Visual properties of neurons in inferotemporal cortex of the macaque, J. Neurophysiol., 35:96–111. Heeger, D. J., 1992. Normalization of cell responses in cat striate cortex, Vis. Neurosci., 9:181–197. Kovács, I., and B. Julesz, 1993. A closed curve is much more than an incomplete one: effect of closure in figure-ground segmentation, Proc. Natl. Acad. Sci. USA, 90:7495–7497. Kulikowski, J. J., and P. E. King-Smith, 1973. Spatial arrangement of line, edge, and grating detectors revealed by subthreshold summation, Vis. Res., 13:1455–1478. Malach, R., Y. Amir, M. Harel, and A. Grinvald, 1993. Relationship between intrinsic connections and functional architecture revealed by optical imaging and in vivo targeted biocytin injections in primary striate cortex, Proc. Natl. Acad. Sci. USA, 90:10469–10473. McKee, S. P., and G. Westheimer, 1978. Improvement in vernier acuity with practice, Percept. Psychophys., 24:258–262. McLaughlin, D. C., R. Shapley, J. Shelley, and D. J. Wielaard, 2000. A neuronal network model for macaque primary visual cortex (V1): orientation selectivity and dynamics in the input layer 4Ca, Proc. Natl. Acad. Sci. USA, 97:8087–8092. Nachmias, J., and R. V. Sansbury, 1974. Grating contrast: discrimination may be better than detection, Vis. Res., 14:1039–1042. Naka, K. I., and W. A. Rushton, 1966. S-potentials from colour units in the retina of fish, J. Physiol., 185:584–599. Pantle, A., and R. Sekuler, 1968. Size detecting mechanisms in human vision, Science, 162:1146–1148. Parkes, L., J. Lund, A. Angelucci, J. A. Solomon, and M. Morgan, 2001. Compulsory averaging of crowded orientation signals in human vision, Nat. Neurosci., 4:739–744. Phillips, G. C., and H. R. Wilson, 1984. Orientation bandwidths of spatial mechanisms measured by masking, J. Opt. Soc. Am. A, 1:226–232. Polat, U., K. Mizobe, M. W. Pettet, T. Kasamatsu, and A. M. Norcia, 1998. Collinear stimuli regulate visual responses depending on cell’s contrast threshold, Nature, 391:580–584. Polat, U., and D. Sagi, 1993. Lateral interactions between spatial channels: suppression and facilitation revealed by lateral masking experiments, Vis. Res., 33:993–999. Polat, U., and D. Sagi, 1994. The architecture of perceptual spatial interactions, Vis. Res., 34:73–78. Saarinen, J., and D. M. Levi, 1995. Perceptual learning in vernier acuity: what is learned? Vis. Res., 35:519–527.

  :     

1067

Sachs, M. B., J. Nachmias, and J. G. Robson, 1971. Spatialfrequency channels in human vision, J. Opt. Soc. Am., 61: 1176–1186. Schade, O. H., 1958. On the quality of color television images and the perception of color detail, J. Soc. Motion Pict. Television Eng., 67:801–819. Sclar, G., J. H. R. Maunsell, and P. Lennie, 1990. Coding of image contrast in central visual pathways of the macaque monkey, Vis. Res., 30:1–10. Snowden, R. J., and S. T. Hammett, 1998. The effects of surround contrast on contrast thresholds, perceived contrast and contrast discrimination, Vis. Res., 38:1935–1945. Somers, D. C., E. V. Todorev, A. G. Siapas, L. J. Toth, D. S. Kim, and M. Sur, 1998. A local circuit approach to understanding integration of long range inputs in primary visual cortex, Cereb. Cortex, 8:204–217. Swanson, W. H., and H. R. Wilson, 1985. Eccentricity dependence of contrast matching and oblique masking, Vis. Res., 25:1285– 1295. Thomas, J. P., 1970. Model of the function of receptive fields in human vision, Psych. Rev., 77:121–134. Vidyasagar, T. R., X. Pei, and M. Volgushev, 1996. Multiple mechanisms underlying the orientation selectivity of visual cortical neurones, TINS, 19:272–277. von Békésy, G., 1960. Experiments in Hearing (E. G. Wever, trans.), New York: McGraw-Hill. Wilkinson, F., T. W. James, H. R. Wilson, J. S. Gati, R. S. Menon, and M. A. Goodale, 2000. An fMRI study of the selective activation of human extrastriate form vision areas by radial and concentric gratings, Curr. Biol., 10:1455–1458. Wilkinson, F., and R. Peterson, 1989. Spatial limits to the perception of textural coherence, Invest. Ophthalmol. Vis. Sci. Suppl., 30:254.

1068

, ,   

Wilkinson, F., H. R. Wilson, and D. Ellemberg, 1997. Lateral interactions in peripherally viewed texture arrays, J. Opt. Soc. Am. A, 2057–2068. Wilson, H. R., 1975. A synaptic model for spatial frequency adaptation, J. Theoret. Biol., 50:327–352. Wilson, H. R., 1980. A transducer function for threshold and suprathreshold human vision, Biol. Cybernet., 38:171–178. Wilson, H. R., 1991. Psychophysical models of spatial vision and hyperacuity, in Spatial Vision (D. Regan ed.), London: MacMillan, pp. 64–86. Wilson, H. R., 1999. Spikes, Decisions, and Actions: Dynamical Foundations of Neuroscience, Oxford: Oxford University Press. Wilson, H. R., R. Blake, and S.-H. Lee, 2001. Dynamics of travelling waves in visual perception, Nature, 412:907–910. Wilson, H. R., and S. Giese, 1977. Threshold visibility of frequency gradient patterns, Vis. Res., 17:1177–1190. Wilson, H. R., and R. Humanski, 1993. Spatial frequency adaptation and contrast gain control, Vis. Res., 33:1133–1149. Wilson, H. R., D. K. McFarlane, and G. C. Phillips, 1983. Spatial frequency tuning of orientation selective units estimated by oblique masking, Vis. Res., 23:873–882. Wilson, H. R., and F. Wilkinson, 1997. Evolving conncepts of spatial channels in vision: from independence to nonlinear interactions, Perception, 26:939–960. Wilson, H. R., and F. Wilkinson, 1998. Detection of global structure in Glass patterns: implications for form vision, Vis. Res., 38:2933–2947. Wilson, H. R., F. Wilkinson, and W. Asaad, 1997. Concentric orientation summation in human form vision, Vis. Res., 37: 2325–2330.

70

Contour Integration and the Lateral Connections of V1 Neurons DAVID J. FIELD AND ANTHONY HAYES

T    repeated claim that the only man-made object that can be seen from space is the Great Wall of China (Fig. 70.1): a structure that is in some sections over 2500 years old, and that snakes over some 6000 km of countryside. In reality, at the altitude of orbital flights, astronauts report seeing a variety of man-made structures, including roads and large seaway projects such as the Suez and Panama Canals. What characterizes many of these structures is the length and smooth continuity of the contours they create across the surface of the earth. The notion that continuity is important to visual perception was a central idea of the Gestalt psychologists, who, in the first half of the twentieth century, described a set of perceptual grouping principles that included the law of good continuation. In formulating their laws, the Gestalt psychologists had rebelled against the belief that perception could be described as the consequence of simple accretion of visual elements. Over the past 10 years, cognitive neuroscience has renewed its interest in the representation of contours and continuity. Researchers in visual anatomy, neurophysiology, computer science, and visual psychophysics have combined their approaches to develop models of how contours are perceived and integrated by the visual system. The reasons for this interest are several. Perhaps of primary importance is the fact that up to recently, much of the work on vision has concentrated on the properties of single neurons. Neuroscience data have provided considerable insight into the properties of the individual neurons that occur along the visual pathway. This work suggests that in the early stages of visual processing, the image of our visual environment is transformed into the responses of large arrays of neurons, each selective to properties such as orientation, position, spatial frequency, and direction of motion. Indeed, it has been argued that these basic properties of the visual system may produce a solution that is close to optimal for describing our natural environment (e.g., Field, 1987; see Chapter 108, for review). However, the question remains of how this information, encoded by different neurons, is integrated into the perception of whole objects and scenes. One common theme is that the visual system does so by building a hierar-

chy of ever more complex receptive fields through series of feedforward connections. In this chapter, we review recent work that takes a different approach. This work suggests that, as early as primary visual cortex, neurons cannot be treated as simple feedforward devices that merely receive input from the retina. Their response properties depend on a complex relationship between the neighboring neurons and their input. In particular, this recent work demonstrates that neurons in primary visual cortex make use of long-range lateral connections that allow integration of information from far beyond the classical receptive field, and the evidence suggests that these connections are involved in associating neurons that respond along the length of a contour. The classical description of a cortical neuron in primary visual cortex is that of a neuron, with feedforward inputs from the lateral geniculate nucleus, whose pattern of connections produces the receptive field profiles described in the 1960s by Hubel and Weisel (see Hubel, 1988, for review). This classical receptive field of a visual neuron is defined as the area of the visual field that has the capacity to modify the resting potential of the neuron. However, while this basic feedforward linear model of the simple-cell receptive field has been invoked to explain a wide variety of perceptual phenomena—and is at the heart of a wide range of modeling studies—it is essentially wrong. Some of the earliest studies that measured receptive field properties of cortical neurons recognized that stimuli presented outside the classical receptive field can modify the activity of the neuron, even if those regions by themselves cannot effect a response (e.g., Maffei and Fiorentini, 1976). The neurons in primary visual cortex show a variety of interesting nonlinearities, with many occurring within the classical receptive field. However, the nonlinearities that are of interest to us here are the responses to regions outside the classical receptive field. Stimulation of these areas typically does not produce a response but can modulate the activity of the neuron. This modulation in activity has commonly been described as inhibitory, and a variety of theories have been proposed (e.g., Allman et al., 1985). One popular account has argued that this inhibition can serve to

1069

B

A F 70.1. The Great Wall of China is one of a small number of man-made structures visible from space. The length and continuity of the contour etched on the surface of the earth by the wall allow the structure to be visible at considerable distances. The figure in B is an edge map of the picture of the Great Wall (A) using a simple (Sobel) edge detector. In the classical view, a neuron in

primary visual cortex responds to only a limited region of the visual field and responds to a restricted range of stimulus orientation. To see the contour formed by the wall as a single entity, some process must integrate the different pieces of the contour. (®Christopher Liu/ChinaStock, All Rights Reserved.)

normalize the neuron’s response and make more efficient use of the neuron’s limited dynamic range (Heeger, 1992; Schwartz and Simoncelli, 2001). In this chapter, we concentrate on a new theory to account for some of these nonlinear effects. This theory proposes that the nonclassical surrounds of receptive fields are intimately involved in a process called contour integration. We do not mean to imply that contour integration is their only role; however, the evidence suggests that it is one role. Indeed, the evidence suggests that some of the effects that have given rise to the notion of nonclassical surrounds are generated by the active grouping or association of cells in neighboring regions of the visual field. In accord with the term receptive field, we have used the term association field to describe the region of associated activity (Field et al., 1993), while others have used the term integration field (e.g., Chavane et al., 2000) or contextual field (e.g., Phillips and Singer, 1997; see also Chapter 113). In the following pages, we address four questions and explore some of the research that is providing answers. The questions are as follows: (1) What is contour integration, and why is it important? (2) What do the anatomy and physiology suggest about the underlying mechanism? (3) What does the behavior of individuals—humans and nonhuman primates—suggest about the underlying mechanism? (4) What insights are provided by computational models of the process? We should note that when putting this review together, we discovered over 500 papers published in the past 10 years that bear directly on these issues of integration. Recently, a number of excellent reviews and discussions have been published on the topic or on associated topics. We recommend Fitzpatrick (2000), Gilbert (1998), and Callaway (1998) for discussions of anatomy and physiology; Polat (1999) and Hess and Field (1999) for reviews of psychophysics; and Li (1998) and Yen and Finkel (1998) for their comprehensive

discussions of the computational issues. In the limited space of this chapter, therefore, we will concentrate on a few issues which we feel have not received the primary attention of the above authors.

1070

, ,   

What is contour integration? Consider the image shown in Figure 70.1A. Because reflection and illumination vary across the different surfaces, occlusions between surfaces commonly produce a luminance discontinuity (i.e., an edge), as shown in the edge map in Figure 70.1B. However, edges in scenes do not occur only at occlusions. They may also arise from textures within surfaces, as well as from shading discontinuities. In the 1980s, a number of modeling studies were published that proposed computational strategies that would help to identify which of the edges in a scene made up the principal boundaries of an object. Under the assumption that boundary edges were likely to extend over large regions of the visual field, the computations were designed to extract only those edges that were continuous over an extended area. The algorithms that were developed were based on the assumption that the problem could be at least partially solved by integrating over neighboring regions that had similar orientations. However, although some of these integration models included, or were derived from, known physiology (e.g., Grossberg and Mingolla, 1985; Parent and Zucker, 1989), the evidence that an integration algorithm of this kind was actually performed by the visual system was not widely accepted. Two lines of research have recently helped to support the plausibility of a scheme such as the one described above. The first line comes from a series of anatomical and physiological studies that used both cat and primate and suggest that there exist long-range connections between neurons in

primary visual cortex that link neurons with similar orientations. The second line consists of two types of psychophysical study that have provided evidence for the sorts of associations implied by the physiological and anatomical results (Field et al., 1993; Polat and Sagi, 1993, 1994). The results of these studies converge on an account that suggests that neurons in primary visual cortex integrate information from outside the classical receptive field in a way that promotes the integration of contours. Below we review some of these studies.

Physiology and anatomy of lateral connections As noted above, a variety of early studies showed that stimuli outside of the classical receptive field of a neuron in visual cortex can modulate that neuron’s activity. The sources of modulation potentially originate from feedforward connections, feedback connections from neurons farther along the visual pathway, lateral projections from neighboring neurons, or a combination of all three. Although we concentrate here on lateral connections, the modulation activity is almost certainly dependent on a more complex circuit involving all three. What has been remarkable over the past few years, however, has been the close ties found between lateral connections and visual behavior of humans and macaques when completing appropriate psychophysical tasks. Early studies exploring the horizontal connections in visual cortex discovered that pyramidal neurons have connections that extend laterally for 2 to 5 mm parallel to the

surface and have terminations that are patchy and selective (Gilbert and Wiesel, 1979; Rockland and Lund, 1982). Studies on the extent and specificity of lateral projections have now been completed on the tree shrew (e.g., Bosking et al., 1997; Rockland and Lund, 1982), primate (e.g., Malach et al., 1993; Sincich and Blasdel, 2001), ferret (e.g., Ruthazer and Stryker, 1996), and cat (e.g., Gilbert and Wiesel, 1989), with largely good agreement between species but also some important differences. Figure 70.2A provides an example of one of the impressive techniques that reveals the specificity of projections using a combination of optical imaging and anatomical data. These results from Bosking et al. (1997) show an overlay of the orientation columns revealed by optical imaging, with the lateral projections of pyramidal neurons near the injection site synapsing onto the surrounding regions. The lateral projections are revealed through extracellular injections of biocytin which label a small number of neurons near the injection site, along with their projections. The orientation tuning of a particular neuron is estimated by its location within an orientation column. As the figure shows, the orientation column of the injection (shown by the dark areas) has the same orientation as those of the columns where the long range projections project (i.e., they synapse onto neurons that are also in the dark regions). The short-range projections do not show such specificity. Bosking et al. also found that in tree shrew the extent of the long-range projections was significantly greater along the axis corresponding to the orientation of the central neuron.

B

A F 70.2. A, Results modified from Bosking et al. (1997) demonstrating the orientation-specific projections of a set of V1 neurons in the tree shrew. Optical imaging is used to reveal the orientation columns, while injections of biocytin are used to map the projections of a set of neurons taking up the biocytin (shown in white). As can be seen, the location of the orientation column of the injection is the same in most cases as the orientation column of

the projection. B, An experimentally and theoretically derived association field (Field et al., 1993) summarizing our beliefs regarding the underlying projections. Short-range connections are theorized to be largely inhibitory and independent of orientation, while longrange connections are theorized to be orientation specific and largely excitatory.

  :     

1071

Work over the past decade has demonstrated that the lateral projections of these pyramidal neurons are quite specific, projecting to regions of the cortex with iso-orientation columns, as well as similar ocular dominance columns and cytochrome oxidase blobs (e.g., Malach et al., 1993; Yoshioka et al., 1996). Pyramidal cells that are tuned to an orientation aligned with the axes of the projections are shown to project primarily to iso-orientation columns. That is, neurons project primarily to neurons of similar orientation preference. In some species, as exemplified in Figure 70.2A, the projections are considerably longer along the primary axis than orthogonal to the primary axis. For tree shrew (Bosking et al., 1997), owl monkey, and squirrel monkey (Sincich and Blasdel, 2001) the axis of projections is a factor of 2 to 4 longer along the primary axis than along the orthogonal axis. Cat and macaque show a similar specificity of projections (projecting to similar orientation columns). However, there is no clear evidence that the projections in these animals are elongated along the primary axis. As Sincich and Blasdel (2001) point out, the results from these highly binocular animals are confounded by the projections to related ocular-dominance columns and by the anisotropy of the visual field representation. In addition to the anatomical studies, support for these results has come from single-unit studies that have explored the effects of co-oriented stimuli presented outside of the classical receptive field (Ito and Gilbert, 1999; Kapadia et al., 1995, 2000; Polat et al., 1998). The results demonstrate that when a neuron is presented with an oriented stimulus within its receptive field, a second collinear stimulus can increase the response rate of the neuron, while the same oriented stimulus presented orthogonal to the main axis (displaced laterally) will produce inhibition or at least less facilitation. Kapadia et al. (2000) attempted to map out these inhibitory and facilitory effects in awake, behaving macaques, with the results showing good agreement with both the anatomy and human behavior, as described below. Figure 70.2B shows our theoretical depiction of these lateral projections, which we have called an association field (Field et al., 1993). This depiction incorporates results from our psychophysical measurements with the particular example from Figure 70.2A. We will discuss these psychophysical results shortly. However, first we summarize what we see as some of the principal anatomical and neurophysiological findings that are important to our discussion of contour integration. 1. Long-range projections of pyramidal cells are “patchy,” projecting primarily to neurons in iso-orientation columns (i.e., with similar orientation tuning) (Gilbert and Wiesel, 1989; Malach et al., 1993).

1072

, ,   

2. Long-range projections commonly extend to distances two to four times the size of classical receptive fields, and extend primarily in a direction collinear with orientation tuning of the cell (Bosking et al., 1997; Sincich and Blasdel, 2001). 3. Long-range projections to collinear neurons appear to be largely facilitatory (Kapadia et al., 1995; Nelson and Frost, 1985; Polat et al., 1998); however, the neurophysiology results also suggest that facilitation is largely dependent on contrast, with excitation predominant at low contrasts (or with high-contrast, cluttered backgrounds) and inhibition predominant at high contrasts (Kapadia et al., 1999). 4. Long-range connections appear to be reciprocal (Kisvardy and Eysel, 1992). 5. Short-range projections appear to be largely independent of orientation and have been argued to be predominantly inhibitory (Das and Gilbert, 1999). These conclusions are not unequivocal. For example, as noted above, not all species show elongation of projections along the primary axis. Furthermore, there is some debate as to whether these long-range connections are the source of facilitory effects. Kapadia et al.’s (2000) results imply that regions orthogonal to the main axis will produce inhibitory modulation. However, Walker et al. (1999), using patches of gratings at positions adjacent to the classical receptive field, found little evidence for facilitation. Although they did find inhibition, that inhibition was rarely symmetric and was typically distributed unevenly. Kapadia et al. (1995) and Polat et al. (1998) have also shown that when a second line segment is presented collinearly outside of the classical receptive field and is aligned with the preferred orientation, a small majority of neurons produce a stronger response when the line segments are separated by a small gap. Their response is stronger to a discontinuous line than to a continuous line. These results might be expected if we assume that short-range inhibition could, in some neurons, cancel out facilitatory collinear effects. The diagram in Figure 70.2B is a simplified representation of what we believe is the underlying mechanism of contour integration. We have made certain assumptions that lack clear support in the anatomy and neurophysiology. For example, in our model we imply that projections to regions offset from the primary axis will project to orientations that are offset in a regular manner. Figure 70.2A provides a weak suggestion that the off-axis projections are not as centered in the orientation column as those along the main axis, but this suggestion lacks quantitative physiological or anatomical data (the hypothesis has not been adequately tested). The motivation for the arrangement shown in the figure comes not from the anatomy and physiology, but from behavioral data discussed below.

Psychophysics Two lines of psychophysical research using different methodologies have demonstrated effects that correspond to the above-discussed anatomical and physiological data. The first line uses a contour integration task developed by Field et al. (1993), and the second explores the sensitivity of lowcontrast, oriented elements as a function of the surrounding stimuli (Polat and Sagi, 1993). An example of a stimulus used in the first line of research is demonstrated in Figure 70.3. Human observers are presented with arrays of high-contrast, oriented elements in which a subset has been aligned according to one of several alignment rules. Human observers attempt to identify the presence of this subset of elements as a function of the alignment. Figures 70.3A and B provide examples of two different rules. As the figure demonstrates, observers are more sensitive to collinear alignment than to orthogonal alignment, even though the stimuli are equated for their information content (Field et al., 1993). A raft of studies using the contour integration task to investigate the conditions under which integration occurs have been published over the past 10 years. For example, studies have demonstrated that integration is possible with elements that have multiple depth planes (Hess and Field, 1995; Hess et al., 1997), with elements that have different phase or polarity (Field et al., 2000), with elements that have different bandwidths, and rather weakly with elements at multiple scales (Dakin and Hess, 1998, 1999). Hayes (2000) has demonstrated that when a local-motion signal induces an apparent displacement of each whole element, integra-

tion is stronger when the alignment corresponds to the elements’ perceptual location as opposed to their physical location. Mullen et al. (2000) demonstrate that although integration across multiple hues is possible, integration between similar hues is more effective. Lee and Blake (2001) show a similar effect for movement. Hess and Dakin (1997) have suggested that there is a precipitous decline in contour integration in the periphery, although others have found that contour integration declines in the periphery at a rate similar to that of other visual functions, such as acuity (Nugent et al., 2001). The second line of psychophysical research believed to be related to long-range cortical interactions explores the contrast threshold of low-contrast elements surrounded by flanking lines (Kapadia et al., 1995; Polat and Sagi, 1993). We refer the reader to Polat (1999) for a detailed review of this research. The results of these studies demonstrate that contrast thresholds for oriented stimuli are reduced (sensitivity is increased) when the stimuli are flanked by collinear stimuli. One of the difficulties in interpreting this research is that the effects are strongly contrast dependent. At the level of single neurons, Kapadia et al. (1999) demonstrated that at low contrasts the length summation area of a neuron increases on average by a factor of 4 relative to the summation area at high contrasts. The enlarged summation area also occurs when the central stimulus is surrounded by a texture of elements such as that shown in Figure 70.3. However, at high contrasts, the long-range effects appear to be largely inhibitory. The above findings lead to an interesting question concerning whether both contrast-threshold effects and

A

B

F 70.3. An example of two stimuli used to measure human sensitivity to contours. The two images contain a path at the same location in each image (as marked by arrows) but created according to two different rules (after Field et al., 1993). The reader should

be able to see that the contour or path contained in A is more visible than that in B. This figure is one example that demonstrates the human visual system’s increased sensitivity to collinear arrangements.

  :     

1073

contour-integration effects are due to dynamic changes to the classical receptive field. Do these effects simply reflect the expansion of the receptive field? This explanation may account for the effects of the contrast threshold, but there are three reasons why, for the contour integration task, it may not be an adequate explanation. First, the threshold effects are strongly dependent on the phase or polarity of flanks (Williams and Hess, 1998), while contour integration has little phase dependence (Field et al., 2000). Second, contour integration occurs between elements that differ by more than 30 degrees (Field et al., 1993) when the alignment is appropriate (Fig. 70.3A), but there is no evidence that the expansion of the receptive field’s length would allow such integration. Third, there appears to be little evidence that perceived contrast is enhanced by flanking stimuli. We will return to this issue in the next section, but the reader can view Figure 70.3A and ask whether the elements in the path appear to be higher in contrast. Hess et al. (1998) found that large variations in contrast had no effect on the ability to detect the presence of the contour. Xing and Heeger (2001) also found no changes in perceived contrast when grating patches were surrounded by patches of similar contrast. Changes were noted only when the flanking patches were significantly higher in contrast. Other psychophysical techniques have demonstrated intriguing results. Chavane et al. (2000) have demonstrated that the speed of an oriented element appears higher when it moves in a direction collinear to its axis than when it moves in a direction orthogonal to its axis. They argue that long-range connections may be responsible. Kapadia et al. (2000) have demonstrated that elements placed along the ends of a central element can induce a perceived change in the orientation of the central element toward the orientation of the central element. However, when the flanking elements are placed along the opposite axis (adjacent to the central element), the central element can be shifted away from the orientation of the flanking elements. They also demonstrated that the spatial distribution of this effect showed good agreement with the neurophysiology of cortical facilitation produced by the flanking lines. Mareschal et al. (2001) have also demonstrated that with a collinear arrangement, flanking grating patches can significantly increase the orientation discrimination thresholds of the central patch. Furthermore, the threshold increase is significantly higher in the collinear arrangement than when the orientation of the elements is perpendicular to the positions of the three patches. Kovacs and Julesz (1993) demonstrated that when measuring the visibility of a path of elements as a function of the density of the surrounding elements, the path is significantly more visible if the contour forms a closed figure. Pettet et al. (1998) argue that this effect may be related to the directional smoothness of the contours (i.e., a circular

1074

, ,   

figure has all the elements changing orientation in a consistent direction). In either case, both results demonstrate that sensitivity depends on elements farther away than the immediate neighbors. The simple model, based on excitatory effects between neighbors, will not produce this effect. Of course, there is no reason to assume that these psychophysical effects necessarily occur in V1, and these psychophysical results may be an indication of the direction of attention toward features that undergo predictable change. Nonetheless, the results suggest that thresholds for perceiving contours depend on complex relationships. We conclude this section on the psychophysical phenomena with a fascinating study by Kovacs et al. (1996). As noted earlier, Hess and Field (1995) demonstrated that it was possible to integrate contour fragments that had relatively large binocular disparities between them; Kovacs et al. went much further and presented to observers binocular image pairs that would be expected to produce rivalry. Consider the presentation of two completely different natural scenes to each eye. Under such conditions, one would expect one eye or the other to dominate much of the time. Kovacs et al. presented such images to observers and then broke up each pair so that each eye received patches from both images, such that the left eye received the complement of the right (e.g., the right eye gets 1,2,1,2,2,1 with the left eye receiving 2,1,2,1,1,2). As one can readily see by observing their demonstrations, Kovacs et al. found that observers commonly see complete images (1,1,1,1,1,1 or 2,2,2,2,2). The contours and other visual information were successfully integrated between the two eyes into a single perceptual whole. This result implies that the process involved in integrating contours is not eye specific.

Computational modeling In some cases, computational models are simple reflections of the data found experimentally. They can be considered existence proofs demonstrating that it is at least possible to perform the desired task with the proposed architecture. They cannot demonstrate that the visual system necessarily uses the architecture of the model, but they can demonstrate that such a model would work if that architecture did underlie the task. However, at times, these models are most useful when they fail, and that may well be the case in the following studies we discuss. To integrate contours, a variety of algorithms have been proposed that use the technique of integrating similar orientations along collinear directions. Part of the argument for using a collinearity algorithm appears to be that the nature of the task demands it. However, these early studies also went to some lengths to explain how such an algorithm might fit with the known physiology and anatomy (e.g., Grossberg and Mingolla, 1985; Parent and Zucker, 1989;

Shashua and Ulman, 1988). In the past 5 years, as our understanding of the underlying physiology has increased, so has the sophistication of computational models (e.g., Geisler and Super, 2000; Li, 1998, 2000; Yen and Finkel, 1998). These models have demonstrated that the architecture revealed by the physiology and anatomy can be used to provide an efficient means of extracting contours in natural scenes, and it can be used to account for a significant amount of the psychophysical data. Our work on contour sensitivity (Field et al., 1993) was partly motivated by the belief that the properties of natural edges would be more efficiently coded by a linking process rather than by a high-level neuron tuned to the particular contour in question. The difficulty with the high-level neuron model is that the number of possible contours in the natural world, or even in our experiments, is much too large to have a neuron for every contour. Geisler et al. (2001) and Sigman et al. (2001) have taken the ecological approach further and asked whether the contour integration model is an efficient means of coding natural scene contours. They measured the co-occurrence statistics of edge elements in natural scenes and found that the relative orientations of neighboring contour segments match well with those predicted physiologically, and with psychophysically defined association fields. Geisler et al.’s results are particularly interesting because of the requirements needed to measure these co-occurrence statistics. As they argue, these statistics are multidimensional in nature. Given an edge at a particular location with a particular orientation, the region around that location is a threedimensional probability map of x-position by y-position by orientation. Only by mapping out this full probability map does one see the full set of statistical dependencies. And it is in these conditional probabilities that one finds the orientation dependencies that map onto the association field properties. The probability map is much higher in dimension if we include the additional dependencies across scale, chromaticity, motion, and disparity. Indeed, our own work (Hayes and Field, in prep.) suggests that both perceptual integration over scale and the structure of natural edges through scale follow similar rules. A potential difficulty for all recent models (e.g., Geisler and Super, 2000; Li, 1998; Yen and Finkel, 1998), as well as for earlier models (e.g., Grossberg and Mingolla, 1985), is that they generally assume that recurrent activity increases the responses of the neurons along the contour. This assumption is supported by some neurophysiological results which show an increase in response rate with flanking collinear lines (Kapadia et al., 1995; Nelson and Frost, 1985). The difficulty is in understanding how the visual system untangles the relationship between neural activity and contrast. Responses increase with contrast, and they also increase with collinear arrangements. How does the visual

system decipher differences in contrast variation from differences in context (i.e., collinearity)? Using human psychophysical techniques, Hess et al. (1998) found that contrast changes have little effect on the visibility of a contour. Consider the image shown in Figure 70.3. The contrast of the path elements is perceived to be the same as that of the background. Such results suggest that neurons must somehow carry the code for contrast separately from the code for the continuity of the contour. There are various possibilities for how this might be achieved. One possibility is that neurons that code contrast are different from those that code the contour. Under this hypothesis, we would need to assume that both neurons coding for contrast and neurons coding for continuity are present in V1. A second approach proposes that continuity is represented by a temporal code, presumably tied to the synchronous activity of neighboring neurons. This approach to binding has received considerable recent attention and has some experimental support (Singer and Gray, 1995; see also Chapter 113). The difficulty with this model is that it requires a mechanism to detect the synchrony. Hess et al. (1998) suggest a rather different and more basic version of a temporal code. They suggest that contrast information is represented by the initial response generated by the feedforward activity, with the later response determined by the lateral connections and the context of the surrounding regions. The contrast signal could then be extracted from the collinearity signal by simply tracking the timing of the response. This hypothesis was derived from the neurophysiological work of Zipser et al. (1996). Using textures as stimuli, they found results consistent with this theory. However, Kapadia et al. (1999) provide data that are supportive in some ways but also make the story more complex. As noted in the previous section, Kapadia et al. found that collinear facilitation for neurons in V1 occurs only at low contrasts or at high contrasts in complex backgrounds. They also noted that this facilitation occurs after the initial transient response of the neuron during the sustained component of the response. This aspect of the response fits the model proposed by Hess et al. (1998). However, at high contrasts, the neurons do not show this sustained response, but only the sharp transient response. What sort of model predicts this high-contrast behavior? It may involve some degree of contrast normalization (e.g., Heeger, 1992), but at present we are not aware of any model that predicts both the timing of responses and the lack of facilitation at high contrasts. There is also the question of whether lateral feedback has the appropriate timing to account for the neurophysiological findings. Along these lines, Van Rullen et al. (2001) provide an interesting alternative to the above models. They argue that models that iterate toward a solution using recurrent lateral feedback are too slow to explain reaction-time data and neurophysiological responses measured during

  :     

1075

visual recognition. They suggest that the contours might be represented not by the magnitude of the neural response, but by the relative speed at which responses pass through the visual system. They argue that lateral connections may serve to facilitate the initial response allowing the collinear context to push the most “meaningful” information most quickly through the visual system. However, all of the models fail to predict the smoothness constraint described by Pettet et al. (1998), whose results demonstrate that a contour which changes in a consistent direction is more visible than a contour which has multiple changes in direction. Such results suggest that sensitivity is a function of more than immediate neighborhood interactions. Contours changing orientation in a consistent angular direction provide for greater sensitivity. But whether this sensitivity is related to the lateral connections in V1, or to higher-level interactions or higher-level feedback, remains to be seen.

Some remaining questions There remain a number of interesting and fruitful directions for research in this area, as well as a number of problems. Psychophysical research, computational modeling, and measurements on natural scenes all support a particular mapping, such as that shown in Figure 70.2B. They suggest that off-axis projections will project to off-axis orientations along the lines of smooth curves. Our own eyeball estimates of the published anatomical data of Bosking et al. (1997) seem to suggest that the off-axis projections project to orientation columns that are slightly shifted from those along the primary axis. To our knowledge, though, no quantitative study has been conducted to support or dismiss this hypothesis. Another question of interest is how contours are integrated across the midline. In V1, communication across the midline must pass across the corpus collosum, a pathway that is significantly longer and possibly less efficient. However, there appear to be no large differences between the integration across the midline or within a hemifield. Presumably, if integration occurs across the midline, this would show up as a delay in processing or a reduction in sensitivity. In our own unpublished work on this problem, no significant delay was found. Indeed, if no differences were found between contour integration across hemifields versus within hemifields, it would argue that much of the contour integration task (or at least the limiting factors in the task) must be performed by areas beyond V1. There also remain questions regarding the relation between contour integration effects and the wide range of studies on illusory contours. A large number of studies that have investigated the perception of illusory contours, and have explored the conditions which produce the appearance

1076

, ,   

of an illusory surface partially bounded by illusory edges (e.g., see Lesher, 1995, for a review). The perception of the illusion depends on the relatability of the supporting contours (Kellman and Shipley, 1991), meaning that the supporting contours must fall along first-order curves, as shown in Figure 70.4A. However, the illusion also depends on the end junctions of the supporting contours (e.g., Rubin, 2001). Figure 70.4B shows an example where the illusion is blocked by converting the L-junctions inducing corners into T-junctions. Kapadia et al. (1995) have demonstrated that T-junctions can also reduce the sensitivity in a contour integration task when the elements are made of T-elements rather than simple line elements. Kapadia et al. also demonstrated that with single neurons, the effects produced by flanking collinear lines are also reduced with such flanking lines. Although there are clearly some important relationships between illusory contours and contour integration, the illusion is certainly not a necessary component of the integration process. As readers may see for themselves, the perception of the contour in Figure 70.3A does not result in an illusion of luminance or the perception of structure between the elements. We should also note that while lateral connections presumed to underlie the contour integration task are found with V1 neurons, neural responses corresponding to illusory contours are not found earlier than V2 (Peterhans and von der Heydt, 1989; von der Heydt et al., 1984). Zhou et al. (2000) have also found that over half of the neurons in V2 and V4 also show selectivity to border ownership. Given the same local contour information (the same information within the classical receptive field), the majority of neurons were found to respond differentially to larger object properties. For example, a neuron responding to a vertical edge may produce a larger response, depending on whether the contour is part of an object to the left or to the right of the contour. In contrast to V2, only 18% of the neurons in the top layer of V1 show this differential response. Zhou et al. (2000) also noted that the differential response to border ownership occurred within 25 msec of

A

B

F 70.4. A shows a modified Kaniza figure that typically results in the perception of an illusory contour. B demonstrates the importance of the supporting endpoints in this illusion. The Tjunction will typically reduce the strength of the illusory triangle.

response onset, arguing that the solution is generated within the visual cortex. These results imply that lateral connections in V1 are important to the integration of contours, but they are not directly involved in the more complex “object” relationships portrayed by illusory figures and by object identity.

Summary The anatomical, neurophysiological, psychophysical, and computational research of the past decade provides a compelling argument that neurons in area V1 integrate information from beyond the classical receptive field in a manner that assists in the integration of contours. Integrated contours represent a critical component of natural scenes important to early vision. They are important in defining the boundaries and extents of the objects in our world. The lateral connections between neighboring neurons are certain to play a number of roles besides contour integration. As argued by a number of investigators, these connections are likely to play a role in contrast normalization, stereo, motion, and texture segregation, among others. Furthermore, the facilitation with single neurons is contrast dependent, and this may imply that the facilitation at collinear positions is a secondary effect to the inhibition found in much of the nonclassical surround. It is also likely that few of these computational problems are “solved” in V1. V1 neurons receive a large amount of input from higher visual areas, which undoubtedly plays a significant role. Indeed, there is ample evidence that both task outcome and the activity of these neurons can be modulated by attention (e.g., Ito and Gilbert, 1999) suggesting that our final model of V1 will be considerably more complex. Overall, the studies reviewed here call into question the notion that V1 codes the visual world by breaking it down into an array of independent features. Although V1 neurons are differentially selective to a variety of visual features, their lateral connections, and the related perceptual phenomena, suggest that V1 should be considered as a complex web of interactions. Each neuron’s response depends in a complex way on its neighbors, on its inputs, and on feedback from higher levels. With the current surge of studies exploring these interactions, a clearer picture of their role is expected to develop over the next few years. At this time, however, the evidence suggests that the Gestalt psychologists of the early twentieth century had a profound insight with their law of good continuation. The integration of contours represents one task well served by the complex interactions found in early vision.

REFERENCES Allman, J., F. Miezin, and E. McGuinness, 1985. Stimulus specific responses from beyond the classical receptive field: neurophysiological mechanisms for local-global comparisons in visual neurons, Annu. Rev. Neurosci., 8:407–430. Bosking, W. H., Y. Zhang, B. Schofield, and D. Fitzpatrick, 1997. Orientation selectivity and the arrangement of horizontal connections in tree shrew striate cortex, J. Neurosci., 17:2112–2127. Callaway, E. M., 1998. Local circuits in primary visual cortex of the macaque monkey, Annu. Rev. Neurosci., 21:47–74. Chavane, F., C. Monier, V. Bringuier, P. Baudot, L. Borg-Graham, J. Lorenceau, and Y. Fregnac, 2000. The visual cortical association field: a Gestalt concept or a psychological entity? J. Physiol. (Paris), 94:333–342. Dakin, S. C., and R. F. Hess, 1998. Spatial-frequency tuning of visual contour integration, J. Opt. Soc. Am. A, 15:1486–1499. Dakin, S. C., and R. F. Hess, 1999. Contour integration and scale combination processes in visual edge detection, Spatial Vis., 12:309–327. Das, A., and C. D. Gilbert, 1999. Topography of contextual modulations mediated by short-range interactions in primary visual cortex, Nature, 399:655–661. Field, D. J., 1987. Relations between the statistics of natural images and the response properties of cortical cells, J. Opt. Soc. Am. A, 4:2379–2394. Field, D. J., A. Hayes, and R. F. Hess, 1993. Contour integration by the human visual system: evidence for a local “association field,” Vis. Res., 33:173–193. Field, D. J., A. Hayes, and R. F. Hess, 2000. The roles of polarity and symmetry in contour integration, Spatial Vis., 13:51–66. Fitzpatrick, D., 2000. Seeing beyond the receptive field in primary visual cortex, Curr. Opin. Neurobiol., 10:438–443. Geisler, W. S., J. S. Perry, B. J. Super, and D. P. Gallogly, 2001. Edge co-occurrence in natural images predicts contour grouping performance, Vis. Res., 41:711–724. Geisler, W. S., and B. J. Super, 2000. Perceptual organization of two-dimensional patterns, Psych. Rev., 107(4):677–708. Gilbert, C. D., 1998. Adult cortical dynamics, Physiol. Rev., 78:467–485. Gilbert, C. D., and T. N. Wiesel, 1979. Morphology and intracortical projections of functionally characterised neurones in the cat visual cortex, Nature, 280:120–125. Gilbert, C. D., and T. N. Wiesel, 1989. Columnar specificity of intrinsic horizontal and corticocortical connections in cat visual cortex, J. Neurosci., 9:2432–2442. Grossberg, S., and E. Mingolla, 1985. Neural dynamics of perceptual grouping: textures, boundaries, and emergent segmentations, Percept. Psychophys., 38:141–171. Hayes, A., 2000. Apparent position governs contour-element binding by the visual system, Proc. R. Soc. Ser. B, 267:1341–1345. Heeger, D. J., 1992. Normalization of cell responses in cat striate cortex, Vis. Neurosci., 9:181–197. Hess, R. F., and S. C. Dakin, 1997. Absence of contour linking in peripheral vision, Nature, 390:602–604. Hess, R. F., S. C. Dakin, and D. J. Field, 1998. The role of “contrast enhancement” in the detection and appearance of visual contours, Vis. Res., 38:783–787. Hess, R. F., and D. J. Field, 1995. Contour integration across depth, Vis. Res., 35:1699–1711. Hess, R., and D. Field, 1999. Integration of contours: new insights, Trends Cogn. Sci., 12:480–486.

  :     

1077

Hess, R. F., A. Hayes, and F. A. A. Kingdom, 1997. Integrating contours within and through depth, Vis. Res., 37:691–696. Hubel, D. H., 1988. Eye, Brain, and Vision, New York: Scientific American Library. Ito, M., and C. D. Gilbert, 1999. Attention modulates contextual influences in the primary visual cortex of alert monkeys, Neuron, 22:593–604. Kapadia, M. K., M. Ito, C. D. Gilbert, and G. Westheimer, 1995. Improvement in visual sensitivity by changes in local context: parallel studies in human observers and in V1 of alert monkeys, Neuron, 15:843–856. Kapadia, M. K., G. Westheimer, and C. D. Gilbert, 1999. Dynamics of spatial summation in primary visual cortex of alert monkeys, Proc. Natl. Acad. Sci. USA, 96:12073–12078. Kapadia, M. K., G. Westheimer, and C. D. Gilbert, 2000. Spatial distribution of contextual interactions in primary visual cortex and in visual perception, J. Neurophysiol., 84:2048– 2062. Kellman, P. J., and T. F. Shipley, 1991. A theory of visual interpolation in object perception, Cogn. Psychol., 23:141–221. Kisvarday, Z. F., and U. T. Eysel, 1992. Cellular organization of reciprocal patchy networks in layer III of cat visual cortex (area 17), Neuroscience, 46:275–286. Kovacs, I., and B. Julesz, 1993. A closed curve is much more than an incomplete one: effect of closure in figure-ground segmentation, Proc. Natl. Acad. Sci. USA, 90:7495–7497. Kovacs, I., T. V. Papathomas, M. Yang, and A. Feher, 1996. When the brain changes its mind: interocular grouping during binocular rivalry, Proc. Natl. Acad. Sci. USA, 93:15508–15511. Lee, S. H., and R. Blake, 2001. Neural synergy in visual grouping: when good continuation meets common fate, Vis. Res., 41:2057–2064. Lesher, G. W., 1995. Illusory contours: toward a neurally based perceptual theory, Psychonom. Bull. Rev., 2:279–321. Li, Z., 1998. A neural model of contour integration in the primary visual cortex, Neural Comput., 10:903–940. Li, Z., 2000. Pre-attentive segmentation in the primary visual cortex, Spatial Vis., 13:25–50. Maffei, L., and A. Fiorentini, 1976. The unresponsive regions of visual cortical receptive fields, Vis. Res., 16:1131–1139. Malach, R., Y. Amir, M. Harel, and A. Grinvald, 1993. Relationship between intrinsic connections and functional architecture revealed by optical imaging and in vivo targeted biocyting injections in primate striate cortex, Proc. Natl. Acad. Sci. USA, 90:10469–10473. Mareschal, I., M. P. Sceniak, and R. M. Shapley, 2001. Contextual influences on orientation discrimination: binding local and global cues, Vis. Res., 41:1915–1930. Mullen, K. T., W. H. Beaudot, and W. H. McIlhagga, 2000. Contour integration in color vision: a common process for the blue-yellow, red-green and luminance mechanisms? Vis. Res., 40:639–655. Nelson, J. I., and B. J. Frost, 1985. Intracortical facilitation among co-oriented, co-axially aligned simple cells in cat striate cortex, Exp. Brain Res., 61:54–61. Nugent, A. K., R. Keswani, R. L. Woods, and E. Peli, 2001. Contour integration in the peripheral field of normal and low vision observers, Invest. Ophtalmol. Vis. Sci., 42(4):5612. Parent, P., and S. Zucker, 1989. Trace inference, curvature consistency and curve detection, IEEE Tran. Pattern Anal. Machine Intell., 11:823–839.

1078

, ,   

Peterhans, E., and R. von der Heydt, 1989. Mechanisms of contour perception in monkey visual cortex II: contours bridging gaps, J. Neurosci., 9:1749–1763. Pettet, M. W., S. P. McKee, and N. M. Grzywacz, 1998. Constraints on long range interactions mediating contour detection, Vis. Res., 38:865–879. Phillips, W. A., and W. Singer, 1997. In search of common foundations for cortical computation, Behav. Brain Sci., 20:657–722. **Polat, U., 1999. Functional architecture of long-range perceptual interactions, Spatial Vis., 12:143–162. Polat, U., K. Mizobe, M. W. Pettet, T. Kasamatsu, and A. M. Norcia, 1998. Collinear stimuli regulate visual responses depending on cell’s contrast threshold, Nature, 391:580–584. Polat, U., and D. Sagi, 1993. Lateral interactions between spatial channels: suppression and facilitation revealed by lateral masking experiments, Vis. Res., 33:993–999. Polat, U., and D. Sagi, 1994. The architecture of perceptual spatial interactions, Vis. Res., 34:73–78. Rockland, K. S., and J. S. Lund, 1982. Widespread periodic intrinsic connections in the tree shrew visual cortex, Science, 215:1532–1534. Rubin, N., 2001. The role of junctions in surface completion and contour matching, Perception, 30:339–366. Ruthazer, E. S., and M. P. Stryker, 1996. The role of activity in the development of long-range horizontal connections in area 17 of the ferret, J. Neurosci., 16:7253–7269. Schwartz, O., and, E. P. Simoncelli, 2001. Natural signal statistics and sensory gain control, Nat. Neurosci., 4(8):819–825. Shashua, A., and S. Ullman, 1988. Structural saliency: the detection of globally salient structures using 2 locally connected network. In Proc. Int. Conf. Comput. Vis. (ICCV) Tampa, FL, 482– 488. Sigman, M., G. A. Guillermo, C. D. Gilbert, and M. O. Magneasco, 2001. On a common circle: natural scenes and Gestalt rules, Proc. Natl. Acad. Sci. USA, 98:1935–1940. Sincich, L. C., and G. G. Blasdel, 2001. Oriented axon projections in primary visual cortex of the monkey, J. Neurosci., 21:4416–4426. Singer, W., and C. M. Gray, 1995. Visual feature integration and the temporal correlation hypothesis, Ann. Rev. Neurosci., 18:555– 586. Van Rullen, R., A. Delorme, and S. J. Thorpe, 2001. Feed-forward contour integration in primary visual cortex based on asynchronous spike propagation, Neurocomputing, 38:1003–1009. von der Heydt, R., E. Peterhans, and G. Baumgartner, 1984. Illusory contours and cortical neuron responses, Science, 224:1260– 1262. Walker, G. A., I. Ohzawa, and R. D. Freeman, 1999. Asymmetric suppression outside the classical receptive field of the visual cortex, J. Neurosci., 19:10536–10553. Williams, C. B., and R. F. Hess, 1998. Relationship between facilitation at threshold and suprathreshold contour integration, J. Opt. Soc. Am. Ser. A, 15:2046–2051. Xing, J., and D. J. Heeger, 2001. Measurement and modeling of centre-surround suppression and enhancement, Vis. Res., 41:571–583. Yen, S. C., and L. H. Finkel, 1998. Extraction of perceptually salient contours by striate cortical networks, Vis. Res., 38:719–741. Yoshioka, T., G. G. Blasdel, J. B. Levitt, and J. S. Lund, 1996. Relation between patterns of intrinsic lateral connectivity, ocular

dominance, and cytochrome oxidase–reactive regions in macaque monkey striate cortex, Cereb. Cortex, 6:297–310. Zhou, H., H. S. Friedman, and R. von der Heydt, 2000. Coding of border ownership in monkey visual cortex, J. Neurosci., 20:6594–6611.

Zipser, K., V. A. F. Lamme, and P. H. Schiller, 1996. Contextual modulation in primary visual cortex, J. Neurophysiol., 16:7376– 7389.

  :     

1079

71

Shape Dimensions and Object Primitives CHARLES E. CONNOR

T   produced by an object is infinitely variable due to changes in viewpoint and illumination. Somehow, this infinity of images must be mapped to a single internal representation in order to identify the object consistently. Moreover, this feat must be accomplished for a virtual infinity of different objects. Neurobiological systems handle this object recognition problem on a fast time scale with apparent ease. In primates, including humans, object recognition is carried out by the ventral pathway of extrastriate visual cortex. The general structure and function of this pathway are reviewed in Chapters 76 and 77. This chapter addresses a particular solution to the object recognition problem—structural representation—and evaluates the evidence that this solution is utilized in the primate ventral pathway. Neural representation schemes can be broadly divided into two classes, local and distributed. In local (or labeled-line) representations, each stimulus is encoded by activity in a single neuron or neural group that functions only to represent that stimulus. In distributed representations, each stimulus is encoded by a pattern of activity across a population of neurons, and each neuron participates in representing multiple stimuli. Distributed representation schemes are thus much more efficient, and they seem to be the rule in sensory systems, where individual neurons typically respond to a range of stimuli. The greater efficiency of distributed representations is critical for encoding object shape, because the infinite space of potential stimuli makes a local scheme impractical. If shape representation is distributed, then how exactly is shape information decomposed and parceled out across neurons? In other words, what elements of shape information do individual neurons represent? At the receptor level in the retina, shape information is partitioned across space. Each neuron represents information about luminance and/or color at a discrete spatial location in the visual image. In this form, shape information is far too implicit to be useful. Moreover, as discussed above, the spatial patterns of luminance produced by a given object are constantly changing. The visual system must transform this pixelated, spatiotopic representation into another type of distributed representation—one in which the pattern for a given object is relatively consistent (facilitating recognition) and patterns for similar objects are themselves similar (facilitating

1080

comparison and categorization; this property is known as second-order isomorphism; Edelman, 1999).

Structural representation: theory According to many theories (Milner, 1974; Selfridge, 1959; Sutherland, 1968), this transformation is based on an alphabet of simple shape elements or primitives that correspond to common real-world object components. Each neuron would represent one type of primitive, responding whenever that primitive was present within its receptive field. A given object would be represented by the combined activity of a number of such neurons, each signaling one of the primitives constituting the object. A complete representation would also require information about the relative position and connectivity between primitives. This is a structural representation in the sense that neurons explicitly encode the geometric composition of the object. The idea is also referred to as representation by parts or representation by components (Biederman, 1987), since the object is described in terms of its parts or primitives. Parts-based representation satisfies the requirement for consistency, since the list of parts making up an object does not change when the retinal image changes. The particular parts that are visible may change when the object rotates (due to self-occlusion), but a familiar object is recognizable from a subset of its parts. Structural or parts-based representation also satisfies the requirement for similarity or second-order isomorphism: Explicitly encoding the geometrical structure of objects ensures that similar objects will have similar neural representations. Finally, parts-based coding has the efficiency and capacity required to represent the infinite space of object shape. A finite number of neurons encoding basic shape elements can represent any combination of those elements in the same way that letters of the alphabet can represent any word. The discrete form of the theory described above is convenient for conveying the basic coding principle, and it analogizes to letters encoding words and DNA triplets encoding proteins. But the notion of stereotyped shape primitives signaled by all-or-nothing neural responses is a simplification. The shapes of real-world object components vary continuously. Correspondingly, visual neurons respond in a graded fashion across a range of shapes. Thus, the shape

alphabet is really a set of shape dimensions suitable for describing object components—a multidimensional feature space (Edelman, 1999; Edelman and Intrator, 2000). [Some authors would reserve the label “structural” for the discrete form of the theory (Edelman and Intrator, 2000); I use it here to encompass all schemes in which part identity and position are explicitly represented.] Neurons with graded tuning in those dimensions would provide an analog signal (in spikes per second) related to how closely shape elements in the current image match their tuning peaks. Neural tuning peaks would be distributed across shape dimensions, so that any value could be represented. A given object would be represented by a constellation of population activity peaks corresponding to its constituent parts. E: C F What would a structural representation look like? Figure 71.1 illustrates a structural scheme for encoding two-dimensional (2-D) outline and silhouette-like shapes such as alphanumeric characters. The shape to be represented is a bold numeral 2 (Fig. 71.1A). There are a number of ways in which this shape could be decomposed into parts. The decomposition shown here is based on curved contour fragments. (The theoretical and empirical reasons for proposing contour fragments as parts are discussed below.) The lowercase letters label contour fragments with different curvature values. These fragments can be represented in four dimensions, two describing shape (Fig. 71.1B) and two describing relative position (Fig. 71.1C ). The two shape dimensions shown here are curvature and orientation. Curvature (radial axis in Fig. 71.1B) can be either positive (convex, projecting outward) or negative (concave, indented inward). Curvature is defined mathematically as the rate of change in tangent angle per unit contour length. For a circle, curvature is inversely related to

radius. Thus, larger values signify tighter, sharper, more acute curvature. Extremely large values correspond to curvatures so tight that we perceive them as tangent discontinuities, that is, angles or corners. In Figure 71.1B, the curvature scale is squashed so that very sharp curves or angles have a value of 1.0 (convex) or -1.0 (concave). Thus, the sharp convex angle labeled b has a curvature value of 1.0, and the sharp concave angle g has a value of -1.0. The broader-convexity a has a value near 0.5, and the broaderconcavity c has a value near -0.5. The straight contour segments (not labeled) would have curvature values of 0. The other shape dimension is orientation (angular axis in Fig. 71.1B), which in this context means the direction in which the curved fragment “points.” More precisely, this is the direction of the surface normal—the vector pointing away from the object, perpendicular to the surface tangent—at the center of the curved contour fragment. Thus, the sharp-convexity b points toward the lower left (225 degrees), the broad-convexity a points toward the upper right (45 degrees), and so on. Note that this definition of orientation differs from the standard definition for straight lines or edges. The standard definition is orientation of the surface tangent rather than the surface normal. The orientation of the normal is more useful because it also indicates figure/ground direction. Under the convention used here, in which the surface normal points away from the figure interior, 45 degrees specifies a contour with the figure side on the lower left (e.g., a), while 225 degrees specifies a contour with the figure side on the upper right (e.g., b). The tangent in these two cases would be the same. The relative position dimensions are shown in Figure 71.1C. The coordinate system used here is polar; the two dimensions are angular position and radial position with respect to the object center. (The radial position scale is relative to object height.) Polar coordinates are convenient

F 71.1. A structural (parts-based) shape-coding scheme based on contour fragments. A, The example shape, a bold numeral 2, can be decomposed into contour fragments (a–g) with different curvatures, orientations, and positions. B, The curvature and

orientation of each contour fragment is plotted on a 2-D domain. C, The positions of the contour fragments (relative to the object center) are plotted on a 2-D domain. Together, plots B and C represent a 4-D domain for describing contour fragments.

:     

1081

for representing shape because changes in object size do not affect angular position and produce a uniform scaling of radial position. Contour section a is at the upper right with respect to the object center, so it is plotted near 45 degrees; b is at the upper left and is plotted at 135 degrees. There are many other ways in which the necessary position information could be parameterized. These four dimensions capture much of the information needed to specify a simple shape like the bold numeral 2. A few important shape dimensions, like contour fragment length and connectivity, have been left out for simplicity. Also, the bold 2 exemplifies just one class of 2-D objects. A much higher dimensionality would be needed to represent three-dimensional (3-D) objects, objects with internal structure, objects of greater shape complexity, and objects defined by color and texture variations. In a neural representation, the four dimensions in Figure 71.1 would constitute the tuning space for a large population of cells. Figures 71.1B and 71.1C can be thought of as 2-D projections of a single four-dimensional (4-D) domain. Each cell would have a tuning peak somewhere in the 4-D space, and tuning peaks would be distributed across the entire space. Each contour fragment would be represented by an activity peak in the population response. In other words, if all the neurons’ responses were plotted, using a color scale, at their tuning peak locations in Figures 71.1B and 71.1C, there would be hot spots at the points corresponding to the object’s contour fragments. Fragment a, for example, would be represented by strong activity in the tuning range labeled a in Figures 71.1B and 71.1C, that is, strong activity in neurons tuned for broad convexity oriented near 45 degrees and positioned near the upper right of the object. The bold 2 as a whole would be represented by the constellation of peaks indicated by all the lowercase letters. The population response pattern would consist not only of punctate peaks. Regions of constant or gradually changing curvature would be represented by continuous ridges in curvature space. For example, the broad convex region labeled a in Figure 71.1A would be represented by an arcshaped ridge running clockwise from 135 to 315 in Figure 71.1B, because it would stimulate cells sensitive to broad convex curvature at all those orientations. The sharp angle at b, on the other hand, would be represented by a punctate peak. The entire pattern of ridges and peaks would characterize the sequence of gradual and abrupt curvature changes in the shape. Neural representations are often thought of as single population activity peaks, but one study of motion coding has shown that the visual system can be sensitive to aspects of the population response pattern other than peak position (Treue et al., 2000). Neural representation in terms of contour fragments would have some of the important characteristics required

1082

, ,   

for object perception. It would be relatively robust to variations in an object’s retinal image such as size and position changes. The population pattern would be stable in the orientation and angular position dimensions, and it would scale uniformly in the curvature and radial position dimensions. If curvature and radial position were represented relative to object size, the pattern would be stable in those dimensions as well. Contour fragment coding would also meet the requirement of similar representations for similar objects (secondorder isomorphism). The numeral 2 rendered in other fonts (2, 2, 2, 2, 2, 2) would retain key features in the curvature representation, such as the broad convexity near the upper right and the sharp convexity near the lower left. In other words, all 2s would evoke a ridge somewhere near a and a peak somewhere near b in the 4-D population response space (Fig. 71.1). In fact, it is that kind of curvature pattern that defines the numeral 2 and allows us to generalize across the entire category of printed and handwritten 2s. Learning a shape category is a process of finding the characteristic features that define that category. It is critical that the neural representations of those features be consistent or at least grouped in neural tuning space. Finally, because of its combinatorial, alphabet-like coding power, the scheme shown in Figure 71.1 would have the capacity and versatility to represent a virtual infinity of shapes composed of standard contour fragments. This could be accomplished by a reasonable number of neurons with tuning functions spanning the 4-D contour curvature space. As noted above, however, a higher-dimensional space would be required to represent more complex objects. S R M The coding scheme in Figure 71.1 is just one way to parameterize shape. There are a number of theoretical ideas about shape primitives or shape dimensions (both of which can be grouped under the general heading of shape descriptors). Most theories posit a hierarchical progression of parts complexity, with each stage in the processing pathway receiving input signals for simpler parts and synthesizing them into output signals for more complex parts (Barlow, 1972; Hubel and Wiesel, 1959, 1968). In almost all models, the first level of shape description is local linear orientation (i.e., orientation of straight edges and lines). This choice is dictated by the overwhelming evidence that linear orientation is accurately and explicitly represented by cells at early stages in the ventral pathway (V1 and V2) (Baizer et al., 1977; Burkhalter and Van Essen, 1986; Hubel and Livingstone, 1987; Hubel and Wiesel, 1959, 1965, 1968). Theories diverge concerning higher-level shape descriptors. Marr distinguished two general possibilities: boundary or surface-based descriptors and axial or volumetric descriptors (Marr and Nishihara, 1978). The contour

curvature scheme illustrated in Figure 71.1 is a surface-based description; it encodes the shape boundary, specifically the 2-D outline. This is a complete description of a flat silhouette shape like the numeral 2. It would also capture much of the important information about a 3-D shape and could even be used to infer 3-D surface shape (Koenderink, 1984). The potential importance of contour curvature was recognized by Attneave (1954), who pointed out that shape information is concentrated in contours at regions of high curvature, including angles. Angles may be particularly significant, because they are invariant to transformations in scale and can be easily derived by summing inputs from cells tuned for edge orientation (Milner, 1974). Contour curvature could serve as a final description level (Hoffman and Richards, 1984) or it could be used to infer the structure of more complex parts (Biederman, 1987; Dickinson et al., 1992; Hummel and Biederman, 1992). Axial or volumetric descriptors constitute the ultimate level of representation in many theories. A volumetric primitive is a solid part, defined by the shape (straight or curved) and orientation (2-D or 3-D) of its medial axis. A complete object description in terms of medial axes is like a stickfigure drawing. That description can be refined with other parameters to represent how object mass is disposed about the axes—what the cross-sectional shape is and how width varies along the axis. A volumetric description of the bold numeral 2 would involve three medial axes, one curved and two straight, with corresponding width functions to specify the thick/thin structure of the font. (Cross-section would not be an issue for a flat 2-D shape.) Volumetric primitives are also known as generalized cones (Marr and Nishihara, 1978; a generalized cone is constructed by sweeping a cross-section of constant shape but smoothly varying size along an axis) or geons (Biederman, 1987). Marr argued that a volumetric description would be more compact and stable than a surface-based description. For most alphanumeric symbols, the axis-defining dimensions would capture the stable, category-defining characteristics. More variable (e.g., font-specific) contour information would be segregated into the width and cross-sectional dimensions. However, medial axes must initially be inferred from surface or boundary information. This requires first segmenting the surface contour into parts (Marr and Nishihara, 1978), probably at regions of high concave curvature, because these represent joints between interpenetrating volumes (Hoffman and Richards, 1984). The medial axis for each part would then be derived from its contours. Some authors have proposed mechanisms for inferring 3-D volumetric structure from certain characteristic 2-D contour configurations (Biederman, 1987; Dickinson et al., 1992; Lowe, 1985). As discussed above, a complete structural representation requires not just a list of primitives but also a description of

their spatial arrangement (as in Fig. 71.1C ). Spatial information is initially available to the visual system in retinotopic coordinates. Because the retinal image of an object is so variable, it would be useful to transform spatial information into an object-centered reference frame (but see Edelman and Intrator, 2000, who argue that coarse retinotopy would suffice). At the least, the object could define the center of the reference frame, so that changes in position on the retina would not alter the neural representation. In other words, neural shape responses would be position invariant at the final level of representation. The object might also define the scale of the reference frame, meaning that shape responses would be size invariant at the final level. This would make the neural representation stable across changes in viewing distance. Some theories would limit the spatial transformation to these two changes—position and scale (e.g., Fig. 71.1C). The orientation of the reference frame would still be defined by the retina or by the head, body, or world (which are usually aligned with the retina). If so, the neural representation would change when the object rotated in either 2-D space (around an axis pointing toward the viewer, e.g., turning upside down) or 3-D space (around any other axis, e.g., rotating around a vertical axis). Dealing with viewpoint changes of this kind is one of the most difficult aspects of shape recognition. One idea is that neural shape representations are viewpoint-dependent—different views of the same object are represented differently—and the visual system learns to recognize an object by storing a limited set of canonical views (Poggio and Edelman, 1990; Tarr and Pinker, 1989; Vetter et al., 1995). Intermediate views would be handled by neural mechanisms for interpolating between the canonical views (Poggio and Edelman, 1990). A more absolute solution to the viewpoint problem is to transform the structural description into a 3-D reference frame defined completely (in position, scale, and orientation) with respect to the object (Biederman, 1987; Dickinson et al., 1992). This would yield a more stable neural representation, but it would require complex mechanisms for inferring and synthesizing 3-D structure. Mel and Fiser (2000) have described an alternative to encoding each part position in a single spatial reference frame. Units sensitive to part (or feature) conjunctions can represent not only identity but also local connectivity between parts. In the example in Figure 71.2, some neurons might be tuned for the conjunction of fragments a and b, others for b-c, others for c-d, and so on. The response pattern across a population of such units would constitute a unique representation for the bold numeral 2. In effect, the local conjunctions would be concatenated to specify the entire sequence of contour fragments. As discussed below, recent neurophysiological results provide support for this idea (Pasupathy and Connor, 2001).

:     

1083

are some results that directly support the theory. As in the first section, an example will be considered in detail before we move on to a general description of the literature. E: C F R  A V4 If the visual system explicitly encodes shape in terms of parts-level structural dimensions, as in Figure 71.1, at least three empirical results can be predicted:

F 71.2. Tuning of a single macaque area V4 neuron in the curvature ¥ orientation domain. The gray circular backgrounds indicate the neuron’s average responses to each of the stimuli (white icons). The scale bar shows that light backgrounds correspond to response rates near zero, dark backgrounds to response rates near 30 spikes per second.

The general alternative to parts-based or structural representation is holistic representation—coding schemes in which each signal carries information about an entire object or an entire scene rather than just one part. Fourier-like decomposition, with basis functions extending across the entire domain, is an example. Edelman proposes that shapes are represented in a multidimensional space defined by pointwise information across whole objects. The high dimensionality of a point-based representation can be reduced by describing a novel shape in terms of its distances from a limited number of learned reference shapes (Edelman, 1999). A recent extension of this theory posits that the reference shapes could be object fragments and that the positions of those fragments could be represented in retinotopic or object-centered space (Edelman and Intrator, 2000). This would constitute a flexible, learning-based (and explicitly continuous) version of structural representation. Ullman (1996) proposes that an object is recognized by virtual (neural) alignment of its complete retinal image (through appropriate translation, scaling, and rotation) and pointwise matching with a shape template stored in memory. He notes that this approach could also be integrated with structural decomposition mechanisms.

Structural representation: evidence Much of the available neuroscientific evidence is at least consistent with structural representation of shape. There

1084

, ,   

1. Individual neurons should exhibit response rate tuning in those dimensions. There are other ways in which part structure might be encoded, but this would be the most explicit and useful. 2. Tuning function peaks should be distributed across the entire tuning space. This would be necessary for representing all possible part structures. 3. Tuning for parts-level structure should be consistent within different global shapes. For example, a cell tuned for sharp convex curvature oriented at 90 degrees and positioned at 90 degrees should respond to arrows (≠), triangles (), diamonds (), and so on. This is the most novel prediction, since shape-tuning functions are usually conceived in terms of a single optimal global shape. This section will describe how these three predictions are met for contour fragment representation in area V4. V4 is an intermediate stage in the ventral, objectprocessing pathway of the primate visual system (Felleman and Van Essen, 1991; Ungerleider and Mishkin, 1982). It has been identified in humans through retinotopic mapping in functional magnetic resonance imaging (fMRI) experiments (DeYoe et al., 1996; Sereno et al., 1995). In the macaque monkey, V4 receives feedforward inputs from V1 and V2 and sends feedforward outputs to posterior and central parts of inferotemporal cortex (PIT and CIT) (Felleman and Van Essen, 1991). Because of its intermediate position in the shape-processing hierarchy, V4 has the potential to exhibit mechanistic details that may be obscured at higher levels. Many V4 neurons are tuned for linear orientation in the same way that V1 and V2 cells are (Desimone and Schein, 1987), but some are selective for more complex shape characteristics (Gallant et al., 1993, 1996; Kobatake and Tanaka, 1994). It is this latter group that is considered here. Many V4 neurons are tuned for contour fragment shape, as hypothesized in Figure 71.1B (Pasupathy and Connor, 1999, 2001, 2002). The example cell in Figure 71.2 was studied in an awake macaque monkey performing a fixation task while contour stimuli were flashed in the cell’s receptive field. The stimuli are depicted as small white icons against circular gray backgrounds. In the actual experiment, the

stimuli were presented in the optimal color for the cell against a uniform gray background covering the rest of the display screen. The stimuli were convex or concave contour fragments. The relevant contour fragment was defined by a sharp luminance transition, but the rest of the stimulus faded gradually into the background gray, as though one part of a larger object was illuminated with a spotlight. The stimuli are plotted on a polar curvature ¥ orientation domain, as in Figure 71.1B. Curvature gradually progresses from sharp concave (negative) at the center of the plot through flat (0) to sharp convex (positive) at the periphery. The average response to each stimulus is indicated by the gray level of its background circle (see the scale bar). This neuron was tuned for acute convex curvature oriented toward the left. The sharpest convexity (outer ring) at 180 degrees evoked a response of 30 spikes per second. Stimuli nearby (in curvature space) evoked somewhat weaker responses, and distant stimuli evoked little or no response. Approximately one-third of V4 neurons exhibit this kind of graded tuning for curvature and orientation (Pasupathy and Connor, 1999). Tuning peaks are distributed across the entire extent of both dimensions, although there is a population-level bias toward sharper convex curvature. These cells are also tuned for contour fragment position, as hypothesized in Figure 71.1C. The example cell in Figure 71.3 was shown in previous tests to be tuned for sharp convex curvature at orientations in the 0 to 270 degree range (lower right quadrant). Figure 71.3 shows how the cell responded to stimuli containing sharp convexities at a variety of positions relative to object center. Stimulus shape is again represented by the white icons, and average response is indicated by the surrounding gray circles (see the scale bar at the bottom). Relative position was varied by changing the configuration of the elliptical base from which the sharp convexity projected. Stimuli are plotted on a polar grid (as in Fig. 71.1C) according to the relative position of the sharp convexity. Relative position tuning was tested for three curvature orientations: 270 degrees (Fig. 71.3A), 315 degrees (Fig. 71.3B), and 0 degrees (Fig. 71.3C). In all three cases, the neuron displayed gradual position tuning with a peak near 315 degrees. Control tests in which the overall stimulus position was varied (not shown) proved that this kind of tuning depended on relative position, not absolute position on the retina. Also, the response pattern does not merely reflect a global shape preference. The optimum shape in the top plot is similar to a right apostrophe (’), while the optimum shape in the bottom plot is similar to a left apostrophe on its side. The consistent factor is the position of the sharp convexity relative to the object center. Most curvature-sensitive V4 neurons exhibit this kind of relative position tuning for contour fragments (Pasupathy and Connor, 2001). In the example in Figure 71.3, the curvature orientation and angular position peaks are similar (315 degrees), but exper-

A

90 135

45

180

0

225

315 270

B

90 135

45

180

0

225

315 270

C

90 135

45

180

0

225

315 270 Angu lar Position

10

20

30

40

50

Spikes/s

F 71.3. Tuning of a single macaque area V4 neuron for the position of a sharp convexity relative to object center. The convexities are oriented at 270 degrees (A), 315 degrees (B), and 0/360 degrees (C ). The gray circular backgrounds indicate the neuron’s average responses (see the scale bar) to each of the stimuli (white icons). The stimuli are plotted on a polar domain representing object-centered position.

:     

1085

iments with more complex shapes in PIT/CIT have shown that orientation and angular position can be dissociated. For example, a neuron can be tuned for curvature oriented at 90 degrees (pointing upward) but positioned at 0 degrees (to the right of object center; Brincat and Connor, 2001). Thus, V4 neurons are tuned for contour fragment shape and position, and tuning peaks are distributed across shape space, fulfilling predictions 1 and 2. Moreover, this kind of tuning is consistent within different global shape contexts (prediction 3). The example cell in Figure 71.4 was responsive to sharp convex curvature oriented at 225 degrees. This cell was tested with a large set of shapes constructed by systematically combining contour fragments at various curvatures and orientations. Each stimulus is represented by a white icon within a gray circle indicating the average response of the cell (see the scale bar on the right). The stimuli are plotted in order of response strength. The stimuli that evoked the strongest responses all contained relatively acute convex curvature oriented at 225 degrees located near the lower left of the object (top rows). Stimuli without this feature evoked little or no response (bottom rows). The effective stimuli varied widely in global shape; the top row of Figure 71.4 includes crescents, triangles, raindrops, and so on. Similar results were obtained for the majority of curvature-sensitive cells in area V4 (Pasupathy and Connor, 2001), and population analysis revealed multipeaked shape representations comparable to that hypothesized in Figure 71.1 (Pasupathy and Connor, 2002). Significantly, many cells were sensitive to conjunctions of two or three curvature fragments. The cell in Figure 71.4 was most responsive to sharp convexities conjoined in the counterclockwise direction to broader concavities (top rows). Shapes with sharp convexities conjoined to broad convexities evoked weaker

responses. The majority of V4 neurons (94/109) were significantly tuned for conjunctive curvature fragments. This could imply that spatial configuration is represented in terms of local connectivity (Mel and Fiser, 2000; see above). Thus, V4 neurons have the basic response characteristics necessary to represent simple 2-D silhouette shapes in terms of their contour components. The precise choice of dimensions is arbitrary; the same tuning properties could be captured more or less effectively by a number of equivalent parameterizations (e.g., in terms of medial axis shape and width functions). Higher dimensionality might capture response properties more precisely, since some V4 cells appear to be tuned for contour fragments of greater complexity. The cell in Figure 71.4, for example, is most responsive to sharp convexities flanked by a concavity near the bottom of the shape (for further details, see Pasupathy and Connor, 2001). The important points, regardless of exact tuning dimensions, are that 1. Area V4 neurons are tuned for shape and relative position of object components. 2. Tuning functions are distributed across the entire range of shape and position. 3. Component-level tuning is consistent across different global shape contexts. These properties imply that shape representation in area V4 is structural—that objects are represented in terms of their constituent parts. S R   V P The ventral, object-processing pathway originates in primary visual cortex (V1) and continues (in the macaque monkey) through areas V2, V4, PIT (posterior inferotem-

20 10

Spikes/s

30

0

F 71.4. Tuning of a single macaque area V4 neuron for sharp convexity near the lower left (relative to object center) in a variety of global shape contexts. Shape stimuli (white icons) were constructed by systematically combining convex and concave frag-

1086

, ,   

ments into continuous closed contours. The gray circular backgrounds indicate the neuron’s average responses (see the scale bar) to each of the stimuli.

poral cortex, also called TE), CIT, and AIT (central and anterior inferotemporal cortex, also called TE). Anatomical studies have shown that these areas are densely interconnected. The pattern of feedforward and feedback projections suggests a hierarchy in which V1 is the earliest processing stage and IT the latest and presumably most advanced (Felleman and Van Essen, 1991). Receptive field size and response complexity increase gradually through the pathway. These properties are consistent with the notion of hierarchical shape processing—the idea that simpler partslevel information is extracted at early processing stages, then synthesized at later stages to infer more complex parts structure (Barlow, 1972; Hubel and Wiesel, 1959, 1968). If parts-based shape processing is hierarchical, it is clear that the first stage involves local orientation of lines, edges, and other extended image elements. This is the primary shape-related tuning dimension in V1 and V2 (Baizer et al., 1977; Burkhalter and Van Essen, 1986; Hubel and Livingstone, 1987; Hubel and Wiesel, 1959, 1965, 1968). It is also prominent at intermediate stages in the ventral pathway— V4 (Desimone and Schein, 1987) and PIT (Tanaka et al., 1991). Orientation tuning has often been interpreted as the first stage in a parts-processing hierarchy, though it has also been viewed as the first stage in a Fourier-like representation (Shapley and Lennie, 1985). If the parts-based hypothesis is correct, orientation signals could be combined to infer more complex contour information of the sort represented in area V4 (see above). Orientation tuning could also lead toward a medial axis-based representation. Lee and colleagues (1998) demonstrated a small but explicit signal for medial axes in area V1, and this could foreshadow more complex axial representation at later stages. According to some theories, orientation signals would be combined to infer the structure of complex 3-D primitives (Biederman, 1987; Dickinson et al., 1992). This has usually been envisioned as a process of detecting unusual configurations (nonaccidental properties) in the 2-D image that imply 3-D structure (Lowe, 1985). However, a recent study of area V4 revealed strong signals for 3-D orientation that could be used directly for deriving 3-D parts structure (Hinkle and Connor, 2002). More complex contour fragments, such as angles and curves, could constitute a second level of structural complexity. The evidence for this in area V4 was discussed in the preceding section. Tuning for contour fragment shape and relative position appears to be even more prominent in PIT/CIT (Brincat and Connor, 2001). Selectivity for angles and curves has also been studied at earlier stages in the ventral pathway: V1 and V2 (Dobbins et al., 1987; Hammond and Andrews, 1978; Hegde and Van Essen, 2000; Heggelund and Hohmann, 1975; Hubel and Wiesel, 1965; Versavel et al., 1990). It should be noted that curvature tuning could also be consistent with holistic representation. Many V4 neurons are selective for curvilinear grating

stimuli (concentric, hyperbolic, and spiral), suggesting that curvature information may be summed in a global fashion (Gallant et al., 1993, 1996; Wilson, 1999). Shape selectivity at higher stages in IT cortex is clearly complex and often related to behaviorally relevant object categories (Desimone et al., 1984). There are some results suggesting that this complex selectivity relates to part structure rather than global or holistic shape. Fujita and colleagues (1992) described a columnar (patchy) arrangement of responses to shape features. Examples presented in that report and others (Tanaka et al., 1991) show selectivity for parts-level features in a variety of global shape contexts (meeting prediction 3 in the previous section). Tsunoda et al. (2001) recently used optical imaging of surface cortex in IT to compare columnar activation by different stimuli. They presented striking examples in which shapes with partially overlapping part structures also evoked activity in partially overlapping sets of columns. Perrett et al. (1982) tested cells in the fundus of the superior temporal sulcus (STS) that were selectively responsive to face stimuli. They found that many cells responded to multiple component facial features with very different shapes (e.g., eyes and hair), suggesting a parts-based synthesis (see also Chapter 78). Wang et al. (2000) used bicuculline to block inhibitory inputs to shape-selective IT neurons. This enhanced responses to previously effective shape stimuli, and in many cases also revealed responses to components of those stimuli. The revealed responses could reflect subthreshold parts-related inputs that contribute to complex shape selectivity under normal conditions. Several recent parametric studies of shape tuning in IT bear on the issue of structural representation. Sigala and Logothetis (2002) trained monkeys to discriminate two categories of cartoon faces. The two categories were distinguished by variations in eye separation and eye height. Mouth height and nose length also varied but were irrelevant to the categorization task. IT neurons showed higher selectivity for the two behaviorally relevant shape parameters. Thus, IT shape tuning functions seem to be modifiable at the level of object parts, suggesting a structural coding scheme. Yamane and colleagues (Yamane et al., 1988; Young and Yamane, 1992) parameterized photographic human faces in terms of distances between facial features. They found that STS and IT responses were best explained in terms of combinations of multiple distance measurements from various parts of the face, implying a holistic representation. Op de Beeck and colleagues (2001) studied variations in silhouette shapes constructed by combining radial frequency components. Radial frequency functions are Fourier-like descriptors for variations in the radius of closed curves. Gradual changes in radial frequency component amplitudes produced parametric variations in overall stimulus shape.

:     

1087

Within groups of gradually varying stimuli, IT neurons showed smooth unimodal tuning along the dimensions defined by radial frequency amplitudes. The entire shape, not just single parts, varied along these dimensions. Thus, smooth tuning could be taken as evidence for global or holistic (rather than structural or parts-based) shape representation. However, smooth tuning might have depended on variations in specific parts of the shapes. Also, individual neurons tended to respond well to very different shapes from multiple groups, perhaps reflecting a common parts-level structure. At the population level, though, responses to the different shape groups were appropriately distinct. Notably, training monkeys to perform different categorization tasks (within groups) produced no discernible differences in neural representation. Baker et al. (2002) trained monkeys to discriminate abstract shapes formed by combining top and bottom parts in a factorial cross. Both parts-level and holistic tuning were enhanced for learned as compared to unlearned shapes, suggesting a mixed coding strategy.

Summary Visual shape recognition is a daunting neurocomputational problem due to the virtual infinity of object shapes and the variability of a given object’s retinal image. Both problems could be overcome by representing objects in terms of their constituent parts or primitives. This kind of structural coding scheme would depend on neurons tuned in a multidimensional space representing part shape and relative position. A number of recent studies of neural responses in the ventral (object-related) pathway of primate visual cortex are consistent with such a scheme. Other results imply a more holistic representation of global shape. Further parametric studies of shape tuning will be needed to fully elucidate the neural coding principles in the ventral pathway. REFERENCES Attneave, F., 1954. Some informational aspects of visual perception, Psychol. Rev., 61:183–193. Baker, C. I., M. Behrmann, and C. R. Olson, 2002. Impact of learning on representation of parts and wholes in monkey inferotemporal cortex, Nat. Neurosci., 5:1210–1214. Baizer, J. S., D. L. Robinson, and B. M. Dow, 1977. Visual responses of area 18 neurons in awake, behaving monkey, J. Neurophysiol., 40:1024–1037. Barlow, H. B., 1972. Single units and sensation: a neuron doctrine for perceptual psychology? Perception, 1:371–394. Biederman, I., 1987. Recognition-by-components: a theory of human image understanding, Psychol. Rev., 94:115–147. Brincat, S. L., and C. E. Connor, 2001. Quantitative characterization of parts-based shape coding in IT cortex, Soc. Neurosci. Abstr., 27. Burkhalter, A., and D. C. Van Essen, 1986. Processing of color, form and disparity information in visual areas VP and V2 of

1088

, ,   

ventral extrastriate cortex in the macaque monkey, J. Neurosci., 6:2327–2351. Desimone, R., T. D. Albright, C. G. Gross, and C. Bruce, 1984. Stimulus-selective properties of inferior temporal neurons in the macaque, J. Neurosci., 4:2051–2062. Desimone, R., and S. J. Schein, 1987. Visual properties of neurons in area V4 of the macaque: sensitivity to stimulus form, J. Neurophysiol., 57:835–868. DeYoe, E. A., G. J. Carman, P. Bandettini, S. Glickman, J. Wieser, R. Cox, D. Miller, and J. Neitz, 1996. Mapping striate and extrastriate visual areas in human cerebral cortex, Proc. Natl. Acad. Sci. USA, 93:2382–2386. Dickinson, S. J., A. P. Pentland, and A. Rosenfeld, 1992. From volumes to views: an approach to 3-D object recognition, CVGP: Image Understanding, 55:130–154. Dobbins, A., S. W. Zucker, and M. S. Cynader, 1987. Endstopped neurons in the visual cortex as a substrate for calculating curvature, Nature, 329:438–441. Edelman, S., 1999. Representation and Recognition in Vision, Cambridge, MA: MIT Press. Edelman, S., and N. Intrator, 2000. (Coarse coding of shape fragments) + (retinotopy) approximately = representation of structure, Spatial Vis., 13:255–264. Felleman, D. J., and D. C. Van Essen, 1991. Distributed hierarchical processing in the primate cerebral cortex, Cereb. Cortex, 1:1–47. Fujita, I., K. Tanaka, M. Ito, and K. Cheng, 1992. Columns for visual features of objects in monkey inferotemporal cortex, Nature, 360:343–346. Gallant, J. L., J. Braun, and D. C. Van Essen, 1993. Selectivity for polar, hyperbolic, and Cartesian gratings in macaque visual cortex, Science, 259:100–103. Gallant, J. L., C. E. Connor, S. Rakshit, J. W. Lewis, and D. C. Van Essen, 1996. Neural responses to polar, hyperbolic, and Cartesian gratings in area V4 of the macaque monkey, J. Neurophysiol., 76:2718–2739. Hammond, P., and D. P. Andrews, 1978. Collinearity tolerance of cells in areas 17 and 18 of the cat’s visual cortex: relative sensitivity to straight lines and chevrons, Exp. Brain Res., 31:329– 339. Hegde, J., and D. C. Van Essen, 2000. Selectivity for complex shapes in primate visual area V2, J. Neurosci. (online), 20:RC61. Heggelund, P., and A. Hohmann, 1975. Responses of striate cortical cells to moving edges of different curvatures, Exp. Brain Res., 23:211–216. Hinkle, D. A., and C. E. Connor, 2002. Three-dimensional orientation tuning in macaque area V4, Nat. Neurosci., 5:665– 670. Hoffman, D. D., and W. A. Richards, 1984. Parts of recognition, Cognition, 18:65–96. Hubel, D. H., and M. S. Livingstone, 1987. Segregation of form, color, and stereopsis in primate area 18, J. Neurosci., 7:3378–3415. Hubel, D. H., and T. N. Wiesel, 1959. RFs of single neurones in the cat’s striate cortex, J. Physiol. (Lond.), 148:574–591. Hubel, D. H., and T. N. Wiesel, 1965. RFs and functional architecture in two nonstriate visual areas (18 and 19) of the cat, J. Neurophysiol., 28:229–289. Hubel, D. H., and T. N. Wiesel, 1968. Receptive fields and functional architecture of monkey striate cortex, J. Physiol. (Lond.), 195:215–243. Hummel, J. E., and I. Biederman, 1992. Dynamic binding in a neural network for shape recognition, Psychol. Rev., 99:480–517.

Kobatake, E., and K. Tanaka, 1994. Neuronal selectivities to complex object features in the ventral visual pathway of the macaque cerebral cortex, J. Neurophysiol., 71:856–867. Koenderink, J. J., 1984. What does the occluding contour tell us about solid shape? Perception, 13:321–330. Lee, T. S., D. Mumford, R. Romero, and V. A. Lamme, 1998. The role of the primary visual cortex in higher level vision, Vis. Res., 38:2429–2454. Lowe, D., 1985. Perceptual organization and visual recognition, Boston, MA: Kluwer Academic Publishers. Marr, D., and H. K. Nishihara, 1978. Representation and recognition of the spatial organization of three-dimensional shapes, Proc. R. Soc. Lond. B Biol. Sci., 200:269–294. Mel, B. W., and J. Fiser, 2000. Minimizing binding errors using learned conjunctive features, Neural Comput., 12:731–762. Milner, P. M., 1974. A model for visual shape recognition, Psychol. Rev., 81:521–535. Op de Beeck, H., J. Wagemans, and R. Vogels, 2001. Inferotemporal neurons represent low-dimensional configurations of parameterized shapes, Nat. Neurosci., 4:1244–1252. Pasupathy, A., and C. E. Connor, 1999. Responses to contour features in macaque area V4, J. Neurophysiol., 82:2490–2502. Pasupathy, A., and C. E. Connor, 2001. Shape representation in area V4: position-specific tuning for boundary conformation, J. Neurophysiol., 86:2505–2519. Pasupathy, A., and C. E. Connor, 2002. Population coding of shape in area V4, Nat. Neurosci., 5:1332–1338. Perrett, D. I., E. T. Rolls, and W. Caan, 1982. Visual neurones responsive to faces in the monkey temporal cortex, Exp. Brain Res., 47:329–342. Poggio, T., and S. Edelman, 1990. A network that learns to recognize three-dimensional objects, Nature, 343:263–266. Selfridge, O. G., 1959. Pandemonium: a paradigm for learning, in The Mechanization of Thought Processes, London: H. M. Stationary Office. Sereno, M. I., A. M. Dale, J. B. Reppas, K. K. Kwong, J. W. Belliveau, T. J. Brady, B. R. Rosen, and R. B. Tootell, 1995. Borders of multiple visual areas in humans revealed by functional magnetic resonance imaging, Science, 268:889–893. Shapley, R., and P. Lennie, 1985. Spatial frequency analysis in the visual system, Annu. Rev. Neurosci., 8:547–583.

Sigala, N., and N. K. Logothetis, 2002. Visual categorization shapes feature selectivity in the primate temporal cortex, Nature, 415:318–320. Sutherland, N. S., 1968. Outlines of a theory of visual pattern recognition in animals and man, Proc. R. Soc. Lond. B Biol. Sci., 171:297–317. Tanaka, K., H. Saito, Y. Fukada, and M. Moriya, 1991. Coding visual images of objects in the inferotemporal cortex of the macaque monkey, J. Neurophysiol., 66:170–189. Tarr, M. J., and S. Pinker, 1989. Mental rotation and orientationdependence in shape recognition, Cogn. Psychol., 21:233–282. Treue, S., K. Hol, and H. J. Rauber, 2000. Seeing multiple directions of motion-physiology and psychophysics, Nat. Neurosci., 3:270–276. Tsunoda, K., Y. Yamane, M. Nishizaki, and M. Tanifuji, 2001. Complex objects are represented in macaque inferotemporal cortex by the combination of feature columns, Nat. Neurosci., 4:832–838. Ullman, S., 1996. High-Level Vision, Cambridge, MA: MIT Press. Ungerleider, L. G., and M. Mishkin, 1982. Two cortical visual systems, in Analysis of Visual Behavior (D. G. Ingle, M. A. Goodale, and R. J. Q. Mansfield, eds.), Cambridge, MA: MIT Press, pp. 549–586. Versavel, M., G. A. Orban, and L. Lagae, 1990. Responses of visual cortical neurons to curved stimuli and chevrons, Vis. Res., 30:235–248. Vetter, T., A. Hurlbert, and T. Poggio, 1995. View-based models of 3D object recognition: invariance to imaging transformations, Cereb. Cortex, 5:261–269. Wang, Y., I. Fujita, and Y. Murayama, 2000. Neuronal mechanisms of selectivity for object features revealed by blocking inhibition in inferotemoral cortex, Nat. Neurosci., 3:807–813. Wilson, H. R., 1999. Non-Fourier cortical processes in texture, form, and motion perception, in Cerebral Cortex, vol. XIII (P. S. Ulinski & E. G. Jones, eds.), New York: Plenum, pp. 445– 477. Yamane, S., S. Kaji, and K. Kawano, 1988. What facial features activate face neurons in the inferotemporal cortex of the monkey? Exp. Brain. Res., 73:209–214. Young, M. P., and S. Yamane, 1992. Sparse population coding of faces in the inferotemporal cortex, Science, 256:1327–1331.

:     

1089

72

Shape and Shading JAN J. KOENDERINK AND ANDREA J. VAN DOORN

T   is able to perceive the geometrical structure of objects, and many of their physical and chemical properties, from the radiation scattered toward the eye. The observer also perceives the light field, that is, the primary and secondary sources of radiation and how radiation pervades space. Although the dynamics of the interaction of the observer with the environment is very important, here we deal only with the monocular, static observer. Most of the optical structure available to such an observer is (approximately) available from pictures ( photographs, computer graphics, paintings) of the environment. We abstract even further and will totally ignore spectral structure (color). We will not focus on the perception of material properties or on that of the light field, although these are interesting and highly relevant topics. Thus, we limit the discussion roughly to the domain of pictorial shape from monochrome pictures, photographs or renderings on computer screens. This has been a generic topic in visual psychophysics for over a century (Helmholtz, 1896; Hering, 1878; Kardos, 1934; Katz, 1911).

The light field For the present purpose radiation is sufficiently described through rays, which are directed straight lines, or photons, which propagate via rectilinear orbits. These entities issue forth from primary radiators (the sun, light bulbs) and are scattered from objects that thereby become secondary radiators. Empty space (air will do) does not interact with the radiation. When rays or photons enter the eye, one “sees light” (or rather simply “sees”). Light is an aspect of consciousness; radiation is a physical entity that is never seen. Consider empty space. It is filled with crisscrossing rays or swarms of photons. Consider some fiducial (and imaginary) volume element and add up the lengths of all rays that happen to cross it. When this total is divided by the volume, you obtain the volume (ray) density of radiation. Alternatively, you may count all photons that happen to be inside the volume within the span of 1 fiducial second of time. When this total is divided by the volume, you obtain the volume ( photon) density. The two measures stand in a fixed ratio and need not be distinguished here. Volume density is important in photo kinesis of simple organisms but is rather irrelevant to the human observer.

1090

One may also consider some fiducial (and imaginary) element of area and count all rays or photons that cross it (within 1 fiducial second, say). Add one for a crossing in one direction and subtract one for the crossing in the reverse direction (thus, you need an oriented area). The net count, divided by the area, depends upon the orientation of the area. For a small element, it is proportional to the cosine of the angle subtended by the surface normal and some direction that is characteristic of the light field. The magnitude and direction defined in this way comprise the net flux vector, which is an important descriptor of the light field (Gershun, 1939) (Fig. 72.1). It is the causal factor in the photo taxis of simple organisms. The net flux vector is the entity that photographers and interior designers ( perhaps unknowingly) refer to when they discuss the “quality of the light.” Consider any closed loop in empty space. When you propagate the loop in the direction of the (local) net flux vector, you generate a tube. Area elements of the boundary of such tubes do not pass any net flux by construction (equally, many rays cross in either direction); thus, the light can be said to be transported by way of such light tubes. In contrast to light rays, light tubes are generally curved. They can even be closed. When a photographer refers to (diffuse) light “creeping around” an object, he or she is (unknowingly) referring to the tubes rather than the rays (Hunter and Fuqua, 1990) (Fig. 72.2). The human intuitive understanding of the behavior of diffuse light fields is based upon lifelong experience with net flux vector fields. Finally (after volumes and surfaces), one may consider lines. How many rays go in a certain fiducial direction through some fiducial point? Clearly, none, for if the number of rays is finite, it is infinitely unlikely to meet with any specific possibility. In order to count a finite number of rays, one needs a finite “environment” of the fiducial ray, known as the phase volume element or étendue (Gershun, 1939; Moon and Spencer, 1981). The étendue is like a slender tube, characterized by the product of its (normal) cross-sectional area and its (solid) angular spread, for neighboring rays can be shifted in space or perturbed in direction. The number of rays in the tube divided by the étendue of the tube is the radiance. The radiance is the single most important entity for the human observer; what we refer to as the light field is simply the radiance distribution. Both the volume density and the net flux vector field can be derived (through suitably

F 72.1. This is one method used to observe (or even measure) the net flux vector field. Put a grease spot on a piece of bond paper. The spot appears lighter than the paper (left) if the back of the paper receives more irradiance than the front, whereas it appears darker than the paper (right) in the opposite case. The spot disappears (center) when the net flux vector lies in the plane of the paper. This way, you can easily map out the flux tubes in a room.

understanding were Lambert (1760) and Bouguer (1729) in the eighteenth century]. Very nice introductions are Gershun’s (1939) paper and Moon and Spencer’s (1981) book (which unfortunately suffers from arcane terminology), but the literature tends to associate the notion with Adelson and Bergen’s (1991) plenoptic function. Rays and light tubes originate from primary radiators and may end at absorbing (blackish) surfaces. They typically neither start nor end in empty space. The only exceptions are spatial sources or sinks. For instance, a volume of air may become a secondary source due to scattering of sunlight. This is important in atmospheric perspective [or air light (Koschmieder, 1924)]. Likewise, black smoke may cause the air to kill rays in midflight and thus form a sink. For the sake of simplicity, we will not consider such cases here.

Objects in the light field

F 72.2. A uniform hemispherical diffuse beam is incident from above (the top row of arrows shows the direction of the incident net flux vector). The lower part of the figure shows how the net flux vector field is influenced by a black (fully absorbing) sphere. Both direction and magnitude are indicated. The illuminance on the surface of the sphere is indicated by shading. Notice that the whole surface is illuminated, also the part that faces away from the source. The obstacle influences both the magnitude and the direction of the net flux vector field: the light tubes “bend around the object.”

averaging over space and/or direction) from the radiance. In empty space the radiance is constant along straight lines. If you pick any point and consider all lines (directions) through the point, you obtain an extreme “fish-eye-like” picture: the radiance in all directions. Thus, the radiance contains all pictures you can shoot from any point in any direction. Indeed, an excellent (because very intuitive) way to think of the radiance is as an infinite filing cabinet [ like Borges’s (1970) creepy library] of all possible pictures of your world. Such an intuitive grasp was, for instance, used by Gibson (1950) in his book on ecological optics. The concept of radiance has existed for centuries [ Leonardo da Vinci (1927) understood it; the first to achieve some formal

Most objects in our ken are opaque and scatter photons at their surfaces. When a photon hits the surface from some direction, it is either absorbed or scattered in another direction. The probability of absorption depends on the direction of incidence, the probability of scattering depends on both the direction of incidence and the direction of exit. We are typically interested only in exit directions that send the ray to our eye; thus, we often refer to the exit direction as the viewing direction. The surface of any object is constantly bombarded by photons. The number of incident rays per surface area is the irradiance of the surface. The (scattered) radiance in the viewing direction divided by (because due to) the irradiance caused by rays from a certain direction of incidence is called the bidirectional reflectance distribution function (BRDF ) (Koenderink and van Doorn, 1996a, 1996b, 1998; Koenderink et al., 1999; Nicodemus et al., 1977). The BRDF depends upon two directions and thus four angular parameters. For most common materials, the BRDF is a very complicated function indeed (CURET, 1997) (Figs. 72.3 and 72.4). Wouldn’t it be nice if the radiance depended not on the viewing direction or on the direction of incidence, but only on the irradiance? Then the BRDF would be a constant instead of a four-parameter function. Alas, such surfaces don’t exist. They can be imagined, though, and were first conceived of by Lambert (1760), who noticed the remarkable fact that a whitewashed wall looks pretty much the same from all vantage points. Some matte surfaces ( paper, plaster) actually come somewhat close. Virtually all theory and psychophysics on shape from shading (Horn and Brooks, 1989) applies to these imaginary surfaces (in psychophysics the stimuli are virtual—a computer screen—rather than real). Since actual surfaces are different, this should give rise to some concern. A “white” Lambertian surface scatters all photons and absorbs none. It has a BRDF of 1/p and is said to possess

   :   

1091

q

q

i

e

dWi n

dWs

f

f

e

i

S

dA

F 72.3. The definition of the BRDF. A narrow beam (angular spread indicated by the incident cone) irradiates the surface S (specified via its normal n). The surface is viewed from another direction. The BRDF is the radiance of the scattered beam divided by the irradiance caused by the incident beam. It depends on all the angles indicated in the diagram.

F 72.4. Two spheres on a desk illuminated by a window and several fluorescent overhead lights. Notice that the multiple sources are immediately apparent from the appearance of the metallic sphere, whereas the Lambertian sphere appears shaded by the “average source.” The cluttered objects in its environment largely cause the radiance pattern due to the metallic sphere, whereas that of the Lambertian sphere looks like a fairly standard textbook rendering. The BRDF of surfaces is just as important as the light field in shaping the radiance field.

unit whiteness (the technical term is albedo). If the surface receives unit irradiance (e.g., 1 mole—Avogadro’s number, or 6.0225 ¥ 1023—of photons per square meter in a second), the radiance of the beam scattered in all directions is 1/p mole photons per square meter, per second, and per unit solid angle (steradian). Since the photons are scattered into a half space (subtending a solid angle of 2p), there might be concern that a factor of 2 is missing here. It is not, though; the difference is due to the averaging over the slant of the

1092

, ,   

F 72.5. A white, matte sphere illuminated with a collimated (top row) and with a hemispherical diffuse (bottom row) beam. (Of course, the beam is vignetted by the support.) The direction of illumination is almost frontal (leftmost figures), from the left (center figures) and a contre jour (rightmost figures). Notice the nature of the body and the cast shadows. Because these are actual photographs, many additional effects (such as reflexes) are evident.

beams (the cosine factor), which reduces the “effective” solid angle by a factor of 2. Consider a white Lambertian sphere in sunlight. Since sunlight is highly directional (collimated ), only one hemisphere catches photons, whereas the other hemisphere is enveloped in an “attached” or body shadow. In the wake of the beam behind the object is the cast shadow, a cylindrical volume of vanishing volume density. The cast shadow becomes visible when you introduce a white object into it: it will not be illuminated, but it “catches” the shadow. The cast shadow volume extends for about 100 times the diameter of the object in the case of sunlight. Of the irradiated hemisphere, not every part catches an equal share of irradiance. When the surface is inclined with respect to the direction of the incident rays, the irradiance becomes less; it is proportional to the cosine of the inclination. This is usually referred to as shading; here we call it the (surface) attitude effect. The body shadow (and often the cast shadow too) tends to be far more visually striking than the shading, yet most of the literature has been devoted to shading alone. In terms of numbers, the hemisphere in shadow receives no radiation at all, whereas the average irradiance of the other hemisphere is one-half of the maximum (or normal ) irradiance. In the visual field, the illuminated hemisphere (seen from the direction of the rays) appears as a disk with a dark rim [ photographers speak of the limb effect (Adams, 1950)] (Fig. 72.5). A collimated beam, like a sunbeam, is an extreme case. Apart from a sunbeam, it is not often encountered outside of the physics laboratory, although it figures prominently in theories on shading. In normal scenes, even in sunlight, the body shadow areas are rarely pitch black due to the effect of scattered radiation from the environment. Frequently encountered light fields are even more diffuse. Photographers (and visual artists in general) tend to avoid collimated beams (they fear lunar effects) and seek clearly directional yet diffuse fields

(da Vinci, 1927; Jacobs, 1988). An example is the uniform hemispherical diffuse beam, where the rays enter from one halfspace. This is approximately realized by an overcast sky (Fig. 72.6) or a large, irradiated white wall on one side of the object, the environment on the other side being dark (common in the photographic studio). In such a light field (see Figs. 72.2 and 72.6), the cast shadow volume extends over less than the diameter of the object. There is no body shadow proper; all areas of the spherical object catch some radiation. Of course, the side of the object that faces the luminous half-space will be more strongly irradiated than the opposite side. This is due to the fact that the object occludes part of the extended source, technically referred to as vignetting [from vignette, an ornamental frame which “occludes” part of the framed picture, in the nineteenth century often stylized leaves of the grapevine (Fr. vigne)]. In the visual field the most strongly illuminated hemisphere still appears as a dark-rimmed disk, but the darkest part of the rim still receives one-half of the normal irradiance, and the average radiance seen from the direction of illumination is five-sixths of that of a normally irradiated planar disk. Real sources are often between the range of the collimated beam (vanishing small angular spread), the hemispherical diffuse beam (all rays make at least some progress in the nominal direction) and the Ganzfeld, which is completely diffuse. Although the extremes are rare, they do occur in natural scenes (direct sunlight with hardly any scattering from environmental objects is almost collimated, whereas in a polar whiteout the Ganzfeld is closely approximated). Most beams are somewhere in between. Professionals (such as photographers) can judge the nature of beams quite accurately through their rendering properties. They look for body and cast shadows, specular reflexes, and especially the

F 72.6. Left: A sunny day on the beach. The light field is the collimated sunbeam with secondary (scattered) radiation from the sky above and the sand below. Notice the pronounced body shadows (face) and cast shadows (of the figure on the sand); Right: This sculpture of grayish stone is illuminated from a heavily overcast sky (almost a perfect hemispherical diffuse beam). Most of the apparent shading is actually due to vignetting. There are no body or cast shadows.

nature of the edge of the body shadow where the rays grazing the surface interact strongly with local surface irregularities (e.g., the pores of the skin in the case of a human face) (Fig. 72.7). The case of multiple sources is also of frequent importance. In the shading of Lambertian objects, only the irradiance of the surface is important, which means that multiple sources have the same effect as a virtual average source. Thus, the fact that the sources are multiple is apparent only due to vignetting. Each source causes particular body shadows and particular cast shadows because the occlusion geometry is different for the different sources. The radiance at the eye is simply the linear superposition of these sources. The professional looks for multiple cast shadows (often introducing a test body—hand or pencil—into the beams for the specific purpose of comparing cast shadows; see Fig. 72.8) and telltale specular reflections. In reflection the sources are often resolved. For instance, the portrait photographer will try to avoid multiple glints in the pupils of the eyes through careful placement of the sources (or will remove them by retouching at a later stage). In much of the literature, the stimuli are due either to (simulated) collimated beams or to a collimated beam (often called point source, a rather unfortunate term) with ambient illumination. The reason is the structure of graphics rendering pipelines. Most programs cheerfully allow rays to enter from the wrong side (from the interior of the object), which yields a negative irradiance due to the attitude effect. The ambient term is simply a constant irradiance that cancels the negative value and returns one to the domain of physical possibilities. (If any negative value remains, it is simply clipped away.) This absurd procedure can be shown to yield (by a miraculous coincidence) the correct formula for the irradiance of a convex object in a uniform hemispherical diffuse field, and since it yields a fast algorithm, it is considered an intelligent implementation of the physics (Foley et al., 1990). It isn’t, though; it works only for convex objects. If some corrugations are added to the surface, which should give rise to the all-important texture, the method yields

F 72.7. A spherical candle with a rough surface finish, left in a collimated beam and center in a hemispherical diffuse beam. Note the local light directions with respect to the surface revealed by the surface irregularities. On the right, a pillar covered with rough plasterwork in direct sunlight. The edge of the body shadow reveals the nature of the source.

   :   

1093

F 72.8. On the left, the cast shadows of the hand on the wall reveal the presence of two distinct (fairly collimated) beams. Notice that this is hardly apparent from the shading of the figure. On the right, cast shadows of objects at various distances from a wall are produced by a sunbeam (the hand and distant foliage behind the photographer). This is a single beam; the shadows look different due to the fact that the angular subtend of the sunbeam is about 0.5 degree.

nonphysical results (Koenderink and van Doorn, 1996a). Many psychophysical reports on shape from shading are rendered virtually worthless due to ignorance of these simple facts. (It would be too unkind to give references here; the reader is gently reminded to watch out for this.) There exist a variety of implementations of the physics in graphics (software or hardware) pipelines. Some approximate the physics fairly closely, but most cut corners in various ways for the sake of computational efficiency at the cost of realism. When graphics renderings are to be used for stimuli, one should always check this out very carefully. A discussion of the various common implementations is beyond the scope of this chapter.

Important photometric effects It is usually convenient to distinguish a number of important levels of scale. These are not to be taken in an absolute sense, but depend upon distance, size of the field of view or region of interest, and spatial resolution of the camera or eye. We distinguish (at least) the level of the scene or setting (insofar as it is of current interest), of the object (of present interest), and of significant texture. Texture is to be understood simply as summarized (incompletely described) structure. Anything below the level of texture is summarized in terms of average radiance at the eye. There may be a great deal of complicated physics at this level, but that is captured in terms of material properties (such as the BRDF ). This means that material properties are level dependent; for instance, a treetop at a distance (leaves not resolved) is a material in this sense (CURET, 1997; Koenderink and van Doorn, 1996a, 1998; Koenderink et al., 1999). That is why objects often are perceived to have material attributes for which one finds no equivalents in physics textbooks. At the level of the object, one describes the structure in terms of shape and local luminous atmosphere (direction and diffuseness of the light field, any additional flare or air light). On the level of the scene, one

1094

, ,   

F 72.9. Context is all-important in shape due to shading. The left-hand column depicts abstract dipoles on a uniform field. An “up” dipole tends to look like a protrusion, a “down” dipole like an indentation due to an implicit “illumination from above” default assumption. The effect is weak and ambiguous. In the central and right-hand columns, the same dipoles are embedded in a context that suggests the actual direction of illumination (an illuminated sphere). The context works because a strong default assumption is that chunks of matter are generally convex. Both up and down dipoles look like protrusions in the central contexts and like indentations in the right-hand contexts. This perhaps suggests that extreme stimulus reduction might not be the best strategy in shape from shading studies. (Although most likely familiar, we haven’t encountered a simple figure like this in textbooks we know.)

distinguishes the general layout and global structure of the light field (sky, earth, clouds, sun, nearby whitewashed wall, etc.). Notice that the object is the context for the texture, and the scene is the context for the object. Contexts are visually all-important because they imply global surface attitude and light direction (Fig. 72.9). Psychophysics in context-free conditions tends to lead to sterile results (Erens et al., 1993a, 1993b). Examples include studies in shape from shading in the absence of occluding contours and/or indications of illumination direction. The distribution of shadow and highlighted areas is the single most important cue in scenes. That is why the artist’s chiaroscuro thumbnail sketches, often made to decide on the global composition of paintings, tend to look like convincing, complete scenes. They fully specify layout and light field. Low-resolution images suffice or even surpass fully rendered scenes in visual realism. The artist is used to squinting or screwing up the eyes to see what is there (Baxandall, 1995; Gombrich, 1995). Shading, shadow, and vignetting capture the shape of an object in its setting (Adams, 1950; Baxandall, 1995; Gombrich, 1995; Hunter and Fuqua, 1990; Jacobs, 1988). Typically, one side of objects is light and the other side is dark due to the directionality of the local light field. The nature of the edge of the body shadow reveals the diffuse-

ness of the light field. Texture is important here; if the light field is very directional, visual texture due to surface corrugations is almost completely limited to the edge of the body shadow (Fig. 72.10). When the light field is very diffuse, the edge is hardly apparent and visual texture is due mainly to vignetting. For this reason, cracks and pits in the surface appear dark. For instance, when one looks at humans, the clefts between fingers, the lips, an arm against the body, and so forth appear as dark lines. For the same reason, the eye sockets commonly appear as dark blotches in the face and cheekbones are accentuated due to dark concave areas, effects commonly accentuated by women using cosmetics. The darkness of a concavity is not due to the attitude effect, but to vignetting, that is, to the depth of the cavity reckoned in (solid) angular terms. (Thus, an optically deep pit may actually be only a minor depression in terms of distance. This leads to confusions in the psychophysical literature; the reader should beware.) The pattern described above is often accentuated by specularities, which appear as a kind of “spicy addition” to the general shading pattern because often they are fairly localized (Fig. 72.11). Specularities commonly arise because the BRDF contains a specular lobe through Fresnel reflection at a smooth interface (Longhurst, 1986). Indeed, a diffuse (almost Lambertian) component due to photons scattered in the bulk material after passing the surface with a superimposed mirror-like reflection at the interface is common in most smoothly finished dielectrics such as polished woods, plastics, and polished stone (Klinker et al., 1987; Lu, 2000).

The distinctness of the specularities depends greatly on the diffuseness of the light field. This is because the specularities are simply mirror images of the light sources (or the environment in general), and thus are very broad (and therefore hardly noticeable) for diffuse light fields and almost punctate (extremely noticeable) for collimated ones. When the light field is collimated, the structure of the specularities depends mainly on the microstructure of the surface. Spectacular examples are offered by the reflection of the sun on rippled lakes. The difference in appearance between an orange and a tomato is largely due to the structure of their specularities. When the scene contains more than one object (virtually all scenes do, except in psychophysical stimuli), the objects scatter photons toward each other and thus interact photometrically (Fig. 72.12). This is a very important effect, because it ties the object causally, in a visually obvious manner, to its environment. That is why reflexes are so highly prized by visual artists (and often exaggerated in paintings): they relate disjunct objects in a visually evident manner ( Jacobs, 1988). If reflexes are left out (as in amateurish painting or many psychophysical stimuli), the scene isn’t easily integrated and tends to fall apart visually. Although many objects are at least somewhat Lambertian, many deviations are common. We have already mentioned the specularities, which are ubiquitous. Other effects which are common enough to merit mention are backscatter, asperity scattering, and translucence.

F 72.10. Three photographs of a tree trunk taken only minutes apart on a clear day, early in the morning, with low sun. The shots were taken (left to right) with frontal illumination (note the cast shadow of the photographer), lateral illumination (note the cast and body shadow of the tree), and a contre jour (note the body and cast shadows of the tree). Texture due to three-dimensional

surface irregularities is mainly evident at the edge of the body shadow. This type of typical texture is worthless from the perspective of shape from texture theorists, yet it is an important cue both to global shape (the cylindrical tree trunk helps to determine the local light direction) and to local shape (the irregularities of the bark).

   :   

1095

F 72.11. On the left a nectarine, on the right a peach, both irradiated with a collimated beam from the same direction. The nectarine is glossy, and the specularity and edge of the body shadow reveal the direction and nature of the beam. The nature of the specularity reveals much about the smoothness of the surface. Note the edge darkening [called the limb effect by Ansel Adams (1950)],

F 72.12. On the left, the back of a hand is illuminated by a distant window. The palm of the hand is turned away from the source and consequently appears quite dark. On the right, a piece of paper is held close to the inside of the hand. The paper “throws a reflex” on the hand, that is, acts like a secondary source, which sends a diffuse beam in the opposite direction from the primary beam. In actual scenes reflexes are always present and quite strong, though they go typically unnoticed by naive observers.

Backscatter is typical for rough surfaces that are close to Lambertian on the micro scale (Koenderink et al., 1999; Oren and Nayar, 1995). (Note that the Lambertian property itself is due to structure on a still finer scale.) Backscatter tends to be due to a combination of two important effects. One is occlusion, that is, the fact that near things may obstruct the view of far things. A rough surface will contain

1096

, ,   

which together with the specularity makes the nectarine appear convex. The peach looks very flat in comparison. It misses the specularity, and the limb effect is effectively canceled through asperity scattering by the fibrous envelope of the peach (which indeed makes it look “peachy”).

many cast shadow areas when irradiated with a collimated beam. The shadowed areas make the surface appear darker, even when not resolved (thus invisible), for they simply decrease the average radiance. When one looks from the direction of the beam, the shadowed areas will be occluded by the (almost normally irradiated, thus very bright) structures that throw the shadows. In that case, the surface must appear quite bright. This is the Richardson effect. It can often be observed as a bright halo that appears around the shadow of one’s head when that shadow falls upon a rough surface such as grass or foliage. The second effect has to do with average surface attitude. On a rough surface there will be many areas almost orthogonal to the visual direction, even though the global surface is slanted. The parts that are slanted subtend small parts in the visual field due to foreshortening, whereas there is no foreshortening for the normally viewed (and thus irradiated) parts. Thus, a rough surface will be much brighter than a smooth, slanted surface when illuminated from the viewing direction simply because it contains an overdose of normally irradiated parts. Due to such effects, rough surfaces often have a very pronounced backscatter lobe. This lobe tends to decrease the limb effect due to shading and thus counteracts shading. It has the general effect of making things look flatter than they are. Asperity scattering is very common in animals and plants, whose surfaces are often covered with hairs (Figs. 72.11, 72.13, and 72.14). In the case of furs, the hairs are aligned with the surface and, when in order (combed), lead to specularities with a very distinct pattern. When the hairs stand on end, asperity scattering proper occurs. The hair tips scatter photons and thus lighten up the surface, especially when there are many hair tips per unit area in the visual field. This happens especially at the occluding

F 72.13. Upper row: a Lambertian sphere; lower row: a dandelion pappus. The columns are for frontal, lateral, and contre jour illumination. (Note that the Lambertian sphere is invisible against the dark background in the contre jour condition.) The dandelion pappus is not shaded but scatters photons to the camera by asperity scattering.

F 72.15. Shape from shading doesn’t properly apply to translucent objects. Here a spherical flask filled with milk has been immersed in clear water to eliminate refraction. It is illuminated with a collimated beam from the left. For concentrated milk (left) the sphere appears to have a proper body shadow, though photometric measurement reveals shading quite unlike that of a Lambertian object. When the milk is diluted with water (right), the body shadow disappears altogether because radiation is scattered from throughout the volume. The radiance that reaches the eye no longer depends only upon the local surface irradiance; the concept of shading does not apply to such cases.

Translucent materials are not at all rare. Most materials are translucent (or even transparent) at the micro scale. When a photon enters the bulk matter, it is repeatedly scattered and effectively performs a random walk until it escapes through the surface. This will typically happen at another location, though; thus, the BRDF description (which describes scattering at a surface point) does not work. The radiance you sample when you look at a point on the surface is due to photons that emerge from a column inside the material (Fig. 72.15). Photons in this column may have originated from any location. Thus, translucent materials look quite different from opaque ones, and concepts such as shading strictly do not apply (Koenderink and van Doorn, 2001) (Fig. 72.15). Scale is important here, for while many materials are translucent on the micro scale, the opaque approximation might suffice on a macro scale (think of human skin, for instance). F 72.14. Asperity scattering is very common on plant and animal surfaces, including those of humans. Even young female faces are covered with downy, almost invisible hairs. Although the hairs on the cheeks are typically not apparent (often below resolution of the camera), they do influence the shading significantly, lightening up the contours and body shadow edges. Asperity scattering is the secret of “velvety” skin in female faces.

contours, where the surface is seen at a grazing angle and hair tip density is very high due to foreshortening. Thus, asperity scattering gives rise to a brightening of the contours. The effect can be seen in dark velvets and peaches (Lu, 2000). Asperity scattering gives surfaces a “peachy” or “soft” look. It counteracts the limb effect due to shading and thus tends to interact destructively with shape from shading processes.

The structure of pictures The radiance distribution at any point specifies all pictures one could possibly take (e.g., using a photographic camera, placing an eye) from that point. The actual pictures also contain features due to the camera or eye. Contrast will be affected by flare and the modulation transfer function ( MTF ) of the optical system. Resolution will be effectively limited, as will the dynamic range. The full range of radiances in a scene containing dark shadows and specular highlights cannot be rendered either photographically or on the computer screen. Thus, the stimuli used in psychophysical experiments almost certainly differ in a number of respects from the radiance field. In pictures the field of view is necessarily limited, thus reducing the context. This may often introduce ambiguities,

   :   

1097

such as reflexes thrown by objects from outside of the field of view. Such effects tend to look unnatural (they frequently lead to complaints by customers of photo finishing facilities), and even when unnoticed, they are likely to distort the visual perceptions of a scene.

Shape from shading Shape is given a variety of meanings in the literature. The standard mathematical meaning of “invariant under a group of transformations” is the most useful one (Klein, 1893). In daily life, the shape of a thing is that which does not change when you rotate or displace the thing. Of course, this notion of shape depends on the choice of the transformation group. For instance, if you include magnifications, all spheres have the same shape, irrespective of their size. If you include arbitrary scalings in orthogonal dimensions, all triaxial ellipsoids have the same shape (and in fact are not different from spheres), and so forth. In shape from shading, the shape is the invariant under changes of illumination of a scene as evident from pictures of that scene. Here one needs to specify a variety of additional parameters—for example, that all pictures should be taken from the same point and with the same camera. Shape by no means exhausts the realm of inferences that might be drawn from a picture. Apart from its shape, one might ask for other geometrical parameters (distance, spatial attitude, size, etc.), for material properties (glossy or matte, dry or wet, etc.), or for properties of the light field instead of the scene. Most often, a number of different properties must be expected to be mutually dependent. For instance, inferences concerning the light field, the material properties, and the shape can hardly be expected to be independent. Photographers straighten a crooked nose through suitable illumination, women apply cosmetics in order to change the visual shape of their faces, and so forth (Fig. 72.16). Such interdependencies have been largely ignored in the literature. An exception that has attracted (perhaps too much) attention has been the concave-convex ambiguity (Kardos, 1934). The idea is that the pictures of a hemispherical concavity or a hemispherical boss on a plane are identical if illuminated from opposite directions. If this were true and if the stimulus yielded no cue (by context) concerning the direction of illumination, the observer would be at a loss to decide, and the responses would be completely idiosyncratic. The general consensus is that observers are biased toward the assumption that the illumination comes from above rather than from below. One problem with such experiments is that the stimuli are typically highly stylized; indeed, most often they represent unphysical situations. In actual scenes it is often difficult to obtain “clean” results, because the stimuli are not ambiguous at all; indeed, there

1098

, ,   

F 72.16. The dirt of decades has descended upon this statue, leading to apparent illumination from below (the dirt on top is dark and appears as shadow). The actual illumination is diffuse, though it appears fairly directional. The principle is the same as that used by women when they apply cosmetics to darken eye sockets or accentuate cheekbones.

are many differences. For instance, the concavity has no optical interaction with the plane at all, but the boss has. The boss throws a cast shadow on the plane and receives reflexes from it. The concavity vignettes the source and allows multiple scattering in its interior. All this leads to very different pictures. That does not mean that observers never experience convex-concave confusion, but the situations most likely to lead to such confusion are quite different from what the literature suggests. For instance, the most certain way to have observers report a convexity when a concavity exists is to illuminate the latter normally (thus, the above/below confusion doesn’t come up at all). In that case the center of the concavity appears brighter than the plane (due to internal reflexes), and it becomes very difficult to see the concavity as anything but convex (Fig. 72.17). In the typical setting preferred by psychophysicists, pictures show isolated objects (very little context) in uniform light fields, either collimated (point source) or hemispherical diffuse (point source with ambient illumination). The surface finish tends to be either Lambertian or one of the BRDFs supported by standard graphics pipelines [such as Phong (1975) shading]. When texture is applied it tends to be flat (because the graphics engines allow easy texture mapping), which renders the texture cue worthless from the photometric point of view. In the majority of cases, the rendering fails to take either vignetting or multiple scattering into account. Ray tracing or radiosity renderings will allow for this, but these methods are computationally expensive and only approxi-

F 72.17. A hemispherical pit (left column) and boss (right column) on a plane. Illumination is with a collimated beam in all cases. The configurations are of matte white material. In the top row (actual photographs), the cast and body shadows and the reflexes evident in oblique illumination leave no doubt about which is which. The bottom row shows exact simulations of true Lambertian surfaces. Instead of a hemispherical boss (right), there is a full sphere against an infinitely distant white screen. Illumination is frontal. Note that the cavity at the bottom left looks much like a boss or a full sphere. The reason might be that the center is actually brighter than the plane due to internal reflexes in the cavity.

mately simulate the actual physics (Langer and Bülthoff, 2000). In collimated light fields (sunlight), the most important cues are body shadow, the textural quality of the body shadow edge, the structure of the specularities, and the structure of the visual contour. An orange and a tomato both look spherical due to the overall shape of their contour and their body shadow; they differ in that the orange has a serrated contour, a broken-up specularity, and a roughly textured body shadow edge (Lu, 2000). Apart from the overall shape of the contour and the cast shadow, these cues typically are absent in commonly used paradigms. In diffuse light fields the major cues are the overall shape of the contour, the textural effects due to vignetting, and the shading. Note that in this case the shading is not due to the attitude effect (thus, the standard shape from shading theories don’t apply), but to vignetting. Vignetting makes the direction of the local light field at a surface element vary from place to place (and so does the degree of diffuseness), even if the global light field is nominally uniform (Koenderink and van Doorn, 1996a) (Fig. 72.18). The standard point source with ambient illumination fails to capture this. Classical shape from shading theories assume that surfaces are perfectly Lambertian (thus, the radiance at the eye is perfectly proportional to the irradiance) and that each part

F 72.18. An artificial face painted matte dark gray (bottom row) and matte white (top row), illuminated frontally with a collimated beam (left column), a hemispherical diffuse beam (center column) and from high frontal with a collimated beam (right column). Note the differences, especially in the eye sockets and cleft between the lips, due to vignetting and reflexes. Vignetting occurs with the diffuse illumination (center column). Reflexes occur for the white material (top row) and hardly at all for the dark material (bottom row). Note the contrast of the texture due to the (three-dimensional) roughness of the material. The texture is strongest under collimated illumination, near the shadow boundary, on dark material. Texture contrast is less for diffuse illumination (due to the attitude effect), for light materials (due to reflexes), and for less oblique illumination (due to the attitude effect). Notice that classical shading is only one (and not necessarily the more important) cue.

of the surface is illuminated by the same source (no vignetting) (Horn and Brooks, 1989). Then the radiance received by the eye depends only upon the angle between the local surface normal and the net flux vector. In practice, this is approximately realized in the case of matte surfaces with shallow relief (no vignetting or reflexes). Then all surface elements with the same inclination with respect to the net flux vector will send the same radiance to the eye. For smooth surfaces such elements lie on curves, the isophotes. Shape from shading then derives the shape from the pattern of isophotes. Here shape can be defined locally as the relation between the surface normals of adjacent locations. When adjacent normals are not parallel, the surface is obviously curved. Because the irradiance of the surface depends only on the direction, not on the location, of the surface element, it is convenient to abstract from the locations altogether. The surface normals can be specified as unit vectors or, equivalently, as points on the unit sphere. Thus, one may conceive of the mapping of surface elements upon the unit sphere. This map is known as the spherical image or Gauss map (Gauss, 1827). Consider some Gauss maps: a plane obvi-

   :   

1099

ously maps on a point (all its normals being parallel); thus, the Gauss map can be highly degenerate. Cylindrical and conical surfaces (more generally developable surfaces) map on curves. All other surfaces map on two-dimensional patches of the unit sphere. In that case, one can set up a one-toone relation between the spherical image and the original surface, at least locally. This is extremely useful, because the irradiances of the unit sphere and the surface (in the same light field) must correspond (parallel normals by construction). Since the pattern of isophotes on the sphere is easy to construct (concentric small circles for a uniform light field), one immediately constructs the isophotes on the surface via the aforementioned map. Going in the other direction, the received radiance specifies the irradiance on the surface, thus on the unit sphere, and hence the local normal direction. The radiance distribution thus specifies the distribution of normals, which again (partly) specifies the shape (Koenderink and van Doorn, 1980, 1982). Since the Gauss map may well be degenerated, complications arise. One point on the sphere may easily correspond to many points on the surface. One has to think of the spherical images as thoroughly “wrinkled,” with various folds and cusps, such that the sphere is multiply covered, the degree of covering varying from region to region (Fig. 72.19). This may be conceived of as either a curse or a blessing. It is a curse in the traditional theories because classical calculus is ill fit to deal with singularities. Indeed, many theories owe more to the straitjacket due to the method adopted by their originators than to the phenomenology of the field they are supposed to explain. In this case, the curse may be turned into a blessing, though, when one analyzes the structure of the field of isophotes near such folds. One finds that the wrinkles give rise to critical points (minima, maxima, and saddle points) of the irradiance field on the surface, even where the irradiance field on the sphere has no extrema or saddle points. Thus, such easily identifiable features of the radiance distribution signal the presence of folds of the spherical image. Folds of the Gauss map are due to inflections of the surface, the parabolic curves. These curves are important shape indicators. They are curves on the surface that bound convex, concave, and saddle-shaped regions. The pattern of parabolic curves can serve as a qualitative shape description. This pattern can be inferred from the detection of critical points of the radiance (maxima, minima, and saddle points, which are clearly apparent visually). This is a fairly robust way to make shape inferences, since one may relax a number of the strict assumptions underlying classical shape from shading. This is the case because the critical points are due more to the wrinkles than to the irradiance pattern on the sphere (Koenderink and van Doorn, 1980, 1982, 1993). Although these structures have proven useful in machine vision, they have not yet been researched in the case of human vision.

1100

, ,   

F 72.19. This shallow hill on a planar surface does not generate a contour except when viewed rather obliquely (leftmost column). Clearly, its summit has the same spatial attitude as the ground plane; thus, the Gauss map is multiply covered: there exists a parabolic curve at the locus of inflection of the curves of steepest descent from the hill; it is a closed loop that encircles the hill. When illuminated from above with a collimated beam, the parabolic curve shows up as a dark annulus, the top of the hill being as light as the plane (top center). For a slightly more oblique incidence, we still find an illuminance maximum near the summit but an additional one on the parabolic curve, as well as an additional minimum, also on the parabolic curve (top right). For a rather oblique incidence, the maximum near the summit is lost and we see a pair of extrema at opposite sides on the parabolic loop. When the direction of illumination is varied, these extrema travel along the parabolic curve but never “break loose.” Such dipole patterns are typical of the shading patterns of local protrusions on a surface and can be considered textons that specify protrusions. This is a robust cue because the precise illumination or shape is irrelevant.

Psychophysical results There is remarkably little psychophysical material on shape from shading that might be considered in any way definitive. There are multiple reasons for this, but most in some way have to do either with extreme stimulus reduction or with nonrealistic stimuli. The latter case is utterly trivial and will not be discussed here. It is by no means irrelevant, though. Perhaps unfortunately, much of the literature can safely be skipped unread due to the fact that the authors (no doubt cheerfully and in good faith) relied on graphics software that cut various corners for the sake of computational efficiency, but at the cost of physical realism. It is often impossible to establish the exact nature of the stimuli, since the authors either used the software without any real understanding or because it is impossible to establish the exact nature of the algorithms used in proprietary software. Some authors are apparently aware of possible problems but evade their responsibility by blaming them on computer graphics. This will not do. Worse still, others are blissfully unaware of the pitfalls. It is to be hoped that such problems will (in historical retrospect) turn out to be specific to the 1980s to 2010s period. The former problem is of scientific interest because it does reflect a con-

scious choice on the part of the authors. The idea is that the scientific method implies that one should eliminate as many irrelevant parameters as possible and study the system in as simple a representation as possible (e.g., in vitro). Thus, if the problem is shape from shading, then the stimulus should contain only shading and the response should address only shape. The problem with this simpleminded, dogmatic approach is that such stimuli never evoke a clear impression of shape (Erens et al., 1993a, 1993b) in the first place; thus, such measurements involve forced choices of the kind “Have you stopped beating your wife?” (Answer “yes” or “no”). The stimuli make no sense to the observers (nor do their responses), but the scientist is happy with both stimuli and results. The forced-choice paradigm guarantees scientific respectability. Such studies—which make up the bulk of the literature—can safely be skipped unread. There have been few counterforces in the history of experimental psychology, one notable exception being James Gibson (1950). What is known is evident as much from the writings of professionals in the visual arts and of particularly perceptive art historians as from the technical psychophysical literature. It is clear that shape impressions derive from (Fig. 72.20) Pure contour (silhouettes); Cartoon drawings (perhaps incomplete outline drawings with internal contours); The mere indication of illuminated and shadowed areas; And from pictures that contain all of these. It is also known that Specularities promote shape understanding;

So do additional cues from texture, especially texture due to shadow and shading on a micro scale through threedimensional surface corrugations (not so much from the “shape from texture” stimuli familiar from the psychophysical literature). Shape impressions (we will discuss the operational definition of these entities later) of various observers only partly agree. Indeed, it cannot be otherwise because of the inherent cue ambiguity. Responses can never be expected to agree completely, for what is not specified by the stimulus has to be supplied by the observer when the task calls for it. This “beholder’s share” is by its very nature completely idiosyncratic. Ideally, the task should not call for the creative imagination of observers, but in practice this can hardly be avoided. To some extent (due to the inherent ambiguity), perception has to be “controlled hallucination.” But the responses of observers may be expected to agree (at best) only to the extent to which the shape is specified by the available cues. A rational comparison of responses should allow for this and disregard the beholder’s share. This has very important consequences for the interpretation of psychophysical data. The agreement of results obtained by different observers is a measure of the degree of the efficacy by which the cues are picked up by the observers, but only if one reckons modulo the cue ambiguity. It is indeed possible to show (Koenderink et al., 1996a) that the concordance becomes greater when the bouquet of available cues is enriched from mere silhouette to cartoon drawing and finally to fully shaded renderings. In that respect, human observers can be held to exploit the shape from shading cue. What is also clear is that observers’ shape impressions differ when the shading is varied (Koenderink et al., 1996b) (Fig. 72.21 and 72.22). Although to a first rough approximation constancy prevails (the perceived shapes don’t depend much on the direction of illumination, for instance), it can easily be shown that perceived shape nevertheless significantly and systematically depends upon the direction of illumination. This is nothing new, since visual artists have known for ages that faces illuminated from below are rendered quite unrecognizable. This is not only due to the fact that faces are special; it is the case for abstract sculpture as well.

Open problems

F 72.20. Three stimuli—a silhouette, a cartoon rendering, and a full-scale monochrome photograph—have identical geometry. The responses (depth contours) of a single subject show obvious differences. (The observer was confronted with the stimuli in this sequence.) Different observers vary greatly on the silhouette, much less so on the cartoon, and hardly at all on the photograph.

There remain numerous open problems in this particular domain; indeed, it is our firm conviction that most of the basic science in shape from shading remains to be done. Perhaps the majority of the literature is irrelevant due to incomplete description of the stimuli (the software problems mentioned above), extreme stimulus reduction (which transforms the task from one relating to the visual world into one involving only the visual field), or invalid paradigms due to

   :   

1101

F 72.21. On the left, four geometrically identical photographs of an object with an illumination from four different directions. On the right are the residual depth maps (depth maps minus the average depth map for all four cases) of a single observer. Although the responses are very similar in all four cases (constancy), significant and systematic differences exist. It is as if the shape were “pulled out in the direction of the source.” (From Koenderink et al., 1996b.)

incomplete understanding of the relevant geometry and photometry. It is important that stimuli be sufficiently complex to be considered natural, because only then can normal processing of the available cues be expected. This does not imply that single-cue studies are ruled out. One may use the standard engineering method of varying the cue structure in the neighborhood of a set point and correlating the resulting variations in the responses with those imposed upon the stimulus. When only a single parameter is varied, one effectively studies a single cue, but in a natural setting. Here lies a huge and very important field of endeavor. It is important that stimuli be sufficiently complex to be reckoned generic. This involves an understanding of the ambiguities (Fig. 72.23) left by the cues. For instance, the majority of studies in shape from shading have involved Lambertian (or somewhat glossy) triaxial ellipsoids. These objects fail to be generic, because they are the only objects for which isophotes and shadow boundaries are planar curves. Any ellipsoid with the given contour is a “solution” if the lighting direction is unknown. Many shape from shading algorithms will simply not apply because these

F 72.22. Crumpled paper illuminated from between the frontal and top left, top right, bottom left, and bottom right. Note how difficult it is to obtain a clear notion of the shape of this landscape—for instance, try to identify the hills and dales in all four images. In these examples the directions of illumination are not at all extreme.

1102

, ,   

F 72.23. A single photograph (top left figure) can be “explained” via many different scenes. The depth of relief and obliqueness of illumination can be traded (top right figures), but even more complicated transformations (full and partial convex/concave inversions) are possible.

objects are degenerate. For instance, the Gauss map does not contain folds; thus, the powerful (because topological and thus robust) algorithms simply do not apply. Here stimulus reduction leads to severe constraints on the observer’s possibilities, perhaps excluding the use of the typical mechanisms. It is often thought that simple Euclidean objects (planes, ellipsoids, cylinders, regular polyhedra, etc.) make simple stimuli, but in fact these are often more complicated because they are degenerate and thus singular cases. Simple stimuli are generic and thus complicated from the perspective of high school geometry. There is little excuse to stop at such objects when modern computer techniques allow one to deal with generic cases with ease. Due to a scarcity of results, this represents another large and potentially fruitful field of endeavor. It is important to understand that the very notion of veridicality depends upon the intrinsic ambiguities of the setting and thus on the precise nature of stimulus and response reduction. Part of the response has to be understood as completely idiosyncratic (the beholder’s share); part can at least conceivably be due to the cues. When observers agree in the latter part, they behave identically insofar as their responses are caused by the stimulus. This is the realm of psychophysics proper. When the notion of veridicality (all too often implicitly invoked in the evaluation of the response) depends on entities that are inherently ambiguous, the concept is void. The beholder’s share is a product of the creative imagination and (by definition) is not causally related to the stimulus, at least not in the immediate sense relevant to psychophysics. These topics are subtle and tricky and should be considered explicitly. This is a difficult topic

because the analysis of cue ambiguity remains incomplete for most of the relevant cues. Thus, the concept of veridicality should be used less lightly than is common in the literature. This is a very important and potentially rewarding field. It involves the simultaneous development of the theory and psychophysics. How might one draw on the theories of shape from shading developed by the computer vision community for template theories of human shape from shading performance? The point is a very tricky one, yet typically is skipped (apparently it is thought to be a trivial issue or is not considered at all) in the literature on human vision. Most authors appear satisfied with trite remarks and indeed get away with it. The available material is typically an analysis of the physics (in a suitably simplified setting in order to render the problem soluble), eventually resulting in an algorithm. Ideally, one feeds the algorithm an image of some object; the algorithm crunches on it for a while and finally responds by yielding a three-dimensional description of the object in the image. The description could be a triangulation of the surface, for instance, a computer-aided design (CAD) file or a list of instructions to a computer-controlled machine that would yield a mechanical copy of the object. In order to turn this into a candidate theory for human performance, one needs to relate the algorithm to the observer’s input (the stimulus) and output (the response). The latter part seems difficult because few responses collected in psychophysical experiments are anywhere near triangulations, CAD files, or instructions to machine-controlled milling machines. The difficulties related to the former part should not be overlooked. Should one feed the pixel values of the

   :   

1103

stimulus image or the local brightnesses perceived by the observer into the algorithm, for instance? Perceived brightness or lightness is a tricky area in its own right (Adelson, 2000; Gilchrist, 1994); the relation to radiances or albedos is at best nonlinear and quite likely not one-to-one. These are important topics (one would like to draw upon the no doubt highly relevant work of the computer vision community) that need much attention. The potential rewards are appreciable. It should not be thought that all lightness gradients in artistic renderings are attempts to render shading in the photometric sense. Most artistic products are not so much (passive) renderings of a scene in front of the artist as attempts to create a work that will elicit certain responses in potential viewers. Shading in this latter sense was effectively used from the Stone Age on, and it is still a major component of artistic practice, even by artists who have been trained in photometric shading in art school. Of course, it would make no sense to run shape from shading algorithms on such pictures. Yet these pictures are often extremely effective in evoking vivid pictorial relief in human observers (Hogarth, 1981). Many “superrealistic” renderings are of this kind. Similar instances can often be observed in many images that result from a variety of scientific methods. A well-known example is scanning electron microscopic images. These tend to look very realistic (indeed, much like the products of superrealist draftsmen), though the process by which radiance modulations are obtained differs drastically from those that operate in the generic human environment. Here lies a wide field of potential study that has been completely ignored by the vision community. Finally, it is very important that novel psychophysical tools be developed that allow one to probe perceived shape effectively. In the final analysis, perceived shape can only be defined operationally. In many cases, one evokes responses of observers that are (hopefully) mainly based on shape impressions these observers are believed to have, but these impressions themselves are never operationalized explicitly. This simply will not do. Shape impressions, though aspects of consciousness, are nevertheless geometrical entities that cannot be described via mere “yes/no” answers or magnitude estimations. One needs methods that yield responses that can be treated as geometrical objects, say triangulations, or other forms of surface description. Such descriptions necessarily involve data structures of finite geometry that will include hundreds or thousands of data points. (Think of the size of CAD files that are used to describe the shapes of industrial objects.) This again implies that psychophysical methods should yield at least 1000 bytes per hour instead of a few yes/no answers or 5-point scale ratings if the aim is to describe the structure of a perceived shape. Theories of shape from shading can only be assessed effectively when

1104

, ,   

such methods become readily available. There is no reason why this should be considered impossible, yet very little progress has been made. We consider this to be yet another open problem. Novel advances on this topic may well lead to considerable scientific progress in the future.

REFERENCES Adams, A., 1950. Photography by Natural Light (Basic Photo Series, Book 4), New York: Morgan and Lester. Adelson, E. H., 2000. Lightness perception and lightness illusions, in The New Cognitive Neurosciences, 2nd ed. (M. Gazzaniga ed.), Cambridge, MA: MIT Press, pp. 339–351. Adelson, E. H., and J. R. Bergen, 1991. The plenoptic function and the elements of early vision, in Computational Models of Visual Processing (M. Landy and J. A. Movshon, eds.), Cambridge, MA: MIT Press, pp. 3–20. Baxandall, M., 1995. Shadows and Enlightenment, New Haven, CT: Yale University Press. Borges, J. L., 1970. The library of Babel, in Labyrinths, Selected Stories and Other Writings (D. A. Yates and J. E. Irby, eds.), Harmondsworth, Middlesex, UK: Penguin Books. Bouguer, P., 1729. Essai d’optique sur la Gradation de la Lumière, Paris: Claude Jombert. CURET, Columbia-Utrecht Reflectance and Texture Database, 1997. http://www.cs.columbia.edu/CAVE/curet da Vinci, Leonardo, 1927. Traité du Paysage (Codex Vaticanus), Paris: Librairie Delagrave. Erens, R. G. F., A. M. L. Kappers, and J. J. Koenderink, 1993a. Perception of local shape from shading, Percept. Psychophys., 54:145–156. Erens, R. G. F., A. M. L. Kappers, and J. J. Koenderink, 1993b. Estimating local shape from shading in the presence of global shading, Percept. Psychophys., 54:334–342. Foley, J. D., A. van Dam, S. K. Feiner, and J. F. Hughes, 1990. Computer Graphics, Principles and Practice, 2nd ed., Reading, MA: Addison-Wesley. Gauss, C. F., 1827/1889. Algemeine Flächentheorie (German translation of Disquisitiones generales circa Superficies Curvas), Hrsg. A. Wangerin, Ostwald’s Klassiker der exakten Wissenschaften 5, Leipzig: Engelmann. Gershun, A., 1939. The light field (P. Moon and G. Timoshenko, trans.), J. Math. Phys., 18:51. Gibson, J. J., 1950. The Perception of the Visual World, Boston: Houghton Mifflin. Gilchrist, A., 1994. Lightness, Brightness and Transparency, Hillsdale, NJ: Erlbaum. Gombrich, E. H., 1995. Shadows, the Depiction of Cast Shadows in Western Art, London: National Gallery Publications. Helmholtz, H. von, 1896. Handbuch der physiologischen Optik, 2nd ed., Hamburg: Voss. Hering, E., 1878/1964. Outlines of a Theory of the Light Sense (L. M. Hurvich and D. Jameson, trans.), Cambridge, MA: Harvard University Press. Hogarth, B., 1981. Dynamic Light and Shape, New York: Watson-Guptil. Horn, B. K. P., and M. J. Brooks, 1989. Shape from Shading, Cambridge, MA: MIT Press. Hunter, F., and P. Fuqua, 1990. Light, Science and Magic: An Introduction to Photographic Lighting, Boston: Focal Press. Jacobs, T. S., 1988. Light for the Artist, New York: Watson-Guptill.

Kardos, L., 1934. Ding und Schatten: eine experimentelle Untersuching, Zeitschrift für Psychologie Erg.-Bd. 23, Leipzig: Barth. Katz, D., 1911. Die Erscheinungsweisen der Farben und ihere Beeinflussung durch die individuelle Erfahrung, Zeitschrift für Psychologie Erg.-Bd 7, Leipzig: Barth. Klein, F., 1893. Vergleichende Betrachtungen über neue geometrische Forschungen (Erlanger Programm), Math. Ann., 43:63–100. Klinker, G. J., S. A. Shafer, and T. Kanabe, 1987. Using a color reflection model to separate highlights from object color, in Proceedings of the First International Conference on Computer Vision (ICCV), (J. M. Brady and A. Rosenfeld, eds.), London: Computer Society Press, pp. 145–150. Koenderink, J. J., and A. J. van Doorn, 1980. Photometric invariants related to solid shape, Opt. Acta, 27:981–996. Koenderink, J. J., and A. J. van Doorn, 1982. Perception of solid shape and spatial layout through photometric invariants, in Cybernetics and Systems Research (R. Trappl ed.), Amsterdam, North Holland. Koenderink, J. J., and A. J. van Doorn, 1993. Illuminance critical points on generic smooth surfaces, J. Opt. Soc. Am., A10:844– 854. Koenderink, J. J., and A. J. van Doorn, 1996a. Illuminance texture due to surface mesostructure, J. Opt. Soc. Am., A13:452– 463. Koenderink, J. J., A. J. van Doorn, and M. Stavridi, 1996b. Bidirectional reflection distribution function expressed in terms of surface scattering modes, in Computer Vision—ECCV’96, vol. II (B. Buxton and R. Cipolla, eds.), Berlin: Springer. Koenderink, J. J., and A. J. van Doorn, 1998. Phenomenological description of bidirectional surface reflection, J. Opt. Soc. Am., A15:2903–2912.

Koenderink, J. J., and A. J. van Doorn, 2001. Shading in the case of translucent objects, in Human Vision and Electronic Imaging VI, (B. E. Rogowitz and T. N. Pappas, eds.), SPIE vol. 4299, Bellingham, Washington, pp. 312–320. Koenderink, J. J., A. J. van Doorn, C. Christou, and J. S. Lappin, 1996a. Shape constancy in pictorial relief, Perception, 25:155– 164. Koenderink, J. J., A. J. van Doorn, C. Christou, and J. S. Lappin, 1996b. Perturbation study of shading in pictures, Perception, 25:1009–1026. Koenderink, J. J., A. J. van Doorn, K. J. Dana, and S. Nayar, 1999. Bidirectional reflection distribution function of thoroughly pitted surfaces, Int. J. Comput. Vis., 31(2/3):129–144. Koschmieder, H., 1924/1968. see Middleton, W. E. K., Vision Through the Atmosphere, Toronto: University of Toronto Press. Lambert, J. H., 1760. Photometria sive de mensure de gradibus luminis, colorum et umbræ, Augsburg, Germany Eberhard Klett. Langer, M., and H. H. Bülthoff, 2000. Depth discrimination from shading under diffuse lighting, Perception, 29:649–660. Longhurst, R. S., 1986. Geometrical and Physical Optics, London: Longman. Lu, R., 2000. Ecological optics of materials, Ph.D. thesis, Utrecht University. Moon, P., and D. E. Spencer, 1981. The Photic Field, Cambridge, MA: MIT Press. Nicodemus, F. E., J. C. Richmond, and J. J. Hsia, 1977. Geometrical considerations and nomenclature for reflectance, Natl. Bur. Stand. (U.S.), Monograph 160. Oren, M., and S. K. Nayar, 1995. Visual appearance of matte surfaces, Science, 267:1153–1156. Phong, B.-T., 1975. Illumination for computer generated images, Commun. ACM, 18(6):311–317.

   :   

1105

73

Visual Perception of Texture MICHAEL S. LANDY AND NORMA GRAHAM

W   , and how might a study of the visual perception of texture help us to better understand human vision? In this chapter we will attempt to give the reader a feel for how the study of texture perception is useful in understanding the impact of texture, as well as in providing a better understanding of basic visual mechanisms that respond not only to texture but to all visual stimuli. This review will be relatively brief and, of necessity, incomplete. We hope to give an overview of the different research areas concerned with texture perception and of the current issues. For a longer early review, we refer the reader to Bergen (1991). Consider the scene in Figure 73.1. The border between the sky and the trees/grass involves a difference in luminance, one that would easily be signaled by a linear mechanism such as a simple cell in primary visual cortex. The boundary between the zebras and the background also involves a change in chromaticity (although not visible in the black-and-white image in Fig. 73.1), which might be signaled by color-opponent mechanisms. But the borders between pairs of zebras involve neither a difference in color nor a difference in average luminance. These borders include stretches of boundary that are black on one side and white on the other, stretches where the colors are reversed, and stretches where there is no local visual information to signal the boundary (where black abuts black or white abuts white). Nevertheless, we perceive a smooth, continuous occlusion boundary at the edge of each animal. It is as if the visual system possesses the capability of segmenting regions of the image based on a local textural property, such as separating “vertical stuff ” from “horizontal stuff.” Thus, texture is a property that is statistically defined. A uniformly textured region might be described as “predominantly vertically oriented,” “predominantly small in scale,” “wavy,” “stubbly,” “like wood grain,” or “like water.” As Adelson and Bergen (1991) put it, texture is a property of stuff in the image, in contrast to visual features such as lines and edges, the things in the image (analogous to the linguistic difference between mass nouns like water and count nouns like mouse). Another way of characterizing visual texture is by the uses to which it might be put. Texture is a property of an image region. Regions in the visual field can be characterized by differences in texture, brightness, color, or other attributes. Relatively early processes in the visual system can use texture

1106

information to perform a tentative segmentation of the visual image into regions to ease the processing load on subsequent computational stages. The analysis of a single textured image region can lead to the perception of categorical labels for that region (“This looks like wood” or “This surface looks slippery”). The appearance of texture allows the observer to determine whether two textured regions appear to be made of the same or different stuff. If two abutting image regions have different surface texture, this may lead to the detection of the intervening texture border (like the border between adjacent zebras in Fig. 73.1). Such texture-defined boundaries may then be used to segment figure from ground and for two-dimensional shape identification. Finally, continuous changes in texture properties may result in the percept of three-dimensional shape (Gibson, 1950). A purpose of much research in this area is to define the mechanisms and representational schemes used to characterize texture, and thus to determine whether the same underlying mechanisms are responsible for each of the above perceptual capabilities.

Texture segregation T F Much of the work on perception concerns the ability of observers to discriminate certain texture pairs effortlessly. For example, Figure 73.2 shows rectangular regions of Xs and Ts on a background of Ls. Observers can perceive effortlessly that there is a region of Xs different from the background, that this region has smooth, continuous borders, and that these borders form a rectangular shape. This is referred to as the segregation of figure from ground or segmentation of the image into multiple homogeneous regions. At the same time, none of these observations may be made about the region of Ts without the use of effortful scrutiny of the individual texture elements one by one. This sort of observation led a number of investigators to consider what aspects of image structure led to preattentive segregation of textures. Beck and Attneave and their colleagues (Beck, 1972, 1973; Olson and Attneave, 1970) hypothesized that textural segmentation is based on the distribution of simple properties of texture elements, where the simple properties are things like the brightness, color, size, the slopes of contours, and other elemental descriptors of a texture. Marr (1976) added contour terminations as an important feature.

F 73.1. content.

Types of image borders. A natural image containing borders signaled by differences in luminance, color, and/or textural

F 73.2. Texture segregation. Note that the region of Xs on the left is easily segregated from the background of Ls. One immediately perceives the borders between the two regions and the shape of the region containing the Xs. By contrast, the border between the Ts and Ls is difficult to see, and the shape of the region of Ts can only be discerned slowly, effortfully, and with item-by-item scrutiny.

Julesz’s early efforts centered on image statistics. He first suggested ( Julesz et al., 1973) that differences in dipole statistics were most important for texture pairs to segregate. (These are the joint image statistics of the gray levels found at the opposite ends of a line segment of a particular length and orientation, as it is placed at all possible image locations, gathered for all possible pairs of gray levels, dipole lengths, and orientations.) But counterexamples to this were found (e.g., Caelli and Julesz, 1978). It was then suggested that textures with identical third-order statistics would prove indiscriminable. (Analogous to dipole statistics, these are joint image statistics of the gray levels found at the three corners of a triangle with a particular size, shape, and orientation as it is placed at all possible image locations, gathered for all possible triplets of gray levels, triangle shapes, sizes, and orientations.) Again, counterexamples to this hypothesis were found ( Julesz et al., 1978). Julesz noted that the counterexamples were suggestive of an alternative explanation for texture segregation similar to those of Beck and Marr. Julesz found that texture pairs that segregated easily but had identical third-order statistics also differed in the amount of an easily discernible image feature (e.g., Caelli et al., 1978). The task then became one of identifying the list of image features, which Julesz (1981) dubbed textons, that were sufficient to explain segregation performance. The initial list of textons included such features as size, orientation, line terminations, and line crossings. It has been noted that the third-order statistics used by Julesz were population statistics. That is, the counter-

  :    

1107

examples to Julesz’s various conjectures never had identical second- or third-order statistics within the actual finite images observed. Rather, the identity was over all possible images that could have been generated by the process that generated the particular instantiation of texture currently in view. In fact, for continuous images, image pairs with identical third-order statistics must be identical images, rendering that version of the conjecture trivial (Yellott, 1993), and finite, discrete images are determined by their dipole statistics (Chubb and Yellott, 2000). On the other hand, Victor (1994) makes the case for the appropriateness of the use of population statistics for theorizing about texture segregation. The feature-based theories were echoed in research in the visual search field (Treisman, 1985). A target pattern in a field of distracter patterns was easily found whenever the target and distracters differed in a feature (e.g., size, orientation) similar to the texton features that led to effortless texture segregation. For example, a target X was effortlessly and immediately located in a field of distracter Ls. However, when the target was a T, the task became effortful and required serial scrutiny of the texture elements, requiring more time with every additional distracter added to the stimulus (Bergen and Julesz, 1983). When the choice of target and distracters requires the observer to attend to a specific combination of two features, the search becomes difficult and observers often perceive illusory conjunctions between features of neighboring objects (Treisman and Schmidt, 1982). Somewhat analogous effects using texture elements having combinations of two features have been noted in texture segregation as well (Papathomas et al., 1999). However, Wolfe (1992) suggests that texture segregation and parallel visual search do not always follow the same rules. A number of other observations have been made concerning when texture element stimuli do or do not segregate. Beck (1982) has pointed out that textures segregate based not only on the particular texture elements used but also on their arrangement, reminiscent of the Gestalt laws of figural goodness. As in the search literature (Treisman and Gormican, 1988), texture segregation may show asymmetries (Beck, 1973; Gurnsey and Browse, 1989). For example, a patch of incomplete circles will easily segregate from a background of circles, whereas the reverse pattern results in poor segregation. It has been suggested that this is due to a difference in the variability of responses of underlying visual mechanisms to the two possible texture elements (Rubenstein and Sagi, 1990). Nothdurft (1985) suggested that finding an edge between two textures is analogous to finding a luminance-defined edge. To determine a luminance boundary involves locating large values of the derivative of luminance (the luminance gradient) across an image. Finding texture boundaries might involve the determination of other aspects of image struc-

1108

, ,   

ture (local scale, local orientation, etc.), and segregation would then result from large values of the structure gradient. Finally, much of the literature assumes that effortless texture segregation and parallel visual search are truly effortless. That is, they require no selective attention to operate (demonstrated by, e.g., Braun and Sagi, 1990). However, Joseph et al. (1997) had observers perform an effortful secondary task and noted a large decrement in search performance in a search task that typically yields performance independent of the number of distracters. Thus, it is possible that even parallel search and, by extension, effortless texture segregation still require selective visual attention. Alternatively, texture segregation may not require focal visual attention, but attention may be used to alter the characteristics of visual mechanisms responsible for texture segregation (e.g., Yeshurun and Carrasco, 2000). Early literature also assumed that texture segregation was effortless in the sense of being immediate. However, at least some textures take substantial time to process (e.g., Sutter and Graham, 1995), thus undermining the notion that preattentive texture segregation is always immediate and effortless. We have treated texture as if it is somehow an isolated cue that can signal the presence, location, and shape of an edge. However, texture can co-occur in a stimulus with other cues to edge presence such as luminance, color, depth, or motion. Rivest and Cavanagh (1996) showed that perceived edge location was a compromise between the position signaled by texture and by other cues (motion, luminance, color). In addition, localization accuracy was better for two-cue than for single-cue stimuli. Landy and Kojima (2001) found that different textural cues to edge location were combined using a weighted average, with greater weight given to the more reliable cues. This is analogous to the cue combination scheme that has been seen with multiple cues to depth (including depth from texture) by Landy et al. (1995), among others. C M  T S How might one model the aspects of texture segregation performance we have just surveyed? If an edge is defined by a difference in luminance (a typical light/dark edge), then a bandpass linear spatial filter similar to a cortical simple cell can detect the edge by producing a peak response at the location of the edge. But, a typical texture-defined edge (e.g., Figs. 73.2 and 73.4A) has the same average luminance on either side of the edge and thus will not be detected by any purely linear mechanism. Several early investigators (e.g., Beck, 1972; Julesz, 1981) suggested that observers calculate the local density of various image features, and that differences in these texton or feature statistics on either side of a texture-defined edge result in effortless texture segregation. However, it was never clearly described exactly what an image feature was and how

it would be computed from the retinal image. The image features discussed (e.g., lines of different slopes, line terminations and crossings) were clearly tied to the kinds of stimuli employed in most texture studies of the period (basically, pen-and-ink drawings) and would not be applied easily to natural gray-scale images. An alternative line of modeling suggests that we need look no further than the orientation- and spatial frequency–tuned channels already discovered in the spatial vision literature through summation, identification, adaptation, and masking experiments using sine wave grating stimuli (De Valois and De Valois, 1988; Graham, 1989, 1992). For example, Knutsson and Granlund (1983) suggested that the distribution of power in different spatial frequency bands might be used to segregate natural textures, and ran such a computational model on patchworks of textures drawn from the Brodatz (1966) collection (a standard collection of texture images often used in the computational literature). Bergen and Adelson (1988) pointed out that even the example of Xs, Ls, and Ts (Fig. 73.2) could be accounted for by the distribution of power in isotropic channels similar in form to cells found in the lateral geniculate nucleus (LGN) and layer 4 of primary visual cortex. Further, they showed that if the size of the Xs was increased to effectively equate the dominant spatial frequency or scale of the different texture elements, the segregation of Xs from a background of Ls could be made difficult. This was strong evidence against the texton or feature theories. A plethora of similar models based on filters selective for spatial frequency and orientation have been investigated (Bovik et al., 1990; Caelli, 1985; Fogel and Sagi, 1989; Graham, 1991; Landy and Bergen, 1991; Malik and Perona, 1990; Sutter et al., 1989; Turner, 1986; for an alternative view, see Victor, 1988). These models are so similar in basic design that Chubb and Landy (1991) referred to this class as the back pocket model of texture segregation, as texture perception researchers pull this model from their back pocket to explain new phenomena of texture segregation. The basic back pocket model consists of three stages (Fig. 73.3). First, a set of linear spatial filters, akin to the simple cells of primary visual cortex, is applied to the retinal image. Second, the outputs of the first-stage linear filters are transformed in a nonlinear manner (by half- or full-wave rectification, squaring, and/or gain control). Finally, another stage of linear filtering is used to enhance texture-defined contours. If this third stage consisted only of spatial pooling, the resulting outputs would resemble those of cortical complex cells. But often this linear filter is modeled as bandpass and orientation-tuned, so that it enhances texture-defined edges much as an orientation-tuned linear spatial filter enhances luminance-defined edges. This process is illustrated in Figure 73.4. Figure 73.4A shows an orientation-defined texture border (Wolfson and

First-Order Seccond-Order Linear Spatial Pointwise Linear Spatial Nonlinearity Filter Filter

Input Image

Pooling and Decision Rule

F 73.3. The back pocket model of texture segregation. The retinal image is first processed by a bank of linear spatial filters. Then some form of nonlinearity is applied. Here, a pointwise fullwave rectification is indicated. Next, a second stage of linear spatial filtering is applied to enhance the texture-defined edge. Subsequent decision processes are dependent on the particular psychophysical task under study.

Landy, 1995). In Figure 73.4B a vertically oriented spatial filter has been applied. The responses are larger to the vertically oriented portion of the image, but these responses are both strongly positive (when the filter is centered on a texture element) and negative (when the filter is positioned off to the side of a texture element). As a result, the average value of the output is identical on either side of the texture border, but on the left the response variability is greater. In Figure 73.4C the responses of Figure 73.4B have been rectified, resulting in larger responses in the area of vertically oriented texture. Finally, in Figure 73.4D, a second-order, larger-scale, vertically oriented spatial filter has been applied, resulting in a peak response at the location of the texture-defined edge. For a detection experiment (“Was there a texture-defined edge in this briefly-flashed stimulus?” or “Were there two different texture regions or only one?”), a model would try to predict human performance by the strength of the peak response in Figure 73.4D as compared to peaks in responses to background noise in stimuli not containing texture-defined edges. For further examples, see Bergen (1991) and Bergen and Landy (1991). A wide variety of terminology has been used to describe the basic model outlined in Figure 73.3, making the literature difficult for the neophyte. The basic sequence of a spatial filter, a nonlinearity, and a second spatial filter has been called the back pocket model (Chubb and Landy, 1991), an LNL (linear, nonlinear, linear) model, an FRF (filter, rectify, filter) model (e.g., Dakin et al., 1999), second-order processing (e.g., Chubb et al., 2001), or a simple or linear channel (the first L in LNL) followed by a comparison-and-decision stage (e.g., Graham et al., 1992).

  :    

1109

A

B

C

D

F 73.4. Back pocket model. A, An orientation-defined edge. B, The result of the application of a linear, vertically oriented spatial filter. C, The result of a pointwise nonlinearity (squaring). D, A second, large-scale, vertically oriented spatial filter yields a peak response at the location of the texture-defined border in A.

About the term “second-order” The term second-order can be particularly troublesome. In some hands, and as we will use it here, it merely refers to the second stage of linear filtering following the nonlinearity in a model like that of Figure 73.3. As such, it has been applied to models in a wide variety of visual tasks (Chubb et al., 2001). But second-order has another technical definition that has also been used in similar contexts. If the nonlinearity in Figure 73.3 is a squaring operation, then the pixels in the output image (after the second stage of linear filtering) are all computed as second-order (i.e., quadratic) polynomials of the pixels in the model input. In this chapter, we will refer to the model of Figure 73.3 as a second-order model, meaning that it contains a second-order linear spatial filter. Of necessity, this second-order linear filter must follow an intervening

1110

, ,   

nonlinearity. Otherwise, there would simply be two sequential linear filters, which are indistinguishable from a single, lumped linear spatial filter. We will use this term regardless of the polynomial order of the intervening nonlinearity. There is also a more general use of second-order. In this usage, a second-order entity (e.g., a neuron) pools, after some intervening nonlinearity, the responses from a number of other entities (called first-order) but, in this more general usage, the first-order entities do not form a linear filter characterized by a single spatial weighting function, as they do in Figure 73.3. Rather, the first-order entities can be an assortment of neurons sensitive to various things (e.g., different orientations or different spatial frequencies). See the introduction to Graham and Sutter (1998) for a brief review of such general suggestions.

Third-order models Second-order models are not the end of the story. For example, Graham et al. (1993) used an element-arrangement texture stimulus consisting of two types of elements, arranged in stripes in one region and in a checkerboard in another region. Consider the case where each texture element is a high-frequency Gabor pattern (a windowed sine wave grating) and the two types of elements differ only in spatial frequency. Consider a second-order model like that just described, with the first linear filter tuned to one of the two types of Gabor patches and the second linear filter tuned to the width and orientation of stripes of elements. This second-order model would yield a response to these element-arrangement textures that is of the same average level, although of high contrast in the striped region and low contrast in the checked region. To reveal the texture-defined edge between the checkerboard and striped regions, therefore, requires another stage of processing, which could be a pointwise nonlinearity followed by an even larger-scale linear spatial filter (another NL), thus producing a sequence LNLNL. For an illustration of such a model’s responses, see Graham et al. (1993), Figure 4. Here we will call this LNLNL sequence a third-order model. But, to avoid confusion, let us note that Graham and her colleagues refer to the first LNL as a complex channel or second-order channel and the final NL is an instance of what they call the comparison-and-decision stage. About the terms “Fourier” and “non-Fourier” There is also possible confusion about the terms Fourier and non-Fourier. A stimulus like that in Figure 73.4A, in which the edge can be found by the model in Figure 73.3, has been referred to as non-Fourier (first applied to motion stimuli by Chubb and Sperling, 1988). The term was used because the Fourier spectrum of this stimulus does not contain components that correspond directly to the texture-defined edge. But some others (e.g., Graham and Sutter, 2000) have used the term Fourier channels for the first linear filters (the simple channels) in Figure 73.3 and reserved the term non-Fourier for the complex channels (the initial LNL) in what we called thirdorder models above (LNLNL). This confusing terminology is the result of a difference in emphasis. In this chapter, we concentrate on models that localize (i.e., produce a peak response at) edges between two abutting textures. But, others (e.g., Graham and Sutter, 2000; Lin and Wilson, 1996) have emphasized response measures that can be used to discriminate between pairs of textures (whether simultaneously present and abutting or not) by any later, nonlinear decision process. Thus, finding the edge in an orientation-defined texture like that of Figure 73.3 is, in Graham and Sutter’s terms, Fourier-based, as the power spectra of the two constituent textures differ, whereas finding the edge in a Gabor-patch element-arrangement texture like that of Graham et al. (1993) is non-Fourier-based,

as the power spectra of the two constituent textures do not differ. M S The models of texture segregation just described are complicated, with many details that require elucidation. Are the initial linear filters of a secondorder pathway the same spatial filters as the spatial frequency channels that have been described using grating experiments? What is the nature of the following nonlinearity? Are there fixed, second-order linear filters, and what is their form? This is an area of current active research, and most of these issues have not been convincingly decided. Graham et al. (1993) and Dakin and Mareschal (2000) provide evidence that the initial spatial filters in a secondorder pathway used to detect contrast modulations of texture are themselves tuned for spatial frequency and orientation. In the same article, Graham and colleagues also demonstrated that the initial spatial filters in a third-order pathway (their complex channels) were orientation- and spatial-frequency-tuned as well. The back pocket model includes a nonlinearity between the two stages of linear spatial filtering that is required to demodulate the input stimuli. For small first-order spatial filters, Chubb et al. (1994) provided a technique called histogram contrast analysis that allowed them to measure aspects of the static nonlinearity, showing that it included components of higher order than merely squaring the input luminances. Graham and Sutter (1998) found that this nonlinearity must be expansive. They also (Graham and Sutter, 2000) suggested that a gain control mechanism acts as an inhibitory influence among multiple pathways of the types called second-order and third-order here. First-order spatial frequency channels were first measured using sine wave grating stimuli and various experimental paradigms including adaptation, masking, and summation experiments (reviewed in Graham, 1989). Recently, researchers used analogous experiments to examine the second-order linear filters. To do so, researchers hope to deliver to the second-order filter something like the sine wave grating stimuli of classical spatial frequency channel studies. The usual ploy is to use a stimulus that has a sine wave (or Gabor) pattern to modulate some aspect of textural content across the stimulus. The assumed first-order filter and the subsequent nonlinearity demodulate this stimulus, providing as input to the second-order linear filter a noisy version of the intended grating or Gabor pattern. Studies of texture modulation detection have revealed a very broadband second-order texture contrast sensitivity function (CSF) using a variety of texture modulations including contrast (Schofield and Georgeson, 1999, 2000; Sutter et al., 1995), local orientation content (Kingdom et al., 1995), and modulation between vertically and horizontally oriented, filtered noise (Landy and Oruç, 2002). This

  :    

1111

A 10

Modulation Contrast Sensitivity

MSL 5

.0625

.125

.25

.5

1

2

Modulation Frequency (cpd) B F 73.5. The second-order contrast sensitivity function. A, This figure is constructed using a modulator image to additively combine vertical and horizontal noise images (Landy and Oruç, 2002). The modulator, shown as a function above the texture, has a spatial frequency that increases from left to right, and its contrast increases from bottom to top. Large modulator values result in a local texture dominated by vertically oriented noise and small values by horizontally oriented noise. Note that threshold modulation contrast is nearly independent of spatial frequency. B, Example data from a forced-choice modulation contrast detection experiment using sine wave modulators of noise patterns.

function is far more broadband than the corresponding luminance CSF. A demonstration of this effect is shown in Figure 73.5A. A modulator pattern is used to combine additively a vertical and a horizontal noise texture. The modulator increases in spatial frequency from left to right and in contrast from bottom to top. As you can see, the texture modulation becomes impossible to discern at approximately the same level for all spatial frequencies. The sample data in Figure 73.5B confirm this observation.

1112

, ,   

Evidence for multiple second-order filters underlying this broad second-order CSF has been equivocal, with evidence both pro (Arsenault et al., 1999; Landy and Oruç, 2002; Schofield and Georgeson, 1999) and con (Kingdom and Keeble, 1996). Many studies have found texture discrimination to be scale-invariant, suggesting the existence of a link between the scale of the corresponding first- and secondorder spatial filters (Kingdom and Keeble, 1999; Landy and Bergen, 1991; Sutter et al., 1995). It has also been suggested that the orientation preferences of the first- and second-order filters tend to be aligned (Dakin and Mareschal, 2000; Wolfson and Landy, 1995). This alignment of first- and second-order filters has also been supported for element-arrangement stimuli that require a third-order model to detect the texturedefined edges (Graham and Wolfson, 2001). If there is an obligatory link between the scales of the firstand second-order filters, this suggests that the preferred second-order scale should depend on eccentricity. This was first demonstrated by Kehrer (1989), who noted that performance on an orientation-defined texture-segregation task at first improves as the target texture moves into the periphery and then worsens as the eccentricity increases further. The poor foveal performance was dubbed the central performance drop (CPD). This argument that the CPD is due to the relation between the scale of the second-order pattern and the local scale of the second-order filter was made by Yeshurun and Carrasco (2000), who, in addition, suggested that the second-order spatial filters are narrowed as a consequence of the allocation of selective attention. The temporal properties of the first- and second-order filters are not well understood, although some information is available (Lin and Wilson, 1996; Motoyoshi and Nishida, 2001; Schofield and Georgeson, 2000; Sutter and Graham, 1995, Sutter and Hwang, 1999). The possibility that the wiring between first- and secondorder filters is more complicated than that shown in Figure 73.3 remains open as well (see, e.g., the appendix in Graham and Sutter, 1998; Mussap, 2001), with particular interest in possible lateral excitatory and inhibitory interactions among different positions within the same filter (Motoyoshi, 1999; Wolfson and Landy, 1999). Early filters are not the only visual processes that play an important role in determining the conscious perception of textured stimuli. Consider He and Nakayama (1994), who constructed a series of binocular demonstration stimuli involving both texture and disparity. The foreground surface consisted of a set of textured squares. The background stimuli consisted of a region of I shapes surrounded by L shapes that, monocularly, segregated quite easily. However, when seen in depth with the squares (that abutted the Ls and Is) in front, both the Ls and Is were perceived as occluded by the squares. They underwent surface completion; that is, they were both perceived as larger rectangles

occluded by the squares, and texture segregation became effortful. This suggests that higher-level, surface-based representations are involved in judgments about the objects perceived on the basis of textured regions in the stimulus.

Texture appearance The previous section concentrated on research concerning observers’ ability to detect borders between differing textures. Here we consider research more directly measuring the appearance of textures. If two images both appear to be a grassy field, then at some level of analysis, the representations of the two images must be similar. To understand the appearance of texture might involve developing such a representation, as well as a metric within that representation space so that textures are perceived as similar if their representations are close and dissimilar if far. Indeed, there is even evidence that texture appearance (or, at least, region-based) mechanisms can be responsible for texture segregation in some cases (Wolfson and Landy, 1998), as certain texture pairs can be discriminated just as well when they are separated as when they abut (forming an edge). Using region-based as well as edge-based mechanisms may be optimal for segregation processes (Lee, 1995). One approach to this problem of measuring texture appearance is a classical one: elicit similarity judgments from observers and try to build a representation. Having done so, one can then ask whether the underlying dimensions have any semantic basis or whether dimensions satisfy any of the properties of other perceptual dimensions (such as the additivity and metamerism of color space). Three dimensions appeared to suffice for sets of natural textures (Rao and Lohse, 1996) as well as artificial ones (Gurnsey and Fleet, 2001; Harvey and Gervais, 1978). A texture analogy to color matching experiments with artificial one-dimensional textures provides satisfactory appearance matches with four texture primaries (Richards and Polit, 1974). As with color matching, this technique shows that one can account for texture matches with the four primaries, but it does not explain texture appearance. Color appearance depends on the particular metameric match, as well as on color context. Similarly, texture appearance can depend on context. For example, Durgin (2001) shows that the perceived texture density of a texture patch depends on the density of the surrounding texture. An alternative approach is to analyze an instance of texture to estimate its representation and then use that representation to generate new instances of texture. The proposed representational scheme is considered successful if the newly generated textures are classified as “made of the same stuff as the original” by observers. The first such model, by Heeger and Bergen (1995), represented the input texture image as the histograms of values in each level of an

A

B F 73.6. Texture appearance, representation, and extrapolation. In the technique of Portilla and Simoncelli (2000), a texture is first analyzed using a bank of linear spatial filters varying in preferred spatial frequency and orientation. A set of statistics, both first-order and correlational, on that set of filter responses becomes the representation of the given texture. This representation may be used to generate new instances of the texture. In each panel, the inset square is the original texture, and the rest of the image is new texture generated using the technique.

oriented pyramid representation of the image, that is, as the statistics of the responses from a collection of orientationand spatial frequency–tuned spatial filters. The resulting newly generated texture images were occasionally striking in their similarity to the original. But, in other instances, especially those involving correlations between different image areas at long distances, the results were quite poor. More recent models incorporate higher-order statistics including correlations between pairs of filter responses across space,

  :    

1113

spatial frequency, and orientation (De Bonet and Viola, 1998; Portilla and Simoncelli, 2000; Zhu et al., 1998). Figure 73.6 shows two sample textures (inset squares) that were extrapolated using the technique of Portilla and Simoncelli (2000). Clearly, the technique has captured a good deal of that which defines the appearance of these textures. The technique is somewhat less successful with purely periodic textures (tiles), binary or pen-and-ink textures, or with pseudotextures that are, for example, collections of small objects (e.g., a pile of jellybeans). It remains to be seen whether a metric (Euclidean, Minkowski, or other) applied to one of these texture representation spaces will correlate well with observers’ judgments of the perceptual similarity of textures. Few psychophysical tests of these new statistical characterizations of texture have been carried out. Kingdom et al. (2001), in an analogy to the work of Chubb and colleagues in the luminance domain (1994), found that observers were most sensitive to kurtosis in the histograms of wavelet (that is, multiscale, orientation-tuned) coefficients in artificial textures. Durgin (2001) has suggested that texture density is a separate dimension from either mean (luminance) or variance (root-mean-squared contrast). The texture representation schemes just discussed are image-based. That is, all content of the representation is based on simple statistics based on responses of filters to the texture. A complete theory of texture perception might involve recognition that natural textures are associated with real-world materials, and the appearance of texture may well relate to perception of the particular material from which the image derived (wood, plastic, water, grassland, etc.) or properties of the real-world material that might relate to actions the observer might wish to take. This is the concept of an affordance (Gibson, 1979). Is this material sticky? Will it crumble in my hand? Will I be able to walk on it in bare feet? A great deal of work has been done, notably in the computer graphics world, to understand image properties of natural materials in order to simulate these materials in virtual displays. By contrast, very little research has been done on the perception of real-world textural properties. Recently, some effort has been made to understand the variety of images one can find of natural textures as viewpoint and lighting conditions are varied (Dana et al., 1999).

Shape from texture Gibson (1950) pointed out that the perspective distortion of surface texture is a cue to surface layout. For example, consider a ground plane that is painted with randomly placed circles. As the surface recedes into the distance, three different texture gradients may be distinguished: size (farther-away texture elements are smaller in the retinal image), density

1114

, ,   

(farther-away texture elements are closer together in the retinal image), and compression (farther-away elements are more slanted relative to the line of sight and hence form more eccentric ellipses in the retinal image). The computational literature is replete with suggested algorithms for the computation of shape from texture. These algorithms vary in how restrictive an assumption is made about the surface texture. The earliest algorithms (e.g., that of Witkin, 1981) assumed an isotropic texture (all orientations were equally represented on the surface, which is true of the above example). More recent algorithms (e.g., that of Aloimonos, 1988) only assume texture homogeneity (i.e., the texture is statistically the same at all positions on the surface). A particularly interesting algorithm is that of Malik and Rosenholtz (1997). This algorithm makes weak assumptions about the underlying surface texture. It looks for affine distortions in image statistics from one location to another, as seen in the responses of a bank of spatial filters varying in orientation and spatial frequency preference, much like the first stage in the current models of texture segregation. Psychophysical research on the perception of shape from texture has followed a similar history. Cutting and Millard (1984) discussed the three possible texture gradients and manipulated them independently in their stimuli. They found that perception of slant for planar stimuli depended mainly on the size gradient, whereas perception of curved stimuli was almost completely determined by the compression gradient. Rosenholtz and Malik (1997) found texture isotropy to be unnecessary for human observers to estimate surface orientation, consistent with their computational theory. Li and Zaidi (2000) examined the types of surface texture that would give a veridical percept of shape when mapped onto a corrugated surface in perspective, and found that several aspects of the Fourier power spectrum were predictive of observer accuracy, corresponding to the availability of oriented energy along lines of maximum and minimum curvature in the surface. A second line of psychophysical research has been to derive ideal (maximum a posteriori) observers and to compare the reliability of human observers’ estimates of surface layout with those of the ideal observer. Blake et al. (1993) derived such a model with the assumption of isotropic, homogeneous surface texture and demonstrated that observers’ estimates of surface curvature must use the compression gradient. Buckley et al. (1996) applied the same strategy to the estimation of surface slant, and found that texture compression dominates observer judgments even for fields of view large enough that, for the ideal, texture density should dominate. Finally, in a series of three papers, Knill (1998a, 1998b, 1998c) derived ideal observers for slant from texture that use the three texture gradient cues and derived the reliability of each cue as a function of slant and field

of view. He found that human observers became more reliable with increasing slant and field of view, just as did the ideal observers. Again, performance was so good that observers must have used texture compression and, at least in part, an assumption of isotropy.

A recent functional magnetic resonance imaging study of static texture segregation (Kastner et al., 2000) concurs, finding little response to texture borders in V1 or V2/VP and increasing responses as one proceeds downstream from V3 to V4 and TEO.

Neurophysiology

Conclusions

The physiological substrate for the first-stage linear filters in texture segregation models is likely to be the spatial frequency and orientation-selective cells in cortical area V1. Further, V1 is sufficiently complicated that other attributes of the current models, such as the normalization or other nonlinearities and subsequent spatial pooling, could certainly also occur in V1. There are also lateral interactions between neurons in V1 (both excitatory and inhibitory) that go beyond the classical receptive field. There has been some controversy over the function of these lateral interactions in V1. Some have suggested that lateral interactions enhance responses to popout stimuli (Kastner et al., 1997, 1999; Nothdurft et al., 1999), to texture elements near texture borders (Nothdurft et al., 2000), to orientation contrast (Knierim and Van Essen, 1992; Sillito et al., 1995), and to figure rather than ground (Lamme, 1995; Zipser et al., 1996). Li (2000) even described a neural network model of segmentation that includes such processes. However, the responses to orientation contrast stimuli are a complex function of the contrasts of the figure and ground (Levitt and Lund, 1997), suggesting that these V1 responses are primarily the result of a gain control mechanism that is only an initial stage of the computation of texture borders and figure-ground. Consistent with this view, several groups have found that input from outside the classical receptive field is mainly suppressive and suggest that it is not involved with figure-ground analysis (Freeman et al., 2001; Rossi et al., 2001; Sceniak et al., 2001; Walker et al., 2000). An in-depth review of a large range of results from areas V1 up through MT, and V4 (Lennie, 1998) concludes that it may be too much to attribute such functions as popout and figureground segregation to area V1, and that these functions probably occur in V2 through V4 or even at higher levels. Lennie suggests that “Spatial interactions in V1 probably have a less exotic role; they provide lateral inhibition in the domain of local structure so that, by analogy with lateral inhibition in the luminance domain, signals from regions of common structure are suppressed and contrasts in structure are made salient.” In this view, it is not until area V4 that the system has even grouped regions of similar structure to find contours, regions, and surfaces and, perhaps, computed surface slant. And thus, in this view, many of the processes called into play by texture stimuli (e.g., the conscious perception of a surface as having a particular texture) would be determined predominantly by still higher-level cortical areas.

The perception of texture is a rich and varied area of study. In the early coding of texture borders, there is some common ground between current psychophysical data and models and the physiology of primary visual cortex, such as the suggestion that texture border coding involves a succession of linear spatial filters and nonlinearities that include static nonlinearities as well as contrast gain control mechanisms. Less well understood, however, are such higher-level computations involving texture as the calculation of figureground, the coding of texture appearance, and the determination of depth and three-dimensional shape from texture cues.

Acknowledgments Michael Landy was supported by National Eye Institute Grant EY08266 and Human Frontier Science Program Grant RG0109/1999-B. Norma Graham was supported by National Eye Institute Grant EY08459. We would like to acknowledge the helpful comments of Sabina Wolfson over a period of many years. REFERENCES Adelson, E. H., and J. R. Bergen, 1991. The plenoptic function and the elements of early vision, in Computational Models of Visual Processing (M. S. Landy and J. A. Movshon, eds.), Cambridge, MA: MIT Press, pp. 3–20. Aloimonos, J., 1988. Shape from texture, Artificial Intelligence, 38:345–360. Arsenault, A. S., F. Wilkinson, and F. A. A. Kingdom, 1999. Modulation frequency and orientation tuning of second-order texture mechanisms, J. Opt. Soc. Am. A, 16:427–435. Beck, J., 1972. Similarity grouping and peripheral discriminability under uncertainty, Am. J. Psychol., 85:1–19. Beck, J., 1973. Similarity grouping of curves, Percept. Motor Skills, 36:1331–1341. Beck, J., 1982. Textural segmentation, in Organization and Representation in Perception ( J. Beck ed.), Hillsdale, NJ: Erlbaum, pp. 285–317. Bergen, J. R., 1991. Theories of visual texture perception, in Vision and Visual Dysfunction, vol. 10B (D. Regan ed.), New York: Macmillan, pp. 114–134. Bergen, J. R., and E. H. Adelson, 1988. Early vision and texture perception, Nature, 333:363–364. Bergen, J. R., and B. Julesz, 1983. Parallel versus serial processing in rapid pattern discrimination, Nature, 303:696–698. Bergen, J. R., and M. S. Landy, 1991. Computational modeling of visual texture segregation, in Computational Models of Visual

  :    

1115

Processing (M. S. Landy and J. A. Movshon, eds.), Cambridge, MA: MIT Press, pp. 253–271. Blake, A., H. H. Buelthoff, and D. Sheinberg, 1993. Shape from texture: ideal observers and human psychophysics, Vis. Res., 33:1723–1737. Bovik, A. C., M. Clark, and W. S. Geisler, 1990. Multichannel texture analysis using localized spatial filters, IEEE Trans. Pattern Anal. Machine Intelligence, 12:55–73. Braun, J., and D. Sagi, 1990. Vision outside the focus of attention, Percept. Psychophys., 48:45–58. Brodatz, P., 1966. Textures, New York: Dover. Buckley, D., J. P. Frisby, and A. Blake, 1996. Does the human visual system implement an ideal observer theory of slant from texture? Vis. Res., 36:1163–1176. Caelli, T., 1985. Three processing characteristics of visual texture segmentation, Spatial Vis., 1:19–30. Caelli, T., and B. Julesz, 1978. On perceptual analyzers underlying visual texture discrimination: Part I, Biol. Cybern., 28:167–175. Caelli, T., B. Julesz, and E. N. Gilbert, 1978. On perceptual analyzers underlying visual texture discrimination: Part II, Biol. Cybern., 29:201–214. Chubb, C., J. Econopouly, and M. S. Landy, 1994. Histogram contrast analysis and the visual segregation of IID textures, J. Opt. Soc. Am. A, 11:2350–2374. Chubb, C., and M. S. Landy, 1991. Orthogonal distribution analysis: a new approach to the study of texture perception, in Computational Models of Visual Processing (M. S. Landy and J. A. Movshon, eds.), Cambridge, MA: MIT Press, pp. 291–301. Chubb, C., L. Olzak, and A. Derrington, 2001. Second-order processes in vision: introduction, J. Opt. Soc. Am. A, 18:2175– 2178. Chubb, C., and G. Sperling, 1988. Drift-balanced random stimuli: a general basis for studying non-Fourier motion perception, J. Opt. Soc. Am. A, 5:1986–2007. Chubb, C., and J. I. Yellott, Jr., 2000. Every discrete, finite image is uniquely determined by its dipole histogram, Vis. Res., 40:485–492. Cutting, J. E., and R. T. Millard, 1984. Three gradients and the perception of flat and curved surfaces, J. Exp. Psychol. Gen., 113:198–216. Dakin, S. C., and I. Mareschal, 2000. Sensitivity to contrast modulation depends on carrier spatial frequency and orientation, Vis. Res., 40:311–329. Dakin, S. C., C. B. Williams, and R. F. Hess, 1999. The interaction of first- and second-order cues to orientation, Vis. Res., 39:2867–2884. Dana, K. J., B. van Ginneken, S. K. Nayar, and J. J. Koenderink, 1999. Reflectance and texture of real-world surfaces, ACM Trans. Graphics, 18:1–34. De Bonet, J. S., and P. Viola, 1998. A non-parametric multi-scale statistical model for natural images, in Advances in Neural Information Processing Systems 9 (M. I. Jordan, M. J. Kearns, and S. A. Solla, eds.), Cambridge, MA: MIT Press, pp. 773–779. De Valois, R. L., and K. K. De Valois, 1988. Spatial Vision, New York: Oxford University Press. Durgin, F. H., 2001. Texture contrast aftereffects are monocular; texture density aftereffects are binocular, Vis. Res., 41:2619– 2630. Freeman, R. D., I. Ohzawa, and G. Walker, 2001. Beyond the classical receptive field in the visual cortex, Prog. Brain Res., 134:157–170. Fogel, I., and D. Sagi, 1989. Gabor filters as texture discriminator, Biol. Cybern., 61:103–113.

1116

, ,   

Gibson, J. J., 1950. The Perception of the Visual World, Boston: Houghton Mifflin. Gibson, J. J., 1979. The Ecological Approach to Visual Perception, Boston: Houghton Mifflin. Graham, N., 1989. Visual Pattern Analyzers, New York: Oxford University Press. Graham, N., 1991. Complex channels, early local nonlinearities, and normalization in perceived texture segregation, in Computational Models of Visual Processing (M. S. Landy and J. A. Movshon, eds.), Cambridge, MA: MIT Press, pp. 273–290. Graham, N., 1992. Breaking the visual stimulus into parts, Curr. Dir. Psychol. Sci., 1:55–61. Graham, N., J. Beck, and A. Sutter, 1992. Nonlinear processes in spatial-frequency channel models of perceived texture segregation: effects of sign and amount of contrast, Vis. Res., 32:719–743. Graham, N., and A. Sutter, 1998. Spatial summation in simple (Fourier) and complex (non-Fourier) texture channels, Vis. Res., 38:231–257. Graham, N., and A. Sutter, 2000. Normalization: contrast-gain control in simple (Fourier) and complex (non-Fourier) pathways of pattern vision, Vis. Res., 40:2737–2761. Graham, N., A. Sutter, and C. Venkatesan, 1993. Spatialfrequency- and orientation-selectivity of simple and complex channels in region segmentation, Vis. Res., 33:1893–1911. Graham, N., and S. S. Wolfson, 2001. A note about preferred orientations at the first and second stages of complex (secondorder) texture channels, J. Opt. Soc. Am. A, 18:2273–2281. Gurnsey, R., and R. A. Browse, 1989. Asymmetries in visual texture discrimination, Spatial Vis., 4:31–44. Gurnsey, R., and D. J. Fleet, 2001. Texture space, Vis. Res., 41:745–757. Harvey, L. O. Jr., and M. J. Gervais, 1978. Visual texture perception and Fourier analysis, Percept. Psychophys., 24:534–542. He, Z. J., and K. Nakayama, 1994. Perceiving textures: beyond filtering, Vis. Res., 34:151–162. Heeger, D., and J. R. Bergen, 1995. Pyramid-based texture analysis/synthesis, in Proceedings of ACM SIGGRAPH 1995. New York: Association for Computing Machinery, pp. 229–238. Joseph, J. S., M. M. Chun, and K. Nakayama, 1997. Attentional requirements in a “preattentive” feature search task, Nature, 387:805–807. Julesz, B., 1981. Textons, the elements of texture perception, and their interactions, Nature, 290:91–97. Julesz, B., E. N. Gilbert, L. A. Shepp, and H. L. Frisch, 1973. Inability of humans to discriminate between visual textures that agree in second-order statistics—revisited, Perception, 2:391–405. Julesz, B., E. N. Gilbert, and J. D. Victor, 1978. Visual discrimination of textures with identical third-order statistics, Biol. Cybern., 31:137–140. Kastner, S., P. de Weerd, and L. G. Ungerleider, 2000. Texture segregation in the human visual cortex: a functional MRI study, J. Neurophysiol., 83:2453–2457. Kastner, S., H. C. Nothdurft, and I. N. Pigarev, 1997. Neuronal correlates of pop-out in cat striate cortex, Vis. Res., 37:371–376. Kastner, S., H. C. Nothdurft, and I. N. Pigarev, 1999. Neuronal responses to orientation and motion contrast in cat striate cortex, Vis. Neurosci., 16:587–600. Kehrer, L., 1989. Central performance drop on perceptual segregation tasks, Spatial Vis., 4:45–62. Kingdom, F. A. A., A. Hayes, and D. J. Field, 2001. Sensitivity to contrast histogram differences in synthetic wavelet-textures, Vis. Res., 41:585–598.

Kingdom, F. A. A., and D. R. T. Keeble, 1996. A linear systems approach to the detection of both abrupt and smooth spatial variations in orientation-defined textures, Vis. Res., 36:409–420. Kingdom, F. A. A., and D. R. T. Keeble, 1999. On the mechanism for scale invariance in orientation-defined textures, Vis. Res., 39:1477–1489. Kingdom, F. A. A., D. R. T. Keeble, and B. Moulden, 1995. Sensitivity to orientation modulation in micropattern-based textures, Vis. Res., 35:79–91. Knierim, J. J., and D. C. Van Essen, 1992. Neuronal responses to static texture patterns in area V1 of the alert macaque monkey, J. Neurophysiol., 67:961–980. Knill, D. C., 1998a. Surface orientation from texture: ideal observers, generic observers and the information content of texture cues, Vis. Res., 38:1655–1682. Knill, D. C., 1998b. Discrimination of planar surface slant from texture: human and ideal observers compared, Vis. Res., 38:1683–1711. Knill, D. C., 1998c. Ideal observer perturbation analysis reveals human strategies for inferring surface orientation from texture, Vis. Res., 38:2635–2656. Knutsson, H., and G. H. Granlund, 1983. Texture analysis using two-dimensional quadrature filters, in Proceedings of the IEEE Computer Society Workshop on Computer Architecture for Pattern Analysis and Image Database Management, Silver Spring, MD: IEEE Computer Society, pp. 206–213. Lamme, V. A. F., 1995. The neurophysiology of figure-ground segregation in primary visual cortex, J. Neurosci., 15:1605– 1615. Landy, M. S., and J. R. Bergen, 1991. Texture segregation and orientation gradient, Vis. Res., 31:679–691. Landy, M. S., and H. Kojima, 2001. Ideal cue combination for localizing texture-defined edges, J. Opt. Soc. Am. A, 18: 2307–2320. Landy, M. S., L. T. Maloney, E. B. Johnston, and M. J. Young, 1995. Measurement and modeling of depth cue combination: in defense of weak fusion, Vis. Res., 35:389–412. . Landy, M. S., and I. Oruç, 2002. Properties of 2nd-order spatial frequency channels, Vis. Res., 42:2311–2329. Lee, T. S., 1995. A Bayesian framework for understanding texture segmentation in the primary visual cortex, Vis. Res., 35:2643–2657. Lennie, P., 1998. Single units and cortical organization, Perception, 27:889–935. Levitt, J. B., and J. S. Lund, 1997. Contrast dependence of contextual effects in primate visual cortex, Nature, 387:73–76. Li, A., and Q. Zaidi, 2000. Perception of three-dimensional shape from texture is based on patterns of oriented energy, Vis. Res., 40:217–242. Li, Z., 2000. Pre-attentive segmentation in the primary visual cortex, Spatial Vis., 13:25–50. Lin, L. M., and H. R. Wilson, 1996. Fourier and non-Fourier pattern discrimination compared, Vis. Res., 36:1907–1918. Malik, J., and P. Perona, 1990. Preattentive texture discrimination with early vision mechanisms, J. Opt. Soc. Am. A, 7:923– 932. Malik, J., and R. Rosenholtz, 1997. Computing local surface orientation and shape from texture for curved surfaces, Int. J. Comput. Vis., 23:149–168. Marr, D., 1976. Early processing of visual information, Philos. Trans. R. Soc. Lond. B, 275:483–519. Motoyoshi, I., 1999. Texture filling-in and texture segregation revealed by transient masking, Vis. Res., 39:1285–1291.

Motoyoshi, I., and S. Nishida, 2001. Temporal resolution of orientation-based texture segregation, Vis. Res., 41:2089–2105. Mussap, A. J., 2001. Orientation integration in detection and discrimination of contrast-modulated patterns, Vis. Res., 41:295– 311. Nothdurft, H. C., 1985. Sensitivity for structure gradient in texture discrimination tasks, Vis. Res., 25:1957–1968. Nothdurft, H. C., J. L. Gallant, and D. C. Van Essen, 1999. Response modulation by texture surround in primate area V1: correlates of “popout” under anesthesia, Vis. Neurosci., 16:15– 34. Nothdurft, H. C., J. L. Gallant, and D. C. Van Essen, 2000. Response profiles to texture border patterns in area V1, Vis. Neurosci., 17:421–436. Olson, R. K., and F. Attneave, 1970. What variables produce similarity grouping? Am. J. Psychol., 83:1–21. Papathomas, T. V., A. Gorea, A. Feher, and T. E. Conway, 1999. Attention-based texture segregation, Percept. Psychophys., 61:1399– 1410. Portilla, J., and E. P. Simoncelli, 2000. A parametric texture model based on joint statistics of complex wavelet coefficients, Int. J. Comput. Vis., 40:49–71. Rao, A. R., and G. L. Lohse, 1996. Towards a texture naming system: identifying relevant dimensions of texture, Vis. Res., 36:1649–1669. Richards, W., and A. Polit, 1974. Texture matching, Kybernetik, 16:155–162. Rivest, J., and P. Cavanagh, 1996. Localizing contours defined by more than one attribute, Vis. Res., 36:53–66. Rosenholtz, R., and J. Malik, 1997. Surface orientation from texture: isotropy or homogeneity (or both)? Vis. Res., 16:2283– 2293. Rossi, A. F., R. Desimone, and L. G. Ungerleider, 2001. Contextual modulation in primary visual cortex of macaques, J. Neurosci., 21:1698–1709. Rubenstein, B. S., and D. Sagi, 1990. Spatial variability as a limiting factor in texture-discrimination tasks: implications for performance asymmetries, J. Opt. Soc. Am. A, 7:1623–1643. Sceniak, M. P., M. J. Hawken, and R. Shapley, 2001. Visual spatial characterization of macaque V1 neurons, J. Neurophysiol., 85:1873–1887. Schofield, A. J., and M. A. Georgeson, 1999. Sensitivity to modulations of luminance and contrast in visual white noise: separate mechanisms with similar behavior, Vis. Res., 39:2697– 2716. Schofield, A. J., and M. A. Georgeson, 2000. The temporal properties of first- and second-order vision, Vis. Res., 40:2475– 2487. Sillito, A. M., K. L. Grieve, H. E. Jones, J. Cudeiro, and J. Davis, 1995. Visual cortical mechanisms detecting focal orientation discontinuities, Nature, 378:492–496. Sutter, A., J. Beck, and N. Graham, 1989. Contrast and spatial variables in texture segregation: testing a simple spatial-frequency channels model, Percept. Psychophys., 46:312–332. Sutter, A., and N. Graham, 1995. Investigating simple and complex mechanisms in texture segregation using the speed-accuracy tradeoff method, Vis. Res., 35:2825–2843. Sutter, A., and D. Hwang, 1999. A comparison of the dynamics of simple (Fourier) and complex (non-Fourier) mechanisms in texture segregation, Vis. Res., 39:1943–1962. Sutter, A., G. Sperling, and C. Chubb, 1995. Measuring the spatial frequency selectivity of second-order texture mechanisms, Vis. Res., 35:915–924.

  :    

1117

Treisman, A. M., 1985. Preattentive processes in vision, Comput. Vis., Graphics Image Processing, 31:156–177. Treisman, A. M., and S. Gormican, 1988. Feature analysis in early vision: evidence from search asymmetries, Psychol. Rev., 95:15–48. Treisman, A. M., and H. Schmidt, 1982. Illusory conjunctions in the perception of objects, Cogn. Psychol., 14:107–141. Turner, M. R., 1986. Texture discrimination by Gabor functions, Biol. Cybern., 55:71–82. Victor, J. D., 1988. Models for preattentive texture discrimination: Fourier analysis and local feature processing in a unified framework, Spatial Vis., 3:263–280. Victor, J. D., 1994. Images, statistics, and textures: implications of triple correlation uniqueness for texture statistics and the Julesz conjecture: comment, J. Opt. Soc. Am. A, 11:1680–1684. Walker, G. A., I. Ohzawa, and R. D. Freeman, 2000. Suppression outside the classical cortical receptive field, Visual Neurosci., 17:369–379. Witkin, A. P., 1981. Recovering surface shape and orientation from texture, Artificial Intelligence, 17:17–45.

1118

, ,   

Wolfe, J. M., 1992. “Effortless” texture segmentation and “parallel” visual search are not the same thing, Vis. Res., 32: 757–763. Wolfson, S. S., and M. S. Landy, 1995. Discrimination of orientation-defined texture edges, Vis. Res., 35:2863–2877. Wolfson, S. S., and M. S. Landy, 1998. Examining edge- and region-based texture mechanisms, Vis. Res., 38:439–446. Wolfson, S. S., and M. S. Landy, 1999. Long range interactions between oriented texture elements, Vis. Res., 39:933–945. Yellott, J. I., Jr., 1993. Implications of triple correlation uniqueness for texture statistics and the Julesz conjecture, J. Opt. Soc. Am. A, 10:777–793. Yeshurun, Y., and M. Carrasco, 2000. The locus of attentional effects in texture segmentation, Nat. Neurosci., 3:622–627. Zhu, S. C., Y. Wu, and D. Mumford, 1998. Filters, random fields and maximum entropy (FRAME)—towards a unified theory for texture modeling, Int. J. Comput. Vis., 27:107–126. Zipser, K., V. A. F. Lamme, and P. H. Schiller, 1996. Contextual modulation in primary visual cortex, J. Neurosci., 16:7376–7389.

74

Visual Segmentation and Illusory Contours ROBERT SHAPLEY, NAVA RUBIN, AND DARIO RINGACH

T    transformation that takes place in visual perception between the analog representation of the visual image in the retina and surfaces and objects as they appear to us. The retinal image represents numerous different brightness levels and colors at a very large number of different points in space, with no explicit representation of which points belong together. But the image we perceive consists of a much smaller number of surfaces and objects that are segregated from the background and from each other. There are probably many stages of this transformation, but one stage of this process is known to be of great importance: visual segmentation. Segmentation is a process of parsing the different surfaces in an image, as well as grouping together the parts of the same surface that are separated from each other in the image by another, occluding surface. Segmentation therefore involves resolving the depth relationships between surfaces and objects in a scene. Understanding segmentation will lead to insight about one of the major theoretical problems in visual neuroscience, namely, how neural signals about small, localized pieces of the visual image are combined into a complete representation of the spatially extended visual image. This is a particularly important example of a general problem in neuroscience: how to go from the local to the global level of object representation in the brain. In this chapter, we review some of the large body of work on visual segmentation in human subjects and animals. The aim is to present a coherent body of experimental results that all relate to the question How is fragmentary visual information completed and made into wholes? There is another significant body of work, on the theory of segmentation computations in perception, that is beyond the scope of this chapter. That would make another interesting separate chapter of visual neuroscience.

Illusory contours As with many brain computations, we can understand segmentation better by observing its action when it deals with an exceptionally difficult task. Usually segmentation is done so efficiently by the brain that we (as observers) are unaware that it is happening. But for certain special visual images, the segmentation process becomes evident. This is the reason for the fascination with these special images, the so-called illusory contours (ICs). An example of such a visual image is

Figure 74.1, an image referred to as a Kanizsa triangle, named after the Italian Gestalt psychologist Gaetano Kanizsa, who made this image famous (Kanizsa, 1979). In this figure, the perception of a bright white triangle is very strong, but if one scrutinizes the boundaries of the triangle, it becomes evident that there is no difference in the amount of light coming to the eye from the regions inside and outside the perceived triangle. Yet we see the inside as a bright surface segmented from its background by sharp contours along the boundary of the triangle. In this sense, the boundary between the inside and outside of the triangle is an IC. This image is a classical example in favor of the basic concept of the Gestalt psychologists, also echoed in the work of Donald Hebb, that the brain is “searching” for meaningful patterns. In this case, the brain manufactures a perceptual triangle from fragmentary information because a meaningful pattern, an occluding triangle, is consistent with the available image information even though other perceptions are possible. It is reasonable to believe that the segmentation computations the visual system performs on these exceptional Kanizsa images are the same as for more typical images. One of the main points of scientific investigation of ICs is the nature and location of the brain area that performs the segmentation of the illusory figure from its background. Some psychologists have favored an explanation in terms of perceptual problem solving and think of ICs as cognitive contours (e.g., Gregory, 1987). Such cognitive approaches do not usually specify or even speculate about the brain areas involved in the perception. However, we could speculate that such a cognitive explanation would involve both visual cortical areas in the posterior cerebral cortex, as well as frontal and temporal cortex. In opposition to the top-down cognitive approach, more bottom-up, stimulus-driven approaches have been proposed (e.g., Grossberg, 1997; Heitger and von der Heydt, 1993). The bottom-up explanation would seem to imply the involvement in IC perception of early visual areas in which visual signals are still arranged retinotopically. There are psychophysical as well as neurophysiological and brain imaging studies of the nature of IC processing and also of localization of IC-evoked signals. The results of these different studies provide a fairly compelling case for the concept that IC perception is the result of the combined and cooperative action of early and later, or more retinotopic and more abstract, visual cortical areas. In this chapter, we

1119

F 74.1. Kanizsa triangle. The occluding triangle that appears in front of the three circles and the three line segments has the same physical brightness as the surroundings. But it appears somewhat brighter, and appears to be a solid surface in front, because of perceptual processes.

begin by discussing psychophysical studies we have done on these problems, followed by a consideration of neurophysiological and brain imaging results on IC perception and segmentation.

Psychophysics of ICs We have developed a psychophysical technique which was designed to provide an objective measure of the perceptual strength of ICs. This technique has yielded many new and interesting results that may enable us to forge a link between the perception and the neural mechanisms of perception related to segmentation. Figure 74.2 illustrates the technique: a shape discrimination task with ICs. The shapes are formed by Kanizsa-style Pacmen that are rotated around their centers by an angle a (see the figure legend for details). (Pacman is a term that refers to the shape of an agent in a video game from the early 1980s. The shape of Pacman was exactly the same as the cut-off circles used by Kanizsa in the IC figures he originated much earlier.) Two categories of shapes are formed: thin when a > 0 and fat when a < 0. The subject in the experiment must classify the shape. The pattern is flashed for about 100 msec, and then a mask follows presentation. With a series of control experiments, we showed that performance on this task is facilitated significantly when the subject sees the ICs compared to her or his performance when it is based on the local inducers’ orientation. One control experiment was done to measure discrimination performance when all the inducers face outward. Then performance on the task was quite poor. Another control experiment proved that it was contour completion in the blank spaces between the pacmen inducers that was crucial for task performance (rather than the symmetrical rotation of the inducers, for instance). This second control experiment involved the placement of lines arranged along the boundaries of a virtual square that overlapped the thin or fat illusory figures. Such masking that is remote from the inducers does degrade performance by a factor of 2 or more. Thus, a high level of performance on the shape

1120

, ,   

F 74.2. The thin-fat task used for psychophysics of ICs. A, A family of Kanizsa-like figures. The rotation parameter a specifies the angle of rotation of the upper-left inducer around its center, as shown in B. Adjacent inducers are rotated in opposite directions to achieve a consistent shape deformation of the illusory surface. (From Ringach and Shapley, 1996.)

discrimination task does, we believe, require perception of the illusory figures. We have used the task performance to answer quantitatively several important questions about illusory contours. S S Ringach and Shapley (1996) found that ICs can span 15 degrees of visual angle, and thus must be formed by integration of neural signals over large distances in visual cortex. They also investigated the issue of spatial scale invariance. Scale-invariant properties of ICs were suggested by prior studies. Shipley and Kellman (1992) presented subjects with Kanizsa squares which varied in their absolute size and the radii of the inducing pacmen. They found that ratings of IC clarity were approximately scale invariant. In other words, the rating of a figure depended mainly on the ratio between the radius of the inducer and the side of the square. This ratio, termed the support ratio, is the crucial spatial parameter for ICs. In Ringach and Shapley’s experiments on spatial scale, they collected shape discrimination data, as described above, with ICs at five different scales but always with the same support ratio. Figure 74.3 shows the variation in IC strength as a function of scale, and it is seen to be a relatively flat function. This is direct evidence for the spatial scale invariance of IC perception. D Ringach and Shapley found that ICs can be formed by inducing elements that flash for a period of 100 msec, but that neural integration must proceed for longer than 250 msec for the contours to be completed. This is the conclusion of backward masking experiments in which

Threshold, deg

1

0.8 ICs Real contours

0.6

0.4

0.2

0 0

4

8

12

Eccentricity of Inducers (deg visual angle)

F 74.3. Scale invariance of IC perception. Thresholds for seeing ICs and real contours are plotted versus separation between the inducers. The real contours were drawn in the image connecting the corners of the inducers. Thresholds are expressed in terms of the rotation angle a that enables the observer to reach a criterion number of correct responses in the thin-fat task. (Figure derived from the data in Ringach and Shapley, 1996.)

the authors blocked the perception of a curved IC by the later presentation of a Kanisza square. Control experiments with real contours show that shape discrimination is much faster with real contours than with ICs. That more time is needed to create the ICs than is required to perceive shapes defined by luminance contours suggests that recurrent neural networks that require some time to compute the shapes may be involved in IC perception. Receptive field models like the ones proposed by Heitger and von der Heydt (1993) or Grossberg (1997) would produce ICs without any cost in time, and this seems to be disconfirmed by the behavioral data. U  L V F A prominent anatomical property of retinotopic visual areas such as V1/V2 is that the representation of the visual field is split into disjoint portions of the cortical sheet. There is the well-known anatomical break between the left and right hemispheres, each of which represents the contralateral visual fields in early visual areas, as discussed below. In addition, there is a wide separation between the upper and lower hemifield representations in extrastriate visual cortex such as V2 or V3. Introspectively, we are not aware of these discontinuities in the cortical retinotopic representation. Surfaces that cross the horizontal or vertical meridians appear unitary and whole. Nevertheless, under careful experimental conditions, it is possible to uncover behavioral effects that may be the result of the anatomical discontinuities of the cortex. Rubin et al. (1996) found such a behavioral effect: human observers exhibit a greater tendency to perceive ICs when the inducing stimuli fall on their lower visual field. There were two experiments that led to this conclusion, both illustrated in Figure 74.4. The stimulus used in the first experiment was a stereogram similar to that depicted in Figure 74.4A. When subjects fixated the upper cross in that figure, they perceived

a bright illusory horizontal stripe, bounded by ICs, linking the two filled rectangles. In contrast, when they fixated the lower cross, the illusory stripe “faded away” and the two filled rectangles were perceived to be disjoint. The stimulus is symmetric with respect to reflection about its horizontal midline, and therefore the only difference is that the (identical) stimulation falls on the upper versus lower visual hemifields. This is the first experimental result that indicates that ICs are perceived more easily in the lower visual field. Rubin et al.’s second experiment utilized the thin-fat task to measure IC strength in the upper and lower visual fields, as depicted in Figure 74.4B with the data from a single subject. The left panels in Figure 74.4B show the results of an individual observer on the IC task, while the right panels show the results for a complete (or luminance) contour task. The upper and lower graphs show the subject’s psychometric functions as a measure of performance when the stimulus fell on the upper and lower hemifields, respectively. The lower hemifield shows a marked advantage for the performance of the IC task, as can be seen by examining the psychometric functions on the left side of Figure 74.4B. The psychometric function for the lower visual field is much steeper, indicating better performance. Also shown in this figure in the right-hand panels are the psychometric functions for filled-in Kanizsa figures, for which the support ratio was 1.0—that is, these were real contours entirely defined by luminance difference. Defining threshold performance as the amount of rotation of the inducing elements needed for the subject to reach 82% correct discrimination, the thresholds for the IC figures were 2 degrees and 7.8 degrees for the lower and upper hemifields, respectively. For figures that were completely defined by luminance contours, the thresholds were not different in the different visual fields: thresholds for the lower and upper hemifields were, respectively, 1.1 and 0.9 degrees. Thus, a performance-based measure also showed that the lower visual field segmented the ICs more easily than the upper visual field. V M Perceptual completion can link widely separated contour fragments and interpolate ICs between them, but can it cross the “seams” in visual cortex, the vertical meridian representation that demarcates the boundary between the visual field representations in the left and right cerebral cortical hemispheres, and the horizontal meridian representation that separates upper and lower field representations? As illustrated in Figure 74.5, Pillow and Rubin (2002) answered this question using a variant of the thin-fat shape discrimination task, but with the variation that only one arm of the angle in an inducer was varied in a single presentation, either the arm that faced horizontally or the one that faced vertically. Thus, with inducers arranged symmetrically around the fovea, they could test whether hor-

, ,  :     

1121

A

B

F 74.4. Upper/lower field asymmetry in IC perception. A, Stereogram that can be placed in the upper or lower visual field, depending on the placement of fixation. If the reader free-fuses the stereogram, she or he should observe a qualitative difference in the percept, as described in the text. B, Psychometric functions for the thin/fat task for ICs (left panels) and filled-in luminance contours (right panels). The inducer rotation angle a is the coordinate, is

labeled on the horizontal axis as “Inducer rotation.” The fraction of thin responses is plotted on the vertical axis. The upper panels are for the upper visual field; the lower panels are for the lower visual field. Steeper psychometric functions are indicative of better discrimination performance. The shallowest psychometric function is for IC perception in the upper visual field. (From Rubin et al., 1996.)

izontal contours that crossed the vertical meridian could be perceived as well as contours that were contained within a single hemifield. They found that completion is much poorer when ICs cross the vertical meridian than when they reside entirely within the left or right visual hemifield and cross the horizontal meridian. This deficit reflects limitations in crosshemispheric integration. The authors also showed that the

sensitivity to the interhemispheric divide is unique to perceptual completion: a comparable task which did not require completion showed no across-meridian impairment. Pillow and Rubin proposed that these findings support the existence of specialized completion mechanisms in early visual cortical areas (V1/V2), since those areas are likely to be more sensitive to the interhemispheric divide.

1122

, ,   

F 74.5. Meridional asymmetry in IC formation. A, Psychometric functions for the modified thin-fat task, as described in the text. Plotting conventions are as in Figure 74.4B. The steeper psychometric function for the within-hemifield IC (left panel)

indicates that performance is better for ICs that do not cross the vertical meridian. B, Summary of thresholds for within- and acrosshemifield contour completion for all subjects tested. (From Pillow and Rubin, 2002.)

P L  IC Perceptual learning also has been used to investigate the cortical mechanisms of IC perception. From a study of human perceptual learning and IC perception, Rubin et al. (1997) concluded that the abrupt, insight-like perceptual learning observed in that study demanded that high-level visual areas, in which templates of remembered visual objects are stored, must interact with lower-level visual areas that have an analog representation of the visual image. Such multilevel interactions were needed to explain the stimulus size dependence of IC perceptual learning and at the same time its abrupt, insight-like onset. These results may help to reconcile psychophysical and neurophysiological results, which suggest that early retinotopic visual areas must be involved in IC perception, with brain imaging results that found IC-related activity in higher, nonretinotopic cortical regions. This issue is brought up again in the “Conclusions.”

Neurophysiology of ICs The first neurophysiology work we discuss tends to support a bottom-up, stimulus-driven explanation of IC perception. From electrophysiological single-cell recordings in awake monkeys, Peterhans and von der Heydt and their colleagues found that Kanizsa-type images and other IC images could excite spike activity in neurons in early visual cortex (see Chapter 76). Peterhans and von der Heydt (1989) recorded from single neurons in area V2 of the macaque visual cortex. Such a neuron responds with excitation to a luminance contour that crosses its receptive field. When an IC (as perceived by us) crosses its receptive field, the cell produces a slightly delayed excitatory response resembling the same cell’s response to a real contour. As a control to ensure that the response is not merely a weak response to the remote features of the IC stimulus, the investigators made a small image manipulation (closing the inducing boundary), and this eliminated the neuron’s response. Peterhans and von der Heydt also performed several quan-

, ,  :     

1123

titative studies on these IC-responsive V2 neurons, in particular measuring the orientation tuning for ICs and real contours on the same population of V2 neurons; they found that real contours and ICs produced similar orientation tuning in IC-responsive neurons in V2. Thus, these neurons seem to be a candidate neural substrate for IC perception. There also have been reports of IC responses in neurons in V1. This is a controversial point since von der Heydt and Peterhans and their colleagues maintained that they observed very few V1 neurons that produced IC responses. In part the discrepancy may occur perhaps because of the use of different stimuli and in part because of different views of what constitutes an IC. For our present purposes, it is enough to conclude that IC responses can be observed in retinotopic areas in the monkey’s brain, areas that are traditionally thought of as stimulus driven. The connection of the monkey V2 neurons with IC perception in humans needs to be established more firmly. We do not know the nature or quality of IC perception in monkeys because there are insufficient animal data using rigorous experiments to test for IC perception. Once we know that monkeys can exhibit behavior that proves that they see ICs in Kanizsa-type images, further experiments will be necessary to find out whether or not the V2 neurons have the same sort of parameter dependence on size, contrast, and retinal location as the behavior. One important question is, how do the V2 neurons respond to an IC that crosses the horizontal meridian? Humans respond as well to such ICs as they do to ICs that do not cross the horizontal meridian. The reader can observe this by fixating the middle of the Kanizsa triangle in Figure 74.1 and observing the robust lateral ICs that traverse the horizontal meridian in her or his visual field. But in V2 there is a marked separation between neurons that represent the visual field just above and just below the meridian (see Horton and Hoyt, 1991). So one might expect some deleterious effect on IC responses for meridian-crossing ICs in V2 neurons. If that dropoff in IC sensitivity in V2 neurons were observed, it might cast doubt on the role of V2 alone in IC perception. Moreover, as we will discuss below, the human brain imaging data point to other brain areas as the major processing sites for ICs in humans. The monkey results could be interpreted to indicate that similar IC-related activity is going on in human V2, but the functional magnetic resonance imaging (fMRI) and other techniques used on humans are too insensitive to measure it. A second possibility is that the V2 activity seen in monkeys is related to IC perception but that it is not the central mechanism involved in the percept. Another possibility is that human and monkey perception and neural mechanisms are fundamentally different at this midlevel stage of visual processing. Human responses to ICs have been measured with fMRI techniques. Most fMRI studies have involved the measure-

1124

, ,   

ment of the activation of Kanizsa squares or diamonds compared with the same pacman-shaped inducers rotated outward or all in the same direction. An earlier study by Hirsch and colleagues (1995) found that there was activation of the occipital cortex lateral to V1 by Kanizsa-type figures, but they could not pinpoint the cortical location because the IC experiments were not combined with retinotopic mapping. Therefore, these studies established that signals related to segmentation were present in occipital cortex, but further work is needed to be more precise about localization. The extensive research of Mendola and colleagues (1999) at Massachusetts General Hospital established that ICrelated signals were observed in retinotopic area V3 and also in LO, the lateral occipital area previously discovered by Malach and colleagues (Malach et al., 1995). Figure 74.6 is an fMRI image from the Mendola paper indicating the large region of cortical activation evoked by the Kanizsa diamonds used as stimuli in that study. The early retinotopic areas V1 and V2 did not produce statistically significant activation, as seen in the figure. Mendola and her colleagues also used different inducers for ICs, such as aligned line endings, and found a similar pattern of brain activation in V3 and LO. These results are important in implicating extrastriate cortex in the process of visual segmentation in humans. But it is important to note the apparent conflict between these results and the findings of Peterhans and von der Heydt (1989) that implicated V2 in IC processing in monkeys. The brain imaging results on humans suggest that higher-level visual areas produce the major response to ICs.

Figure-ground and border ownership While ICs are often chosen for studying visual segmentation, there are other visual phenomena that can also lead to an understanding of segmentation. The assignment of an image region as figure or ground is one such phenomenon. As Edgar Rubin, the famous perceptual psychologist, pointed out, such assignment is automatic and inescapable (Rubin, 1921). But ambiguous figures exist in which figure and ground assignments flip back and forth, and perception changes when that happens. Rubin’s familiar face/vase figure is the most widely reproduced example, but there are other examples from E. Rubin that illustrate the consequences of figure-ground assignments even more. One of these is the Maltese cross figure in Figure 74.7. This example is described in Koffka’s (1935) book but not depicted there. The diamond-shaped arms of the cross can appear to be grouped in fours, with a vertical and a horizontal pair grouped together as figures in front (resembling a propeller in shape) and then two diagonal pairs grouped together as figures in front (the vertical-horizontal pairs are then in back). The brightness contrasts in the figure are arranged

 F 74.6. Mapping of IC responses in human cortex with fMRI. A, Map in human visual cortex of retinotopic visual areas and the interhemispheric region (labelled bi and colored green) using phase mapping. B, The region of differentially high activity when Kanizsa ICs are compared with activation produced by the unorganized pacman inducers. The main activation is in V3A and in the interhemispheric region. C, Activation produced by a square defined by luminance contours. (From Mendola et al., 1999, with permission.) (See color plate 48).

F 74.7. E. Rubin’s Maltese cross figure. There are two sets of vanes, white and mid-gray. They group together to produce propeller figures that alternate front and back. When the propeller of a given color is seen in back, it tends to complete, into a gray diamond or a white square, respectively. Some observers report that the perceived value of the mid-gray changes, becoming darker when the gray regions form a propeller in front and lighter when they form a diamond in back. (Drawn from a description in Koffka, 1935.)

such that the vertical-horizontal propeller shape looks darker in front than it does when it is perceived in back, looking like a light gray diamond behind the white tilted propeller. This is because of the enhanced effect of brightness contrast across borders that define a figure and on the regions to which such borders are attached, as E. Rubin noted (cited in Koffka, 1935). Similar effects can be seen in color. This is only one of many illustrations of the deep consequences of figure-ground assignment. For instance, another consequence of the importance of figure-ground is that people remember the shapes of figures, not grounds. Thus, understanding the neural basis for this phenomenology is likely to be an important clue to the function of the visual system. Figure-ground assignment is a special case of a more general problem in vision, the assignment of border ownership. Assignment of a region as figure or ground is all one

, ,  :     

1125

has to do if there is only one figure surrounded by the background. But if there are many figures, and if one is in front of another so that it partly occludes the shape of the second figure in the visual image, then the visual system must decide on the basis of image information which surface is in front along the boundary between the two figures in the image. Briefly, the brain has to decide which figural region owns the border between them; that is the front surface. Assignment of border ownership is a problem that must be solved in almost every visual image. There have been only a few investigations of neural mechanisms for border ownership and figure-ground assignments. One study is by Zhou and colleagues (2000) on single cells in V1 and V2 cortex of macaque monkeys. By keeping local edge contrast the same but varying the global stimulus so that different regions own the boundary between them perceptually, Zhou et al. tested sensitivity to border ownership in single cortical neurons. The experimental design and results in an archetypal border-ownership cell are shown in Chapter 76 in this book from the work of Zhou et al. (2000). A substantial fraction of border-ownership cells like that cell are encountered in monkey V2 cortex. Baylis and Driver (2001) reported recently that many neurons in monkey inferotemporal (IT) cortex respond differentially to figure or ground, and thus these also must reflect signals about border

ownership. Since IT cortex is supposedly involved in object recognition, it is very reasonable that neurons in this area should be affected by border ownership that is necessary for accurate object recognition in the real world. Studies in human cortex of figure-ground reversals using fMRI, by Kleinschmidt and his colleagues (1998) at the Wellcome Imaging Center in London, revealed activation over a number of areas in occipital, temporal, parietal, and even frontal cortex. The involvement of temporal, parietal, and frontal cortex seems to imply that activation of topdown influences from high-level cortical areas could be necessary for figure-ground reversal of border ownership. However, as in the case of ICs, it is also possible that there also may be signals associated with figure-ground assignment in “early” retinotopic areas like V1 or V2 that are undetectable with fMRI.

F 74.8. Sugita-amodal completion V1. Responses from a neuron in an awake, behaving monkey. The neuron responds to a long contour in its receptive field, as in a. It responds to both eyes (b) and to the right eye alone (c) but not to the left eye alone (d ). The interesting manipulation is in f to h. In f the neuron does not

respond to two unconnected segments. In g it does not respond to the same two segments when they are perceived as being in front of a gray region. But there is a response when the retinal disparity is such that the gray region is in front, occluding the two line segments (h). (From Sugita, 1999, with permission.)

1126

, ,   

Amodal completion An important part of segmentation in human visual perception is the phenomenon of amodal completion, that is, completion and grouping together of the parts of a partially occluded object that are visible. This completion process is crucial for normal object perception in the real world. Evidence that amodal completion affects the firing rates of

V1 neurons in macaque V1 was obtained by Y. Sugita by manipulating apparent occlusion using stereopsis, as shown in Figure 74.8. Only a small fraction of V1 neurons were affected by amodal completion, but still it is a significant result. Amodal completion in human perception was studied by Ringach and Shapley (1996) using the thin-fat task. This was done by enclosing each pacman inducer with an annulus of the same gray scale as the inducer and investigating how accurately and with what speed the observers could perform the task. The accuracy of amodal completion was almost the same as for ICs. The speed of the processes leading to amodal versus IC completion was measured with a backward masking paradigm. The results of these experiments indicated that amodal completion was significantly slower than IC formation, up to 50 msec slower. However, the only masks that were used in this experiment were local pinwheel masks that overlapped spatially with the pacman, or modified pacman, inducers. This suggests that the difference in timing between amodal and IC completion may be caused by the neural computations that are used to define the junctions at the corners of the inducers. In other work not discussed here, psychophysical experiments on ICs and occlusion of the inducer corners indicated the great importance of neural signals about local junctions for global segmentation (Rubin, 2001).

Conclusions Some of the psychophysical results reviewed here indicate that the process of segmentation requires recurrent networks for completion in early visual cortex. These are the results on dynamics, visual field dependencies, and the size specificity of perceptual learning of ICs. Such results tend to rule out models based simply on collection of signals by large receptive fields in higher visual areas. However, there are other results, such as size scale invariance and the abruptness of perceptual learning, that point to the necessary involvement of higher-level, more abstract representations only in higher-level visual areas. This duality is also reflected in the pattern of neurophysiological and brain imaging results. The full spectrum of results about ICs and segmentation that we have reviewed here indicates that such perception is not simply the result of a single unitary process in the brain. A reasonable deduction from this pattern of results is that multiple visual areas collaborate on producing the IC percept and visual segmentation generally. Multistage processing and cooperation in vision have been suggested before in general terms, for instance, by Ullmann (1998). The picture that emerges from all the studies of segmentation we have reviewed is of a visual cortical network with intense feedback between higher-level visual areas and

lower-level areas that are exciting the higher-level areas. This cortical network acting as a feedback loop is hunting for structure in the visual image. Related ideas have been proposed more than once before, but now the weight of neurophysiological, brain imaging, and psychophysical evidence points more and more strongly to the visual cortex (that is, primary and extrastriate cortex taken altogether) as an array of feedback loops that cooperate in the segmentation of the visual image.

REFERENCES Baylis, G. C., and J. Driver, 2001. Shape-coding in IT cells generalizes over contrast and mirror reversal, but not figure-ground reversal, Nat. Neurosci., 4:937–942. Gregory, R. L., 1987. “Illusory Contours and Occluding Surfaces,” in The Perception of Illusory Contours (S. Petry and G. Meyer, eds.), New York: Springer-Verlag. pp. 81–89. Grossberg, S., 1997. Cortical dynamics of three-dimensional figure-ground perception of two-dimensional pictures, Psychol. Rev., 104:618–658. Heitger, F., and R. von der Heydt, 1993. A computational model of neural contour processing: figure-ground segregations and illusory contours, in Proceedings of the International Conference on Computer Vision, pp. 32–40. Hirsch, J., R. De La Paz, N. Relkin, J. Victor, K. Kim, T. Li, P. Borden, N. Rubin, and R. Shapley, 1995. Illusory contours activate specific regions in human visual cortex: evidence from functional magnetic resonance imaging, Proc. Natl. Acad. Sci. USA, 92:6469–6473. Horton, J. C., and W. F. Hoyt, 1991. Quadrantic visual field defects. A hallmark of lesions in extrastriate (V2/V3) cortex, Brain, 114:1703–1718. Kanizsa, G., 1979. Organization in Vision, New York: Praeger. Kleinschmidt, A., C. Buchel, S. Zeki, and R. S. Frackowiak, 1998. Human brain activity during spontaneously reversing perception of ambiguous figures, Proc. R. Soc. Lond. B Biol. Sci., 265:2427– 2433. Koffka, K., 1935. Principles of Gestalt Psychology, San Diego, CA: Harcourt, Brace. Malach, R., J. B. Reppas, R. R. Benson, K. K. Kwong, H. Jiang, W. A. Kennedy, P. J. Ledden, T. J. Brady, B. R. Rosen, and R. B. Tootell, 1995. Object-related activity revealed by functional magnetic resonance imaging in human occipital cortex, Proc. Natl. Acad. Sci. USA, 92:8135–8139. Mendola, J. D., A. M. Dale, B. Fischl, A. K. Liu, and R. B. Tootell, 1999. The representation of illusory and real contours in human cortical visual areas revealed by functional magnetic resonance imaging, J. Neurosci., 19:8560–8572. Peterhans, E., and R. von der Heydt, 1989. Mechanisms of contour perception in monkey visual cortex. II. Contours bridging gaps, J. Neurosci., 9:1749–1763. Pillow, J., and N. Rubin, 2002. Perceptual completion across the vertical meridian and the role of early visual cortex, Neuron, 33:805–813. Ringach, D., and R. Shapley, 1996. Spatial and temporal properties of illusory contours and amodal boundary completion, Vis. Res., 36:3037–3050. Rubin, E., 1921. Visuell wahrgenommene Figuren, Copenhagen: Gyldendal.

, ,  :     

1127

Rubin, N., 2001. The role of junctions in surface completion and contour matching, Perception, 30:339–366. Rubin, N., K. Nakayama, and R. Shapley, 1996. Enhanced perception of illusory contours in the lower vs. the upper visual hemifields, Science, 271:651–653. Rubin, N., K. Nakayama, and R. Shapley, 1997. Abrupt learning and retinal size specificity in illusory contour perception, Curr. Biol., 7:461–467.

1128

, ,   

Shipley, T., and P. Kellman, 1992. Strength of visual interpolation depends on the ratio of physically specified to total edge length, Percept. Psychophys., 48:259–270. Sugita, Y., 1999. Grouping of image fragments in primary visual cortex, Nature, 401:269–272. Ullmann, S., 1998. High Level Vision, Cambridge, MA: MIT Press. Zhou, H., H. S. Friedman, and R. von der Heydt, 2000. Coding of border ownership in monkey visual cortex, J. Neurosci., 20:6594–6611.

75

Global Yet Early Processing of Visual Surfaces YUKIYASU KAMITANI AND SHINSUKE SHIMOJO

F   optics viewpoint (Gibson, 1950), the input to the eyes is basically ambient light, that is, a lattice of lights that reflects geometry and perspectives between the observer and the environment. Since the lights are reflected from objects’ surfaces, the lattice may be considered a summary of the environmental surfaces which is created according to the rules of optics. More importantly, the representation of visual surfaces is an essential step for the representation of visual objects, the materials the observer’s action is aimed at and performed with. Thus, the task of the visual system is to retrieve from the retinal input the information about surfaces and objects that is ecologically relevant. It is primarily for this reason that understanding of visual surface representation is critical to our understanding of visual processing. In spite of its functional significance, only recently has surface representation begun to be a subject of neurophysiological studies. This may be partly because the traditional view of visual processing derived from electrophysiological studies of single visual neurons is at odds with the fundamental nature of perceptual surfaces. In this chapter, we first illustrate the essential characteristics of perceptual surfaces, emphasizing the following points: 1. Local information given to limited areas of the visual field can lead to the global percept of surfaces, including the three-dimensional surface layout. 2. Surface representation occurs early in visual processing in the sense that it is not mediated by conscious/cognitive processes and even precedes other perceptual processes such as motion perception. These characteristics may appear to be contradictory, but only if one takes the physiologically driven classical notion of visual processing as a hierarchical, unidirectional relay of signals through local feature detectors, whose size and complexity increase with the level in the hierarchy. According to the traditional view, global perception is achieved only at high/late stages of visual processing, involving even problemsolving processes. We attempt to reconcile these two seemingly incompatible characteristics of perceptual surfaces in the light of recent physiological evidence for dynamic and global processing in the early visual cortex.

From local inputs to global percept of surfaces In classical illusions such as the Kanizsa figure (Fig. 75.1A) (Kanizsa, 1979) and the Varin configuration for neon color spreading (Fig. 75.1B) (Varin, 1971), we can observe completed surfaces even though the physical information is spatially limited. In the Kanizsa figure, in addition to illusory contours, the pacman-shaped stimuli create an illusory rectangular surface where the entire area appears slightly darker than the background. In the Varin figure, the color in the wedge-shaped portions appears to fill the square region. The term filling-in refers to the situation where a property such as brightness and color propagates beyond the region of physical stimulation to form a clear percept of a delineated surface. Filling-in phenomena are important because they provide a psychophysical paradigm to explore, as well as give insights into, the mechanism that integrates local inputs to form a global surface representation. What is noteworthy is that filling-in effects often involve an impression of one surface in front of another (or others). An example may be seen in a white disc that appears to be partly occluded by the illusory rectangular surface in the Kanizsa figure (Fig. 75.1A). This is sometimes called amodal completion because the occluded surface is perceived but is not locally visible in the literal sense. In the same Kanizsa figure, the rectangular illusory surface can be considered modally completed because it is perceived as an entirely visible, occluding surface. Another example of modal completion is the illusory colored surface in the Varin figure (Fig. 75.1B). In this case, the colored portions are seen as parts of a single semitransparent colored surface through which the back surface is also visible, thus allowing visibility of two surfaces along the same line of sight. Thus, in short, filling-in and multiple surface layouts characterize visual surface perception from spatially sparse inputs. Furthermore, it has been shown that minor changes in local visual inputs often lead to a drastic global change in surface perception. Figure 75.2 is an example demonstrating the effect of local disparity (Nakayama et al., 1989). The left-middle pair and the middle-right pair, which can fuse, contain opposite signs of local disparity, while the retinally stimulated areas are largely identical. Three discs through

1129

A

B

A

B F 75.1. A, Kaniza square. In addition to illusory contours forming a square, the pacman-shaped stimuli give rise to the percept of an illusory surface in which the entire area appears slightly darker than the background, and amodally completed white discs. B, Varin figure for neon color filling-in. By adding colored wedge portions to A, a transparent, colored surface and white discs seen through it are induced. (See color plate 49.)

F 75.2. Effect of local disparity information on global surface layouts. The observer is expected to see either three discs through one window or one disc and three windows, depending on the direction of disparity. The converging fuser should fuse the left and middle images to see three discs through one window and the middle and right images to see one disc through three windows. The diverging fuser should fuse the opposite pairs. (From Nakayama et al., 1989.)

one window or one disc through three windows can be perceived, depending on the disparity, that is, on which pair is fused. Thus, the relationship between modal (occludding) and amodal (occluded) can be reversed by a local change in disparity. Nakayama et al. (1989) also showed that global surface layouts defined by local disparity influences recognition of a face occluded by stripes. The performance was better when the face was seen behind the stripes than in front, indicating that amodal completion of the face facilitated recognition. Nakayama et al. (1990) studied the effect of local disparity on color filling-in using the Varin and other configurations. When the colored portions were defined as front (by a crossed disparity), color filling-in, subjective contours, and transparency were all enhanced. By contrast, when they were defined as behind, all these became amodal, thus suppressed. Figure 75.3 presents examples of color filling-in derived from very limited colored areas (the original was developed by Ken Nakayama; an unpublished observation). Here,

1130

, ,   

F 75.3. Effect of local edges on global surface completion. The converging fuser should fuse the left and middle images; the diverging fuser should fuse the middle and right images. The difference in the small colored regions leads to remarkably different filled-in surfaces: a diamond in A and a cross in B. Note that one of the colored portions is given only for one eye (the colored region on the left of the middle image is missing). Global filled-in surfaces can be formed even in the absence of binocularly matched inputs. (Courtesy of Ken Namayama.) (See color plate 50.)

colored areas, as well as disparity and collinearity cues, are presented very locally and sparsely. Note also that one of the colored areas is even unpaired between the left- and righteye images (Nakayama and Shimojo, 1990). Yet, a microscopic difference in edge orientation alone gives rise to a global difference in the completed surface (compare the top and the bottom stereograms when fused). Whereas collinearity itself may be considered a global property, the information that defines this property is edge location and orientation, which are given only very locally. Local changes in luminance-related cues, such as contrast at edges (Nakayama et al., 1990) and background luminance (Anderson, 1999), are also known to be critical in determining global surface properties. It should be noted that the local factors determining global surface properties, such as edge orientation, contrast at edges, disparity, and so on, are typical features that are detected in the early visual cortex (such as areas V1 and V2). Hence, the global aspects of surface perception do not exclude the critical role of local feature detection in the early visual cortex. As we will discuss later, one of the possible mechanisms underlying global surface representation is a propagation-like process starting from local features, which in effect can fill in a big gap in space to establish a surface representation. In such a mechanism, a local feature detector is local in the sense that it is activated by an isolated stimulus presented in a limited area (classical receptive field) but also global in the sense that the activity can be modulated by the global context outside the receptive field (Gilbert et al., 1990). Likewise, a local feature

F 75.4. Effect of amodal completion on the barber pole illusion. In this illusion, the movement of stripes (moving orthogonal to their orientation) is seen through a window. They appear to move vertically (horizontally) through a vertically (horizontally) elongated frame. A, If the panels separating the diagonal stripes are seen in

back by adding disparity, the stripes, seen in three horizontally elongated regions, appear to move horizontally. B, If the panels are placed in front, the diagonal stripes are completed behind them, forming a vertically elongated region. Then the movement of the stripes is seen as vertical. (From Shimojo et al., 1989.)

is local in that it is given in a limited retinal area, but it can also be global in that it can modulate the activity of distant local feature detectors.

pleted surfaces, indicating the precedence of surface completion over at least some types of motion processing. For example, it is known that the perceived direction of a drifting grating is ambiguous when presented in a rectangular window, with the direction along the longer axis being dominant (the barber pole illusion; Wallach, 1935). This dominance is preserved even when the window is partly occluded such that the longer axis of the visible (nonoccluded) areas is different from that of the entire window (Shimojo et al., 1989) (Fig. 75.4B). These results demonstrate that visual surface representation is established at a level earlier than cognitive and some perceptual processes as a result of preattentive or unconscious operations. On the other hand, it has been argued that surface perception can be regarded as a process of finding a statistically optimal solution to the inverse optics problem with regard to the real-world constraint (Nakayama and Shimojo, 1992; Poggio et al., 1985), thus analogous to cognitive inference at the functional level. It should be noted, however, that the early and inference-like aspects are not necessarily mutually exclusive. The real issue is whether the inference-like process is implemented at an early level, not mediated by conscious/cognitive processes, or at a late level, mediated by conscious/cognitive processes. The evidence above seems to support the former conclusion.

Surface formation in early visual processing Surface completion may be considered an example of problem solving or reasoning. As an alternative, it may be achieved by bottom-up mechanisms based on local feature detection at the very early level of cortical processing. In the case of subjective contours (and surface), for example, topdown, cognitive theories (Gregory, 1972; Rock and Anson, 1979) and bottom-up mechanistic theories (Grossberg and Mingolla, 1985; Peterhans and von der Heydt, 1989) have been proposed. More recent evidence indicates that the process of surface completion precedes other cognitive/perceptual processes. For example, Davis and Driver (1994) demonstrated that Kanizsa subjective figures can be detected without focal attention at parallel (thus early) stages of the human visual system. He and Nakayama (1994a) showed that visual search occurs only after the stage of amodal completion. Furthermore, a clinical study suggested that in a parietally damaged patient who suffered from extinction (a condition in which a visual stimulus is neglected when another stimulus is present in the intact visual field), the condition was less severe when bilateral stimuli formed a common surface, such as an illusory Kanizsa figure and a surface completed behind an occluder (Mattingley et al., 1997). Likewise, the perceived direction of ambiguous motion, both smooth (Shimojo et al., 1989) (Fig. 75.4) and apparent (He and Nakayama, 1994b; Shimojo and Nakayama, 1990), is affected by amodally com-

Aftereffects induced by perceptual surfaces Aftereffects, visual sensations that persist after prolonged viewing of stimuli, have been widely used to characterize mechanisms underlying visual perception. The percept is thought to reflect adaptation of neural subunits which

  :  

1131

respond to the adapting stimuli (see also Chapter 60). Aftereffects can be highly specific with respect to the features of adapting stimuli, such as orientation and spatial frequency, corresponding to the feature selectivity found in single neuronal responses in the visual system. Thus, aftereffects have been called the psychologist’s microelectrode (Frisby, 1979), which can be used to probe the relationship between visual perception and underlying neural mechanisms. In this and the following sections, we highlight the global and early nature of visual surface representation in light of its role in the formation of aftereffects. The orientation-contingent color aftereffect (the McCollough, effect; McCollough, 1965) is one of the striking examples of highly specific sensory adaptation. In the paradigm of the McCollough effect, an observer views alternating horizontal and vertical stripes with different colors, such as red vertical stripes alternating with green horizontal stripes, for several minutes. After adaptation, a test stimulus consisting of achromatic horizontal or vertical stripes is perceived to be tinged with the color complementary to that of the adapting stripes with the same orientation. Watanabe et al. (1992) replaced the test stripes with a grid pattern that appears to consist of overlapping horizontal and vertical stripes due to its appropriate luminance combination for perceptual transparency and surface segregation. They found that the orientation-contingent color aftereffect is perceived in the subjective overlapping stripes. Watanabe (1995) also showed that a test pattern made of largely occluded but perceptually completed stripes can elicit the McCollough effect. These results demonstrate that the McCollough effect, which is generally thought to involve early visual processes specific to orientation and color (Stromeyer, 1978), can be mediated or preceded by the global processing of perceptual surfaces. In the example described above, visual patterns inducing subjective surfaces were used as test stimuli on which the effects of adaptation were perceived. Is it possible to adapt to perceptually filled-in surfaces, resulting in aftereffects observed in the region that is not retinally stimulated during adaptation? It is known that prolonged viewing of a grating leads to a decrease in apparent contrast or to the elevation of threshold in a test grating with a similar orientation and spatial frequency (Blakemore and Cambell, 1969). Instead of a fully visible grating, Weisstein (1970) used a grating partially occluded by an object as an adapting stimulus and measured the apparent contrast of a small grating patch within the region where the occluding object had been presented during adaptation. The apparent contrast of the grating patch was significantly lower than that seen after adaptation to a blank screen or to the object alone. This suggests that the representation of amodally completed gratings can undergo adaptation in a manner similar to that of nonoccluded gratings.

1132

, ,   

Afterimages of filled-in surfaces Afterimages are often distinguished from other aftereffects (tilt, motion, size, etc.) in that they do not require a particular test stimulus to observe the effect. Afterimages are modulation of the first-order visual features, luminance and color, which are defined at each point of an image. Thus, a homogeneous background, having first-order features (luminance/color), is sufficient to observe their modulation. By contrast, other aftereffects are modulation of higher-order visual features, such as orientation and motion, which are defined by the relations among more than one point, and thus require a patterned (higher-order) test stimulus to observe their modulation. Hence, afterimages have been considered to reflect the most primitive point-by-point visual processing, and their origin has been believed to arise from either bleaching of photochemical pigments or neural adaptation in other retinal cells (Brindley, 1962; Craik, 1940, 1966; Virsu and Laurinen, 1977). Can afterimages, which have been thought to be concerned with primitive point-by-point processing, be created by adaptation to perceptually completed global surfaces? Shimojo et al. (2001) showed that perceptually filled-in surfaces can give rise to afterimages, using color filling-in displays such as the Varin figure (Figs. 75.1B, 75.5A). After prolonged adaptation, the adapting stimulus was replaced with a dark homogeneous background (Fig. 75.5A). The observers perceived not only the afterimages of the local inducers (pacmen/wedges or discs), but also a global afterimage of the perceptually filled-in surface (Fig. 75.5B). The global afterimage cannot be attributed to the general fuzziness and leaky edges of the afterimage, because adaptation to the inner wedge portions alone (without the outer pacmen) leads to the four parts corresponding to the wedges separately visible in the afterimage. The global afterimage is distinct from conventional afterimages in that it is visible at a portion that has not been retinally stimulated, but corresponds to a perceptually filled-in surface. The observation described above, however, does not necessarily imply that the global afterimage arises from the adaptation of the neural mechanism producing the perceptually filled-in surface. It is possible that the global afterimage originates from local afterimages of the inducers: the color of the global afterimage may be merely a result of the ordinary filling-in mechanism that may equally treat real stimuli and signals due to local adaptation. Shimojo et al. (2001) performed a series of experiments to demonstrate that the global afterimage is indeed due to adaptation of the representation of the filled-in surface (surface adaptation hypothesis), as opposed to local adaptation followed by an ordinary filling-in process (element adaptation hypothesis). First, the time course of the afterimage was analyzed. During the test period, different types of afterimage

A

B

F 75.5. Afterimage formed by adaptation to a perceptually filled-in surface. A, After adaptation to the Varin figure for neon color filling-in (left image; fixating to the white dot for about 30 seconds), the afterimage is observed on a blank screen (right). B, Typical afterimages. During the observation period, the appearance of the afterimage changes dynamically. The afterimages for the discs (left), for the filled-in surface (right), and for both (middle) appear to alternate several times. (Adapted from Shimojo et al., 2001.) (See color plate 51.)

appeared and disappeared several times in different time courses. Subjects were asked to monitor the visibility of two types of afterimage separately using two buttons: the local afterimage, which corresponded to elements of the inducer (pacmen, wedges, discs, or their combinations), and the global afterimage, which extended out of them toward the central portion to form a color-filled rectangle (Fig. 75.5B). They pressed one button while the local afterimage was visible and another while the global afterimage was visible. The results show that there were significant time periods during which the subjects reported visibility of the global but not the local afterimages (average, 20% of the total test period of 20 seconds). Furthermore, the likelihood analysis for the global afterimage in the presence or absence of the local afterimage revealed that the global afterimage tended to be more visible when the local afterimage was not visible than when it was visible. These observations indicate that the visibility of the local afterimage is not a necessary or even a favorable condition for the visibility of the global afterimage, in disagreement with the element adaptation hypothesis. One prediction of the surface adaptation hypothesis is that since the global afterimage is due to the perceptually filled-in surface during adaptation, the strength of the global

afterimage should be correlated with that of the perceptual filling-in during adaptation. The element adaptation hypothesis, on the other hand, predicts that the strength of the global afterimage should be determined solely by that of the local afterimages, and thus remain constant as long as the local afterimages are the same. In another experiment, the strength of perceptual fillingin during adaptation was manipulated by alternating two frames (667 msec each) composed of complementary parts of the inducer, as shown in Figure 75.6A (except condition 5, where only the colored wedge portions were turned on and off ). Since the strength of perceptual filling-in decreases approximately in the order of the conditions shown in Figure 75.6A, the surface adaptation hypothesis would predict that the strength of the global afterimage also decreases in the same order. On the other hand, since the total duration of adaptation was equal across portions of the stimulus in all the conditions except condition 5, the strength of the local afterimage should be the same across conditions 1 to 4. Thus, the element adaptation hypothesis would predict that the relative strength of the global afterimage would be approximately the same across conditions 1 to 4 (and perhaps weaker in condition 5). As shown in Figure 75.6B and 75.6C, both the estimated strength and the visible

  :  

1133

(2)

B

6

n=6 4

2

0

1

2

3

(4)

(3)

4

5

Condition

(5)

C Relative duration

(1)

Estimated strength

A

n=6 0.2

0.1

0

1

2

3

4

5

Condition

F 75.6. Surface versus element adaptation I. A, The inducer for the color filling-in was divided into complementary pairs (except in condition 5, where only the wedge portions were shown), and they were alternated (667 msec for each frame) for adaptation. The strength of perceptual filling-in during adaptation was varied (decreasing approximately in this order), while the total duration of adaptation to each portion of the inducer was held constant (except in condition 5). B, Estimated strength of the global afterimage for

the conditions depicted in A. The data for six subjects are pooled (error bar, standard error of the mean). The estimated value 10 corresponds to the strength of the global afterimage for the complete Vairn figure (no alternation or decomposition), which had been observed before this experiment. C, Duration of the global afterimage relative to the total test period. (From Shimojo et al., 2001.) (See color plate 52.)

duration of the global afterimage decreased according to the order of strength of perceptual filling-in, supporting the surface adaptation hypothesis but not the element adaptation hypothesis. In the last experiment, a new type of dynamic stimulus for color filling-in was employed to further dissociate the predictions of these hypotheses. In the static condition (Fig. 75.7A), line segments were placed sparsely and statically, so that the impression of color filling-in would be minimal, while the local afterimage of the line segments formed strongly. In the dynamic condition (Fig. 75.7B), the line segments were displaced up and down to create an impression of motion, while the disc-shaped area within which the line segments were colored blue was fixed. It appeared as though a set of white line segments moved up and down behind a semitransparent, stationary colored disc. This condition was designed to enhance the impression of color filling-in during adaptation while minimizing the local afterimage of line segments by the constant displacement. The surface adaptation hypothesis predicts that the duration of the local afterimage would be reduced, whereas that of the global afterimage would be increased, relative to the static condition. The element adaptation hypothesis, on the other hand, predicts that both local and global afterimages would be attenuated in the dynamic condition, since the duration of the global

afterimage should depend strictly on that of the local afterimage. The results of the visible duration of the local and global afterimages clearly indicate the enhancement of the global afterimage in spite of the attenuation in the local afterimage, consistent only with the surface adaptation hypothesis. These results support the idea that afterimages can arise from the adaptation of the neural mechanism representing perceptually filled-in surfaces. Since, unlike other aftereffects, afterimages are concerned with most primitive visual attributes, it may suggest that filled-in surfaces are represented at some very early stage of visual processing. In fact, Shimojo et al. (2001) also reported that interocular transfer of adaptation did not occur to induce the global afterimage in the unadapted eye, indicating that the adaptation occurred in monocular units, which exist only in the primary visual cortex or earlier (Hubel and Wiesel, 1962). In addition, when one eye was suppressed by pressure-blinding the retina after adaptation, the global as well as the local afterimages became invisible. Although these observations suggest contributions of retinal adaptation to the global afterimage, it is unlikely that retinal processing explicitly represents filled-in global surfaces, since each retinal cell deals with inputs only from a small area and lacks extensive longrange connections with other cells and projections from

1134

, ,   

A

Relative duration

C

B

SS

YK

0.4

Local

0.2

0

DW

0.4

RN

Global 0.2

0

Static

Dynamic

Static Dynamic

F 75.7. Surface versus element adaptation II. A, Static condition. The line segments were presented sparsely and statically. B, Dynamic condition. The line segments underwent vertical apparent motion while their portions in the fixed disc-shaped area were colored blue. The broken lines indicate positions for displacement and were invisible in the actual stimulus. The moving inducer pro-

duces more vivid filling-in but less local adaptation compared to the static condition. C, The duration of visible afterimages relative to the total observation time is plotted for the static and dynamic conditions in each subject. The white and black bars indicate local and global afterimages, respectively. (From Shimojo et al., 2001.) (See color plate 53.)

higher visual areas, which visual cortical neurons have. Therefore, the earliest level of visual cortical processing, which can handle both local primitive features and global connections, would need to be involved in the representation of perceptual surfaces and the adaptation that gives rise to the global afterimage. In the next section, we will discuss possible neural correlates in the early visual cortex.

actions beyond classical receptive fields (Spillmann and Werner, 1996; see also Chapter 106). Neural responses correlated with global surface perception have been demonstrated by manipulating the global context determining perceived surface attributes while holding the local features inside the receptive field constant. Rossi et al. (1996) studied the activity of neurons whose receptive field fell within a gray square surrounded by a background with changing luminance. This display produces illusory modulation of brightness of the central gray square: the brightness correlates negatively with the background luminance (brightness induction; Hering, 1964). They found that in many neurons in the primary visual cortex, the activity was modulated by the background luminance in accordance with the perceived brightness. Lamme (1995) showed that the activity of neurons in the primary visual cortex depends on the figure-ground context, that is, on which region appears as a surface in front of the other. A significantly larger response was observed when the receptive field was in the figure region than when it was in the background, while the local feature within the receptive field was constant. Zhou et al. (2000) (see also Chapter 76) reported another type of figure-ground selectivity of early visual cortical neurons. A local light-dark edge, for instance, could be the left side of a dark square or the right side of a light square. They found that neural responses to

Neural substrates for perceptual surfaces It is challenging to explain the global aspects of surface perception in terms of topographically organized, small receptive fields of visual neurons (Barlow, 1953; Hubel and Wiesel, 1962; Kuffler, 1953). This is the case not only for illusory surfaces, which we have focused on in the previous sections, but also for real homogeneous surfaces whose edges and surface attributes are physically available, because visual neurons generally respond better to variation in luminance or color than to their absolute values (homogeneous surface attributes). It is now widely accepted, however, that the response of visual neurons can be substantially and selectively modulated by stimuli presented outside the classical receptive field (Allman et al., 1985; see also Chapter 45). Surface perception may be one of the global perceptual phenomena that require such long-range inter-

  :  

1135

the same edge can be modulated by the side to which the border belongs to (border ownership), while the context used to determine border ownership was provided outside the receptive field. Neural responses to illusory or completed contours/lines that emerge in association with surface perception have also been found in early visual areas. Cells in area V1 as well as in V2 are known to be responsive to illusory contours, such as those perceived in the Kanizsa figure (e.g., Lee and Nguyen, 2001; von der Heydt et al., 1984; see also Chapter 76). Sugita (1999) demonstrated that cells in V1 respond to a bar occluded by a small patch (surface) when a disparity is given outside the receptive field such that the patch is seen in front. Bakin et al. (2000) also found that responses in the early visual areas (more in V2 than V1) are correlated with modally and amodally completed lines while depth cues for surface segregation are varied outside the receptive field. Most of these electrophysiological data are consistent with the view that perceptual surfaces are topographically represented: the response of each cell reflects the perceived surface attributes at the point/area corresponding to its receptive field. The notion of topographical representation is often criticized for assuming “redundant processes of painting an internal screen” (Dennett, 1991). It is logically possible that perceptual surfaces are represented in a “symbolic” way, without being mediated by topographical representation. For instance, a neuron that is selectively activated by images of faces can be thought to represent faces symbolically, but such representation may not require a map of activity similar to faces formed in a visual cortical area. The coding of border ownership described above (Zhou et al., 2000) may be regarded as a semisymbolic representation in early visual areas, since it enables a determination of surface attributes and configuration by tracing the activity corresponding to the border without looking at the whole twodimensional map of activity. It has been argued, however, that in order to determine surface attributes from local ambiguous information given within receptive fields, the brain may make effective use of the topographically organized circuits in early visual areas (Pessoa et al., 1998). A solution to the problem of determination of surface attributes may be achieved by propagation of activity from cells receiving critical information such as edges through synaptic cascades of local connections. This propagation mechanism may be considered more consistent with the topographical as opposed to the symbolic representation. A recent human functional magnetic resonance imaging study provides further support for topographical representation of perceptual surfaces. When a color fillingin stimulus similar to the Varin figure was presented, clear activation was produced in the cortical region in the primary visual cortex corresponding to the filled-in area in the visual

1136

, ,   

field (Sasaki et al., 2001). Thus, perceptually filled-in surfaces could be represented by filling-in in the cortical map. This finding is highly consistent with the demonstration of global afterimages induced by filled-in surfaces, described in the previous section in detail. A similar implication comes from a totally different angle and method. Kamitani and Shimojo (1999) showed that by transcranial magnetic stimulation (TMS) of the human occipital cortex, a hole, or scotoma, can be created in a flashed, large-field visual pattern, in accordance with the anatomy of early cortical maps (Fig. 75.8). They also found distortion of the scotoma in grating patterns (Figs. 75.8A and 75.8B), which can be interpreted in terms of the local connection among orientationdetection units observed in the primary visual cortex (Kapadia et al., 1995). The distortion may reflect a completion process that operates against the inhibitory effect of TMS by propagation of activity through local synaptic connections (Fig. 75.8D). While further studies must be done to determine whether, and to what degree, perceptual surfaces are topographically represented, functional characteristics of early visual cortical areas seem to be ideal to handle global as well as local information required for the processing of visual surfaces. It has been proposed that horizontal intrinsic connections within areas (Gilbert et al., 1990) and recurrent inputs from higher areas (Lamme and Roelfsema, 2000) result in dynamic changes in tuning in early visual areas, including the emergence of the global properties correlated with surface perception (Spillmann and Werner, 1996). It should be noted that recurrent inputs from higher areas are not necessarily derived from conscious/cognitive commands. They can be fast, automatic processes not mediated by conscious/cognitive processes. Furthermore, representation at early visual areas would remain early even after modulation by recurrent inputs, as long as it is available for later perceptual or cognitive processes. Although contributions of possible symbolic representation at higher areas, as well as global processing at subcortical levels (e.g., Pöppel, 1986), should not be excluded, the global and early aspects of visual surfaces discussed in the previous sections could be best explained by the global processing of early visual areas.

Conclusions In this chapter, we have provided a framework to integrate psychophysical and physiological findings around the key concept of visual surface representation. We have shown that in light of recent findings, the classical view of visual processing as a serial relay of signals through local feature detectors with progressively increasing size and complexity needs to be fundamentally reconsidered. The classical view assumes certain analogies among the notions, such as local/global and early/late: local processing is early, and

A

B

C

D +

+

- - F 75.8. A–C, Reconstructed percepts of TMS-induced scotomas seen on flashed visual patterns: a horizontal grating (A), a vertical grating (B), and a grid pattern (C). The visual pattern was presented for 40 msec, and the magnetic stimulation was delayed for 106 msec. D, Hypothetical mechanism for the anisotropic distortion of the scotoma. A horizontal grating is shown with horizontally tuned neural units aligned vertically (right) and horizontally

(bottom). The central gray circle represents the visual space corresponding to the cortical area directly affected by the magnetic stimulation. Long-range facilitatory connections among collinearly aligned units (+ in the bottom row) mask inhibition by inputs caused by TMS (-); thus, the suppressed region appears compressed along the stripes. (From Kamitani and Shimojo, 1999.)

global processing is late. However, we have demonstrated that visual surface representation is achieved by global yet early processing. We have also seen that the dichotomies of local versus global and early versus late themselves also need to be carefully applied to describe psychophysical phenomena and physiological processes involved in surface perception. For instance, a global surface representation can be created by local retinal inputs, and the activity of a neuron with a local receptive field is modulated by the global context. Early perceptual solutions look similar to late cognitive solutions based on inference, and recurrent processing in the visual cortex makes the distinction between early and late stages less meaningful. Such complications derive from the fact that that surface representation is beyond the scope of the classical view of visual processing as sequential, local-to-global feature detection.

Bakin, J. S., K. Nakayama, and C. D. Gilbert, 2000. Visual responses in monkey areas V1 and V2 to three-dimensional surface configurations, J. Neurosci., 20:8188–8198. Barlow, H. B., 1953. Summation and inhibition in the frog’s retina, J. Physiol., 119:69–88. Blakemore, C., and F. W. Cambell, 1969. On the existence of neurons in the human visual system selectively sensitive to the orientation and size of retinal images, J. Physiol., 203:237–260. Brindley, G. S., 1962. Two new properties of foveal afterimages and a photochemical hypothesis to explain them, J. Physiol., 164:168–179. Craik, K. J. W., 1940. Origin of visual after-images, Nature, 145:512. Craik, K. J. W., ed., 1966. The Nature of Psychology: A Selection of Papers, Essays and Other Writings by the Late K. J. W. Craik, Cambridge: Cambridge University Press. Davis, G., and J. Driver, 1994. Parallel detection of Kanizsa subjective figures in the human visual system, Nature, 371:791–793. Dennett, D. C., 1991. Consciousness Explained, Boston: Little, Brown. Frisby, J. P., 1979. Seeing: Illusion, Brain, and Mind, Oxford: Oxford University Press. Gibson, J. J., 1950. The Perception of the Visual World, Boston: Houghton Mifflin. Gilbert, C. D., J. A. Hirsch, and T. N. Wiesel, 1990. Lateral interactions in visual cortex, Cold Spring Harb. Symp. Quant. Biol., 55:663–677. Gregory, R. L., 1972. Cognitive contours, Nature, 238:51–52. Grossberg, S., and E. Mingolla, 1985. Neural dynamics of form perception: boundary completion, illusory figures, and neon color spreading, Psychol. Rev., 92:173–211.

REFERENCES Allman, J., F. Miezin, and E. McGuinness, 1985. Stimulus specific responses from beyond the classical receptive field: neurophysiological mechanisms for local-global comparisons in visual neurons, Annu. Rev. Neurosci., 8:407–430. Anderson, B. L., 1999. Stereoscopic surface perception, Neuron, 24:919–928.

  :  

1137

He, Z. J., and K. Nakayama, 1994a. Perceiving textures: beyond filtering, Vis. Res., 34:151–162. He, Z. J., and K. Nakayama, 1994b. Perceived surface shape not features determines correspondence strength in apparent motion, Vis. Res., 34:2125–2135. Hering, E., 1964. Outlines of a Theory of the Light Sense, Cambridge, MA: Harvard University Press. Hubel, D. H., and T. N. Wiesel, 1962. Receptive fields, binocular interaction and functional architecture in the cat’s visaul cortex, J. Physiol., 160:106–154. Kamitani, Y., and S. Shimojo, 1999. Manifestation of scotomas created by transcranial magnetic stimulation of human visual cortex, Nat. Neurosci., 2:767–771. Kanizsa, G., 1979. Organization in Vision: Essays on Gestalt Perception, New York: Praeger. Kapadia, M. K., M. Ito, C. D. Gilbert, and G. Westheimer, 1995. Improvement in visual sensitivity by changes in local context: parallel studies in human observers and in V1 of alert monkeys, Neuron, 15:843–856. Kuffler, S. W., 1953. Discharge patterns and functional organization of mammalian retina, J. Neurophysiol., 16:37–68. Lamme, V. A., 1995. The neurophysiology of figure-ground segregation in primary visual cortex, J. Neurosci., 15:1605–1615. Lamme, V. A., and P. R. Roelfsema, 2000. The distinct modes of vision offered by feedforward and recurrent processing, Trends Neurosci., 23:571–579. Lee, T. S., and M. Nguyen, 2001. Dynamics of subjective contour formation in the early visual cortex, Proc. Natl. Acad. Sci. USA, 98:1907–1911. Mattingley, J. B., G. Davis, and J. Driver, 1997. Preattentive fillingin of visual surfaces in parietal extinction, Science, 275:671–674. McCollough, C., 1965. Color adaptation of edge-detectors in the human visual system, Science, 149:1115–1116. Nakayama, K., and S. Shimojo, 1990. DaVinci stereopsis: depth and subjective occluding contours from unpaired image points, Vis. Res., 30:1811–1825. Nakayama, K., and S. Shimojo, 1992. Experiencing and perceiving visual surfaces, Science, 257:1357–1363. Nakayama, K., S. Shimojo, and V. S. Ramachandran, 1990. Transparency: relation to depth, subjective contours, luminance, and neon color spreading, Perception, 19:497–513. Nakayama, K., S. Shimojo, and G. H. Silverman, 1989. Stereoscopic depth: its relation to image segmentation, grouping, and the recognition of occluded objects, Perception, 18:55–68. Pessoa, L., E. Thompson, and A. Noe, 1998. Finding out about filling-in: a guide to perceptual completion for visual science and the philosophy of perception, Behav. Brain Sci., 21:723–748. Peterhans, E., and R. von der Heydt, 1989. Mechanisms of contour perception in monkey visual cortex. II. Contours bridging gaps, J. Neurosci., 9:1749–1763.

1138

, ,   

Poggio, T., V. Torre, and C. Koch, 1985. Computational vision and regularization theory, Nature, 317:314–319. Pöppel, E., 1986. Long-range colour-generating interactions across the retina, Nature, 320:523–525. Rock, I., and R. Anson, 1979. Illusory contours as the solution to a problem, Perception, 8:665–681. Rossi, A. F., C. D. Rittenhouse, and M. A. Paradiso, 1996. The representation of brightness in primary visual cortex, Science, 273:1104–1107. Sasaki, Y., T. Watanabe, A. M. Dale, and R. B. Tootell, 2001. V1 involvement for color filling-in revealed by human fMRI, Soc. Neurosci. Abstr., 12.11. Shimojo, S., Y. Kamitani, and S. Nishida, 2001. Afterimage of perceptually filled-in surface, Science, 293:1677–1680. Shimojo, S., and K. Nakayama, 1990. Amodal representation of occluded surfaces: role of invisible stimuli in apparent motion correspondence, Perception, 19:285–299. Shimojo, S., G. H. Silverman, and K. Nakayama, 1989. Occlusion and the solution to the aperture problem for motion, Vis. Res., 29:619–626. Spillmann, L., and J. S. Werner, 1996. Long-range interactions in visual perception, Trends Neurosci., 19:428–434. Stromeyer, C. F. I., 1978. Form-colour aftereffects in human vision, in Handbook of Sensory Physiology: VIII. Perception (R. Held, H. W. Leibowitz, and H. L. Teuber, eds.), Berlin: Springer-Verlag, pp. 97–142. Sugita, Y., 1999. Grouping of image fragments in primary visual cortex, Nature, 401:269–272. Varin, D., 1971. Fenomini di contrasto e diffusione cromatica nell’organizzazione spaziale del campo percettivo, Rivista di Psicologia, 65:101–128. Virsu, V., and P. Laurinen, 1977. Long-lasting afterimages caused by neural adaptation, Vis. Res., 17:853–860. von der Heydt, R., E. Peterhans, and G. Baumgartner, 1984. Illusory contours and cortical neuron responses, Science, 224: 1260–1262. Wallach, H., 1935. Über visuell wahrgenommene Bewegungsrichtung, Psychol. Forsch., 20:325–380. Watanabe, T., 1995. Orientation and color processing for partially occluded objects, Vis. Res., 35:647–655. Watanabe, T., G. L. Zimmerman, and P. Cavanagh, 1992. Orientation-contingent color aftereffects mediated by subjective transparent structures, Percept. Psychophys., 52:161–166. Weisstein, N., 1970. Neural symbolic activity: a psychophysical measure, Science, 168:1489–1491. Zhou, H., H. S. Friedman, and R. von der Heydt, 2000. Coding of border ownership in monkey visual cortex, J. Neurosci., 20: 6594–6611.

76

Image Parsing Mechanisms of the Visual Cortex RÜDIGER VON DER HEYDT

I  , I  discuss visual processes that are often labeled as intermediate-level vision. As a useful framework, we consider vision as a sequence of processes, each of which is a mapping from one representation to another. Understanding vision, then, means analyzing how information is represented at each stage and how it is transformed between stages (Marr, 1982). It is clear that the first stage, the level of the photoreceptors, is an image representation. This is a twodimensional (2-D) array of color values resembling the bitmap format of digital computers. Retinal processes transform this representation into a format that is suitable for transmission through the optic nerve to central brain structures. A radical transformation then takes place in the primary visual cortex. At the output of area V1 we find visual information encoded as a feature map, a representation of local features. The two dimensions of retinal position are encoded in the locations of the receptive fields of cortical neurons, but each neuron now represents not a pixel but a small patch of the image, and several new dimensions are added to the three color dimensions, such as orientation, spatial frequency, direction of motion, and binocular disparity. While the local feature representation of V1 is known in detail, we still do not have a good understanding of the nature of processing in the extrastriate areas. At first glance, the visual properties of the neurons in area V2 are rather similar to those of their input neurons of V1, except that V2 neurons have larger receptive fields (Burkhalter and Van Essen, 1986; Zeki, 1978). Many neurons of area V4 again have properties that can be found in V1 and V2, such as selectivity for color, orientation, and spatial frequency (Desimone et al., 1985; Schein and Desimone, 1990; Zeki, 1978). Some neurons in the extrastriate areas show selectivity for increasingly complex features (see Chapter 71). Here I will review evidence that the extrastriate areas provide a new stage of processing that may be described as image parsing. This stage appears as a mediator between the local feature representation of V1 and the processes of attentional selection and object recognition. From the enormous amount of information that streams in through the optic nerves at each moment, the visual system selects a small fraction, and in general precisely what is relevant for a given

task. This amazing performance indicates powerful mechanisms for organizing the incoming information. The Gestalt psychologists first pointed out that the visual system tends to organize elemental visual units (such as points and lines) into larger perceptual units, or figures, according to certain rules called Gestalt laws (see Chapter 106). Much of this organization occurs independently of what the subject knows or thinks about the visual stimulus. Gaetano Kanizsa illustrated this “autonomy of perception” with a painting that is supposed to show a knife behind a glass but is perceived instead as a transparent knife passing in front of the stalk of the glass (Kanizsa, 1979, Fig. 2.19). Grouping together features that belong to an object is a general task of perception. Specific visual problems arise from the fact that vision is based on 2-D projections of a three-dimensional (3-D) world. Due to spatial interposition, parts of the scene are occluded and features of near and far objects are cluttered in the image. Projections of the same objects vary with the viewing angle, and the true 3-D shape of the objects and their relations in space can only be inferred from the images. In principle, any image has an infinite number of possible interpretations in 3-D; vision is an “ill-posed problem” (Poggio and Koch, 1985). In spite of this fundamental ambiguity, vision is the most reliable of our senses. Apparently, through evolution and experience, biological vision systems have learned to make efficient use of the regularities present in images and to infer the missing information (Attneave, 1954; Barlow, 1961; Helmholtz, 1866; Marr, 1982; Poggio and Koch, 1985; Ullman, 1996).

Illusory contours: creative mechanisms A prominent example of this creative process is the phenomenon of illusory contours (Fig. 76.1). In A, the system “visibly” fills in the missing contours of an overlaying triangle. Note that the illusory contours are not just interpolations between given contrast borders, as they might seem to be in A, but form also in the absence of contrast borders that could be interpolated (C). In fact, when the corners of the overlying triangle are defined by lines which could be interpolated, illusory contours do not form (B). What all illusory contour figures have in common is the presence of occlusion

1139

F 76.1. Perception of illusory contours. (A and C, after Kanizsa, 1979. D, René Magritte, Paysage de Baucis, etching, 1966.)

cues, such as terminations of lines and edges (Coren, 1972). Thus, the system seems to infer an occluding object. However, this is not an inference in abstract terms. The mere expectation of a contour does not lead to perception of illusory contours (Fig. 76.1D). Apparently, in forming the contours, the system combines evidence from occlusion cues with rules such as the Gestalt principle of good continuation. Interestingly, illusory contours are represented in the visual cortex at a relatively early stage. In monkey area V2, many cells respond to illusory contour stimuli as if the contours were contrast borders (von der Heydt et al., 1984). Figure 76.2 shows an example of a cell that was tested with a moving illusory bar. The raster plot in B shows that the cell responds when the illusory contour traverses a small region that was determined before as the cell’s minimum response field (ellipse; see legend). Figure 76.2C shows a control in which the two bars were moved exactly as in B, but the open ends were closed off with thin lines. Closing lines weaken the perceptual illusion (see the figure at the bottom), and they also reduce the responses of the neuron. Cells in V2 respond not only to figures with illusory bars,

1140

, ,   

F 76.2. Illusory contour responses in a neuron of area V2. Each line of dots in the raster plots represents a sequence of action potentials fired in response to the stimulus shown on the left. A, Responses to a moving dark bar; B, to a figure in which a moving illusory bar is perceived; C, to a modified figure in which the illusion is abolished by adding line segments. Note the reduction of responses. The figures at the bottom illustrate the perceptual effect of adding lines. D, Spontaneous activity. Ellipses indicate the minimum response field of the neuron (i.e., the minimum region outside of which a bar does not evoke a response); the cross indicates the fixation point. (From Peterhans and von der Heydt, 1989, with permission.)

but also to other figures that produce illusory contours, such as a pattern of two abutting line gratings (Fig. 76.3B). It can be seen that the cell of Figure 76.3 responds at the same orientations for the illusory contour as for the bar stimulus; thus, it signals the orientation of an illusory contour. Using the criteria of consistent orientation tuning and response reduction by the closing lines, 30% to 40% of the cells of V2 were found to signal illusory contours of one or the other type, and the results obtained with the two types of contour were highly correlated (Peterhans and von der Heydt, 1989; von der Heydt and Peterhans, 1989). As shown in Figure 76.2, illusory contour responses can be evoked by stimuli which are devoid of contrast over the excitatory center of the receptive field. The inducing contrast features can be restricted to regions from which an optimized bar stimulus would not evoke any response. The cells seem to integrate occlusion features over a region larger than the conventional receptive field (Peterhans et al., 1986).

F 76.3. Illusory contour responses in another neuron of V2. In A, bars, and in B, the border between two gratings were moved across the receptive field at 16 different orientations spanning 180 degrees. The neuron responds at the same orientations for bars and illusory contours. Bottom right, control: grating without a border of discontinuity. (Modified from von der Heydt and Peterhans, 1989, with permission.)

Nevertheless, the extent of spatial integration is limited; for neurons with near-foveal receptive fields, the responses declined if the gap of the stimulus (Fig. 76.2B) was made wider than about 3 degrees visual angle. V2 is one of the largest areas of the monkey cerebral cortex (Felleman and Van Essen, 1991), and the fact that so many cells in this area respond this way indicates that illusory contour stimuli probe a basic function of the visual cortex. V2 is an early stage of processing where responses are fast and highly reproducible. Illusory contour responses arise as early as 70 msec after stimulus onset (Lee and Nguyen, 2001; von der Heydt and Peterhans, 1989). This indicates that illusory contours are probably not the result of object recognition processes at higher levels but are generated within the visual cortex. Computational models have shown how such contours might be generated (e.g., Finkel and Sajda, 1992; Grossberg and Mingolla, 1985; Heitger et al., 1998). P S Representation of illusory contours has also been demonstrated in V1 of cat (Redies et al., 1986; Sheth et al., 1996) and monkey (Grosof et al., 1993; Lee and Nguyen, 2001; Ramsden et al., 2001). However, it is not clear if cells in V1 also generalize over the various types of illusory contour figures and if they signal the contour orientation. Sheth et al. (1996) and Ramsden et al. (2001) used a combination of optical imaging and single-unit recording to identify the illusory contour representation with the abutting-grating type of stimulus. Sheth et al. found cells with consistent orientation tuning for illusory contours in V1

of the cat. In the monkey, Ramsden et al. found that the representation of illusory contours in V1 is different from that of V2. Illusory contours reduced activity in columns of the corresponding orientation but increased activity in columns of the orthogonal orientation, in contrast to V2, where the same columns were activated by illusory contours and contrast borders. They conclude that V1 deemphasizes illusory contours. Studies that compared both areas invariably found marked differences between V1 and V2 in the frequency of cells that signaled illusory contours, the signaling of orientation, and the degree of cue invariance (Bakin et al., 2000; Leventhal et al., 1995; Ramsden et al., 2001; Sheth et al., 1996; von der Heydt and Peterhans, 1989). C  P  P Varying the configurations and spatial parameters of the displays shows a tight correspondence between human perception and neural responses for illusory contours generated by abutting gratings (Fig. 76.3B) (Soriano et al., 1996). However, in discriminating the shape of illusory figures, the human visual system shows larger spatial integration than the neurons of monkey V2 (Ringach and Shapley, 1996). Because neurons that signal illusory contours are only a subset of the cells that signal contrast edges, orientation-dependent adaptation aftereffects should transfer from contrast-defined to illusory contours, but not in the reverse direction, and the discrimination of orientation should be less accurate for illusory contours than for contrast-defined contours. Both predictions were borne out in psychophysical experiments (Paradiso et

  :       

1141

al., 1989; Westheimer and Li, 1996). Illusory contours are usually associated with perception of overlay (Coren, 1972), and some neurons in V2 are selective for the implied direction of occlusion of illusory contours (Baumann et al., 1997). Thus, the illusory contour mechanisms may be related to the coding of border ownership, discussed below. I C  U Perception of illusory contours has been demonstrated in a variety of nonhuman species, including the cat, owl, and bee (Bravo et al., 1988; De Weerd et al., 1990; Nieder and Wagner, 1999; Srinivasan et al., 1987; for a review, see Nieder, 2002). Most elegant is the combination of behavioral experiments with single-cell recordings (Nieder and Wagner, 1999).

Border ownership: image context integration Illusory contours and related visual phenomena are only the tip of an iceberg of cortical processes involved in perceptual organization. Kanizsa’s figure (Fig. 76.1A) suggests that illusory contours are the product of mechanisms in figureground segregation. The system takes the peculiar arrangement of the black elements as evidence for an occluding triangle and hence creates a representation of its contours. In fact, it also creates the representation of a white opaque surface, as one can see from the subtle difference in brightness relative to the background. The illusory contours appear as the edges of this surface. Similar linking of contour and surface can also be observed for sharp contrast borders. Perception tends to interpret such borders as occluding contours and assigns them to a surface on one or the other side of the border. This compulsion of the visual system is demonstrated by Rubin’s vase figure (Fig. 76.4A). The borders are perceived either as the contours of a vase or as the contours of two faces. Each border is perceived as belonging to one or the other side, but rarely to both. In the case of a simple figure such as the white square of Figure 76.4B, the contrast borders are “of course” perceived as the contours of the square. They seem to belong to the enclosed light-textured region. The surrounding gray, which does not “own” these borders, is perceived as extending behind the square, forming the background. Perception of border ownership is a subtle phenomenon that remained long unnoticed until it was discovered by the Gestalt psychologists (Koffka, 1935; Rubin, 1921). One could argue that even the display of Figure 76.4B is ambiguous. With some effort, the square can also be perceived as a window, and the border then appears as the edge of the frame. Completely unambiguous displays can be produced by means of random-dot stereograms, as shown in Figure 76.4C. When binocularly fused by crossing the eyes (see the legend), the top pair shows a tipped square floating in front of a background plane, while the bottom pair shows

1142

, ,   

F 76.4. Perception of border ownership. A, Physiologist’s version of Rubin’s vase. The black-white borders are perceived either as contours of the vase or as contours of the faces. B, White square. The contrast borders are generally perceived as contours of the square. C, Stereograms. Left and right textured square fields can be fused, for example, by squinting (try crossing the lines of sight of the two eyes until three fields are perceived instead of two; the center field then shows the result of binocular fusion). On fusion with crossed eyes, the top pair shows a square figure, while the bottom pair shows a square window. In the former, the 3-D edges belong to the figure; in the latter, to the surround.

a square window through which a background plane can be seen. In the first case, the stereoscopic borders are perceived as the edges of the square; in the second case, as edges of the window frame. Perception of border ownership cannot be reversed in these stereograms. In perceptual experiments we observe the tip of the iceberg. By recording signals in visual cortex, we should also be able to explore the depth of it. Contrast borders are represented in the visual cortex by signals of the orientationselective cells discovered by Hubel and Wiesel. Do these signals also represent the relationship between border and

F 76.5. Selectivity for side of figure in a neuron of area V2. Edges of squares were tested so that the square was either on the left side (A) or on the right side (B) of the receptive field (ellipses show the minimum response field, cross marks the fixation point). Note that corresponding displays in A and B are identical over the

combined area of the two squares. Tests with square sizes of 4, 10, and 15 degrees are shown. The bar graph represents mean firing rates with the standard error. In every case, the neuron responds more strongly when the figure is on the left side, despite locally identical stimulation. (From Zhou et al., 2000, with permission.)

surface? This idea can be tested with a simple experiment (Zhou et al., 2000). Light-dark borders are placed in the receptive field of a neuron at optimal orientation (Fig. 76.5), and the same border is either presented as the right side of a light square (e.g., A1) or as the left side of a dark square (B1). A2 and B2 show a similar test with displays of reversed contrast, and columns 3 and 4 and 5 and 6 show the same kind of test with larger squares. The bar graph at the bottom represents the responses of a cell of V2. If we compare the responses to the corresponding displays in A and B, we see that in every case the neuron responds more strongly when the edge in the receptive field belonged to a square to the left than a square to the right, despite locally identical stimulation. Note that the corresponding displays in rows A and B are identical over the entire region occupied by the two squares (as one can see by superimposing them). Thus, if a neuron responds differently, it must have information from outside this region. Therefore, by varying the size of the square, we can reveal the extent of image context integration. In this example, square sizes of 4, 10, and 15 degrees were tested, and in each case the responses differed, depending on the location of the figure. By contrast, the size of the minimum response field of this cell was only 0.4 degree, which is typical for V2 neurons of the foveal representation. Thus, although the cell can “see” only a small piece of contrast border through the aperture of its receptive field, its responses reveal processing of an area of at least 15 degrees in diameter.

What might be the mechanism of side-of-figure selectivity? For a single square figure on a uniform background, relatively simple algorithms would be able to discriminate figure and ground. The convexity of the figure area could be used, or simply the orientation of the L-junctions (corners) on either side of the receptive field, or the fact that the figure is a region of one color enclosed by a region of a different color (surroundedness). Any of these strategies would work for the isolated square. However, for other displays in which border ownership is also perceptually clear, mechanisms based on one simple strategy would fail to produce the right answer. We have used two other configurations besides squares to see how well the neural responses correlated with perception, a C-shaped figure shown in columns 3 and 4 of Figure 76.6, and a pair of overlapping squares as shown in columns 5 and 6 of the same figure. For the C shape, convexity is not valid, and the L-junctions next to the receptive field are reflected to the other side in comparison with the square, but surroundedness would still be a valid cue. For the overlapping squares, surroundedness is violated, while convexity and orientation of L-junctions are valid. Figure 76.6 shows data from another neuron of V2. Columns 1 and 2 show the same test described in Figure 76.5. This cell was selective for contrast polarity, responding to light-dark edges as shown in A1 and B1 but hardly at all to dark-light edges as shown in A2 and B2 (the actual colors in the experiment were violet, shown here as light gray and

  :       

1143

F 76.6. Example of a V2 neuron tested with squares, Cshaped figures, and overlapping figures. The neuron was color selective, with a preference for violet (depicted here as light gray). In the test with single squares (1 and 2) the neuron is selective for side of figure and local contrast polarity, responding best to the edge of a violet square located on the lower-left-hand side of the

receptive field (A1). With C-shaped figures (3 and 4), the neuron responds better to B3 than to A3, pointing to the lower left as the figure side, in agreement with perception. With overlapping figures (5 and 6), the neuron responds better to A5 than to B5, assigning the edge to the figure that is perceived as overlaying. (From Zhou et al., 2000, with permission.)

gray; the cell was strongly color selective). The cell was also side-of-figure selective, with a preference for figure location on the left side of the receptive field (display A1). Columns 3 and 4 show a test with a C-shaped figure. It can be seen that the cell “correctly” preferred the display in which the C-shaped figure was located on the left of the field (display B3), although the L-junctions next to the receptive field suggest a figure on the opposite side. Columns 5 and 6 show a test with two overlapping figures. These displays are fairly symmetric about the receptive field as far as size of regions and distribution of colors are concerned, and neither of the figures is surrounded by uniform color. Nevertheless, the cell preferred display A5, in which the border in the receptive field belongs to the lower left figure. In this case, the T-junctions might account for the emergence of the occluding square as a figure (but convexity might also contribute because the overlapped region has a concavity, whereas the overlapping region does not). Thus, the responses of this cell are entirely consistent with the perception of border ownership. Not all cells tested showed this pattern, but the example is not unusual. About half of the cells with a side-of-figure effect for single squares exhibited the corresponding side preference for overlapping figures, while the others showed no significant response difference. Vice versa, the overlapping figure test predicted the single-figure result in about half of the cases. When tested with the concave side of C-shaped figures, about one-third of the cells with a side-of-figure effect for

single squares showed preference for the C on the same side; the others were indifferent. Cases in which the side preferences were “contradictory” (as judged by perception) were rare. Cells with a response preference for one or the other side of the figure were found for any location and orientation of receptive field, and side-of-figure preference was invariant throughout the recording period. The responses of these cells seem to carry information not only about the location and orientation of the contours, but also about the side to which they belong. About half of the orientation-selective cells of areas V2 and V4 were found to be side-of-figure selective by the test of Figure 76.5. In 32% of the V2 cells, the ratio of the responses to preferred and nonpreferred sides was greater than 2, and ratios as high as 10 were not unusual. For comparison, by the same criterion, 29% of V1 cells are direction selective (De Valois et al., 1982) and 50% of upper-layer V1 cells are opponent color selective (from Fig. 76.9 of Leventhal et al., 1995). We found side-of-figure selectivity also in V1, but in a smaller fraction of the cells. These results show that the side-of-figure test probes an important aspect of the cortical representation. Experiments as shown in Figures 76.5 and 76.6 suggest the existence of cortical mechanisms that use figure-ground cues to assign border ownership. In other words, the signals of orientation-selective cells in V2 might represent not only the location, orientation, and contrast of pieces of contour, but also the side of ownership.

1144

, ,   

F 76.7. Stereoscopic edge selectivity. Responses of a neuron of V1 (left) and two neurons of V2. Contrast-defined squares are compared with squares portrayed in random-dot stereograms (Julesz, 1971). Each figure is presented at 16 different positions relative to the receptive field, as shown schematically at the bottom. Ellipses represent the minimum response fields; squares represent the test figures. Data points in the graphs correspond to the dotted

positions below. The plots show that cell 1 is edge selective for contrast-defined figures but responds all over the surface of the figure for random-dot stereograms, whereas cells 2 and 3 are edge selective for both types of figures. Most of these cells respond asymmetrically to the two sides, like cell 3. (From von der Heydt et al., 2000, with permission.)

Stereoscopic depth and monocular form cues

erence should agree with the preferred depth order. That is, the preferred figure side should be the near side of the preferred step edge. This experiment is illustrated in Figure 76.8. With the random-dot stereogram, the cell is activated by the left edge of the figure and the right edge of the window, but not by the right edge of the square or the left edge of the window. From this we conclude that activation of this cell means border assignment to the surface on the right. Therefore, the responses to the contrast-defined square (A stronger than B) show that the cell “correctly” assigns the border to the square, so the square is interpreted as a figure. If the cell responded more strongly to B than to A, this would mean that it assigns the border to the frame and the square would be interpreted as a window. Of 27 cells recorded in V2 that signaled depth order for random-dot edges and side of figure for contrast-defined figures (p < .05 in each case), 21 responded according to the “figure” interpretation and 6 according to the “window” interpretation (Qiu et al., 2001). This result is in agreement with the tendency in human perception to interpret compact, uniform regions in the image as objects. We speculate that the minority of “window” responses might be not just aberrant signals, but the representation of a valid alternative parsing solution. Occasional dissident votes were also recorded when the side preferences for single squares were compared with those for overlapping figures.

A crucial test of the hypothesis of border ownership coding is to examine the responses of orientation-selective cells to contrast-defined and disparity-defined figures. A contrastdefined square is generally perceived as a figure, with the borders assigned to the square (Fig. 76.4), while a corresponding region in a random-dot stereogram is perceived either as a figure, if its disparity is “near,” or as a window, if its disparity is “far,” relative to that of the surrounding region. In the stereogram, the nearer surface always owns the border. Thus, the random-dot stereogram is the “gold standard” of border ownership perception. Binocular disparity is represented extensively in the monkey visual cortex (Cumming and DeAngelis, 2001; Poggio, 1995), and cells that signal edges in randomdot stereograms exist in area V2 (von der Heydt et al., 2000). These cells are orientation selective and respond to disparity-defined edges as well as to contrast borders (Fig. 76.7). Most of them are selective for the depth order of the stereoscopic edge, responding, for example, to a vertical edge if the front surface is on the right side, but not if the front surface is on the left side (cell 3 of Fig. 76.7). If border ownership is represented in V2, then some of the cells there should combine monocular shape cues with binocular disparity information, and the side-of-figure pref-

  :       

1145

F 76.8. A critical test of the hypothesis of border ownership coding. Side-of-figure preference and stereo edge preference are assessed for each neuron. For contrast-defined squares, the “figure” interpretation is perceptually more compelling than the “window” interpretation. Therefore, if neurons code for border ownership, the preferred near side should be also the preferred side of the figure. This fictitious neuron prefers the nearer surface to the right for the stereogram (regardless of whether a square or a window is displayed) and prefers the figure to the right (A) for the contrast-defined display, in accordance with the hypothesis.

Monocular form cues are usually ambiguous; a square can be perceived as a window, and even the display of two overlapping squares can alternatively be perceived as an Lshaped object adjacent to a square. It seems plausible that the visual cortex represents several alternative 3-D interpretations if the image is ambiguous. L C  S A The convergence of stereoscopic edge mechanisms and side-of-figure processing in single cells strongly supports the conclusion that side-of-figure selective cells code for border ownership. Figure 76.9 illustrates, for a pair of overlapping squares, how border ownership information is represented together with information about other contour features such as orientation, and color and luminance contrast. Each piece of contour is represented by two pools of neurons, one for each side of ownership. By analogy to the opponent coding of direction of motion, we assume that border ownership is encoded in the relative strength of activity in pairs of neurons of opposite side preference but otherwise identical receptive fields. This scheme of coding allows the linking of contour and surface attributes. Location and orientation of contour are coded by virtue of orientation selectivity and the small size of response fields. Color and brightness of object surface are coded by means of color and contrast polarity selectivity of cells with the corresponding border ownership pointer. Depth of surface is encoded similarly in the activity of stereo edge-selective cells.

A case for low-level mechanisms An interesting point is the time course of the border ownership signals. Figure 76.10 compares the averaged neuronal

1146

, ,   

F 76.9. Schematic illustration of the cortical representation of a pair of overlapping squares. Ellipses indicate receptive fields. Each piece of contour is represented by two pools of orientationselective neurons with opposite border ownership preference, as indicated by arrows. Filled symbols indicate the neurons whose activity would be enhanced for this stimulus. Border ownership is thought to be encoded in the relative activation of the two pools.

responses for a figure on the preferred side (thick line) and a figure on the nonpreferred side (thin line). Data from areas V1, V2, and V4 are shown. It can be seen that a differentiation of responses occurs soon after stimulus onset and well before the responses peak. Note also that the response difference in V2 neurons remains constant during the remainder of stimulus presentation. If the effect of side of figure were due to feedback from areas of much higher order, we would probably see a delay. The immediate differentiation suggests that the mechanisms reside in these lower-order visual areas. Also, Bakin et al. (2000), who studied neural correlates of contour salience, illusory contours, and depth capture in V1 and V2, found no increase of latency for these image-parsing processes. Lee and Nguyen (2001) found illusory contour responses in V2 with latencies as low as 50 msec, and differentiation between modal and amodal completion by 70 msec. All these results argue for fast processes, implicating highly parallel mechanisms in the lower-order cortical areas. Because of the ambiguity of monocular form cues and the ill-posed nature of the vision problem in general, image segmentation is usually regarded as a task that cannot be solved by low-level computations but that requires the use of stored representations in memory. Why, then, would the visual system use low-level mechanisms at all to resolve figure-ground relationships? Since memory will eventually be used to recognize objects, one may wonder what is the advantage of low-level mechanisms.

F 76.10. The time course of the border ownership signal. The figure shows the averaged normalized responses of neurons in three cortical areas. Squares of 4 or 6 degree size were presented, as in Figure 76.5. Zero on the time scale refers to the onset of the display. Thick and thin lines represent responses to preferred and nonpreferred sides, averaged over both contrast polarities. The delay between onset of response and differentiation of side of figure was less than 25 msec. (From Zhou et al., 2000, with permission.)

In the case of a single border, as in Figure 76.6, columns 5 and 6, assigning figure-ground direction reduces the number of shapes that have to be compared with memory from two to one (by eliminating the inverted L shape created by occlusion), which may not appear as a great saving. However, it is important to recognize that the problem is generally more complex. As an example, consider the display of Figure 76.11A, which might be perceived as two elongated objects occluding one another or, in keeping with the macaque perspective, as a branch in front of a tree stem. Contrast borders divide the display into seven regions of different shapes. Since the contrast borders may be occluding contours, most of these shapes are meaningless because they are surfaces of partly occluded objects, that is, regions that do not own the borders. There are 10 segments of borders (not counting the frame), each of which could belong to one of the adjacent regions, creating a total of 210 = 1024 possible depth configurations. Each depth configuration defines a different set of shapes. To help the reader to see this, I have illustrated two of the possible 3-D decompositions in Figure 76.11B. Most of these configurations are generally not perceived. The point is that there is a large number of shapes

F 76.11. The interpretation of the overlay structure of images is highly ambiguous. A, Example of a display that would generally be perceived as two elongated objects occluding one another. B, Two of about 1000 possible interpretations of A.

that could give rise to an image like that of Figure 76.11A. All of these would have to be searched in memory if the system were not able to assign the borders beforehand. If borders are assigned, only the two bars in front of a white background have to be processed further. Thus, low-level border assignment reduces the load on the memory matching process enormously in this example; this is probably similar in images of natural scenes, which are generally complex. Note that the advantage of low-level processes does not depend on their ability to find a unique solution for each parsing problem. On the contrary, the results on border ownership coding sketched above indicate that the visual cortex can simultaneously represent alternative solutions. A representation like that of area V2, which contains on the order of 100 million cells, has an enormous capacity. V1 and V2 can represent more than one orientation per image point (see, for example, simultaneous representation of two superimposed gratings in V1; Movshon et al., 1985). Therefore, it is plausible to hypothesize that the visual cortex can represent several alternative parsing results from which cognitive routines can select. Compared to the huge number of possible interpretations of an image, this would still be a very specific representation. There are theoretical and empirical arguments for this hypothesis. On the theoretical side, it was pointed out that it would be a disadvantage if the system locked in on one solution, given the ambiguous information available at the

  :       

1147

precognitive level (principle of least commitment; Marr, 1982). Experimental studies of binocular rivalry show that the duration of dominance of the stimulus in one eye depends on the suppressed stimulus in the other eye (Levelt, 1968). Accordingly, the unperceived stimulus in rivalry is represented in the neural activity in visual cortex (Leopold and Logothetis, 1996; Logothetis and Schall, 1989).

Conclusions The studies reviewed above demonstrate a wealth of processes of visual organization in the cortex. While it was previously thought that contour mechanisms serve to make contrast borders explicit and “fill in the gaps,” recent findings indicate that cortical processing goes beyond representation and completion of contrast borders. The goal of this processing apparently is to make the 3-D structure of a scene explicit—specifically, to represent occluding contours and how they belong to surfaces. Assigning contours to surfaces is perhaps the most important first step on the way from image to object representation, because it specifies the surfaces in the scene and their ordering in depth. Threedimensional surface organization influences perceptual grouping of elemental features in many ways and modulates responses of V1 and V2 neurons accordingly (Bakin et al., 2000; Sugita, 1999; Zipser et al., 1996). Changes in 3-D surface organization that alter perception of border ownership can also influence the perceived direction of motion, and this influence can be traced in the motion signals of area MT (Duncan et al., 2000). The most surprising aspect of the new findings is that neural signals at early cortical levels, despite the small size of the conventional receptive fields of the cells, reflect information about the global image context. Assigning border ownership generally requires the application of algorithms that involve image regions of at least the size of the projection of the object to be processed. The same is true for labeling textured regions as figure or ground (Lamme, 1995; Lee et al., 1998; Zipser et al., 1996). These algorithms might implement Gestalt rules such as Prägnanz of form, contour closure, and common fate, as well as the rules that relate patterns of junctions to 3-D shape and layout of objects (Adelson, 1993; Nakayama et al., 1995; Waltz, 1975). Thus, figure-ground segregation and assignment of border ownership are evidence for image parsing mechanisms and suggest networks of global scope. The organization of visual representations in terms of 3-D surfaces may be a first stage in the construction of object representations (Marr, 1982), or it may provide a structure for specific processing of selected detail information which is required by most visual tasks, when attention is directed to a specific object in a scene, or to a specific feature of an object (He and Nakayama, 1992; He and Ooi, 2000; Nakayama et al.,

1148

, ,   

1995; see also Egeth and Yantis, 1997). Perhaps the same networks that achieve image parsing serve also for the selection of information. The exact nature of those image parsing mechanisms still needs to be clarified. What kinds of cues are used, and how are they combined? What is the output of this stage? Probably the output is not a globally coherent representation of 3-D relationships (because perception is not globally coherent; see the Penrose triangle and the work of the artist M. C. Escher), but rather a patchwork of coherent domains. How large are these domains? Does the parsing stage provide a unique solution for a given image (or domain) or multiple solutions? The mechanisms might either home in on the most likely interpretation of the given optical information or pass on several choices to the next stage. The results sketched out above on the combination of stereoscopic depth and monocular form cues suggest that V2 can represent multiple solutions in parallel, leaving the final decision to a later stage. It has often been pointed out that vision (and any perception) is highly selective. An enormous amount of information streams in through the eyes, but little of it is used. A common demonstration is the “two pictures with 10 differences.” Viewers generally have to look back and forth many times between the two pictures before they find the differences. This shows that the amount of information the system can store and compare between two looks is only a small fraction of the total, even in those relatively simple drawings. This phenomenon of change blindness, which has been documented by formal experiments (Rensink et al., 1997; see also Chapter 102), has led to the conclusion that the visual system processes only the selected information and does not represent much more than the image. Whatever information is needed at a given moment, it is thought, can be retrieved from the image representation. However, psychological tests can only reveal processing that leads either to a motor response or to a retrievable representation in memory. As I have argued, this is only the tip of the iceberg. The recordings from the visual cortex reveal sophisticated processing of gigantic amounts of information. In the experiments on border ownership coding, for example, the animal was attending to a visual task at the fixation spot and thereby—probably—trying to ignore all other visual stimuli as much as possible, but border ownership processing was nevertheless obvious in half of the signals recorded in V2. Thus, it seems that many locations of the retinal image are processed automatically and in parallel all the time. Whenever our eyes saccade to a new point of fixation, area V2 recomputes the figure-ground relationships, parsing a new image into object-like chunks of information. All this occurs three or four times per second on average. Without this extensive preprocessing, the system would not be able to select information as efficiently as it does. The relatively

large size of the early visual areas V1, V2, and V4 is certainly related to the computational difficulty of the image parsing task, which still defies the power of supercomputers. The imperceptible function of these areas makes vision appear effortless.

REFERENCES Adelson, E. H., 1993. Perceptual organization and the judgment of brightness, Science, 262:2042–2044. Attneave, F., 1954. Some informational aspects of visual perception, Psychol Rev., 61:183–193. Bakin, J. S., K. Nakayama, and C. D. Gilbert, 2000. Visual responses in monkey areas V1 and V2 to three-dimensional surface configurations, J. Neurosci., 20:8188–8198. Barlow, H. B., 1961. Possible principles underlying the transformations of sensory messages, in Sensory Communication (W. A. Rosenblith ed.), Cambridge, MA: MIT Press, pp. 217– 257. Baumann, R., R. van der Zwan, and E. Peterhans, 1997. Figureground segregation at contours: a neural mechanism in the visual cortex of the alert monkey, Eur. J. Neurosci., 9:1290–1303. Bravo, M., R. Blake, and S. Morrison, 1988. Cats see subjective contours, Vis. Res., 28:861–865. Burkhalter, A., and D. C. Van Essen, 1986. Processing of color, form and disparity information in visual areas VP and V2 of ventral extrastriate cortex in the macaque monkey, J. Neurosci., 6:2327–2351. Coren, S., 1972. Subjective contours and apparent depth, Psychol. Rev., 79:359–367. Cumming, B. G., and G. C. DeAngelis, 2001. The physiology of stereopsis, Annu. Rev. Neurosci., 24:203–238. De Valois, R. L., E. W. Yund, and N. Hepler, 1982. The orientation and direction selectivity of cells in macaque visual cortex, Vis. Res., 22:531–544. De Weerd, P., E. Vandenbussche, B. Debruyn, and G. A. Orban, 1990. Illusory contour orientation discrimination in the cat, Behav. Brain Res., 39:1–17. Desimone, R., S. J. Schein, J. Moran. and L. G. Ungerleider, 1985. Contour, color and shape analysis beyond the striate cortex, Vis. Res., 25:441–452. Duncan, R. O., T. D. Albright, and G. R. Stoner, 2000. Occlusion and the interpretation of visual motion: perceptual and neuronal effects of context, J. Neurosci., 20:5885–5897. Egeth, H. E., and S. Yantis, 1997. Visual attention: control, representation, and time course, Annu. Rev. Psychol., 48:269–297. Felleman, D. J., and D. C. Van Essen, 1991. Distributed hierarchical processing in the primate cerebral cortex, Cereb. Cortex., 1:1–47. Finkel, L. H., and P. Sajda, 1992. Object discrimination based on depth-from-occlusion, Neural Comput., 4:901–921. Grosof, D. H., R. M. Shapley, and M. J. Hawken, 1993. Macaque-V1 neurons can signal illusory contours, Nature, 365: 550–552. Grossberg, S., and E. Mingolla, 1985. Neural dynamics of form perception: boundary completion, illusory figures, and neon color spreading, Psychol. Rev., 92:173–211. He, Z. J., and K. Nakayama, 1992. Surfaces versus features in visual search, Nature, 359:231–233. He, Z. J., and T. L. Ooi, 2000. Perceiving binocular depth with reference to a common surface, Perception, 29:1313–1334.

Heitger, F., R. von der Heydt, E. Peterhans, L. Rosenthaler, and O. Kübler, 1998. Simulation of neural contour mechanisms: representing anomalous contours, Image Vis. Comput., 16:409–423. Helmholtz, H. V., 1866. Handbuch der physiologischen Optik, Hamburg: Voss. Julesz, B., 1971. Foundations of Cyclopean Perception, Chicago: University of Chicago Press. Kanizsa. G., 1979. Organization in Vision. Essays on Gestalt Perception, New York: Praeger. Koffka, K., 1935. Principles of Gestalt Psychology, New York: Harcourt, Brace and World. Lamme, V. A. F., 1995. The neurophysiology of figure-ground segregation in primary visual cortex, J. Neurosci., 15:1605– 1615. Lee, T. S., D. Mumford, R. Romero, and V. A. F. Lamme, 1998. The role of the primary visual cortex in higher level vision, Vis. Res., 38:2429–2454. Lee, T. S., and M. Nguyen, 2001. Dynamics of subjective contour formation in the early visual cortex, Proc. Natl. Acad. Sci. USA, 98:1907–1911. Leopold, D. A., and N. K. Logothetis, 1996. Activity changes in early visual cortex reflect monkeys percepts during binocular rivalry, Nature, 379:549–553. Levelt, W. J. M., 1968. On Binocular Rivalry, The Hague: Mouton. Leventhal, A. G., K. G. Thompson, D. Liu, Y. Zhou, and S. J. Ault, 1995. Concomitant sensitivity to orientation, direction, and color of cells in layers 2, 3, and 4 of monkey striate cortex, J. Neurosci., 15:1808–1818. Logothetis, N. K., and J. D. Schall, 1989. Neuronal correlates of subjective visual perception, Science, 245:761–763. Marr, D., 1982. Vision. A Computational Investigation into the Human Representation and Processing of Visual Information, San Francisco: Freeman. Movshon, J. A., E. H. Adelson, M. S., Gizzi, and W. T. Newsome, 1985. The analysis of moving patterns, Exp. Brain. Res. Suppl., 11:117–151. Nakayama, K., Z. J. He, and S. Shimojo, 1995. Visual surface representation: a critical link between lower-level and higher-level vision, in Invitation to Cognitive Science (S. M. Kosslyn and D. N. Osherson, eds.), Cambridge, MA: MIT Press, pp. 1–70. Nieder, A., 2002. Seeing more than meets the eye: processing of illusory contours in animals, J. Comp. Physiol. [A], 188:249– 260. Nieder, A., and H. Wagner, 1999. Perception and neuronal coding of subjective contours in the owl, Nat. Neurosci., 2:660–663. Paradiso, M. A., S. Shimojo, and K. Nakayama, 1989. Subjective contours, tilt aftereffects, and visual cortical organization, Vis. Res., 29:1205–1213. Peterhans, E., and R. von der Heydt, 1989. Mechanisms of contour perception in monkey visual cortex. II. Contours bridging gaps, J. Neurosci., 9:1749–1763. Peterhans, E., R. von der Heydt, and G. Baumgartner, 1986. Neuronal responses to illusory contour stimuli reveal stages of visual cortical processing, in Visual Neuroscience ( J. D. Pettigrew, K. J. Sanderson, and W. R. Levick, eds.), Cambridge: Cambridge University Press, pp. 343–351. Poggio, G. F., 1995. Mechanisms of stereopsis in monkey visual cortex, Cereb. Cortex, 3:193–204. Poggio, T., and C. Koch, 1985. Ill-posed problems in early vision: from computational theory to analogue networks, Proc. R. Soc. Lond. B, 226:303–323. Qiu, F. T., T. J. Macuda, and R. von der Heydt, 2003. Neural correlates for Gestalt rules in figure-ground organization. In press.

  :       

1149

Ramsden, B. M., C. P. Hung, and A. W. Roe, 2001. Real and illusory contour processing in area V1 of the primate: a cortical balancing act, Cereb. Cortex, 11:648–665. Redies, C., J. M. Crook, and O. D. Creutzfeldt, 1986. Neuronal responses to borders with and without luminance gradients in cat visual cortex and dorsal lateral geniculate nucleus, Exp. Brain Res., 61:469–481. Rensink, R. A., J. K. O’Regan, and J. J. Clark, 1997. To see or not to see: the need for attention to perceive changes in scenes, Psychol. Sci., 8:368–373. Ringach, D. L., and R. Shapley, 1996. Spatial and temporal properties of illusory contours and amodal boundary completion, Vis. Res., 36:3037–3050. Rubin, E., 1921. Visuell wahrgenommene Figuren, Copenhagen: Gyldendal. Schein, S. J., and R. Desimone, 1990. Spectral properties of V4 neurons in the macaque, J. Neurosci., 10:3369–3389. Sheth, B. R., J. Sharma, S. C. Rao, and M. Sur, 1996. Orientation maps of subjective contours in visual cortex, Science, 274: 2110–2115. Soriano, M., L. Spillmann, and M. Bach, 1996. The abutting grating illusion, Vis. Res., 36:109–116. Srinivasan, M., M. Lehrer, and R. Wehner, 1987. Bees perceive illusory contours induced by movement, Vis. Res., 27:1285– 1290.

1150

, ,   

Sugita, Y., 1999. Graping of image fragments in primary visual cortex, Nature, 401:269–272. Ullman, S., 1996. High-level Vision, Cambridge, MA: MIT Press. von der Heydt, R., and E. Peterhans, 1989. Mechanisms of contour perception in monkey visual cortex. I. Lines of pattern discontinuity, J. Neurosci., 9:1731–1748. von der Heydt, R., E. Peterhans, and G. Baumgartner, 1984. Illusory contours and cortical neuron responses, Science, 224: 1260–1262. von der Heydt, R., H. Zhou, and H. S. Friedman, 2000. Representation of stereoscopic edges in monkey visual cortex, Vis. Res., 40:1955–1967. Waltz, D. I., 1975. Understanding line drawings of scenes with shadows, in The Psychology of Computer Vision (P. H. Winston ed.), New York: McGraw-Hill, pp. 19–91. Westheimer, G., and W. Li, 1996. Classifying illusory contours by means of orientation discrimination, J. Neurophysiol., 75:523–528. Zeki, S. M., 1978. Uniformity and diversity of structure and function in rhesus monkey prestriate visual cortex, J. Physiol. (Lond.), 277:273–290. Zhou, H., H. S. Friedman, and R. von der Heydt, 2000. Coding of border ownership in monkey visual cortex, J. Neurosci., 20: 6594–6611. Zipser, K., V. A. F. Lamme, and P. H. Schiller, 1996. Contextual modulation in primary visual cortex, J. Neurosci., 16:7376–7389.

77

Inferotemporal Response Properties KEIJI TANAKA

A TE    cortex represents the final purely visual stage of the occipitotemporal (or ventral visual) pathway (Fig. 77.1). The occipitotemporal pathway starts at the primary visual cortex (V1) and leads to TE after relays at V2, V4, and TEO. Although skipping projections also exist, such as those from V2 to TEO and those from V4 to the posterior part of TE, the step-by-step projections are more numerous. TE, in turn, projects to various polymodal brain sites, including the perirhinal cortex, prefrontal cortex, amygdala, and striatum of the basal ganglia. The projections to these targets are more numerous from TE, particularly from the anterior part of TE, than from areas at earlier stages. Therefore, there is a sequential cortical pathway from V1 to TE, and output from the pathway originates mainly from TE. In monkeys, bilateral TE ablation or their complete deafferentation by bilateral ablation of more than one stage in the occipitotemporal pathway resulted in severe and selective deficits in tasks that required visual discrimination or recognition of objects (Dean, 1976; Gross, 1973; Yaginuma et al., 1993). Thus, the occipitotemporal pathway is essential for object vision, and because TE is the final common stage of the pathway, it is expected that the mechanisms underlying flexible properties of primate object vision can be found in properties of neuronal responses in TE. In this chapter, I will discuss the properties of neuronal responses and the functional architecture of TE, the effects of learning in the adult on the response properties and functional architecture, and their relationship to object vision.

Moderately complex features There is a principal difficulty in determining the stimulus selectivity of individual cells in TE. There is a great variety of object features in the world, and it remains to be determined how the brain scales down this variety. There have been studies that used mathematically perfect sets of shapes (Gallant et al., 1993, 1996; Richmond et al., 1987; Schwartz et al., 1983). However, the generality of these sets would hold only if the system were linear, which is hardly expected in higher visual centers. We have used an empirical reduction method that involves the real-time modification of stimulus images on an imageprocessing computer system (Fujita et al., 1992; Ito et al., 1994, 1995; Kobatake and Tanaka, 1994; Tanaka et al.,

1991; Wang et al., 1998). After spike activities from a single cell were isolated, many three-dimensional animal and plant models were first presented manually to find the effective stimuli. Different aspects of the objects were presented in different orientations. Second, images of several most effective stimuli were taken with a video camera and displayed on a TV monitor by a computer to determine the stimulus that evoked the maximal response. Third, the image of the most effective stimulus was simplified step by step in the direction in which the maximal activation was maintained. Finally, the minimal requirement for maximal activation was determined as the critical feature for the cell, as exemplified in Figure 77.2. Even starting at the same object image, the effective direction of simplification varied from cell to cell. Thus, images used in the simplification procedure were made in real time while the activity of the cell was recorded. The procedure is time-consuming, and it usually takes 2 to 4 hours to determine the critical feature for one TE cell. The magnitude of responses often increased as the complexity of an image was reduced. This may be due to the adjustment of size, orientation, and shape, as well as the removal of other features, which may suppress the activation by the critical feature (Missal et al., 1997; Sato, 1989, 1995; Tsunoda et al., 2001). Additional examples of the reduction of complexity of images for 12 other TE cells are shown in Figure 77.3. The pictures to the left of the arrows are the original images of the most effective object stimuli, and those to the right are the critical features determined after the reduction process. Some of the critical features are moderately complex shapes, while others are combinations of such shapes with color or texture. After determining the critical features for hundreds of cells in TE, we concluded that most cells in TE required moderately complex features for their maximal activation. The critical features for TE cells were more complex than just the orientation, size, color, or simple textures, which are known to be extracted and represented by cells in V1, but at the same time were not sufficiently complex to represent the image of a natural object through the activity of single cells. The combined activation of multiple cells, which represent different features contained in the object image, is necessary. Although the reduction method appears to be the best among currently available methods of determining the stimulus selectivity of TE cells, it has limitations. The initial

1151

F 77.1. Occipitotemporal pathway (ventral visual pathway). Lateral view (top) and bottom view (bottom) of the monkey brain. The shadow indicates the extent of TE.

F 77.3. Further examples of reductive determination of optimal features for 12 other TE cells. The images to the left of the arrows represent the original images of the most effective object stimulus, and those to the right of the arrows represent the critical features determined by the reduction process. (See color plate 54).

F 77.2. Example of reductive determination of optimal features for a cell recorded in TE.

1152

, ,   

survey of effective stimuli cannot cover the entire variety of objects existing in the world. We may miss some very effective features. In addition, the tested methods of reducing the complexity of effective object images are limited by the time of continuous recording from a single cell and also by the imagination of the experimenter. Because of these limitations, the objectiveness of the determined optimal features has sometimes been doubted. It is desirable that the reduction procedure be automated. Pasupathy and Connor (2001) have developed a method of presenting a large number of shapes made by combining several arcs of different curvatures. They have shown the usefulness of this method in studying the selectivity of V4 cells, but it may not be useful for TE cells, which respond to more complicated shapes than do V4 cells. Keysers et al. (2001) developed a method of analyzing responses to more than 1000 stimulus images in a fixation task. The stimulus images were presented individually for a short time (e.g., 100 msec) without an interstimulus

interval. The following stimulus presented may inhibit the response to the previous stimulus, but because the order of stimulus presentation is randomized and because TE cells tend to respond to a small part of the stimuli, there are no inhibitory interactions in the majority of repetitions. These two methods may be combined to explore systematically a large feature space of complex shapes—sufficiently complex for the activation of most TE cells.

Faces and other extensively learned objects Although the critical features for the activation of TE cells are only moderately complex in general, there are cells that respond to faces and critically require nearly all the essential features of the face. Such cells were originally found deep in the superior temporal sulcus (Bruce et al., 1981; Perrett et al., 1982), but they were also found in TE (Baylis et al., 1987). Thus, there is more convergence of information to single cells for representations of faces than for those of nonface objects. This difference may be due to the fact that discrimination of faces from other objects is not the final goal, and further processing of facial images is needed to discriminate among individuals and expressions, while distinguishing a nonface object from other objects is close to the final goal. Optical imaging experiments showed that there is a cluster of such cells responding selectively to faces in a small region of TE (Wang et al., 1996, 1998). There are suggestions that responses to whole objects will develop in TE if the subject is extensively trained in fine discrimination of similar objects. Logothetis et al. (1995) trained adult monkeys to recognize wire-frame objects against many other similar wire-frame objects, and recorded from cells in TE of the monkeys during the same task. About 20% of cells responded to wire-frame objects more strongly than to any other tested objects. Some of them responded to some parts of the objects as well as to the entire image of the objects, while others did not respond to parts of the objects (Logothetis, 1998). Based on these results, Logothetis (1998) proposed that some TE cells respond to whole objects with which the subjects have conducted fine discriminations, while a majority of TE cells respond to features present in images of multiple different objects. However, this remains to be studied further, because the examination of selectivity (described in Logothetis, 1998) for object parts was rather preliminary. There are rare cases of brain-damaged patients whose capability of making fine discrimination among faces is specifically deteriorated (prosopagnosia). Their deficiency is specific in that they can normally discriminate a nonface object from other objects. They can also distinguish faces from other objects. Brain imaging studies showed that a region in the fusiform gyrus of normal human subjects is

more activated by faces than by other objects ( fusiform face region) (Allison et al., 1994; Kanwisher et al., 1997; Puce et al., 1995). The damage of the prosopagnosic patients extends over the ventral surface of the occipitotemporal lobes. It includes the fusiform face region, although the damaged region is usually much larger. The fusiform face region may correspond to the small region of the monkey TE in which cells responding selectively to faces cluster. However, there is no consensus about the degree of stimulus selectivity of the fusiform face region (Haxby et al., 2001; Ishai et al., 1999; Kanwisher, 2000; Tarr and Gauthier, 2000), and its extent, as determined in previous studies, was larger (>5 mm) than the monkey face region (~1.5 mm). The nature of the fusiform face region should be studied further before we determine its relation with cells in the monkey TE responding selectively to faces.

Depth structure of object surfaces Some of the critical features for the activation of TE cells include the gradient of luminosity (e.g., top right in Fig. 77.2). The gradient of luminosity often provides depth structure of object surfaces with an assumption of the direction of illumination. In this sense, the features represented by TE cells are not necessarily purely two-dimensional (2D). These features may be described in 2D space but reflect depth structures. Moreover, recent studies have found that some TE cells respond selectively to the horizontal disparity in addition to the 2D shape of stimuli. The horizontal disparity between images projected to the left and right eyes is a strong cue for perception of depth. Although it was once assumed that the selectivity for disparity is more predominant in the occipitoparietal (or dorsal visual) pathway, which is responsible for visuomotor control or spatial vision, than in the occipitotemporal (or ventral visual) pathway, recent studies have shown that many cells in TE are selective to the disparity of stimuli, as well as to their 2D shapes in the frontoparallel plane. Uka et al. (2000) recorded activity of TE cells in monkeys performing a fixation task and examined their responses to 2D shape stimuli presented at different depths. The depth was defined relative to that of the fixation point, as in other such experiments. They used 11 2D shapes, and cells that responded to at least one of them at zero disparity were examined for disparity selectivity. Responses of more than one-half (63%) of the cells showed statistically significant dependence on depth. Most of the disparity-selective cells were either near or far neurons according to the classification of Poggio and Fisher (1977). This is in contrast to the primary visual cortex and area MT, in which the tuned excitatory cells constitute a large part (2/3 in V1 and 2/5 in MT) of the disparity-selective cells (Maunsell and Van Essen, 1983; Poggio and Fisher, 1977).

:   

1153

The stimuli used by Uka et al. (2000) were flat in the depth direction, that is, there were no depth structures within their contours. Many objects in nature have surfaces tilted or curved in the depth direction, and such a depth gradient of the surface is an important feature of the object image. Janssen et al. (1999, 2000a, 2000b) used a stimulus set composed of stimuli having several different depth profiles in combination with several different 2D shapes. About onehalf of the cells recorded from the ventral bank of the anterior part of the superior temporal sulcus exhibited selectivity for depth profile. Some of them responded to a linear gradient of depth, some to a combination of opposite linear gradients (or wedge profile), and others to a smooth concave or a convex depth curvature. These cells were selective for both 2D shape and depth profile. The selectivity for the depth profile was not explained by the selectivity for the depth position of a particular part of the stimulus, because the stimuli of the opposite depth profile did not activate the cells at any depth. The proportion of such cells was much lower in the ventrolateral surface (i.e., area TE) (about 10%) than in the ventral bank of the superior temporal sulcus. Based on the cytoarchitectural criteria, the ventral bank of the anterior part of the superior temporal sulcus (TEa and TEm) was distinguished from the ventrolateral surface (TE). However, H. Tanaka and I. Fujita (personal communication) found that cells in the ventral bank were as selective for complex 2D shapes as were cells in TE. Moreover, the cells in the ventral bank were much more sensitive to the direction of the disparity gradient or curvature (e.g., concave versus convex), than to the quantitative values of curvature or gradient ( Janssen et al., 2000b). Therefore, responses of cells in the ventral bank of the superior temporal sulcus do not represent a full reconstruction of the three-dimensional (3D) structure of the objects. Rather, it may be the case that the representation in this area is still mainly 2D, and the qualitative information of disparity gradient or curvature simply renders the 2D representation richer. I  R Our ability to recognize objects is retained even in the event of many different kinds of translation of the objects in space. These invariances can, in part, be explained by invariant properties of single-cell responses in TE. Using a set of shape stimuli composed of individually determined critical features and several other shape stimuli obtained by modifying the critical features, we have observed that the selectivity for shape is preserved over the receptive fields (Ito et al., 1995), which usually range from 10 to 30 degrees in a one-dimensional (1D) size. However, the maximum response is usually obtained around the geometrical center of the receptive field, and the magnitude of response decreases toward the edges of the receptive field (Ito et al., 1995; Op de Beeck and Vogels, 2000). The center of receptive fields is scattered around the fovea (Kobatake

1154

, ,   

and Tanaka, 1994; Op de Beeck and Vogels, 2000). Therefore, responses of individual TE cells carry coarse information on the position of stimuli as well as detailed information on their shape, color, and texture. The effects of changes in stimulus size varied among cells (Ito et al., 1995; Tanaka et al., 1991). Twenty-one percent of the TE cells tested responded to a size range of more than 4 octaves of the critical features with more than 50% maximum responses, whereas 43% responded to a size range of less than 2 octaves. TE cells with considerable invariance for the location and size of stimuli were also found by Lueschow et al. (1994) and Logothetis et al. (1995). The tuned cells may be immature, and those with various optimal sizes may converge to yield the size-invariant responses. Alternatively, both size-dependent and -independent processing of images may occur in TE. A definite number of TE cells tolerated reversal of the contrast polarity of the shapes. Contrast reversal of the critical feature evoked more than 50% of the maximum responses in 40% of tested cells (Ito et al., 1994). Sary et al. (1993) found that some TE cells responded similarly to shapes defined by differences in luminosity, direction of motion of texture components, and coarseness of texture while maintaining their selectivity for shape. Tanaka et al. (2001) found that about one-fourth of TE cells responded similarly to shapes defined by difference in horizontal disparity of texture components, to those defined by difference in the size of texture components, and to those defined by difference in luminosity. Another kind of invariance of TE cells was found with regard to the aspect ratio of shapes. The aspect ratio is the ratio of the size along one axis of the stimulus to that along the orthogonal axis. When an object rotates in depth, the features contained in the image change their shapes. Unless occlusion occurs, changes occur mainly in the aspect ratio. For individual TE cells, we first determined the critical feature using the reduction method and then tested the effects of changes in the aspect ratio of the critical feature. We observed that one-half of cells responded to an aspect ratio range of more than 3 octaves with more than 50% of the maximum responses (Esteky and Tanaka, 1998). In Figures 77.2 and 77.3 (and in our previous studies), we show the features determined to be critical for the activation of individual TE cells as 2D images. However, this is for the sake of description, and it does not necessarily mean that the cells were tuned to 2D images. Selectivity can only be defined as a list of tested stimulus deformations and their associated response reductions. The above-described invariances of TE cells suggest that they are actually more sensitive to certain types of deformation than others. The types of deformation that often occur when an object moves around appear to be better tolerated.

Columnar organization in TE We examined the spatial distribution of the cells responding to various critical features in TE. By recording two TE cells simultaneously with a single electrode, we found that cells located close together in the cortex had similar stimulus selectivities (Fujita et al., 1992). The critical feature of one isolated cell was determined using the procedure described above, while the responses of another isolated cell, or nonisolated multiunits, were simultaneously recorded. In most cases, the second cell responded to the optimal and suboptimal stimuli of the first cell. The selectivities of the two cells differed slightly, however, in that the maximal response was evoked by slightly different stimuli, or the mode of the decrease in response differed when the stimulus was changed from the optimal stimulus. To determine the spatial extent of the clustering of cells with similar selectivities, we examined the responses of cells recorded successively along long penetrations that were made vertical or oblique to the cortical surface (Fujita et al., 1992). The critical feature for a cell located in the middle of the penetration was first determined. A set of stimuli, including the critical feature for the first cell, its rotated versions, and ineffective control stimuli, was constructed, and cells recorded at different positions along the penetration were tested with the fixed set of stimuli. Cells recorded along the vertical penetrations commonly responded to the critical feature for the first cell or to some related stimuli. Such clusters of cells with similar stimulus selectivity covered nearly the entire thickness from layer 2 to layer 6. In the case of penetrations that were made oblique to the cortical surface, however, the cells that commonly responded to the critical feature of the first cell or to related stimuli were limited to within a short span around the first cell. The horizontal extent of the span was on average, 400 mm. Cells outside the span did not respond to any of the stimuli included in the set, or they responded to some stimuli that were not effective in activating the first cell and were included in the set as ineffective control stimuli. Based on these results, we proposed that TE is composed of columnar modules, cells in each of which respond to similar features (Fig. 77.4). It should be noted that the precise determination of the optimal features is essential to observing the similarity of stimulus selectivities between neighboring cells clustered in a columnar region. Several studies, which used a fixed set of arbitrarily selected object images, failed to find the similarity. The optimal features for the activation of TE cells are complex and defined by many dimensions. The preference of cells within a column is similar in some dimensions but different in other dimensions. For example, cells in a column respond to star-like shapes or shapes with multiple protrusions. They are similar in that they respond to star-like shapes, but they may differ in the preferred number of pro-

F 77.4.

Schematic of the columnar organization in TE.

trusions or the amplitude of the protrusions. Therefore, if only a fixed set of object images is used, because star-like shapes with different numbers of protrusions appear in different objects, cells within the column will respond to different objects. The same is true for the primary visual cortex. Cells within an orientation column have the same preferred orientation, while they differ in the preferred width and length of stimuli, binocular disparity, and the sign of contrast. If a set of stimuli that vary not only in orientation but also in all other parameters is used, cells within an orientation column will not show clear similarity in selectivity.

Spatial arrangement of columns To further study the spatial properties of the columnar organization in TE, we used optical imaging with intrinsic signals (Wang et al., 1996, 1998). The cortical surface was exposed and illuminated with red light tuned to 605 nm, and the reflected light image was recorded by a CCD video camera. The reflected images for different visual stimuli were compared. The region of the cortex with elevated neuronal activities appears darker than other regions in the reflected image. We first recorded the responses of single cells with a microelectrode to determine the critical feature and then conducted optical imaging. In the experiment, the results of which are shown in Figure 77.5, the critical feature determined for a cell recorded at the cortical site indicated by a cross was the combination of white and black horizontal bars. The peri stimulus time (PST) histograms on the left side represent the responses of the cell. The combination evoked a strong response in the cell, but a white bar alone or a black bar alone did not activate the cell. The images on the right side were taken from the same 1 ¥ 1.5 mm cortical region. A dark spot appeared around the penetration site when the monkey saw the combination of the two bars, whereas there

:   

1155

F 77.5. Correspondence of optical signals with neuronal activity. The histograms on the left show the responses of a cell recorded at the site indicated by the crosses in the optical images. The cell responded selectively to the combination of a white bar and a black bar. The white bar alone or the black bar alone evoked much smaller responses. Correspondingly, the optical image showed a black spot covering the recording site during the time the monkey was viewing the combination shape, whereas there were no dark spots for the control stimuli. (From Wang et al., 1996, with modification.)

F 77.6. Map of activation evoked by the presentation of eight moderately complex features. To obtain the map, each image was subtracted by the reference image averaged over the images obtained for all the different stimuli combined in the experiment to remove the global darkening. The activation spots were delineated at 1/e of the maximum intensity in individual images, and the contours of spots in the images for different stimuli are indicated by different line types. (From Wang et al., 1998, with modification.)

were no dark spots around the site when the monkey saw the simpler features. Similar results were obtained in 11 out of 13 cases. Tsunoda et al. (2001) further confirmed the correlation of optical signals with neuronal responses in TE. Although the critical feature was determined for a single cell, a large proportion of cells in the region must be activated to produce an observable metabolic change. Therefore, the localized and specific occurrence of dark spots indicates a regional clustering of cells with similar stimulus selectivities. However, when we observed a larger area of the cortical surface, we found that the presentation of a single feature activated multiple spots. In Figure 77.6, the spots activated by eight moderately complex features are indicated by different kinds of lines and superimposed, that is, spots activated by four features are shown in the upper half and those by the other four features in the lower half. For example, feature 1 evoked six spots, and feature 2 evoked two spots. A single feature is processed in multiple columns in TE.

Another interesting observation here is the partial overlapping between activation spots evoked by different features. Some of the overlapping regions, which were activated by many stimuli, likely represent columns of nonselective cells. However, others that were activated by only two of the stimuli may represent overlapping between selective columns. For many of these overlapping regions, we can find similarity between the two features, although the judgment of similarity is only subjective. The partial overlapping of columns responding to different but related features was most clearly observed for faces presented in different views (Fig. 77.7). This experiment was performed using optical imaging guided by unit recording. We recorded five cells in one electrode penetration around the center of the imaged region, and all of them responded selectively to faces. Three of them responded maximally to the front view of the face, and the remaining two responded to the profile, that is, the lateral view of the face. In an optical imaging session, five different views of the face of the same

1156

, ,   

F 77.7. Systematic movement of the activation spot with rotation of the face. The images were obtained for five different views of the face of the same doll shown on top. The reference image obtained by averaging the five images was subtracted. The contours circumscribing the pixels with t-values at p < .05, compared with the reference image, are superimposed at the bottom. (From Wang et al., 1996, with modification.)

doll were presented in combination with 14 nonface features. All of the faces evoked activation spots around the center of the illustrated 3 ¥ 3 mm region. However, their central positions were slightly different. The contours of the dark spots are superimposed at the bottom of Figure 77.7. The activation spot moved in one direction as the face was rotated from the left to the right profile through the front view of the face. Individual spots were 0.4 to 0.8 mm in diameter, and the overall region was 1.5 mm. These regions were not activated by the 14 nonface features. Similar results, namely, selective activation by faces and systematic shift of the activation spot with the rotation of the face, were obtained for three other monkeys. In these three monkeys, optical imaging was not guided by unit recording. The recording chamber with an inner diameter of 18 mm was placed in the same region of TE, and the face-selective activation was found at approximately the same location (approximately the posterior third of TE on the lateral surface close to the lip of the superior temporal sulcus). The effects of rotating the face around a different axis (chin up and down) and of changing the facial expression were also determined in some of the experiments, but neither of these caused a shift in the activation spot. Only two faces were tested: a human face and a doll’s face. The two faces activated regions that mostly overlapped. There are two possible interpretations of this result. One is that the variations other than those with horizontal rotation are represented at different sites not covered by the recording chamber in the

experiments. Alternatively, it is possible that only the variations along the horizontal rotation are explicitly mapped along the cortical surface as the first principal component, and other variations are embedded in overlapping cell populations. Data for the nonface features are few, but I hypothesize that there are similar structures, and I propose a modified model of the columnar organization of neurons in TE as shown in Figure 77.8. The borders between neighboring columns are not necessarily distinct. Instead, multiple columns that represent different but related features partially overlap with one another and as a whole compose a larger scale unit. At least in some cases, some parameter of the features is continuously mapped along the cortical surface. This systematic arrangement of related columns can be used for various computations necessary for object recognition. One simple possible computation is the generalization of activation by the horizontal excitatory connections to nearby columns representing related features. We may call it the selective blurring of activation. The blurring is selective in that the spread of activation in TE results in activation of related features but not blurring in the image plane. Another possible simple processing is the mutual inhibition among nearby columns for the winner-take-all-type selection. The continuous mapping of different views of faces cannot be generalized to nonface objects. Because the critical features for TE cells are only moderately complex except for faces, the image of a nonface object has to be represented

:   

1157

F 77.8. TE.

Revised schematic of the columnar organization in

by a combination of activations at multiple cortical sites. Rotation of a nonface object causes changes in activation at multiple cortical sites, each of which corresponds to a partial change of a feature. The mechanisms underlying the viewinvariant representation of nonface objects should be more widely distributed over a large region of TE than those for faces.

From features to objects Since most inferotemporal cells represent features of object images but not the whole-object images, the representation of the image of an object requires a combination of multiple cells representing different features contained in the image of the object. This process of combination provides unique scientific problems. Objects often appear in a clutter. A part of features belonging to one object may be mistakenly combined with a part of features belonging to another object. This erroneous combination causes a false perception of an object that is not visually present. How does the brain avoid such an erroneous combination? Previously, the synchronization of spiking activity between cells was proposed as the mechanism for binding the features belonging to one object. Some experiments found a correspondence between the synchronization and object borders (reviewed by Singer, 1999), while others did not (Lamme and Spekreijse, 1998). Another possible means of avoiding the erroneous combination is to have features partially overlapping with one another (Mel and Fiser, 2000). Suppose we are to represent four-letter strings. There will be an erroneous combination if we use only representation units coding single letters (e.g., ABCD is not discriminated from BADC, CDAM, and so on, if units code A, B, and C), while there will be no erroneous combinations if we use units specifying two consecutive letters and those specifying letters at the

1158

, ,   

beginning and end of three consecutive letters (e.g., ABCD is the only four-letter string that contains AB, CD, and AC). Tsunoda et al. (2001) compared activation of the inferotemporal cortex by object images and that by features included in the object images using a combination of optical imaging and single-cell recordings. The image of an object usually activated several spots within the imaged region (6 ¥ 8 mm), and a feature contained in the object image activated part of the spots in some cases. This result was consistent with the idea that different spots were activated by different features contained in the object image. However, in other cases, the activation by a feature often included new spots that had not been activated by the whole-object image. Single-cell recordings revealed that cells within such spots were activated by a feature while inhibited by another feature included in the original object image. Previous single-cell recording studies had also shown that the response of inferotemporal cells to the optimal stimulus was suppressed by the simultaneous presentation of a second stimulus (Missal et al., 1997, 1999; Sato, 1989, 1995). These results indicate that the stimulus selectivity of inferotemporal columns should be described by both the simplest feature for the maximum activation and features that suppress the activation. Even with the same optimal feature for the excitation, the range of features that suppresses the excitation can vary from column to column and probably also from cell to cell. This complexity of the overall stimulus selectivity of inferotemporal columns and cells may help to reduce the chance of erroneous detection of nonexisting objects. Yamane et al. (2001) also used a combination of optical imaging and single-cell recordings, and found that some of the columns activated by an object image were activated not by local features, but by a global feature of the object image. These columns were more sensitive to the global arrangement of object parts than to the properties of the parts. For example, one column responded to two vertically aligned black parts, regardless of the shape of either part. These columns representing global features also help reduce the possibility of erroneous detection of nonexisting objects.

Intrinsic horizontal connections within TE Intrinsic horizontal connections span up to 8 mm in TE. Projection terminals are more or less continuously distributed within 1 mm from cells of origin, whereas they are clustered in patches in more distant regions (Fujita and Fujita, 1996; Tanigawa et al., 1998). The cells of origin of these horizontal connections contain inhibitory neurons within 1 mm, but they are composed exclusively of excitatory cells (mostly pyramidal cells) for longer connections (Tanigawa et al., 1998). Iontophoretic injection of bicuculline methiodide, an antagonist of an inhibitory synaptic transmitter GABA,

reduced the stimulus selectivity of TE cells; in particular, the stimuli optimal for nearby cells turned out to evoke excitatory responses during the blockage of inhibition (Wang et al., 2000). Inhibitory components of horizontal connections contribute to the formation of stimulus selectivity. The functional roles of excitatory components are not known. It is possible that they connect columns responding to similar features, as is the case in the primary visual cortex (Gilbert and Wiesel, 1989). The combination of optical imaging and anatomical tracing will provide insights into this issue.

Functions of TE columns Representation by multiple cells in a columnar module, in which the precise selectivity varies from cell to cell while effective stimuli largely overlap, can satisfy two apparently conflicting requirements in visual recognition: one is the ability to disregard subtle changes in input images; the other is the preciseness of representation. A cluster of cells having overlapping and slightly different selectivities may work as a buffer to absorb changes. Although single cells in TE tolerate some changes in size, contrast polarity, and aspect ratio, these invariant properties at the single-cell level are not sufficient to explain the entire range of flexibility of object recognition. In particular, responses of TE cells are generally selective for the orientation of the shape in the frontoparallel plane. Cells preferring different orientations and other parameters of the same three-dimensional shape may be packed in a column to provide invariant output. Whether signals from these selective cells converge to a group of single cells exhibiting invariant responses is a matter for further investigation. One possibility is that output of cells preferring different orientations, sizes, aspect ratios, and contrast polarities of the same shape overlaps in the target structure, thereby evoking the same effects. One of our anatomical studies involving injection of an anterograde tracer into a focal site in TE suggested that projections from TE to the ventrocaudal striatum of the basal ganglia exhibit this property (Cheng et al., 1997). Another possibility is that activation of cells may be transmitted to other cells within a column and to nearby columns that represent related features through horizontal excitatory connections in the presence of top-down signals from other brain sites, for example, the prefrontal cortex. The representation by multiple cells with overlapping selectivities can be more precise than a mere summation of representations by individual cells. A subtle change in a particular feature, which does not markedly change the activity of individual cells, can be coded by differences in activities of cells with overlapping and slightly different selectivities. Projections from the ventroanterior part of TE to the perirhinal cortex extensively diverge (Saleem and Tanaka, 1996). Projection terminals from a single site of the ven-

F 77.9. The 28 shapes used for the training (top) and the paradigm (bottom).

troanterior TE cover about one-half of the perirhinal cortex. This divergence in projections may distribute subtle differences in images over a larger area of the perirhinal cortex so that objects recognized at individual levels can be distinctively associated with other kinds of information. Subtle differences can also be emphasized by mutual inhibition between cells or nearby columns for winner-take-alltype selection. The inhibition may also be under top-down control.

Changes of selectivity in the adult The selectivity of inferotemporal cells can be changed in adult animals by long-term training. We found this by training two adult monkeys to recognize 28 moderately complex shapes shown in the upper half of Figure 77.9 and recording from inferotemporal cells after the training (Kobatake et al., 1998). The training paradigm was a kind of delayed matching to sample shown in the lower half of Figure 77.9. One stimulus that was randomly selected from the set of 28 stimuli appeared on a television monitor as a sample, the monkey touched it, and the sample disappeared. After a delay period, the sample appeared again on the display, but this time together with four other stimuli that were randomly selected from the stimulus set. The monkey selected the sample and touched it to get a drop of juice as a reward. After an intertrial interval, the trial was repeated with a different sample. The monkey performed the task on a

:   

1159

F 77.10. Responses of one TE cell to the image of the most effective stimulus of the object stimulus set (top) and its responses to the 28 shape stimuli used for the training. This cell was recorded from a monkey that had been trained with the 28 stimuli. Statistically significant responses (p < .05) are labeled with their relative response magnitudes. (From Kobatake et al., 1998, with modification.)

stand-alone apparatus placed in front of the home cage. The monkey came to the apparatus and practiced the task whenever it wanted. The training was performed starting with a 1 second delay, and the delay was increased gradually to 16 seconds. At the end of the training, the monkey performed 500 successful trials with greater than 80% performance. After the training was completed, we prepared the monkeys for repeated recordings and conducted recordings from TE under anesthesia, once a week for 3 to 4 months. The training was continued on the days when the recordings were not conducted. We determined the most effective

1160

, ,   

stimulus for individual cells from the same set of animal and plant models that we used for the reduction experiment, and the response to this best object stimulus was compared with the responses of the same cell to the training stimuli. In this experiment, we did not conduct the reduction process, but just took the images of several most effective object stimuli with a video camera and presented them under computer control in combination with the training stimuli. The cell illustrated in Figure 77.10 responded maximally to the sight of a watermelon among the object stimuli. However, the cell responded more strongly to the cross

shape, which was one of the training stimuli. There were also responses to several other training stimuli. Twenty-five percent of responsive cells recorded from the two trained monkeys responded more strongly to some of the training stimuli than to the best object stimulus, as in this example, but only 5% of responsive cells in the control monkeys responded maximally to some of the stimuli. These results indicate that the number of cells maximally responsive to training stimuli increased during the period of the discrimination training. Sakai and Miyashita (1994) and Logothetis et al. (1995) had trained adult monkeys to discriminate among Fourier descriptors or wire-frame objects and found that many TE cells responded to the learned stimuli after the training. A unique contribution of our study is the demonstration that training increases the proportion of TE cells that respond to the learned stimuli as measured against untrained controls. These results appear to be inconsistent with previous findings that responses of cells in TE and the perirhinal cortex decreased as the same stimuli were repeatedly presented. Miller et al. (1991) and Riches et al. (1991), by recording cells in TE and the perirhinal cortex during a serial delayed-matching-to-sample task, found that responses to the second (match) presentation of a stimulus were significantly smaller than those to the first (sample) presentation of the same stimulus in 10% to 30% of visually responsive cells. Miller et al. (1991) and Li et al. (1993) further found that, for stimuli that were new to the monkey at the beginning of the recording of a cell, the decrement of response did not recover from one trial to a sample presentation of the same stimulus in a new trial, even with several tens of intervening trials; thus, the decrement was accumulated through several trials repeatedly presenting the initially new, identical sample stimulus within a day. Riches et al. (1991) and Fahy et al. (1993) found that a definite proportion (about 15%) of TE and perirhinal cells responded to novel stimuli significantly more strongly than to familiar stimuli. This difference was found even when the individual familiar stimuli had not been presented since the previous day. These findings indicate that responses of TE and perirhinal cells decrease as stimuli become familiar, both within a day and across days. The two sets of apparently conflicting findings can be explained as follows. As the familiarity of stimuli increases with repetition of their presentation, a small proportion of TE cells become more responsive to the stimuli, while responses of the remaining majority of TE cells to the same stimuli become weak. In other words, the representation of stimuli in the entire TE changes from a widely distributed one to a sparse one. Although sparse representations are not necessarily superior over distributed representations in discrimination accuracy, sparse representations have advan-

tages in memory capacity and connectional efficiency. Because the studies that found the decrease in the level of responses used a few arbitrarily selected stimuli (fewer than 10) for individual cells or did not pay much attention to the stimulus selectivity of cells, they found only the decrease that occurred in a majority of TE cells. In our study (Kobatake et al., 1998) and the study of Logothetis et al. (1995), only 10% to 25% of TE cells responded more strongly to some of the stimuli used in the training (n = 28 in our study) than to many reference stimuli. It remains to be examined whether the columnar organization in TE changes in adults due to training. It is possible that the columns do not change once they are established in early development after birth. If this is the case, the changes in the selectivity of individual cells, like those observed in the previous studies, should occur in cells distributed across many columns, without emergence of columns specifically tuned to the stimuli used in the task training.

Relationship between task requirements and TE cells’ selectivity It is now established that the responsiveness of TE cells to particular stimuli changes according to long-term training with the stimuli even in adults. However, it is not yet clear whether the selectivity of TE responses reflects the behavioral requirements of the task in which the monkeys have been trained. To answer this question, the task should include something beyond a simple discrimination among stimuli. Sigala and Logothetis (2002) trained monkeys in a classification of many line drawings of faces into two groups. The monkeys reported their classification by pressing either the left or the right bar. Faces were varied by changing four parameters of the face: eye height, eye separation, nose length, and mouth height. The faces were thus distributed in a four-dimensional feature space. The two groups of faces were separated by a linear plane in the feature space, and the plane was defined by only two of the parameters (diagnostic parameters). Sigala and Logothetis (2002) recorded TE cells during the task after the monkeys had completely learned the classification. They found that responses of TE cells were statistically more selective for the two diagnostic parameters than for the other two parameters. The monkeys were also trained to classify line drawings of fishes, and similar results were obtained. These results may suggest that the selectivity of TE cells specifically develops through the task training to make the task requirement easier. In this study, however, the diagnostic parameters were not exchanged with the nondiagnostic parameters between monkeys. Therefore, it is possible that the parameters that were selected as diagnostic in the task happened to be more influential on responses of TE cells than the parameters

:   

1161

selected as nondiagnostic, even without the classification training. Vogels (1999a, 1999b) pioneered the study of the relationship between the stimulus selectivity of TE cells and categorization behavior of monkeys. The monkeys were trained to distinguish trees from other objects and fishes from other objects. The ability of the monkeys to generalize the categorizations to novel trees and novel fishes was demonstrated (Vogels, 1999a). Vogels (1999b), by recording cells from TE after the monkey learned the categorization of trees versus nontrees, found that 10% of TE cells responded exclusively to subpopulations of trees and that none of them responded to all the trees. However, it is difficult to draw a conclusion from the data on whether TE cells as a whole were more sensitive to differences between trees and other objects than to variations within trees, or whether the TE cells responding only to trees emerged as a result of categorization training. Sakai and Miyashita (1991) trained monkeys in a paired associate task. A trial of the task was composed of a sample presentation, delay, and choice periods, as in the delayedmatching-to-sample tasks. However, differences from the conventional delayed-matching-to-sample task were that two stimuli presented in the choice period did not include the sample stimulus, and the monkey had to select the one paired with the sample. Because the pairing was randomly determined by the experimenter, the paired stimuli were not more similar than unpaired stimuli. The pairing was consistent throughout the training, which lasted for several months. Sakai and Miyashita recorded activity of TE cells during the task after the monkeys had learned the associations and found that some TE cells responded selectively to both stimuli composing a single pair. However, Higuchi and Miyashita (1996) found that this trace of associative memory in TE cells depended on the perirhinal cortex, to which TE projects. They first cut the anterior commissure to disconnect the left and right TE from each other. After the monkeys were trained in the paired associate task, the perirhinal cortex in one hemisphere was destroyed by injecting a toxic agent. TE cells on the lesioned side did not show the correlated responsiveness to paired stimuli, although their responses to individual stimuli were intact. These results indicate that the correlated responsiveness was generated in the perirhinal cortex and projected to TE through feedback input from the perirhinal cortex to TE. In summary, it is now established that responsiveness of TE cells in an adult monkey to particular stimuli increases through training with the stimuli, but it is not clear whether the selectivity of individual TE cells reflects the behavioral requirements of the task in which the monkey has been trained. The trace of associative memory in TE found by Sakai and Miyashita (1991) was a reflection of a phenomenon that occurred primarily in the perirhinal cortex.

1162

, ,   

Conclusions Neurons in area TE of the monkey inferotemporal cortex respond to moderately complex visual features of objects. Different TE cells respond to different features. Because the features to which individual TE cells respond are only moderately complex, combinations of several to several tens of cells responding to different features are necessary to specify particular objects. This coding of object images by combinations of features (combination coding) may underlie the generalization capability of our visual perception. Cells with similar, but slightly different, stimulus selectivities cluster in local columnar regions. This clustering with certain variety suggests the presence of two schemes in representation of object images in TE. In one of them, an object image is represented by a combination of activated columns. In this coarse representation, the variety of selectivity within individual columns may work as a tool to disregard subtle changes in input images. The other representation depends on the distribution of activities among the cells within individual columns. Because cells within individual columns have overlapping and slightly different selectivities, the representation can be more precise than a simple summation of activities of the cells. These two types of representation may be working in parallel, with some graded emphasis changing in accordance to the behavioral context. There is some evidence for regularities in arrangement of columns at nearby positions, which may also have functional significance. The stimulus selectivity of TE cells changes in adults as the monkey learns particular visual tasks with object images or complex patterns. These neuronal changes in the adult brain likely underlie the long-term memory of the monkey. By examining the relationship between neuronal changes and requirements of the tasks that the monkey has learned—or, in other words, changes in behavioral performance of the monkey through the learning—we will be able to clarify further the functional roles of the neuronal stimulus selectivity and columnar organization in TE. REFERENCES Allison, T., H. Ginter, G. McCarthy, A. C. Nobre, A. Puce, M. Luby, and D. D. Spencer, 1994. Face recognition in human extrastriate cortex, J. Neurophysiol., 71:821–825. Baylis, G. C., E. T. Rolls, and C. M. Leonard, 1987. Functional subdivisions of the temporal lobe neocortex, J. Neurosci., 7:330–342. Bruce, C., R. Desimone, and C. G. Gross, 1981. Visual properties of neurons in a polysensory area in superior temporal sulcus of the macaque, J. Neurophysiol., 46:369–384. Cheng, K., K. S. Saleem, and K. Tanaka, 1997. Organization of corticostriatal and corticoamygdalar projections arising from the anterior inferotemporal area TE of the macaque monkey: a Phaseolus vulgaris Leucoagglutinin study, J. Neurosci., 17:7902–7925.

Dean, P., 1976. Effects of inferotemporal lesions on the behavior of monkeys, Psychol. Bull., 83:41–71. Esteky, H., and K. Tanaka, 1998. Effects of changes in aspect ratio of stimulus shape on responses of cells in the monkey inferotemporal cortex, Soc. Neurosci. (Abstr.), 24:899. Fahy, F. L., I. P. Riches, and M. W. Brown, 1993. Neuronal activity related to visual recognition memory: long-term memory and the encoding of recency and familiarity information in the primate anterior and medial inferior temporal and rhinal cortex, Exp. Brain Res., 96:457–472. Fujita, I., and T. Fujita, 1996. Intrinsic connections in the macaque inferior temporal cortex, J. Comp. Neurol., 368:467–486. Fujita, I., K. Tanaka, M. Ito, and K. Cheng, 1992. Columns for visual features of objects in monkey inferotemporal cortex, Nature, 360:343–346. Gallant, J. L., J. Braun, and D. C. Van Essen, 1993. Selectivity for polar, hyperbolic, and Cartesian gratings in macaque visual cortex, Science, 259:100–103. Gallant, J. L., C. E. Connor, S. Rakshit, J. W. Lewis, and D. C. Van Essen, 1996. Neural responses to polar, hyperbolic, and Cartesian gratings in area V4 of the macaque monkey, J. Neurophysiol., 76:2718–2739. Gilbert, C. D., and T. N. Wiesel, 1989. Columnar specificity of intrinsic horizontal and corticocortical connections in cat visual cortex, J. Neurosci., 9:2432–2442. Gross, C. G., 1973. Visual functions of inferotemporal cortex, in Handbook of Sensory Physiology, vol. 7, part 3B (R. Jung, ed.), Berlin: Springer-Verlag, pp. 451–482. Haxby, J. V., M. I. Gobbini, M. L. Furey, A. Ishai, J. L. Schouten, and P. Pietrini, 2001. Distributed and overlapping representations of faces and objects in ventral temporal cortex, Science, 293:2425–2430. Higuchi, S., and Y. Miyashita, 1996. Formation of mnemonic neuronal responses to visual paired associates in inferotemporal cortex is impaired by perirhinal and entorhinal lesions, Proc. Natl. Acad. Sci. USA, 93:739–743. Ishai, A., L. G. Ungerleider, A. Martin, J. L. Schouten, and J. V. Haxby, 1999. Distributed representation of objects in the human ventral visual pathway, Proc. Natl. Acad. Sci. USA, 96:9379–9384. Ito, M., I. Fujita, H. Tamura, and K. Tanaka, 1994. Processing of contrast polarity of visual images in inferotemporal cortex of the macaque monkey, Cereb. Cortex, 5:499–508. Ito, M., H. Tamura, I. Fujita, and K. Tanaka, 1995. Size and position invariance of neuronal responses in monkey inferotemporal cortex, J. Neurophysiol., 73:218–226. Janssen, P., R. Vogels, and G. A. Orban, 1999. Macaque inferior temporal neurons are selective for disparity-defined threedimensional shapes, Proc. Natl. Acad. Sci. USA, 96:8217–8222. Janssen, P., R. Vogels, and G. A. Orban, 2000a. Selectivity for 3D shape that reveals distinct areas within macaque inferior temporal cortex, Science, 288:2054–2056. Janssen, P., R. Vogels, and G. A. Orban, 2000b. Three-dimensional shape coding in inferior temporal cortex, Neuron, 27:385–397. Kanwisher, N., 2000. Domain specificity in face perception, Nat. Neurosci., 3:759–763. Kanwisher, N., J. McDermott, and M. M. Chun, 1997. The fusiform face area: a module in human extrastriate cortex specialized for face perception, J. Neurosci., 17:4302–4311. Keysers, C., D.-K. Xiao, P. Foldiak, and D. L. Perrett, 2001. The speed of sight, J. Cogn. Neurosci., 13:90–101. Kobatake, E., and K. Tanaka, 1994. Neuronal selectivities to complex object features in the ventral visual pathway of the macaque cerebral cortex, J. Neurophysiol., 71:856–867.

Kobatake, E., G. Wang, and K. Tanaka, 1998. Effects of shapediscrimination training on the selectivity of inferotemporal cells in adult monkeys, J. Neurophysiol., 80:324–330. Lamme, V. A., and H. Spekreijse, 1998. Neuronal synchrony does not represent texture segregation, Nature, 396:362–366. Li, L., E. K. Miller, and R. Desimone, 1993. The representation of stimulus familiarity in anterior inferior temporal cortex, J. Neurophysiol., 69:1918–1929. Logothetis, N. K., 1998. Object vision and visual awareness, Curr. Opin. Neurobiol., 8:536–544. Logothetis, N. K., J. Pauls, and T. Poggio, 1995. Shape representation in the inferior temporal cortex of monkeys, Curr. Biol., 5:552–563. Lueschow, A., E. K. Miller, and R. Desimone, 1994. Inferior temporal mechanisms for invariant object recognition, Cereb. Cortex, 5:523–531. Maunsell, J. H. R., and D. C. Van Essen, 1983. Functional properties of neurons in middle temporal visual area of the macaque monkey. II. Binocular interactions and sensitivity to binocular disparity, J. Neurophysiol., 49:1148–1167. Mel, B. W., and J. Fiser, 2000. Minimizing binding errors using learned conjunctive features, Neural Comput., 12:731–762. Miller, E. K., L. Li, and R. Desimone, 1991. A neural mechanism for working and recognition memory in inferior temporal cortex, Science, 254:1377–1379. Missal, M., R. Vogels, L. Chao-yi, and G. A. Orban, 1999. Shape interactions in macaque inferior temporal neurons, J. Neurophysiol., 82:131–142. Missal, M., R. Vogels, and G. A. Orban, 1997. Responses of macaque inferior temporal neurons to overlapping shapes, Cereb. Cortex, 7:758–767. Op de Beeck, H., and R. Vogels, 2000. Spatial sensitivity of macaque inferior temporal neurons, J. Comp. Neurol., 426:505–518. Pasupathy, A., and C. E. Connor, 2001. Shape representation in area V4: position-specific tuning for boundary conformation, J. Neurophysiol., 86:2505–2519. Perrett, D. I., E. T. Rolls, and W. Caan, 1982. Visual neurones responsive to faces in the monkey temporal cortex, Exp. Brain Res., 47:329–342. Poggio, G. F., and B. Fischer, 1977. Binocular interaction and depth sensitivity in striate and prestriate cortex of behaving rhesus monkey, J. Neurophysiol., 40:1392–1405. Puce, A., T. Allison, J. C. Gore, and G. McCarthy, 1995. Facesensitive regions in human extrastriate cortex studied by functional MRI, J. Neurosci., 74:1192–1199. Riches, I. P., F. A. W. Wilson, and M. W. Brown, 1991. The effects of visual stimulation and memory on neurons of the hippocampal formation and the neighboring parahippocampal gyrus and inferior temporal cortex of the primate, J. Neurosci., 11:1763–1779. Richmond, B. J., L. M. Optican, M. Podell, and H. Spitzer, 1987. Temporal encoding of two-dimensional patterns by single units in primate inferior temporal cortex. I. Response characteristics, J. Neurophysiol., 57:132–146. Sakai, K., and Y. Miyashita, 1991. Neural organization for the long-term memory of paired associates, Nature, 354:152–155. Sakai, K., and Y. Miyashita, 1994. Neuronal tuning to learned complex forms in vision, Neuroreport, 5:829–832. Saleem, K. S., and K. Tanaka, 1996. Divergent projections from the anterior inferotemporal area TE to the perirhinal and entorhinal cortices in the macaque monkey, J. Neurosci., 16:4757–4775.

:   

1163

Sary, G., R. Vogels, and G. A. Orban, 1993. Cue-invariant shape selectivity of macaque inferior temporal neurons, Science, 260:995–997. Sato, T., 1989. Interactions of visual stimuli in the receptive fields of inferior temporal neurons in awake macaques, Exp. Brain Res., 77:23–30. Sato, T., 1995. Interactions between two different visual stimuli in the receptive fields of inferior temporal neurons in macaques during matching behaviors, Exp. Brain Res., 105:209– 219. Schwartz, E. L., R. Desimone, T. D. Albright, and C. G. Gross, 1983. Shape recognition and inferior temporal neurons, Proc. Natl. Acad. Sci. USA, 80:5776–5778. Sigala, N., and N. K. Logothetis, 2002. Visual categorization shapes feature selectivity in the primate temporal cortex, Nature, 415:318–320. Singer, W., 1999. Neuronal synchrony: a versatile code for the definition of relations? Neuron, 24:49–65. Tanaka, K., H. Saito, Y. Fukada, and M. Moriya, 1991. Coding visual images of objects in the inferotemporal cortex of the macaque monkey, J. Neurophysiol., 66:170–189. Tanaka, H., T. Uka, K. Yoshiyama, M. Kato, and I. Fujita, 2001. Processing of shape defined by disparity in monkey inferior temporal cortex, J. Neurophysiol., 85:735–744. Tanigawa, H., I. Fujita, M. Kato, and H. Ojima, 1998. Distribution, morphology, and g-aminobutyric acid immunoreactivity of horizontally projecting neurons in the macaque inferior temporal cortex, J. Comp. Neurol., 401:129–143. Tarr, M. J., and I. Gauthier, 2000. FFA: a flexible fusiform area for subordinate-level visual processing automatized by expertise, Nat. Neurosci., 3:764–769.

1164

, ,   

Tsunoda, K., Y. Yamane, M. Nishizaki, and M. Tanifuji, 2001. Complex objects are represented in macaque inferotemporal cortex by the combination of feature columns, Nat. Neurosci., 4:832–838. Uka, T., H. Tanaka, K. Yoshiyama, M. Kato, and I. Fujita, 2000. Disparity selectivity of neurons in monkey inferior temporal cortex, J. Neurophysiol., 84:120–132. Vogels, R., 1999a. Categorization of complex visual images by rhesus monkeys. Part 1: behavioral study, Eur. J. Neurosci., 11:1223–1238. Vogels, R., 1999b. Categorization of complex visual images by rhesus monkeys. Part 2: single-cell study, Eur. J. Neurosci., 11:1239–1255. Wang, Y., I. Fujita, and Y. Murayama, 2000. Neuronal mechanisms of selectivity for object features revealed by blocking inhibition in inferotemporal cortex, Nat. Neurosci., 3:807–813l. Wang, G., K. Tanaka, and M. Tanifuji, 1996. Optical imaging of functional organization in the monkey inferotemporal cortex, Science, 272:1665–1668. Wang, G., M. Tanifuji, and K. Tanaka, 1998. Functional architecture in monkey inferotemporal cortex revealed by in vivo optical imaging, Neurosci. Res., 32:33–46. Yaginuma, S., Y. Osawa, K. Yamaguchi, and E. Iwai, 1993. Differential functions of central and peripheral visual field representations in monkey prestriate cortex, in Brain Mechanisms of Perception and Memory: From Neuron to Behavior (T. Ono, L. R. Squire, R. E. Raichle, D. Perrett, and M. Fukuda, eds.), New York: Oxford University Press, pp. 1–33. Yamane, Y., K. Tsunoda, M. Matsumoto, A. Phillips, and M. Tanifuji, 2001. Decomposition of object images by feature columns in macaque inferotemporal cortex, Soc. Neurosci. (Abstr.), 27:1050.

78

Invariant Object and Face Recognition EDMUND T. ROLLS

I   face recognition is a major computational problem that is solved by the end of ventral stream cortical visual processing in the primate inferior temporal visual cortex. The formation of invariant representations is crucial for the areas that receive from the inferior temporal visual cortex, so that when they implement learning based on one view, size, and position on the retina of the object, the learning generalizes later when the same object is seen in a different view, size, and position. Evidence that invariant representations are provided by some neurons in the inferior temporal visual cortex, and hypotheses about how these representations are formed, are described here. Some of the major properties of neurons in the primate inferior temporal visual cortex that will be described are summarized here. Some neurons in the primate temporal cortical visual areas provide representations of faces and objects that are invariant with respect to position, size, and even view. These neurons show rapid processing and rapid learning. The particular face or object being seen is encoded using a distributed representation in which each neuron conveys independent information in its firing rate, with little information evident in the relative time of firing of different neurons. This ensemble encoding has the advantages of maximizing the information in the representation useful for discrimination between stimuli using a simple weighted sum of the neuronal firing by the receiving neurons, of generalization, and of fault tolerance. The invariant representations found may be produced in a hierarchically organized set of visual cortical areas with convergent connectivity from area to area, in which the neurons use a modified Hebb synaptic modification rule with a short-term memory trace to capture at each stage whatever can be captured that is invariant about objects as the objects change in retinal view, position, size, and rotation. The invariant representations formed in the temporal cortical visual areas are ideal as inputs to other brain regions involved in short-term memory, long-term episodic memory, and associative memory of the reward and punishment associations of the visual stimuli. In addition to these neurons, there are separate faceselective neurons coding for face expression, face motion, and face view, which are likely to be involved in social interactions, in which explicitly encoded information of this type is useful. Damage to these or related systems can lead to prosopagnosia, an impairment in recognizing individuals from the

sight of their faces, or in difficulty in identifying the expression on a face.

Neuronal responses found in different temporal lobe cortex visual areas While recording in the temporal lobe cortical visual areas of macaques, Charles Gross and colleagues found some neurons that appeared to respond best to complex visual stimuli such as faces (Bruce et al., 1981; Desimone and Gross, 1979; see also Desimone, 1991). It was soon found that while some of these neurons could respond to parts of faces, other neurons required several parts of the face to be present in the correct spatial arrangement, and that many of these neurons did not just respond to any face that was shown, but responded differently to different faces (Desimone et al., 1984; Gross et al., 1985; Perrett et al., 1982; Rolls, 1984). By responding differently to different faces, these neurons potentially encode information useful for identifying individual faces. This early work showed that there is some specialization of function of different temporal cortical visual areas, and this specialization of function is described next. The visual pathways project from the primary visual cortex V1 to the temporal lobe visual cortical areas by a number of intervening ventral stream cortical stages including V2 and V4 (Baizer et al., 1991; Rolls and Deco, 2002; Seltzer and Pandya, 1978). The inferior temporal visual cortex, area TE, is divided on the basis of cytoarchitecture, myeloarchitecture, and afferent input into areas TEa, TEm, TE3, TE2, and TE1. In addition, there is a set of different areas in the cortex in the superior temporal sulcus (Baylis et al., 1987; Seltzer and Pandya, 1978) (Fig. 78.1). Of these latter areas, TPO receives inputs from temporal, parietal, and occipital cortex; PGa and IPa from parietal and temporal cortex; and TS and TAa primarily from auditory areas (Seltzer and Pandya, 1978). There is considerable specialization of function in these architectonically defined areas (Baylis et al., 1987). Areas TPO, Pga, and IPa are multimodal, with neurons that respond to visual, auditory, and/or somatosensory inputs. The more ventral areas in the inferior temporal gyrus (areas TE3, TE2, TE1, TEa, and TEm) are primarily unimodal visual areas. Areas in the cortex in the anterior and dorsal part of the superior temporal sulcus (e.g., TPO, IPa, and IPg)

1165

F 78.1. Lateral view of the macaque brain showing the different architectonic areas (e.g., TEm, TPO) in and bordering the anterior part of the superior temporal sulcus of the macaque (see text).

have neurons specialized for the analysis of moving visual stimuli. Neurons responsive primarily to faces are found more frequently in areas TPO, TEa, and TEm, where they comprise approximately 20% of the visual neurons responsive to stationary stimuli, in contrast to the other temporal cortical areas, in which they comprise 4% to 10%. The stimuli which activate other cells in these TE regions include simple visual patterns such as gratings and combinations of simple stimulus features (Baylis et al., 1987; Gross et al., 1985; Tanaka et al., 1990). Neurons with responses related to facial expression, movement, and gesture are more likely to be found in the cortex in the superior temporal sulcus, whereas neurons with activity related to facial identity are more likely to be found in the TE areas (see below and Hasselmo et al., 1989a).

The selectivity of one population of neurons for faces Neurons with responses selective for faces respond 2 to 20 times more to faces than to a wide range of gratings, simple geometrical stimuli, or complex three-dimensional objects (Baylis et al., 1985, 1987; Rolls, 1984, 1992b). The responses to faces are excitatory, with firing rates often reaching 100 spikes per second, are sustained, and have typical latencies of 80 to 100 msec. The neurons are typically unresponsive to auditory or tactile stimuli and to the sight of arousing or aversive stimuli. These findings indicate that explanations in terms of arousal, emotional or motor reactions, and simple visual feature sensitivity are insufficient to account for the selective responses to faces and face features observed in this population of neurons (Baylis et al., 1985; Perrett et al., 1982; Rolls and Baylis, 1986). Observations consistent with these findings have been published by Desimone et al. (1984), who described a similar population of neurons

1166

, ,   

located primarily in the cortex in the superior temporal sulcus which responded to faces but not to simpler stimuli such as edges and bars or to complex nonface stimuli (see also Gross et al., 1985). These neurons are specialized to provide information about faces in that they provide much more information (on average, 0.4 bit) about which (of 20) face stimuli is being seen than about which (of 20) nonface stimuli is being seen (on average, 0.07 bit) (Rolls and Tovee, 1995a; Rolls et al., 1997). These information theoretic procedures provide an objective and quantitative way to show what is “represented” by a particular population of neurons (Rolls and Deco, 2002; Rolls and Treves, 1998). Masking out or presenting parts of the face (e.g., eyes, mouth, or hair) in isolation reveals that different cells respond to different features or subsets of features. For some cells, responses to the normal organization of cut-out or linedrawn facial features are significantly greater than to images in which the same facial features are jumbled (Perrett et al., 1982; Rolls et al., 1994b). These findings are consistent with the hypotheses developed below that, by competitive selforganization, some neurons in these regions respond to parts of faces by responding to combinations of simpler visual properties received from earlier stages of visual processing, and that other neurons respond to combinations of parts of faces and thus respond only to whole faces. Moreover, the finding that for some of these latter neurons the parts must be in the correct spatial configuration shows that the combinations formed can reflect not just the features present, but also their spatial arrangement. This provides a way in which binding can be implemented in neural networks (see also Elliffe et al., 2002). Further evidence that neurons in these regions respond to combinations of features in the correct spatial configuration was found by Tanaka et al. (e.g., 1990) using combinations of features that are used by comparable neurons to define objects. Other neurons respond not to face identity but to face expression (Hasselmo et al., 1989a). The neurons responsive to expression were found primarily in the cortex in the superior temporal sulcus. Information about facial expression is of potential use in social interactions. A further way in which some of the neurons in the cortex in the superior temporal sulcus may be involved in social interactions is that some of them respond to gestures—for example, to a face undergoing ventral flexion (Hasselmo et al., 1989a; Perrett et al., 1989a). The interpretation of these neurons as being useful for social interactions is that in some cases these neurons respond not only to ventral head flexion, but also to the eyes lowering and the eyelids closing (Hasselmo et al., 1989b). These two movements (eye lowering and eyelid lowering) often occur together when one monkey is breaking social contact with another. It is also important when decoding facial expression to

retain some information about the direction of the head relative to the observer, for this is very important in determining whether a threat is being made in the observer’s direction. The presence of view-dependent head and body gestures (Hasselmo et al., 1989b), and eye gaze (Perrett et al., 1985b), representations in some of these cortical regions where face expression is represented, is consistent with this requirement. In contrast, the TE areas (more ventral, mainly in the macaque inferior temporal gyrus), in which neurons tuned to face identity (Hasselmo et al., 1989a) and with viewindependent responses (Hasselmo et al., 1989b) are more likely to be found, may be more related to an object-based representation of identity. Of course, for appropriate social and emotional responses, both types of subsystem would be important, for it is necessary to know both the direction of a social gesture and the identity of the individual in order to make the correct social or emotional response. Outputs from the temporal cortical visual areas reach the amygdala and the orbitofrontal cortex, and evidence is accumulating that these brain areas use the representations of faces provided by the inferior temporal cortex visual areas to produce social and emotional responses to faces (Rolls, 1990, 1992a, 1992b, 1999). For example, lesions of the amygdala in monkeys disrupt social and emotional responses to faces, and we have identified a population of neurons with face-selective responses in the primate amygdala (Leonard et al., 1985), some of which respond to facial and body gestures (Brothers et al., 1990). Rolls et al. (2002b) have found a number of face-responsive neurons in the orbitofrontal cortex; these neurons are also present in adjacent prefrontal cortical areas (O’Scalaidhe et al., 1999; Wilson et al., 1993). We have applied this research to the study of humans with frontal lobe damage to try to develop a better understanding of the social and emotional changes that may occur in these patients and that are related to the information these brain areas receive from the temporal cortical visual areas. Impairments in the identification of facial and vocal emotional expression were demonstrated in a group of patients with ventral frontal lobe damage who had behavioral problems such as disinhibited or socially inappropriate behavior (Hornak et al., 1996). A group of patients with lesions outside this brain region, without these behavioral problems, was unimpaired on expression identification tests. These findings suggest that some of the social and emotional problems associated with ventral frontal lobe or amygdala damage may be related to difficulty in identifying correctly facial (and vocal) expression and in learning associations involving such visual stimuli (Hornak et al., 1996, 2003; Rolls, 1990, 1999; Rolls et al., 1994a). Neuroimaging data, while unable to address the details of what is encoded in a brain area or how it is encoded, do provide evidence consistent with the neurophysiology that

there are different face processing systems in the human brain. [Neurophysiology, at the single and multiple single neuron levels, rather than neuroimaging, is needed to understand issues such as the proportions of different types of neuron showing different types of response in a brain area; whether inputs from different sources or produced by different inputs activate the same neuron; the response latency in different areas; how long neurons in a given region must fire for perception to occur; whether the representation is fully distributed, sparse (and if so, how sparse it is and what the firing rate distribution is), or local; the amount of information provided by single neurons and how this scales with the size of the neuronal ensemble; the possibility of neuronal synchronization as a method of neural encoding; and, more generally, exactly how the information represented by the computing elements of the brain, and transmitted between the neurons, changes from stage to stage of neural processing in the brain. Answers to these issues are fundamental for understanding neural computation (Rolls and Deco, 2002).] For example, Kanwisher et al. (1997) and Ishai et al. (1999) have shown activation by faces of an area in the fusiform gyrus; Hoffman and Haxby (2000) have shown that distinct areas are activated by eye gaze and face identity; Dolan et al. (1997) have shown that a fusiform gyrus area becomes activated after humans learn to identify faces in complex scenes; and the amygdala (Morris et al., 1996) and orbitofrontal cortex (Blair et al., 1999) may become activated, particularly by certain face expressions.

The selectivity of a different population of neurons in the temporal cortex visual areas for objects Other neurons in the inferior temporal visual cortex encode view-invariant representations of objects, including wireframe objects (Logothetis and Sheinberg, 1996; Logothetis et al., 1994) and real objects (Booth and Rolls, 1998). For the neurons with view-invariant representations of real objects, it has been shown that no specific training is required to form the view-invariant representations and that experience with the objects in the home cage is sufficient (Booth and Rolls, 1998).

Distributed encoding of face and object identity An important question for understanding brain function is whether a particular object (or face) is represented in the brain by the firing of one or a few (gnostic or grandmother) cells (Barlow, 1972), or whether instead the firing of a group or ensemble of cells, each with a different profile of responsiveness to the stimuli, provides a representation that is distributed across neurons. This issue is of considerable interest with respect to the inferior temporal visual cortex, for this

:     

1167

provides the main output from the ventral visual stream to brain areas such as the amygdala, orbitofrontal cortex, and hippocampal system, and influences how they make use of and read the code. The actual representation found is distributed. Baylis et al. (1985) showed this with the responses of temporal cortical neurons that typically responded to several members of a set of five faces, with each neuron having a different profile of responses to each face. Hasselmo et al. (1989a) applied multidimensional scaling to populations of neurons recorded in the inferior temporal visual cortex and the cortex in the superior temporal sulcus, and indicated that the neural code read from the population indicated that these populations were coding for different aspects of the stimuli (see also Young and Yamane, 1992). In a more recent study using 23 faces and 45 nonface natural images, a distributed representation was found again (Rolls and Tovee, 1995a), with the average sparseness being 0.65. [The sparseness of the representation provided by a neuron can be defined as 2

a = (S s =1,S rs S ) S s =1,S (rs2 S ) where rs is the mean firing rate of the neuron to stimulus s in the set of S stimuli (Rolls and Treves, 1998). If the neurons are binary (either firing or not firing to a given stimulus), then a would be 0.5 if the neuron responded to 50% of the stimuli and 0.1 if the neuron responded to 10% of the stimuli.] If the spontaneous firing rate was subtracted from the firing rate of the neuron to each stimulus, so that the changes of firing rate, that is, the active responses of the neurons, were used in the sparseness calculation, then the response sparseness had a lower value, with a mean of 0.33 for the population of neurons. The distributed nature of the representation can be further understood by the finding that the firing rate distribution of single neurons when a wide range of natural visual stimuli are being viewed is approximately exponentially distributed, with a few stimuli producing high firing rates and increasingly large numbers of stimuli producing lower and lower firing rates (Baddeley et al., 1997; Rolls and Tovee, 1995a; Treves et al., 1999) (Fig. 78.2). This is a clear answer to the question of whether these neurons are grandmother cells: they are not, in the sense that each neuron has a graded set of responses to the different members of a set of stimuli, with the prototypical distribution similar to that of the neuron illustrated in Figure 78.2. On the other hand, each neuron does respond for more to some stimuli than to many others, and in this sense is tuned to some stimuli. The sparseness of such an exponential distribution of firing rates is 0.5. It has been shown that the distribution may arise from the threshold nonlinearity of neurons combined with short-term variability in the responses of neurons (Treves et al., 1999). The distributed properties of the code used are further revealed by applying information theory (see Rolls and

1168

, ,   

F 78.2. Firing rate distribution of a single neuron in the temporal visual cortex to a set of 23 face (F) and 45 nonface images of natural scenes. The firing rate to each of the 68 stimuli is shown. The neuron does not respond to just one of these stimuli. Instead, it responds to a small proportion of stimuli with high rates, to more stimuli with intermediate rates, and to many stimuli with almost no change of firing. This is typical of the distributed representations found in temporal cortical visual areas. P, stimuli that were faces in profile view; B, body parts (e.g., hands). (After Rolls and Tovee, 1995a.)

Deco, 2002; Rolls and Treves, 1998, Appendix 2; Shannon, 1948) to analyze how information is represented by a population of these neurons. The information required to identify which of S equiprobable stimuli were shown is log2 S bits. The information about which of 20 equiprobable faces had been shown that was available from the responses of different numbers of these neurons is shown in Figure 78.3. First, it is clear that the information rises approximately linearly, and the number of stimuli encoded thus rises approximately exponentially, as the number of cells in the sample increases (Abbott et al., 1996; Rolls et al., 1997; see also Rolls and Treves, 1998). This direct neurophysiological evidence thus demonstrates that the encoding is distributed, that the responses are sufficiently independent

(a)

(b)

4

100 90

3.5

80 70 Percent correct

Information (bits)

3

2.5

2

60 50

1.5

40

1

30 20

0.5

10 0 0

2

4 6 8 10 12 Cells in the population

14

0

2

4 6 8 10 12 Cells in the population

14

F 78.3. a, The values for the average information available in the responses of different numbers of these neurons on each trial about which of a set of 20 face stimuli has been shown. The decoding method was dot product (crosses) or probability estimation (pluses), and the effects obtained with cross-validation procedures utilizing 50% of the trials as test trials are shown. The remainder of the trials in the cross-validation procedure were used as training

trials. The solid line indicates the amount of information expected from populations of increasing size, assuming random correlations within the constraint given by the ceiling (the information in the stimulus set, I = 4.32 bits). b, The percentage correct (for estimating which stimulus was shown from the responses of different numbers of neurons using a decoding method) for the data corresponding to those shown in a. (After Rolls et al., 1997.)

and reliable, and that the representational capacity increases exponentially with the number of neurons in the ensemble. The consequence of this is that large numbers of stimuli, and fine discriminations between them, can be represented without having to measure the activity of an enormous number of neurons. [It has been shown that the main reason the information tends to asymptote, as shown in Figure 78.3 as the number of neurons in the sample increases, is that the ceiling on how much information is required to discriminate between the set of stimuli is being approached, which with 20 stimuli is log2 20 = 4.32 bits (Abbott et al., 1996; Rolls et al., 1997).] Second, it is clear that some information is available from the responses of just one neuron—on average, approximately 0.34 bit. Thus, knowing the activity of just one neuron in the population does provide some evidence about which stimulus was present, even when the activity of all the other neurons is not known. Rolls et al. (1997) also showed that information in the firing of individual neurons is made explicit in a way that allows neuronally plausible decoding, in which a receiving neuron simply uses each of its synaptic strengths to weight the input activity being received from each afferent axon and sums the result over all inputs. This is a dot product operation (between an input vector of the firing rates of different neurons and a synaptic weight vector, and the decoding is called dot product decoding).

It has also been shown that for the neurons in the inferior temporal visual cortex that encode view-invariant representations of objects, the same type of representation is found, namely, distributed encoding with independent information conveyed by different neurons (Booth and Rolls, 1998). The analyses just described were obtained with neurons that were not simultaneously recorded, but similar results have now been obtained with simultaneously recorded neurons; that is, the information about which stimulus was shown increases approximately linearly with the number of neurons, showing that the neurons convey information that is nearly independent, with redundancy being on the order of 4% to 10% (Rolls et al., 2002a). Gawne and Richmond (1993) and Rolls et al. (2002a) showed that even adjacent pairs of neurons recorded simultaneously from the same electrode carried information that could be 90% independent. Panzeri et al. (1999a) and Rolls et al. (2003a) developed a method for measuring the information in the relative time of firing of simultaneously recorded neurons, which might be significant if the neurons became synchronized to some but not other stimuli in a set, as postulated by Singer and colleagues (e.g., Singer, 1999, 2000, Chapter 113). We found that for two different sets of inferior temporal cortex neurons, almost all the information was available in the firing rates (or number of spikes) of the cells, and almost no

:     

1169

information was available about which image was shown in the relative time of firing of different simultaneously recorded neurons (Panzeri et al. 1999a; Rolls et al. 2000a; 2000b). If significant cross-correlations between pairs of simultaneously recorded inferior temporal cortex neurons were present, these were usually not stimulus-dependent, and were associated with a small degree of redundancy between the cells. Thus, the evidence shows that most of the information is available in the firing rates (or number of spikes in a short time interval) of the neurons, and not in synchronization, for representations of faces and objects in the inferior temporal visual cortex (this is also the case for spatial neurons in the hippocampus and for olfactory neurons in the orbitofrontal cortex; see Rolls et al., 1996, 1998). The information available from the number of spikes emitted by single neurons in short times is considerable, with the information available in a 50 msec time window being 75% of that available in a long window of 500 msec and 50% of that available in a 20 msec window (Tovee and Rolls, 1995; Tovee et al., 1993). Moreover, backward masking experiments show that sufficient information is available with 30 msec of inferior temporal cortex neuronal firing allowed for 50% correct recognition of the face shown (Rolls and Tovee, 1994; Rolls et al., 1994b; Rolls et al., 1999). Further, within a fraction of an interspike interval, with a distributed representation, much information can be extracted (Panzeri et al., 1999b; Rolls et al., 1997; Treves, 1993; Treves et al., 1997). In effect, spikes from many different neurons can contribute to calculating the angle between a neuronal population and a synaptic weight vector within an interspike interval. Given these facts, it is unnecessary, and indeed introduces a number of artificialities into the situation, to suppose that only the first spike of each neuron after a stimulus, and even the order in which spikes arrive from the different neurons, matters (Thorpe et al., 2001). These results provide evidence that a cortical area can perform the computation necessary for the recognition of a visual stimulus in 20 to 30 msec and emphasizes just how rapidly cortical circuitry can operate. Although this speed of operation does seem fast for a network with recurrent connections (mediated, e.g., by recurrent collateral connections between pyramidal cells or by inhibitory interneurons), recent analyses of integrate-and-fire networks with biophysically modeled neurons which integrate their inputs and have spontaneous activity to keep the neurons close to the firing threshold show that such networks can settle very rapidly (Rolls and Treves, 1998; Treves, 1993; Treves et al., 1997). This approach has been extended to multilayer networks such as those found in the visual system, and again very rapid propagation (in 50 to 60 msec) of information through such a four-layer network with recurrent col-

1170

, ,   

laterals operating at each stage has been found (Panzeri et al., 2001). The advantages of the distributed encoding actually found will now be discussed (see further Rolls and Deco, 2002; Rolls and Treves, 1998). E H C C This property refers to the fact that the information from a population of neurons rises linearly with the number of neurons, so that the number of stimuli that can be discriminated rises exponentially. This property arises from two factors: (1) the encoding is sufficiently close to independent by the different neurons (i.e., factorial), and (2) the encoding is sufficiently distributed. E  W  C C B R  R N The code can be read with a neurally plausible, dot product, decoding method. All that is required of decoding neurons is the property of adding up postsynaptic potentials produced through each synapse as a result of the activity of each incoming axon (Abbott et al., 1996; Rolls et al., 1997) (Fig. 78.4). G, C, G D,  H R  N Because the decoding of a distributed representation involves assessing the activity of a whole population of neurons and computing a dot product or correlation between the set (or vector) of inputs and the synaptic weights (Fig. 78.4), a distributed representation provides more resistance to variation in individual components than does a local encoding scheme. This allows higher resistance to noise (Panzeri et al., 1996), graceful (in that it is gradual) degradation of performance when synapses or input axons are lost, and generalization to similar stimuli. S  R   I The information available in a distributed representation can be decoded by an analyzer more quickly than can the information from a local representation, given comparable firing rates.

Invariance in the neuronal representation of stimuli One of the major problems that must be solved by a visual system is the building of a representation of visual information which allows recognition to occur relatively independently of size, contrast, spatial frequency, position on the retina, angle of view, and so on. This is required so that if the receiving regions such as the amygdala, orbitofrontal cortex, and hippocampus learn about one view, position, or size of the object, the animal generalizes correctly to other views, positions, and sizes of the object. The majority of face-selective neurons in the inferior temporal cortex have responses that are relatively invariant with respect to the size

F 78.4. Neuronal activation and the dot product as used in models of associative memory. When a stimulus is present on the input axons, the total activation hi of a neuron i is the sum of all the activations produced through each strengthened synapse wij by each active axon r¢j. We can express this as hi = Sjr¢jwij where Sj indicates that the sum is over the C input axons (or connections) indexed by j. The synapse wij is the jth synapse (arising from axon j) onto neuron i. The multiplicative form indicates that activation should be produced by an axon only if it is firing and only if it is connected to the dendrite by a strengthened synapse. The sum of all such activations expresses the idea that summation (of synaptic currents in real neurons) occurs along the length of the dendrite to produce activation at the cell body, where the activation hi is converted into firing ri by a function that includes a threshold. Calculation of the neuronal activation by multiplying the vector of input firings by the vector of synaptic weights is an inner or dot product of two vectors which measures the similarity of the two vectors. It is this computation of similarity (very close to the correlation) between the two vectors that enables neurons to show the interesting properties of generalization, completion, graceful degradation, and resistance to noise, provided that the input representations r¢ are distributed (Rolls and Treves, 1998).

of the stimulus (Rolls and Baylis, 1986). The median size change tolerated with a response of more than half of the maximal response was 12 times. Also, the neurons typically responded to a face when the information in it had been reduced from a three-dimensional to a two-dimensional representation in gray on a monitor, with a response which was on average 0.5 of that to a real face. Another transform over which recognition is relatively invariant is spatial frequency, and it has been shown that the processing with respect to spatial frequency is not additive with respect to different bands of spatial frequency (Rolls et al., 1985, 1987).

Inferior temporal visual cortex neurons also often showed considerable translation (shift) invariance, not only under anesthesia (Gross et al., 1985), but also in the awake, behaving primate (Tovee et al., 1994). In most cases, the responses of the neurons were little affected by which part of the face was fixated, and the neurons responded (with a greater than half-maximal response) even when the monkey fixated 2 to 5 degrees beyond the edge of a face which subtended 8 to 17 degrees at the retina. Moreover, stimulus selectivity between faces was maintained this far eccentric within the receptive field. Until recently, research on translation invariance considered the case in which there are only one or two images or objects in the visual field. What happens in a cluttered natural environment? Do all objects that can activate an inferior temporal neuron do so whenever they are anywhere within the large receptive fields of inferior temporal neurons? If so, the output of the visual system might be confusing for structures which receive inputs from the temporal cortical visual areas. To clarify this, Rolls et al. (2003a) recorded from inferior temporal cortex neurons while macaques searched for objects in complex natural scenes as well as in plain backgrounds, as normally used. They found that inferior temporal cortex neuron receptive fields are much smaller in natural scenes (mean radius = 15.7 degrees) than in plain backgrounds (mean radius = 33.7 degrees), and in complex backgrounds can reduce to close to the size of a 9 ¥ 7 degree object. The peak firing rate of the neuron was little affected by whether the effective stimulus was a target to which attention should be shown or not in a complex scene or a plain background. In terms of responsiveness and selectivity, background invariance is thus a property of these neurons. Attention, as influenced by whether the object was to be searched for and was a target for action, had little effect in a complex scene. The main effect of object-based attention on inferior temporal cortex neurons was found in blank displays when it increased the receptive field size of objects that were the target for action. This increase was especially pronounced when there was only one object in a blank scene, but it was also found when there were two objects in a blank scene. It was concluded that inferior temporal cortex neurons, which provide the object-related output of the ventral visual processing stream, have receptive fields which shrink to approximately the size of an object when the object is shown in a natural scene. This helps the inferior temporal visual cortex to provide an unambiguous representation of a potential target, for the output reflects in a natural scene what is shown at the fixation point. The results also showed that attentional effects are mainly demonstrable in inferior temporal cortex neurons when stimuli are shown in an artificial plain background, and that normally the output of the inferior temporal cortex reflects what is shown at the fovea (see further Rolls and Deco, 2002; Trappenberg et al., 2002).

:     

1171

The translation invariance that is a property of these neurons may enable the visual system to compensate for small changes in the position of stimuli on the retina, and to adjust the receptive field size to that appropriate for receiving all the evidence relevant to diagnosing the presence of particular objects independently of their size (Rolls et al., 2003a). These findings are a step toward understanding how the visual system functions in a normal environment (see also Gallant et al., 1998; Stringer and Rolls, 2000) and are complemented by a finding of DiCarlo and Maunsell (2000) that inferotemporal cortex neurons respond similarly to an effective shape stimulus for a cell even if some distractor stimuli are present a few degrees away. A similar result was also found by Rolls and Tovee (1995b), who showed in addition that with two stimuli present in the visual field, the anterior inferior temporal neuronal responses were weighted toward the stimulus that was closest to the fovea. In addition to these types of invariance, some temporal cortical neurons reliably respond differently to the faces of two different individuals independently of viewing angle, although in most cases (16/18 neurons) the response was not perfectly view-independent (Hasselmo et al., 1989b). Mixed together in the same cortical regions are neurons with viewdependent responses. Such neurons might respond, for example, to a view of a profile of a monkey but not to a fullface view of the same monkey (Perrett et al., 1985b). These findings, of view-dependent, partially view-independent, and view-independent representations in the same cortical regions are consistent with the hypothesis discussed below that view-independent representations are being built in these regions by associating together neurons that respond to different views of the same individual. Further evidence that some neurons in the temporal cortical visual areas have object-based rather than view-based responses comes from a study of a population of neurons that responds to moving faces (Hasselmo et al., 1989b). For example, four neurons responded vigorously to a head undergoing ventral flexion, irrespective of whether the view of the head was full face, of either profile, or even of the back of the head. These different views could only be specified as equivalent in object-based coordinates. Further, the movement specificity was maintained across inversion, with neurons responding, for example, to ventral flexion of the head irrespective of whether the head was upright or inverted. In this procedure, retinally encoded or viewer-centered movement vectors are reversed, but the object-based description remains the same. Neurons with view-invariant responses of objects seen naturally by macaques have also been described (Booth and Rolls, 1998). The stimuli were images of 10 real plastic objects which had been in the monkey’s cage for several weeks to enable him to build view-invariant representations of the objects. After this visual experience, it was shown

1172

, ,   

that some inferior temporal cortex neurons have similar responses to views of the same object presented on a monitor and different responses to different objects. This invariance was based on shape in that it remained when the images of the objects were shown in gray scale (i.e., without color). Further evidence consistent with these findings is that some studies have shown that the responses of some visual neurons in the inferior temporal cortex do not depend on the presence or absence of critical features for maximal activation (e.g., Perrett et al., 1982; see also Tanaka, 1993, 1996). For example, Mikami et al. (1994) have shown that some TE cells respond to partial views of the same laboratory instrument(s) even when these partial views contain different features. Using a different approach, Logothetis et al. (1994) reported that in monkeys extensively trained (over thousands of trials) to treat different views of computer-generated wire-frame objects the same way, a small population of neurons in the inferior temporal cortex did respond to different views of the same wire-frame object (see also Logothetis and Sheinberg, 1996). However, extensive training is not necessary for invariant representations to be formed, and indeed, no explicit training in invariant object recognition was given in the experiment by Booth and Rolls (1998), as Rolls’ (1992b) hypothesis is that view-invariant representations can be learned by associating together the different views of objects as they are moved and inspected naturally in a period that may be only a few seconds long.

Learning new representations in the temporal cortical visual areas To investigate the idea that visual experience may guide the formation of the responsiveness of neurons, so that they provide an economical and ensemble-encoded representation of items actually present in the environment, the responses of inferior temporal cortex face-selective neurons have been analyzed while a set of new faces were shown. Some of the neurons studied in this way altered the relative degree to which they responded to the different members of the set of novel faces over the first few (one or two) presentations of the set (Rolls et al., 1989). This evidence is consistent with categorization being performed by selforganizing competitive neuronal networks, as described below and elsewhere (Rolls and Treves, 1998). Further evidence that these neurons can learn new representations very rapidly comes from an experiment in which binarized black and white images of faces which blended with the background were used. These did not activate face-selective neurons. Full gray-scale images of the same photographs were then shown for ten 0.5 second presentations. In a number of cases, if the neuron happened to be responsive to that face, when the binarized version of the same face was shown next the neurons responded to it (Tovee

et al., 1996). This is a direct parallel to the same phenomenon which is observed psychophysically, and it provides dramatic evidence that these neurons are influenced by only a very few seconds (in this case 5 seconds) of experience with a visual stimulus. We have shown a neural correlate of this effect using similar stimuli and a similar paradigm in a positron emission tomography neuroimaging study in humans, with a region showing an effect of the learning found for faces in the right temporal lobe and for objects in the left temporal lobe (Dolan et al., 1997). Such rapid learning of representations of new objects appears to be a major type of learning in which the temporal cortical areas are involved.

Possible computational mechanisms in the visual cortex for object recognition

Layer 4

Layer 3

Layer 2

Receptive Field Size / deg

The neurophysiological findings described above, and wider considerations on the possible computational properties of the cerebral cortex (Rolls, 1989, 1992b; Rolls and Deco, 2002; Rolls and Treves, 1998), lead to the following outline working hypotheses on object (including face) recognition by visual cortical mechanisms. Cortical visual processing for object recognition is considered to be organized as a set of hierarchically connected cortical regions consisting at least of V1, V2, V4, posterior inferior temporal cortex (TEO), inferior temporal cortex (e.g., TE3, TEa, and TEm), and anterior temporal cortical areas (e.g., TE2 and TE1). There is convergence from each small part of a region to the succeeding region (or layer or stage in the hierarchy) in such a way that the receptive field sizes of neurons (e.g., 0.5 to 1 degree near the fovea in V1) become larger by a factor of approximately 2.5 with each succeeding stage (the typical parafoveal receptive field sizes

found would not be inconsistent with the calculated approximations of, e.g., 8 degrees in V4, 20 degrees in TEO, and 50 degrees in inferior temporal cortex; Boussaoud et al., 1991) (Fig. 78.5). Such zones of convergence would overlap continuously with each other (Fig. 78.5). This connectivity would be part of the architecture by which translationinvariant representations are computed. Each stage in the hierarchy is considered to act partly as a set of local self-organizing competitive neuronal networks with overlapping inputs. The operation of competitive networks is described by Kohonen (1989), Rolls and Treves (1998), and Rolls and Deco (2002). They use competition implemented by lateral inhibition and associative modification of active inputs onto output neurons that are left firing after the competition. Competitive networks can be thought of as building feature analyzers, in that each neuron in a competitive network uses associative synaptic modification to learn to respond to a set or combination of coactive inputs to the neuron, which might represent a visual feature (Rolls and Milward, 2000; Rolls and Treves, 1998; Wallis and Rolls, 1997). Increasing complexity of representations could also be built in such a multiple-layer hierarchy by similar competitive learning mechanisms. In order to avoid a combinatorial explosion, low-order combinations of inputs would be learned by each neuron. An important part of the theory is that some local spatial information is inherent in the features being combined. For example, cells might not respond to the combination of an edge and a small circle unless they were in the correct spatial relation to each other. [This is in fact consistent with the data of Tanaka et al. (1990), and with our data on face neurons, in that some faces neurons require the face features to be in the correct spatial configuration and not jumbled (Rolls et al., 1994b).] The local spatial

50

TE

20

TEO

8.0

V4

3.2

V2

1.3

V1

view independence

view dependent configuration sensitive combinations of features

larger receptive fields

LGN Layer 1

0 1.3 3.2 8.0 20 50 Eccentricity / deg

F 78.5. Schematic diagram showing convergence achieved by the forward projections in the visual system, and the types of representation that may be built by competitive networks operating at each stage of the system from the primary visual cortex (V1) to the inferior temporal visual cortex (area TE) (see text). LGN, lateral geniculate nucleus. Area TEO forms the posterior inferior temporal cortex. The receptive fields in the inferior temporal visual cortex (e.g., in the TE areas) cross the vertical midline (not shown).

:     

1173

information in the features being combined would ensure that the representation at the next level would contain some information about the (local) arrangement of features. Further low-order combinations of such neurons at the next stage would include sufficient local spatial information so that an arbitrary spatial arrangement of the same features would not activate the same neuron; this is the proposed, and limited, solution which this mechanism would provide for the feature binding problem (Elliffe et al., 2002). Once such a feature hierarchy scheme has been trained with exemplars in the early layers, training at only some locations is needed at the higher levels to achieve generalization to new locations of new objects trained at some locations (Elliffe et al., 2002). Although hierarchical processing schemes have been investigated before (e.g., Fukushima, 1980, 1989, 1991), Rolls (1992b) suggested that translation, size, and view invariance could be computed in such a system by utilizing competitive learning that operates across short time scales to detect regularities in inputs when real objects are transforming in the physical world (Rolls, 1992a). The idea is that because objects have continuous properties in space and time in the world, an object at one place on the retina might activate feature analyzers at the next stage of cortical processing, and when the object is translated to a nearby position, because this would occur in a short period (e.g., 0.5 second), the membrane of the postsynaptic neuron would still be in its associatively modifiable state, and the presynaptic afferents activated with the object in its new position would thus become strengthened on the still-activated postsynaptic neuron. The neuronal mechanisms that might implement this short-term temporal averaging in modifiability are of interest and include lasting effects of calcium entry as a result of the voltage-dependent activation of NMDA receptors, as well as continuing firing of the neuron implemented by recurrent collateral connections forming a short-term memory. The short temporal window (e.g., 0.5 second) of associative modifiability helps neurons to learn the statistics of objects transforming in the physical world, and at the same time to form different representations of different feature combinations or objects, as these are physically discontinuous and present less regular correlations to the visual system. Foldiak (1991) has proposed computing an average activation of the postsynaptic neuron to assist with translation invariance. Rolls (1992a) suggested that other invariances, for example, size, spatial frequency, rotation, and view invariance, could be learned by mechanisms similar to those just described. To test and clarify the hypotheses just described about how the visual system may operate to learn invariant object recognition, we have performed a simulation, VisNet, which implements many of the ideas just described, and is consistent with and based on much of the neurophysiology sum-

1174

, ,   

marized above. The network simulated can perform object recognition, including face recognition, in a biologically plausible way, and after training shows, for example, translation and view invariance (Rolls and Milward, 2000; Wallis and Rolls, 1997; Wallis et al., 1993). The network can identify objects shown in a cluttered environment (e.g., a natural scene) and can identify partially occluded objects (Stringer and Rolls, 2000). Parga and Rolls (1998) and Elliffe et al. (2000) incorporated the associations between exemplars of the same object in the recurrent synapses of an autoassociative (attractor) network so that the techniques of statistical physics could be used to analyze the storage capacity of a system implementing invariant representations in this way. They showed that such networks did have an “object” phase in which the presentation of any exemplar (e.g., a view) of an object would result in the same firing state as other exemplars of the same object, and that the number of different objects that could be stored is proportional to the number of synapses per neuron divided by the number of views of each object. Rolls and Milward (2000) and Rolls and Stringer (2001) explored the operation of the trace learning rule used in the VisNet architecture further, and showed that the rule operated especially well if the trace incorporated activity from previous presentations of the same object but no contribution from the current neuronal activity being produced by the current exemplar of the object. The explanation is that this temporally asymmetric rule (the presynaptic term from the current exemplar and the trace from the preceding exemplars) encourages neurons to respond to the current exemplar in the same way as they did to previous exemplars. These results with a feature hierarchy network, VisNet, show that the proposed learning mechanism and neural architecture can produce cells with responses selective for stimulus type with considerable position or view invariance (Rolls and Deco, 2002). The network has recently been extended to incorporate back projections and interactions between a dorsal spatial processing stream and a ventral object processing stream, and is enabling extensive modeling of spatial and object attentional processes within a mean-field framework (Deco and Rolls, 2002; Rolls and Deco, 2002).

Conclusions Neurophysiological investigations of the inferior temporal cortex are revealing at least part of the way in which neuronal firing encodes information about faces and objects and are showing that the representation implements several types of invariance. The representation found has clear utility for the receiving networks. These neurophysiological findings are stimulating the development of computational neuronal network models which suggest that part of the cellular processing involves the operation of a modified associative

learning rule with a short-term memory trace to help the system learn invariances from the statistical properties of the inputs it receives. It is a challenge to identify the cellular processes that could implement this short-term memory trace, as well as the processes that might help to maintain the total synaptic strength received by each neuron approximately constant, as is required for competitive networks (Rolls and Treves, 1998).

Acknowledgments The author has worked on some of the investigations described here with P. Azzopardi, G. C. Baylis, M. Booth, P. Foldiak, M. E. Hasselmo, C. M. Leonard, T. J. Milward, D. I. Perrett, S. M. Stringer, M. J. Tovee, A. Treves, and G. Wallis, and their collaboration is sincerely acknowledged. Different parts of the research described were supported by the Medical Research Council, PG8513790 and PG9826105; by a Human Frontier Science Program grant; by an EC Human Capital and Mobility grant; by the MRC Oxford Interdisciplinary Research Centre in Cognitive Neuroscience; and by the Oxford McDonnell-Pew Centre in Cognitive Neuroscience. REFERENCES Abbott, L. F., E. T. Rolls, and M. J. Tovee, 1996. Representational capacity of face coding in monkeys, Cereb. Cortex, 6:498–505. Baddeley, R. J., L. F. Abbott, M. J. A. Booth, F. Sengpiel, T. Freeman, E. A. Wakeman, and E. T. Rolls, 1997. Responses of neurons in primary and inferior temporal visual cortices to natural scenes, Proc. R. Soc. B, 264:1775–1783. Baizer, J. S., L. G. Ungerleider, and R. Desimone, 1991. Organization of visual inputs to the inferior temporal and posterior parietal cortex in macaques, J. Neurosci., 11:168–190. Barlow, H. B., 1972. Single units and sensation: a neuron doctrine for perceptual psychology? Perception, 1:371–394. Baylis, G. C., E. T. Rolls, and C. M. Leonard, 1985. Selectivity between faces in the responses of a population of neurons in the cortex in the superior temporal sulcus of the monkey, Brain Res., 342:91–102. Baylis, G. C., E. T. Rolls, and C. M. Leonard, 1987. Functional subdivisions of temporal lobe neocortex, J. Neurosci., 7:330–342. Blair, R. J., J. S. Morris, C. D. Frith, D. I. Perrett, and R. J. Dolan, 1999. Dissociable neural responses to facial expressions of sadness and anger, Brain, 122:883–893. Booth, M. C. A., and E. T. Rolls, 1998. View-invariant representations of familiar objects by neurons in the inferior temporal visual cortex, Cereb. Cortex, 8:510–523. Boussaoud, D., R. Desimone, and L. G. Ungerleider, 1991. Visual topography of area TEO in the macaque, J. Comp. Neurol., 306:554–575. Brothers, L., B. Ring, and A. S. Kling, 1990. Response of neurons in the macaque amygdala to complex social stimuli, Behav. Brain Res., 4:199–213. Bruce, C., R. Desimone, and C. G. Gross, 1981. Visual properties of neurons in a polysensory area in superior temporal sulcus of the macaque, J. Neurophysiol., 46:369–384.

Deco, G., and E. T. Rolls, 2003. A neurodynamical cortical model of visual attention and invariant object recognition. In preparation. Desimone, R., 1991. Face-selective cells in the temporal cortex of monkeys, J. Cogn. Neurosci., 3:1–8. Desimone, R., T. D. Albright, C. G. Gross, and C. Bruce, 1984. Stimulus-selective properties of inferior temporal neurons in the macaque, J. Neurosci., 4:2051–2062. Desimone, R., and C. G. Gross, 1979. Visual areas in the temporal lobe of the macaque, Brain Res., 178:363–380. DiCarlo, J. J., and J. H. R. Maunsell, 2000. Form representation in monkey inferotemporal cortex is virtually unaltered by free viewing, Nat. Neurosci., 3:814–821. Dolan, R. J., G. R. Fink, E. T. Rolls, M. Booth, A. Holmes, R. S. J. Frackowiak, and K. J. Friston, 1997. How the brain learns to see objects and faces in an impoverished context, Nature, 389:596–599. Elliffe, M. C. M., E. T. Rolls, N. Parga, and N. Renart, 2000. A recurrent model of transformation invariance by association, Neural Net., 13:225–237. Elliffe, M. C. M., E. T. Rolls, and S. M. Stringer, 2002. Invariant recognition of feature combinations in the visual system, Biol. Cybern., 86:59–71. Foldiak, P. 1991. Learning invariance from transformation sequences, Neural Comp., 3:193–199. Fukushima, K., 1980. Neocognitron: a self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position, Biol. Cybern., 36:193–202. Fukushima, K., 1989. Analysis of the process of visual pattern recognition by the neocognitron, Neural Net., 2:413–420. Fukushima, K., 1991. Neural networks for visual pattern recognition, IEEE Trans., E74:179–190. Gallant, J. L., C. E. Connor, and D. C. Van-Essen, 1998. Neural activity in areas V1, V2 and V4 during free viewing of natural scenes compared to controlled viewing, Neuroreport, 9:85–90. Gawne, T. J., and B. J. Richmond, 1993. How independent are the messages carried by adjacent inferior temporal cortical neurons? J. Neurosci., 13:2758–2771. Gross, C. G., R. Desimone, T. D. Albright, and E. L. Schwartz, 1985. Inferior temporal cortex and pattern recognition, Exp. Brain. Res. Suppl., 11:179–201. Hasselmo, M. E., E. T. Rolls, and G. C. Baylis, 1989a. The role of expression and identity in the face-selective responses of neurons in the temporal visual cortex of the monkey, Behav. Brain Res., 32:203–218. Hasselmo, M. E., E. T. Rolls, G. C. Baylis, and V. Nalwa, 1989b. Object-centered encoding by face-selective neurons in the cortex in the superior temporal sulcus of the monkey, Exp. Brain Res., 75:417–429. Hoffman, E. A., and J. V. Haxby, 2000. Distinct representations of eye gaze and identity in the distributed neural system for face perception, Nat. Neurosci., 3:80–84. Hornak, J., E. T. Rolls, and D. Wade, 1996. Face and voice expression identification in patients with emotional and behavioural changes following ventral frontal lobe damage, Neuropsychologia, 34:247–261. Hornak, J., J. Bramham, E. T. Rolls, R. G. Morris, J. O’Doherty, P. R. Bullock, and C. E. Polkey, 2003. Changes in emotion after circumscribed surgical lesions of the orbitofrontal and cingulate cortices, Brain 126, in press. Ishai, A., L. G. Ungerleider, A. Martin, J. L. Schouten, and J. V. Haxby, 1999. Distributed representation of objects in the

:     

1175

human ventral visual pathway, Proc. Natl. Acad. Sci. USA, 96: 9379–9384. Kanwisher, N., J. McDermott, and M. M. Chun, 1997. The fusiform face area: a module in human extrastriate cortex specialized for face perception, J. Neurosci., 17:4302–4311. Kohonen, T., 1989. Self-Organization and Associative Memory, 3rd ed. (1984, 1st ed.; 1988, 2nd ed.), Berlin: Springer-Verlag. Leonard, C. M., E. T. Rolls, F. A. W. Wilson, and G. C. Baylis, 1985. Neurons in the amygdala of the monkey with responses selective for faces, Behav. Brain Res., 15:159–176. Logothetis, N. K., J. Pauls, H. H. Bulthoff, and T. Poggio, 1994. View-dependent object recognition by monkeys, Curr. Biol., 4:401–414. Logothetis, N. K., and D. L. Sheinberg, 1996. Visual object recognition, Annu. Rev. Neurosci., 19:577–621. Mikami, A., K. Nakamura, and K. Kubota, 1994. Neuronal responses to photographs in the superior temporal sulcus of the rhesus monkey, Behav. Brain Res., 60:1–13. Morris, J. S., C. D. Fritch, D. I. Perrett, D. Rowland, A. W. Young, A. J. Calder, and R. J. Dolan, 1996. A differential neural response in the human amygdala to fearful and happy face expressions, Nature, 383:812–815. O’Scalaidhe, S. P., F. A. W. Wilson, and P. S. Goldman-Rakic, 1999. Face-selective neurons during passive viewing and working memory performance of rhesus monkeys: evidence for intrinsic specialization of neuronal coding, Cereb. Cortex, 9:459–475. Panzeri, S., G. Biella, E. T. Rolls, W. E. Skaggs, and A. Treves, 1996. Speed, noise, information and the graded nature of neuronal responses, Network, 7:365–370. Panzeri, S., E. T. Rolls, F. Battaglia, and R. Lavis, 2001. Speed of information retrieval in multilayer networks of integrate-and-fire neurons, Network: Comput. Neural Syst., 12:423–440. Panzeri, S., S. R. Schultz, A. Treves, and E. T. Rolls, 1999a. Correlations and the encoding of information in the nervous system, Proc. R. Soc. B, 266:1001–1012. Panzeri, S., A. Treves, S. R. Schultz, and E. T. Rolls, 1999b. On decoding the responses of a population of neurons from short time epochs, Neural Comp., 11:1553–1577. Parga, N., and E. T. Rolls, 1998. Transform invariant recognition by association in a recurrent network, Neural Comp., 10:1507–1525. Perrett, D. I., E. T. Rolls, and W. Caan, 1982. Visual neurons responsive to faces in the monkey temporal cortex, Exp. Brain Res., 47:329–342. Perrett, D. I., P. A. J. Smith, A. J. Mistlin, A. J. Chitty, A. S. Head, D. D. Potter, R. Broennimann, A. D. Milner, and M. A. Jeeves, 1985a. Visual analysis of body movements by neurons in the temporal cortex of the macaque monkey: a preliminary report, Behav. Brain Res., 16:153–170. Perrett, D. I., P. A. J. Smith, D. D. Potter, A. J. Mistlin, A. S. Head, D. Milner, and M. A. Jeeves, 1985b. Visual cells in temporal cortex sensitive to face view and gaze direction, Proc. R. Soc. B, 223:293–317. Rolls, E. T., 1984. Neurons in the cortex of the temporal lobe and in the amygdala of the monkey with responses selective for faces, Hum. Neurobiol., 3:209–222. Rolls, E. T., 1989. Functions of neuronal networks in the hippocampus and neocortex in memory, in Neural Models of Plasticity: Experimental and Theoretical Approaches (J. H. Byrne and W. O. Berry, eds.), San Diego, CA: Academic Press, pp. 240–265. Rolls, E. T., 1990. A theory of emotion, and its application to understanding the neural basis of emotion, Cogn. Emotion, 4:161–190.

1176

, ,   

Rolls, E. T., 1992a. Neurophysiology and functions of the primate amygdala, in The Amygdala (J. P. Aggleton ed.), New York: Wiley-Liss, pp. 143–165. Rolls, E. T., 1992b. Neurophysiological mechanisms underlying face processing within and beyond the temporal cortical visual areas, Philos. Trans. R. Soc., 335:11–21. Rolls, E. T., 1999. The Brain and Emotion, Oxford: Oxford University Press. Rolls, E. T., N. C. Aggelopoulos, L. Franco, and A. Treves, 2002a. Information encoding in the inferior temporal cortex: contributions of the firing rates and correlations between the firing of neurons. Submitted for publication. Rolls, E. T., N. C. Aggelopoulos, and F. Zheng, 2003a. The receptive fields of inferior temporal cortex neurons in natural scenes, J. Neurosci., 23:339–348. Rolls, E. T., and G. C. Baylis, 1986. Size and contrast have only small effects on the responses to faces of neurons in the cortex of the superior temporal sulcus of the monkey, Exp. Brain Res., 65:38–48. Rolls, E. T., G. C. Baylis, and M. E. Hasselmo, 1987. The responses of neurons in the cortex in the superior temporal sulcus of the monkey to band-pass spatial frequency filtered faces, Vis. Res., 27:311–326. Rolls, E. T., G. C. Baylis, M. E. Hasselmo, and V. Nalwa, 1989. The effect of learning on the face-selective responses of neurons in the cortex in the superior temporal sulcus of the monkey, Exp. Brain Res., 76:153–164. Rolls, E. T., G. C. Baylis, and C. M. Leonard, 1985. Role of low and high spatial frequencies in the face-selective responses of neurons in the cortex in the superior temporal sulcus, Vis. Res., 25:1021–1035. Rolls, E. T., H. D. Critchley, A. S. Browning, and K. Inoue, 2002b. Face-selective and auditory neurons in the primate orbitofrontal cortex. Submitted for publication. Rolls, E. T., H. D. Critchley, and A. Treves, 1996. The representation of olfactory information in the primate orbitofrontal cortex, J. Neurophysiol., 75:1982–1996. Rolls, E. T., and G. Deco, 2002. Computational Neuroscience of Vision, Oxford: Oxford University Press. Rolls, E. T., L. Franco, N. C. Aggelopoulos, and S. Reece, 2003b. Application of an information theoretic approach to analysing the contributions of the firing rates and correlations between the firing of neurons. J. Neurophysiol., 89:2810–2822. Rolls, E. T., J. Hornak, D. Wade, and J. McGrath, 1994a. Emotionrelated learning in patients with social and emotional changes associated with frontal lobe damage, J. Neurol. Neurosurg. Psychiatry, 57:1518–1524. Rolls, E. T., and T. Milward, 2000. A model of invariant object recognition in the visual system: learning rules, activation functions, lateral inhibition, and information-based performance measures, Neural Comput., 12:2547–2572. Rolls, E. T., and S. M. Stringer, 2001. Invariant object recognition in the visual system with error correction and temporal difference learning, Network: Comput. Neural Syst., 12:111–129. Rolls, E. T., and M. J. Tovee, 1994. Processing speed in the cerebral cortex, and the neurophysiology of visual backward masking, Proc. R. Soc. B, 257:9–15. Rolls, E. T., and M. J. Tovee, 1995a. Sparseness of the neuronal representation of stimuli in the primate temporal visual cortex, J. Neurophysiol., 73:713–726. Rolls, E. T., and M. J. Tovee, 1995b. The responses of single neurons in the temporal visual cortical areas of the macaque when more than one stimulus is present in the visual field, Exp. Brain Res., 103:409–420.

Rolls, E. T., M. J. Tovee, and S. Panzeri, 1999. The neurophysiology of backward visual masking: information analysis, J. Cogn. Neurosci., 11:335–346. Rolls, E. T., M. J. Tovee, D. G. Purcell, A. L. Stewart, and P. Azzopardi, 1994b. The responses of neurons in the temporal cortex of primates, and face identification and detection, Exp. Brain Res., 101:474–484. Rolls, E. T., and A. Treves, 1998. Neural Networks and Brain Function, Oxford: Oxford University Press. Rolls, E. T., A. Treves, R. G. Robertson, P. Georges-François, and S. Panzeri, 1998. Information about spatial view in an ensemble of primate hippocampal cells, J. Neurophysiol., 79:1797–1813. Rolls, E. T., A. Treves, and M. J. Tovee, 1997. The representational capacity of the distributed encoding of information provided by populations of neurons in the primate temporal visual cortex, Exp. Brain Res., 114:149–162. Seltzer, B., and D. N. Pandya, 1978. Afferent cortical connections and architectonics of the superior temporal sulcus and surrounding cortex in the rhesus monkey, Brain Res., 149:1–24. Singer, W., 1999. Neuronal synchrony: a versatile code for the definition of relations? Neuron, 24:49–65. Singer, W., 2000. Response sychronisation: a universal coding strategy for the definition of relations, in The New Cognitive Neurosciences, 2nd ed. (M. Gazzaniga ed.), Cambridge, MA: MIT Press, pp. 325–338. Shannon, C. E., 1948. A mathematical theory of communication, ATT Bell Labs. Tech. J., J27:379–428. Stringer, S. M., and E. T. Rolls, 2000. Position invariant recognition in the visual system with cluttered environments, Neural Net., 13:305–315. Tanaka, K., 1993. Neuronal mechanisms of object recognition, Science, 262:685–688. Tanaka, K., 1996. Inferotemporal cortex and object vision, Annu. Rev. Neurosci., 19:109–139. Tanaka, K., C. Saito, Y. Fukada, and M. Moriya, 1990. Integration of form, texture, and color information in the inferotemporal cortex of the macaque, in Vision, Memory and the Temporal Lobe (E. Iwai and M. Mishkin, eds.), New York: Elsevier, pp. 101–109. Thorpe, S. J., A. Delorme, and R. Van Rullen, 2001. Spike-based strategies for rapid processing, Neural Net., 14:715–725. Tovee, M. J., and E. T. Rolls, 1995. Information encoding in short firing rate epochs by single neurons in the primate temporal visual cortex, Visual Cogn., 2:35–58. Tovee, M. J., E. T. Rolls, and P. Azzopardi, 1994. Translation invariance and the responses of neurons in the temporal visual cortical areas of primates, J. Neurophysiol., 72:1049–1060. Tovee, M. J., E. T. Rolls, and V. S. Ramachandran, 1996. Rapid visual learning in neurones of the primate temporal visual cortex, Neuroreport, 7:2757–2760. Tovee, M. J., E. T. Rolls, A. Treves, and R. P. Bellis, 1993. Information encoding and the responses of single neurons in the primate temporal visual cortex, J. Neurophysiol., 70:640– 654. Trappenberg, T. P., E. T. Rolls, and S. M. Stringer, 2002. Effective size of receptive fields of inferior temporal cortex neurons in natural scenes, in Advances in Neural Information Processing Systems (T. G. Dietterich, S. Becker, and Z. Ghahramani, eds.), Cambridge, MA: MIT Press, pp. 293–300. Treves, A., 1993. Mean-field analysis of neuronal spike dynamics, Network, 4:259–284. Treves, A., E. T. Rolls, and M. Simmen, 1997. Time for retrieval in recurrent associative memories, Physica, D107:392–400.

Treves, A., S. Panzeri, E. T. Rolls, M. Booth, and E. A. Wakeman, 1999. Firing rate distributions and efficiency of information transmission of inferior temporal cortex neurons to natural visual stimuli, Neural Comput., 11:611–641. Wallis, G., and E. T. Rolls, 1997. Invariant face and object recognition in the visual system, Prog. Neurobiol., 51:167–194. Wallis, G., E. T. Rolls, and P. Foldiak, 1993. Learning invariant responses to the natural transformations of objects, Int. Joint Conf. Neural Networks, 2:1087–1090. Wilson, F. A. W., S. P. O’Sclaidhe, and P. S. Goldman-Rakic, 1993. Dissociation of object and spatial processing domains in primate prefrontal cortex, Science, 260:1955–1958. Young, M. P., and S. Yamane, 1992. Sparse population encoding of faces in the inferotemporal cortex, Science, 256:1327–1331.

BIBLIOGRAPHY Ballard, D. H., 1993. Subsymbolic modelling of hand-eye coordination, in The Simulation of Human Intelligence (D. E. Broadbent ed.), Oxford: Blackwell, pp. 71–102. Baylis, G. C., and E. T. Rolls, 1987. Responses of neurons in the inferior temporal cortex in short term and serial recognition memory tasks, Exp. Brain Res., 65:614–622. Bishop, C. M., 1995. Neural Networks for Pattern Recognition, Oxford: Clarendon Press. Eckhorn, R., and B. Popel, 1974. Rigorous and extended application of information theory to the afferent visual system of the cat, Biol. Cybern., 16:191–200. Engel, A. K., P. Konig, A. K. Kreiter, T. B. Schillen, and W. Singer, 1992. Temporal coding in the visual system: new vistas on integration in the nervous system., Trends Neurosci., 15:218–226. Hopfield, J. J., 1982. Neurons with graded responses have collective properties like those of two-state neurons, Proc. Natl. Acad. Sci. USA, 81:3088–3092. Koenderink, J. J., and A. J. Van Doorn, 1979. The internal representation of solid shape with respect to vision, Biol. Cybern., 32:211–217. Kohonen, T., 1977. Associative Memory: A System Theoretical Approach, New York: Springer. MacKay, D. M., and W. S. McCullough, 1952. The limiting information capacity of a neuronal link, Bull. Math. Biophys., 14:127–135. Maunsell, J. H. R., and W. T. Newsome, 1987. Visual processing in monkey extrastriate cortex, Annu. Rev. Neurosci., 10:363–401. Miller, E. K., and R. Desimone, 1994. Parallel neuronal mechanisms for short-term memory, Science, 263:520–522. Miyashita, Y., 1993. Inferior temporal cortex: where visual perception meets memory, Annu. Rev. Neurosci., 16:245–263. Olshausen, B. A., and D. J. Field, 1997. Sparse coding with an overcomplete basis set: a strategy employed by V1? Vis. Res., 37:3311–3325. Perrett, D. I., A. J. Mistlin, and A. J. Chitty, 1987. Visual neurons responsive to faces, Trends Neurosci., 10:358–364. Poggio, T., and S. Edelman, 1990. A network that learns to recognize three-dimensional objects, Nature, 343:263–266. Renart, A., R. Moreno, J. de la Rocha, N. Parga, and E. T. Rolls, 2001. A model of the IT-PF network in object working memory which includes balanced persistent activity and tuned inhibition, Neurocomputing, 38–40:1525–1531. Riesenhuber, M., and T. Poggio, 1998. Just one view: invariances in inferotemporal cell tuning, Adv. Neural. Inf. Proc. Syst., 10:215–221.

:     

1177

Rolls, E. T., 1991. Neural organisation of higher visual functions, Curr. Opin. Neurobiol., 1:274–278. Rolls, E. T., 1996. A theory of hippocampal function in memory, Hippocampus, 6:601–620. Rolls, E. T., 2000. Neurophysiology and functions of the primate amygdala, and the neural basis of emotion, in The Amygdala: A Functional Analysis (J. P. Aggleton ed.), Oxford: Oxford University Press. Rolls, E. T., and M. J. Tovee, 1995. The responses of single neurons in the temporal visual cortical areas of the macaque when more than one stimulus is present in the visual field, Exp. Brain Res., 103:409–420. Rolls, E. T., and A. Treves, 1990. The relative advantages of sparse versus distributed encoding for associative neuronal networks in the brain, Network, 1:407–421. Sato, T., 1989. Interactions of visual stimuli in the receptive fields of inferior temporal neurons in macaque, Exp. Brain Res., 77:23–30. Thorpe, S. J., and M. Imbert, 1989. Biological constraints on connectionist models, in Connectionism in Perspective (R. Pfeifer, Z. Schreter, and F. Fogelman-Soulie, eds.), Amsterdam: Elsevier, pp. 63–92.

1178

, ,   

Trappenberg, T. P., E. T. Rolls, and S. M. Stringer, 2002. Effective size of receptive fields of inferior temporal cortex neurons in natural scenes, in Advances in Neural Information Processing Systems (T. G. Diettrich, S. Becker, and Z. Ghahramani eds.), Cambridge, MA: MIT Press, pp. 293–300. Treves, A., and E. T. Rolls, 1991. What determines the capacity of autoassociative memories in the brain? Network, 2:371–397. Treves, A., and E. T. Rolls, 1994. A computational analysis of the role of the hippocampus in memory, Hippocampus, 4:374–391. Ullman, S., 1996. High-Level Vision. Object Recognition and Visual Cognition, Cambridge, MA: Bradford/MIT Press. Williams, G. V., E. T. Rolls, C. M. Leonard, and C. Stern, 1993. Neuronal responses in the ventral striatum of the behaving macaque, Behav. Brain Res., 55:243–252. Willshaw, D. J., O. P. Buneman, and H. C. Longuet-Higgins, 1969. Nonholographic associative memory, Nature, 222:960–962. Xiang, J. Z., and M. W. Brown, 1998. Differential neuronal encoding of novelty, familiarity and recency in regions of the anterior temporal lobe, Neuropharmacology, 37:657–676. Yamane, S., S. Kaji, and K. Kawano, 1988. What facial features activate face neurons in the inferotemporal cortex of the monkey? Exp. Brain Res., 73:209–214.

79

The Ventral Visual Object Pathway in Humans: Evidence from f MRI NANCY KANWISHER

W    an object within a fraction of a second, even if we have never seen that exact object before and even if we have no advance clues about what kind of object it might be (Potter, 1976; Thorpe et al., 1996). The cognitive and neural mechanisms underlying this remarkable ability are not well understood, and current computer vision algorithms still lag far behind human performance. One promising strategy for attempting to understand human visual recognition is to characterize the neural system that accomplishes it: the ventral visual pathway, which extends from the occipital lobe into inferior and lateral regions of the temporal lobe. Here I describe research from neuroimaging on humans that has begun to elucidate the general organization and functional properties of the cortical regions involved in visually perceiving people, places, and things. I will focus on two main questions in this review. First, what is the functional organization of the ventral visual pathway? This pathway has been characterized in some detail in the macaque using single-unit recording. However, very little was known about its organization in humans even a few years ago, when functional magnetic resonance imaging (fMRI) studies of this region began. Other chapters in this volume review the organization of this pathway in macaques, as well as the organization of earlier retinotopic regions in human visual cortex (see Chapters 32 and 34). This review will focus on the segment of the human ventral visual pathway that lies anterior to retinotopic cortex. I will argue that this pathway contains a small number of category-specific regions, each primarily involved in processing a specific stimulus class, in addition to a more general-purpose region that responds to any kind of visually presented object. Second, what is the nature of the representations we extract from visually presented objects? This question is at the heart of any theory of object recognition and has long been addressed using behavioral methods such as priming. f MRI is beginning to provide some clues about the nature of the visual representations that are extracted in each region. The technique of f MRI adaptation (Grill-Spector et al., 2000; Naccache and Dehaene, 2001) enables us to determine the invariances and equivalence classes of neural representations of objects within each region of cortex scanned.

Other techniques are beginning to address the question of whether objects are represented in distributed neural codes that span much of the ventral visual pathway or whether some kinds of objects are represented in focal regions of cortex.

Functional organization: category-selective regions This section describes work that has characterized three distinct regions in the human ventral visual pathway, each of which responds selectively to a single category of visual stimuli (Fig. 79.1). F Faces are enormously rich and biologically relevant stimuli, providing information not only about the identity of a person but also about his or her mood, age, sex, and direction of gaze. Indeed, behavioral studies of normal subjects and neurological patients (see Farah, 2000, for a review), as well as event-related potentials in humans (Allison et al., 1999; Bentin et al., 1996) and single-unit recording in monkeys (Perrett et al., 1982; Chapter 78, this volume), provide evidence that face perception engages cognitive and neural mechanisms distinct from those engaged during the recognition of other classes of objects. Several brain imaging studies (e.g., Haxby et al., 1991; Puce et al., 1995, 1996; Sergent et al., 1992) described cortical regions that were most active during viewing of faces. However, these studies did not include the kinds of control conditions that are necessary for testing whether the activated regions are selectively involved in face perception. Kanwisher et al. (1997) scanned subjects with fMRI while they viewed rapidly presented sequences of faces versus sequences of familiar inanimate objects. We found a region in the fusiform gyrus in most subjects, and a second region in the superior temporal sulcus in about half of the subjects, that produced a stronger MR response during face viewing than object viewing (see also McCarthy et al., 1997). A greater response to faces than to objects could be produced by processes that have nothing to do with face perception per se, including attentional engagement, which may be greater for faces than for nonfaces, a general response to anything animate or anything human, or a response to the

1179

F 79.1. Three category-selective regions in human extrastriate cortex. The brain images at the left show coronal slices in individual subjects; overlaid in color are regions that responded significantly more strongly to faces than to objects (the FFA), to scenes than to objects (the PPA), and to body parts than to object parts (the EBA). Each region responds to a wide variety of exemplars of

the category; for each area, four examples of such preferred stimuli are show in the blue box at the top. Examples of nonpreferred stimuli for each area (that elicit about half the response of preferred stimuli in terms of percent age signal increase from a fixation baseline) are indicated in the red box at the bottom. (See color plate 55.)

low-level visual features present in face stimuli. To test these and other hypotheses, we first identified the candidate faceselective fusiform region individually in each subject with the comparison of faces to objects, and then measured the response in this region of interest (ROI) to a number of subsequent contrasting conditions. After demonstrating that the same region responded at least twice as strongly to faces as to any of the other control stimuli, we concluded that this region is indeed selectively involved in face processing and named it the fusiform face area (FFA) (Fig. 79.1, top, and Fig. 79.2, bottom). The claim that the FFA responds selectively or specifically to faces does not mean that it responds exclu-

sively to faces. Although the FFA responds much more to faces than to objects, it responds more to objects than to a baseline condition such as a fixation point. The standard criterion for neural selectivity (Tovee et al., 1993), adopted here, is that the response must be at least twice as great for the preferred stimulus category as for any other stimulus category. By now, the FFA has been studied extensively in many different experiments and labs. These studies generally agree that the FFA responds more strongly to a wide variety of face stimuli (e.g., front-view photographs of faces, line drawings of faces, cat faces, cartoon faces, and upside-down faces)

1180

, ,   

F 79.2. The top row (adapted from Figure 1 of Wada and Yamamoto, 2001) shows the site of a lesion (outlined in red for greater visibility) that produced a severe deficit in face recognition but not in object recognition. The bottom row shows the author’s FFA (color indicates regions responding significantly more strongly during face viewing than object viewing). Note the similarity in the anatomical locus of the lesion and the FFA activation, suggesting that an intact FFA may be necessary for face but not object recognition. (See color plate 56.)

than to various nonface control stimuli, even when each of these (like faces) constitutes multiple similar exemplars of the same category, including houses (Haxby et al., 1999; Kanwisher et al., 1997), hands (Kanwisher et al., 1997), animals, provided that their heads are not visible (Kanwisher et al., 1999; but see Chao et al., 1999), flowers (McCarthy et al., 1997), or cars (Halgren et al., 1999). These effects are similar when the subject is merely passively viewing the stimuli or carrying out a demanding discrimination task on them (Kanwisher et al., 1997), suggesting that the response does not arise from a greater attentional engagement by faces than by other stimuli. Nor can the FFA response to faces be accounted for in terms of a low-level feature confound, as the response is higher when a face is perceived versus not perceived even when the stimulus is unchanged, as in binocular rivalry (Tong et al., 1998) and face-vase reversals (Hasson et al., 2001). While the basic response properties of the FFA are generally agreed upon, the function of this region is not. The most basic question is whether the function of the FFA is truly specific to faces or whether it involves a domain-general operation that could in principle be applied to other stimuli (despite being more commonly carried out on faces). For example, in our original paper on the FFA, we suggested testing whether it could be activated by inducing holistic encoding on nonface stimuli. Rossion et al. (2000) found that although attending to whole faces, rather than parts of faces, enhanced the right (but not left) FFA response, attending to whole houses, rather than parts of houses, did not. These data argue against the domain-general holistic encoding hypothesis, instead implicating the right FFA in processing holistic/configural aspects of faces. Gauthier and her colleagues have argued for a somewhat different domain-general hypothesis, according to which the right FFA is specialized for discriminating between any structurally similar exemplars of a given category for which the subject is an expert (Tarr and Gauthier, 2000). However, most of her evidence is based on studies using novel stimuli

called Greebles, a suboptimal choice for testing this hypothesis because they have the same basic configuration as a face (i.e., a symmetrical configuration in which two horizontally arranged parts are above two vertically aligned central parts, as in the configuration of eyes, nose, and mouth). Nonetheless, in one study, Gauthier et al. (1999) found that the FFA was activated by cars in car fanatics and birds in bird experts; this result was replicated by Xu et al. (Xu and Kanwisher, 2001). However, in both studies the effect sizes are small, and the response to faces remains about twice as high as the response to cars in car experts, a result that is consistent with both the face-specificity hypothesis and the subordinatelevel-categorization-of-structurally-identical-exemplars-forwhich-the-subject-is-expert1 hypothesis. Stronger evidence on this debate comes from a double dissociation in neurological patients: face recognition impairments can be found in the absence of impairments in the expert discrimination of category exemplars (Henke et al., 1998) and vice versa (Moscovitch et al., 1997). These findings argue that different cortical mechanisms are involved in face perception and in the expert visual discrimination of structurally similar category exemplars (Kanwisher, 2000). If the face specificity of the FFA is granted, the next question is what exactly the FFA does with faces. The FFA appears not to be involved specifically in discriminating the direction of eye gaze, because it is more active during attention to face identity than to gaze direction, while the faceselective region in the superior temporal sulcus responds more strongly in the opposite comparison (Hoffman and Haxby, 2000). Nor is the FFA likely to be specifically involved

1

Note that if any of these descriptors is removed, the hypothesis has already been disproved: the low FFA response to words shows that visual expertise is not sufficient, and the low FFA response during hand or house discrimination shows that subordinate-level discrimination of structurally identical exemplars is not sufficient to explain the high response to faces.

:      

1181

in extracting emotional expressions from faces, given the consistently high response of the FFA during viewing of expressionless faces. In studies directly manipulating the presence or absence of emotional expressions in face stimuli, the greatest activation is in the amygdala (Breiter et al., 1996) or anterior insula (Phillips et al., 1997), not the fusiform gyrus. Another hypothesis is that the FFA represents semantic rather than perceptual information (Martin and Chao, 2001). However, this too seems unlikely because (1) this region does not respond more to a familiar face, for which semantic information about the individual is available, than to an unfamiliar face, for which it is not (Gorno-Tempini and Price, 2001; Shah et al., 2001), and (2) this region does not appear to represent abstract semantic information about people in general, as it responds no more when subjects read paragraphs describing people than when they read paragraphs describing inanimate objects, though this same comparison produces robust activation in the superior temporal sulcus (R. Saxe and N. Kanwisher, unpublished data). Thus, the FFA appears not to be involved specifically in extracting information about gaze direction or emotional expression, or to be involved in representing semantic information about individual people. Evidence that this area may be involved in simply detecting the presence of a face comes from the findings that activity in the FFA is strong even for inverted faces (Aguirre et al., 1999; Haxby et al., 1999; Kanwisher et al., 1998) and for line drawings of faces (A. Harris and N. Kanwisher, unpublished data; see also Halgren et al., 1999; Ishai et al., 1999), both of which support easy face detection but not face recognition. However, another study (K. Grill-Spector and N. Kanwisher, unpublished data) found that activity in the right FFA is correlated with both successful detection and successful categorization of faces (versus nonfaces) and in successful discrimination between individual faces, suggesting that it is involved in both of these abilities. P For navigating social primates like humans, one other visual ability is arguably as important as recognizing faces: determining our location in the environment. A region of cortex called the parahippocampal place area (PPA) appears to play an important role in this ability (Epstein and Kanwisher, 1998). The PPA responds strongly whenever subjects view images of places, including indoor and outdoor scenes, as well as more abstract spatial environments such as urban “scenes” made out of Legos, virtual spaces depicted in video games (Aguirre et al., 1996, Maguire et al., 1998), or close-up photographs of desktop scenes (P. Downing, R. Epstein, and N. Kanwisher, unpublished data). Remarkably, the visual complexity and number of objects in the scenes are unimportant; the response is just as high to bare empty rooms (two walls, a floor, and sometimes a door or window) as it is to complex photos of the same rooms completely fur-

1182

, ,   

nished. The PPA also responds fairly strongly to images of houses cut out from their background (though less than to full scenes), presumably because spatial surroundings are implicit in a depiction of a house. Thus, it is information about the spatial layout of the scene that is apparently critical to the PPA response (Fig. 79.1, middle). Patients with damage to parahippocampal cortex often suffer from topographical disorientation, an impairment in wayfinding (Aguirre and D’Esposito, 1999; Epstein et al., 2001; Habib and Sirigu, 1987). The core deficit in these patients is an inability to use the appearance of places and buildings for purposes of orientation, perhaps implicating the PPA in place recognition. However, we tested a neurological patient with no PPA and largely preserved place perception but an apparent deficit in learning new place information, suggesting that the PPA may be more critical for encoding scenes into memory than for perceiving them in the first place (Epstein et al., 2001). This possibility is consistent with evidence from other laboratories suggesting that parahippocampal cortex is involved in memory encoding of words (Wagner et al., 1998) and scenes (Brewer et al., 1998). The PPA is apparently not engaged in processes that rely on knowledge of the specific environment (such as planning a route to a particular location in one’s stored cognitive map of the world), as it responds with the same strength to familiar versus unfamiliar places: Epstein et al. (1999) presented MIT students and Tufts University students with scenes from the MIT and Tufts campuses, and found no difference in the response to the same images when they depicted familiar rather than unfamiliar places. Interestingly, however, a significantly higher response was found in the PPA to familiar than to unfamiliar buildings cut out from their background, perhaps because the spatial background was more likely to be inferred in a familiar scene. One attractive idea is that the PPA may constitute the neural instantiation of a previously hypothesized system for spatial reorientation (Cheng, 1986; Hermer and Spelke, 1994). When disoriented rats and human infants must search for a hidden object, they rely largely on the shape of the local environment to reorient themselves and find the object (but see Gouteux et al., 2001; Learmonth et al., 2001). Strikingly, they completely ignore informative landmark cues such as the location of a salient visual object or feature. This led Cheng and others to hypothesize the existence of a geometric module that represents the shape (but not other features) of surrounding space for the purpose of reorientation. The exclusive use of spatial layout information, and not object/landmark information, is tantalizingly reminiscent of the much greater activation of the PPA by images of spatial layouts than images of objects. How is the PPA related to the two other neural structures most commonly implicated in spatial encoding and navigation, the hippocampus and the parietal lobe? It has been

hypothesized that the hippocampus contains a cognitive map of the animal’s environment (O’Keefe and Nadel, 1978). In contrast, the parietal lobe has been implicated in representing the specific spatial information that is relevant to guiding current action. In keeping with this division of labor, physiological recordings in animals indicate that the hippocampus contains allocentric (world-centered) representations of place, whereas the parietal lobes contain egocentric (body-centered) representations of spatial locations (Burgess et al., 1999). For example, place cells in the rat hippocampus respond when the animal is in a specific location in its environment, largely independent of which way the animal is facing, while spatial view cells in the primate hippocampus respond when the animal views a given spatial location (Georges-François et al., 1999). In contrast, neurons in the primate parietal cortex apparently represent space in a number of egocentric coordinates tied to the location of the retina, hand, or mouth (Colby and Goldberg, 1999). A recent study found that fMRI adaptation to repeated stimuli in the PPA occurs only when the same view of a scene is repeated, implicating the PPA in egocentric rather than allocentric representations of space (Epstein et al., 2003). In sum, although it is now well established that the PPA responds selectively to information about spatial layouts and places, it remains unclear what exactly the PPA does with this information. Critical questions for future research concern the role of the PPA in reorientation and encoding of spatial information into memory, as well as the nature of the interactions between the PPA, the hippocampus, and the parietal lobe. B Our latest addition to the set of category-selective regions of cortex is the extrastriate body area (EBA) (Downing et al., 2001). This region responds about twice as strongly when subjects view images depicting human bodies or body parts (nothing too interesting!) as when they view objects or object parts (Fig. 79.1, bottom). The EBA is found in all subjects in the right (and sometimes also the left) lateral occipitotemporal cortex on the lower lip of the posterior superior temporal sulcus, just superior to area MT/MST. The EBA’s response profile is unlikely to reflect low-level stimulus confounds, as the same region responded about twice as strongly to body as to nonbody stimuli even when the two stimulus sets were visually similar (e.g., stick figures versus rearranged versions of stick figures that no longer corresponded to body configurations; silhouettes of people versus slightly rearranged silhouettes). Further experiments showed that the EBA does not simply respond to anything living, animate, or known to be capable of motion, or to any object with parts that can move relative to each other: the EBA responds more to human bodies than to trees, mammals, or objects with movable parts such as scissors, staplers, and corkscrews. The one exception to the body

specificity of the EBA is the fact that this region responds no more to faces than to objects. As expected from this result, the EBA does not overlap much, if at all, with the face-selective region in the superior temporal sulcus. At present, the function of the EBA is unknown. It may be involved in recognizing individuals (when their faces are hidden or far away), or in perceiving the body configuration of other people, or even in perceiving the location of one’s own body parts. The EBA is suggestively close to area MT, perhaps implicating it in integrating information about body shape and motion (Grossman et al., 2000). The EBA is also close to other regions that have been shown to be activated during social perception, from discriminating the direction of eye gaze, to perceiving or inferring intentions, to perceiving human voices. Thus the EBA may be part of a broader network of nearby areas involved in social perception and social cognition. W E? How many category-selective regions of cortex exist in the human visual pathway? Other categories including animals and tools have been reported to selectively activate focal regions of cortex (Martin and Chao, 2001). However, the evidence is not as strongly established in these cases. When only a few stimuli have been compared, apparent category selectivity must be treated cautiously. For example, we found a region that responded more strongly to chairs than to faces or places, replicating the findings of Ishai et al. (1999), but the same region responded just as strongly to pictures of food, animals, and flowers. In ongoing work in our lab, we have tested well over a dozen categories (P. Downing and N. Kanwisher, unpublished data); so far, we have found no other regions of cortex that exhibit the strong category selectivity typical of the FFA, PPA, and EBA. Thus, it appears that faces, places, and bodies may be unusual in the way they are processed and represented in the cortex. The apparent lack of other category-selective regions of cortex raises the question of how other kinds of objects are represented.

Functional organization: category-general regions Considerable evidence suggests that in addition to the category-specific regions described previously, human visual cortex contains a region more generally involved in perceiving the shape of any kind of object. A large region of lateral and inferior occipital cortex just anterior to retinotopic cortex [the lateral occipital complex (LOC)] responds more strongly to stimuli depicting shapes than to stimuli with similar low-level features that do not depict shapes (Kanwisher et al., 1996; Malach et al., 1995; see GrillSpector et al., 2001, for a review). Importantly, the response in this region was the same for familiar and unfamiliar shapes, so the response cannot be straightforwardly

:      

1183

accounted for in terms of matching to stored visual representations, or semantic or verbal coding of the stimuli. Common areas within this lateral occipital region are activated by shapes defined by motion, texture, and luminance contours (Grill-Spector et al., 1998a; Grill-Spector et al., 1998b). Several studies have implicated the LOC in visual object recognition by showing that activity in this region is correlated with success on a variety of object recognition tasks (Bar et al., 2001; Grill-Spector et al., 2000; James et al., 2000; Lerner et al., 2002). Thus, an investigation of the response properties of the LOC may provide important clues about the nature of the representations underlying object recognition. Several studies have shown a reduction in the response of a particular region of the LOC (and other regions of cortex) when stimuli are repeated. Grill-Spector et al. (1999) further showed that in the LOC this effect (fMRI adaptation) can be observed even when the repeated shapes vary in size and position, demonstrating that the representations in this area are largely invariant with respect to changes in size and position. While this adaptation effect was not found across changes in object viewpoint or direction of illumination in this study, another recent study by Vuilleumier et al. (2002) found that the left fusiform gyrus (but not the right) exhibited invariance to viewpoint. Kourtzi and Kanwisher (2001) further demonstrated adaptation in this region between stimulus pairs that had different contours but the same perceived shape (because of changes in occlusion), but not between pairs with identical contours that differed in perceived shape (because of a figure-ground reversal). These findings suggest that neural populations in the LOC represent the perceived shape of an object in a fashion invariant to changes in position and size but not viewpoint (at least in the right hemisphere). Given the correlation of the MR signal in this region with successful recognition, representations with these properties are likely to play an important role in human object recognition. Other studies have shown that the response in this region declines as images of familiar objects are cut into pieces and the positions of those pieces are rearranged. However, interestingly, most regions with in the LOC do not show much decline in the magnitude of the MR response until images are broken into at least 16 fragments (Lerner et al., 2001), suggesting that neural populations in these regions are fragmentbased rather than holistic. At the same time, the response of the LOC is strongly affected by more global factors such as object completion, with higher responses to partly occluded line drawings that can be completed compared to those that cannot (Lerner et al., 2002). Another intriguing recent study found that a small region within the LOC responds to objects compared to textures in both visual and haptic modalities, although most of the LOC responds preferentially to only visually presented objects (Amedi et al., 2001).

1184

, ,   

Anatomically, the LOC is close to and sometimes partly overlapping with the FFA (on the ventral surface) and the EBA (on the lateral surface). Note that such overlap does not imply any contradiction in the data; it simply indicates that some voxels respond significantly more strongly to faces than to nonface objects (and hence are included in the FFA) or to bodies than to objects (and hence are included in the EBA), while the same voxels also respond significantly more strongly to nonface objects than to scrambled objects (and hence are included in the LOC). However, such overlap does indicate that functional definitions of this sort do not serve to categorize uniquely each region of cortex. One account of this situation is that the FFA, EBA, and the LOC are in fact part of the same functional region, which is composed of a set of category-selective and/or feature-selective columns (Fujita et al., 1992) at such a fine scale that they cannot be resolved with fMRI, except for a few very large such regions such as the FFA. Another possibility is that the FFA and LOC (and the EBA and LOC) do not in fact overlap anatomically, with the apparent overlap due to limitations in the spatial resolution of fMRI. In sum, it appears that the ventral visual pathway contains one region, the LOC, that responds strongly to object structure but that exhibits little selectivity for specific object categories, along with a small number of category-specific modules (for faces, places, bodies, and perhaps a few others yet to be discovered). Indeed, it would seem a sensible design for the cortex to supplement its general-purpose mechanisms for describing the shape of any kind of visually presented object (i.e., the LOC) with a small number of additional more specialized mechanisms, each of which may be designed to handle the unique computational challenges posed by stimuli of a specific kind.

Important open questions As the previous sections suggest, neuroimaging in humans has taught us much about the functional organization of the ventral visual pathway and about the representations involved in object recognition. However, some of the most important and difficult questions remain to be tackled. Next, I outline some of these questions and the ongoing experimental work that is attempting to address them. D  L R  O Many of the studies described in previous sections follow a common strategy in visual neuroscience of inferring the function of a cortical area, voxel, or neuron from the stimulus that drives it most strongly. However, this strategy is viable only to the extent that maximal responses carry most of the information in a neural representation. Thus, an important unresolved question concerns the functional significance of the “nonpreferred” responses in

the cortical regions discussed above. For example, do the low but nonzero responses to nonfaces in the FFA reflect a critical involvement of the FFA in the detection or recognition of nonface objects? Haxby et al. (1999) have argued that the partial response to nonfaces is “problematic for [Kanwisher et al.’s] hypothesis that face-selective regions . . . constitute a ‘module specialized for face perception’ ” (p. 196). However, there are at least two reasons why it need not be problematic. First, because of limitations on the spatial resolution due to voxel size, blood flow regulation, and other factors, the MR signal intensity from a particular region should not be expected to reflect a pure measure of the activity in a single functional module, but will include contributions from functionally distinct adjacent (or interleaved) neural tissue. Second, there is no reason to expect even a strongly face-selective cortical area to shut itself off completely when a nonface is presented. Indeed, it is hard to imagine how this could occur without an additional gating mechanism that discriminates between faces and nonfaces and allows only face information into the region in question. In the absence of such a gating mechanism, it would be most natural to expect a low but positive response to nonfaces in a region of cortex specialized for face processing. Thus, the mere existence of nonpreferred responses does not argue against the functional specificity of the region they are recorded from. The critical questions we must answer to understand nonpreferred responses are (1) do they carry information? and (2) is this information used? A recent paper by Haxby et al. (2001) addresses the first question. Haxby et al. (2001) used fMRI to scan subjects while they viewed eight different categories of stimuli. The data from each subject were then split in half, with the data from odd runs in one set and the data from even runs in the other set (the same stimuli were used in odd and even runs). In this fashion, two “partner” activation maps were generated for each of the eight stimulus categories (i.e., 16 activation maps per subject). Next, Haxby et al. carried out seven pairwise comparisons for each activation map, each testing whether that activation map was more similar to its partner (in the other data set) than to each of the activation maps from the other seven categories. In this fashion, the performance on activation map categorization was quantified as the percentage of these pairwise comparisons that were categorized “correctly,” that is, in which the target map was more similar to its partner than to the other map. Haxby et al. found high accuracy in activation map categorization, demonstrating that the patterns of activation for each category were highly replicable within individual subjects. More importantly, they argued that when only the region that responded maximally to a given category was included in the analysis, categorization performance in determining which of the nonpreferred categories had been presented was still well above chance. They therefore suggested that “regions such as the ‘PPA’ and

‘FFA’ are not dedicated to representing only spatial arrangements or human faces, but, rather, are part of a more extended representation for all objects.” However, Haxby et al. did not carry out the analyses necessary to support this conclusion. Spiridon and Kanwisher (2002) replicated their main result, and also compared performance levels for discriminations involving faces and houses and discriminations between pairs of inanimate objects. We found that the FFA supports accurate discrimination between faces and nonfaces but performs at nearchance levels on discriminations between inanimate objects. Similarly, the PPA contains sufficient information for accurate discrimination of houses versus other objects but performs at near-chance levels on discriminations between nonpreferred stimuli. Further, on discriminations between small inanimate objects, neither the FFA nor the PPA outperforms retinotopic cortex, suggesting than any small amount of discriminative information concerning nonpreferred stimuli that may exist in these areas is likely to be based on low-level features that are confounded with stimulus category rather than on true abstract category information. Thus, although some object information may be distributed across the ventral visual pathway, we find no evidence that the FFA and PPA carry any real categorical information about nonpreferred stimuli. Of course, these investigations are subject to two important limitations characteristic of all fMRI research. First, each voxel in the fMRI data contains hundreds of thousands of neurons, so it is possible that discriminative information for nonpreferred categories might exist in these regions at a finer spatial scale. Second, fMRI data (like neurophysiological recordings) are purely correlational, so even when information is present in a given cortical region, we cannot be sure that it forms a critical part of the representation. A recent neuropsychological study addresses both problems for the case of the FFA. Wada and Yamamoto (2001) describe a neurological patient with an unusually circumscribed lesion restricted to the region of the right FFA (Fig. 79.2). This man was severely impaired on face recognition but had fully preserved object recognition. If we assume that his lesion included the right FFA, these data suggest that the FFA plays a necessary role in face but not object recognition (see also Barton et al., 2002). Thus, even if a small amount of category-discriminative information for nonfaces exists in the FFA of normal subjects (undetected in the SpiridonKanwisher study), this information appears not to play any necessary role in the recognition of those nonface objects. W D C M T U  V R? Even if we can determine that some categories of objects are primarily recognized within focal regions of cortex selectively responsive to those categories, will this tell us much about how visual recognition works?

:      

1185

Does the mere existence of a specialized cortical region for a given category imply that qualitatively distinct processing mechanisms are involved in recognizing stimuli from that category? One might argue that special-purpose mechanisms for processing a particular stimulus class would be expected only if the recognition of stimuli from that class poses new computational problems that could not be handled by existing general-purpose mechanisms. Connectionist researchers have noted the computational efficiency gained by the decomposition of a complex function into natural parts ( Jacobs, 1999), and cortical specializations for components of visual recognition are plausible candidates for such task decomposition. If visual cortex is organized in such a computationally principled fashion, then each of the modular components of the system we discover with functional imaging could be expected to instantiate a distinct set of computations. However, an alternative hypothesis is that visual cortex contains a large number of stimulus-selective regions (such as the feature columns in inferotemporal cortex reported by Tanaka, 1997), but the computations that go on in each of these regions are very similar. On this view, cortical specialization might be found for virtually any stimulus class, yet these specializations might not imply qualitative differences in the processing of these stimulus classes. A critical goal for future research is to determine whether the functional organization of visual recognition is better characterized by this kind of shallow specialization, or whether it reflects a deeper form of functional decomposition in which each of a small number of functionally specific regions carries out a qualitatively distinct computation in the service of an evolutionarily or experientially fundamental visual process. O  S R   V V P Where do cortical specializations come from? Does functional differentiation within the ventral visual pathway arise from experience-dependent selforganization of cortex ( Jacobs, 1997), or are these cortical specializations partly innately specified? For faces, places, and bodies, this question is hard to answer because both experiential and evolutionary arguments are plausible. Despite recent misattributions to me of innatist claims about the origins of the FFA (Pierce et al., 2001; Tarr and Gauthier, 2000), my view is that we have almost no relevant data on this question and are in no position to make any strong claims about the origins of the FFA. On the one hand, experience must surely play some role in the development of face areas, given the ample evidence that neurons in the ventral visual pathway are tuned by experience. On the other hand, at least some aspects of face perception appear to be innately specified, as newborn infants preferentially track schematic faces compared to visually

1186

, ,   

similar scrambled faces ( Johnson et al., 1991). However, these two observations leave open a vast space of possible ways that genes and environment could interact in the construction of a selective region of cortex such as the FFA. What does seem pretty clear is that the development of normal adult face processing (and thus, by hypothesis, the development of the FFA) is heavily constrained both anatomically and chronologically. First, neuropsychological patients who selectively lose face recognition abilities as a result of focal brain damage are rarely if ever able to relearn this ability, suggesting that the remaining visual cortex (which is adequate for visual recognition of nonface objects) cannot be trained on face recognition in adulthood. Further, this inability to shift face mechanisms to alternative neural structures may be set very early in development, as evidenced by a patient who sustained damage to the fusiform region when only 1 day old, and who as an adult now has severe difficulties in the recognition of faces (and some other object categories) (Farah et al., 2000). Evidence that very early experience is also crucial in the development of normal adult face recognition comes from a remarkable recent study by Le Grand et al. (2001), who tested people born with dense bilateral cataracts. These people had no pattern vision until their cataracts were surgically corrected between 2 and 6 months of age. After surgery, pattern vision was excellent, if not quite normal. Surprisingly, these individuals never developed normal configural processing of faces. As adults, they are impaired at discriminating between faces that differ in the relative positions of facial features, despite being unimpaired at discriminating faces on the basis of individual face parts. (They are also unimpaired or on either task relative to normal controls when the face stimuli are presented upside down.) Thus, pattern vision in the first few months of life is necessary for the development of normal face processing as an adult; years of subsequent visual experience with faces are not sufficient. One intriguing part of the puzzle comes from recent reports of developmental prosopagnosic patients, who have no brain damage discernible from MRI images or life histories but who are severely impaired at face recognition. These individuals generally do not have other cognitive impairments and often have few or no other impairments in other visual tasks (Duchaine, 2000; Nunn et al., 2001). Although many of these people report relatives with similar deficits, it is not known whether this syndrome is heritable. Thus, it is not yet clear whether it arises from subtle brain damage that cannot be detected on MRIs, or from alterations in genes that code specifically for the construction of a normal face recognition system, or from a failure of general developmental mechanisms that normally lead to the functional differentiation of neural tissue based on experience. Another clue comes from two recent studies showing that autistic subjects exhibit different patterns of cortical activation when

they view faces from those found in normal subjects (Pierce et al., 2001; Schultz et al., 2000). But as with developmental prosopagnosia, this finding can be explained in terms of either experience or genetic factors, or both. One way to unconfound genetic and experiential factors in the development of category-specific regions of cortex is to consider a category for which a specific role of genes is unlikely: visual word recognition. People have been reading for only a few thousand years, which is probably not long enough for natural selection to have produced specialized machinery for visual word recognition. Thus, strong evidence for a region of cortex selectively involved in the visual recognition of letters or words would provide proof that experience alone with a given category of stimulus, without a specific genetic predisposition, can be sufficient for the construction of a region of cortex that is selectively involved in the recognition of stimuli of that category. Some evidence has been reported for cortical specializations for visually presented letters (Polk and Farah, 1998) and words (Cohen et al., 2000). However, preliminary work in our lab suggests otherwise: cortical regions that respond to visually presented words do not show the kind of selectivity seen in the FFA, PPA, and EBA ( J. Jovicich and N. Kanwisher, unpublished data). Thus, for the case of visual recognition in humans, it is not yet clear whether it is the experience of the individual or the experience of the species (or both) that is critical for the construction of functionally distinct regions in the ventral visual pathway.

Conclusions In just the past few years, functional neuroimaging has taught us a great deal about the organization of the ventral visual pathway in humans. Three new category-specific regions of cortex (the FFA, PPA, and EBA), as well as another category-general region (the LOC), have been discovered and described in some detail. But despite this rapid progress, fundamental questions remain unanswered. f MRI has taught us little or nothing about the perceptual functions each of these newly described regions is critical for, the connections between each of these areas and the rest of the brain, the developmental origins of these areas, or the actual mechanisms that occur in each and how they collectively accomplish object recognition. On the other hand, new methods are being developed at a rapid rate and hold the promise of real progress in answering many of these questions. The integration of fMRI with event-related potential and magnetoencephalography data may provide the temporal resolution that will be critical for understanding how visual computations unfold over time. The ability to scan children and perhaps even infants should enable us to trace the appearance and function of each of these areas over development. Patient studies and transcra-

nial magnetic stimulation in normal subjects provide methods for testing the necessity of each region for different visual recognition tasks. Finally, in one of the developments I find most pleasing, the once one-way flow of information from primate visual neuroscience to visual cognitive neuroscience in humans has recently become bidirectional, with fMRI studies in macaques now motivated and informed by prior work on humans (Tsao et al., 2001; Vanduffel et al., 2001).

Acknowledgments I thank Paul Downing, Russell Epstein, Winrich Freiwald, Miles Shuman, and Jonathon Winawer for comments on the manuscript and Ellen Goodman for help with the references. This work was supported by Grants MH59150 and EY13455 to N. Kanwisher.

REFERENCES Aguirre, G. K., and M. D’Esposito, 1999. Topographical disorientation: a synthesis and taxonomy, Brain, 122:1613–1628. Aguirre, G. K., J. A. Detre, D. C. Alsop, and M. D’Esposito, 1996. The parahippocampus subserves topographical learning in man, Cereb. Cortex, 6:823–829. Aguirre, G., R. Singh, and M. D’Esposito, 1999. Stimulus inversion and the responses of face and object-sensitive cortical areas, Neuroreport, 10:189–194. Allison, T., A. Puce, D. D. Spencer, and G. McCarthy, 1999. Electrophysiological studies of human face perception. I. Potentials generated in occipitotemporal cortex by face and non-face stimuli, Cereb. Cortex, 9:415–430. Amedi, A., R. Malach, T. Hendler, S. Peled, and E. Zohary, 2001. Visuo-haptic object-related activation in the ventral visual pathway, Nat. Neurosci., 4:324–330. Bar, M., R. B. Tootell, D. L. Schacter, D. N. Greve, B. Fischl, J. D. Mendola, B. R. Rosen, and A. M. Dale, 2001. Cortical mechanisms specific to explicit visual object recognition, Neuron, 29:529–535. Barton, J. J., D. Z. Press, J. P. Keenan, and M. O’Connor, 2002. Lesions of the fusiform face area impair perception of facial configuration in prosopagnosia, Neurology, 58:71–78. Bentin, S., T. Allison, A. Puce, E. Perez, and G. McCarthy, 1996. Electrophysiological studies of face perceptions in humans, JOCN, 8:551–565. Breiter, H. C., N. L. Etcoff, P. J. Whalen, W. A. Kennedy, S. L. Rauch, R. L. Buckner, M. M. Strauss, S. E. Hyman, and B. R. Rosen, 1996. Response and habituation of the human amygdala during visual processing of facial expression, Neuron, 17:875–887. Brewer, J. B., Z. Zhao, J. E. Desmond, G. H. Glover, and J. D. E. Gabrieli, 1998. Making memories: brain activity that predicts how well visual experience will be remembered, Science, 281:1185–1187. Burgess, N., K. Jeffery, and J. O’Keefe, 1999. Integrating hippocampal and parietal functions: a spatial point of view, in The Hippocampal and Parietal Foundations of Spatial Cognition (N. Burgess, K. J. Jeffery, and J. O’Keefe, eds.), New York: Oxford University Press, pp. 3–29. Chao, L. L., A. Martin, and J. V. Haxby, 1999. Are face-responsive regions selective only for faces? NeuroReport, 10:2945–2950.

:      

1187

Cheng, K., 1986. A purely geometric module in the rat’s spatial representation, Cognition, 23:149–178. Cohen, L., S. Dehaene, L. Naccache, S. Lehericy, G. DehaeneLambertz, M. A. Henaff, and F. Michel, 2000. The visual word form area: spatial and temporal characterization of an initial stage of reading in normal subjects and posterior split-brain patients, Brain, 123(Pt 2):291–307. Colby, C. L., and M. E. Goldberg, 1999. Space and attention in parietal cortex, Annu. Rev. Neurosci., 22:319–349. Downing, P., Y. Jiang, M. Shuman, and N. Kanwisher, 2001. A cortical area selective for visual processing of the human body, Science, 293:2470–2473. Duchaine, B. C., 2000. Developmental prosopagnosia with normal configural processing, NeuroReport, 11:79–83. Epstein, R., K. S. Graham, and P. E. Downing, 2003. Viewpointspecific scene representations in human parahippocampal cortex, Neuron, 37:865–876. Epstein, R., E. De Yoe, D. Press, and N. Kanwisher, 2001. Neuropsychological evidence for a topographical learning mechanism in parahippocampal cortex, Cogn. Neuropsychol., 18:481–508. Epstein, R., A. Harris, D. Stanley, and N. Kanwisher, 1999. The parahippocampal place area: recognition, navigation, or encoding? Neuron, 23:115–125. Epstein, R., and N. Kanwisher, 1998. A cortical representation of the local visual environment, Nature, 392:598–601. Farah, M. J., C. Rabinowitz, G. E. Quinn, and G. T. Liu, 2000. Early commitment of neural substrates for face recognition, Cogn. Neuropsychol., 17:117–123. Fujita, I., K. Tanaka, M. Ito, and K. Cheng, 1992. Columns for visual features of objects in monkey inferotemporal cortex, Nature, 360:343–346. Gauthier, I., M. J. Tarr, A. W. Anderson, P. Skudlarski, and J. C. Gore, 1999. Activation of the middle fusiform “face area” increases with expertise in recognizing novel objects, Nat. Neurosci., 2:568–573. Georges-François, P., E. T. Rolls, and R. G. Robertson, 1999. Spatial view cells in the primate hippocampus: allocentric view not head direction or eye position or place, Cereb. Cortex, 9:197–212. Gorno-Tempini, M. L., and C. J. Price, 2001. Identification of famous faces and buildings: a functional neuroimaging study of semantically unique items, Brain, 124:2087–2097. Gouteux, S., C. Thinus-Blanc, and J. Vauclair, 2001. Rhesus monkeys use geometric and nongeometric information during a reorientation task, J. Exp. Psychol. Gen., 130:505–519. Grill-Spector, K., Z. Kourtzi, and N. Kanwisher, 2001. The lateral occipital complex and its role in object recognition, Vis. Res., 41:1409–1422. Grill-Spector, K., T. Kushnir, S. Edelman, G. Avidan-Carmel, Y. Itzchak, and R. Malach, 1999. Differential processing of objects under various viewing conditions in the human lateral occipital complex, Neuron, 24:187–203. Grill-Spector, K., T. Kushnir, S. Edelman, Y. Itzchak, and R. Malach, 1998a. Cue-invariant activation in object-related areas of the human occipital lobe, Neuron, 21:191–202. Grill-Spector, K., T. Kushnir, T. Hendler, S. Edelman, Y. Itzchak, and R. Malach, 1998b. A sequence of object-processing stages revealed by fMRI in the human occipital lobe, Hum. Brain Mapping, 6:316–328. Grill-Spector, K., T. Kushnir, T. Hendler, and R. Malach, 2000. The dynamics of object-selective activation correlate with recognition performance in humans, Nat. Neurosci., 3:837– 843.

1188

, ,   

Grossman, E., M. Donnelly, R. Price, D. Pickens, V. Morgan, G. Neighbor, and R. Blake, 2000. Brain areas involved in perception of biological motion, J. Cogn. Neurosci., 12:711–720. Habib, M., and A. Sirigu, 1987. Pure topographical disorientation: a definition and anatomical basis, Cortex, 23:73–85, 1987. Halgren, E., A. M. Dale, M. I. Sereno, R. B. H. Tootell, K. Marinkovic, and B. R. Rosen, 1999. Location of human face-selective cortex with respect to retinotopic areas, Hum. Brain Mapping, 7:29–37. Hasson, U., T. Hendler, D. Ben Bashat, and R. Malach, 2001. Vase or face? A neural correlate of shape-selective grouping processes in the human brain, J. Cogn. Neurosci., 13:744–753. Haxby, J. V., M. I. Gobbini, M. L. Furey, A. Ishai, J. L. Schouten, and P. Pietrini, 2001. Distributed and overlapping representations of faces and objects in ventral temporal cortex, Science, 293:2425–2430. Haxby, J. V., C. L. Grady, B. Horwitz, L. G. Ungerleider, M. Mishkin, R. E. Carson, P. Herscovitch, M. B. Schapiro, and S. I. Rapoport, 1991. Dissociation of object and spatial visual processing pathways in human extrastriate cortex, Proc. Natl. Acad. Sci. USA, 88:1621–1625. Haxby, J. V., L. G. Ungerleider, V. P. Clark, J. L. Schouten, E. A. Hoffman, and A. Martin, 1999. The effect of face inversion on activity in human neural systems for face and object perception, Neuron, 22:189–199. Henke, K., S. R. Schweinberger, A. Grigo, T. Klos, and W. Sommer, 1998. Specificity of face recognition: recognition of exemplars of non-face objects in prosopagnosia, Cortex, 34:289– 296. Hermer, L., and E. S. Spelke, 1994. A geometric process for spatial reorientation in young children, Nature, 370:57–59. Hoffman, E. A., and J. V. Haxby, 2000. Distinct representations of eye gaze and identity in the distributed human neural system for face perception, Nat. Neurosci, 3:80–84. Ishai, A., L. Ungerleider, A. Martin, J. L. Schouten, and J. V. Haxby, 1999. Distributed representation of objects in the human central visual pathway, PNAS, 96:9379–9384. Jacobs, R., 1997. Nature, nurture, and the development of functional specializations: a computational approach, Psychological Bulletin and Review, 4:299–309. Jacobs, R. A., 1999. Computational studies of the development of functionally specialized neural modules, Trends in Cognitive Sciences, 3:31–38. James, T. W., G. K. Humphrey, J. S. Gati, R. S. Menon, and M. A. Goodale, 2000. The effects of visual object priming on brain activation before and after recognition, Curr. Biol., 10:1017– 1024. Johnson, M. H., S. Dziurawiec, H. Ellis, and J. Morton, 1991. Newborns’ preferential tracking of face-like stimuli and its subsequent decline, Cognition, 40(1–2):1–19. Kanwisher, N., 2000. Domain specificity in face perception, Nat. Neurosci., 3:759–763. Kanwisher, N., J. McDermott, and M. Chun, 1997. The fusiform face area: a module in human extrastriate cortex specialized for face perception, J. Neurosci., 1711:4302–4311. Kanwisher, N., D. Stanley, and A. Harris, 1999. The fusiform face area is selective for faces, not animals, NeuroReport, 10:183–187. Kanwisher, N., F. Tong, and K. Nakayama, 1998. The effect of face inversion on the human fusiform face area, Cognition, 68:B1–B11. Kanwisher, N., R. Woods, M. Iacoboni, and J. Mazziotta, 1996. A locus in human extrastriate cortex for visual shape analysis, JOCN, 91:133–142.

Kourtzi, Z., and N. Kanwisher, 2001. Shapes, not contours, determine fMRI responses in the human lateral occipital complex, Science, 293:1506–1509. Le Grand, R., C. J. Mondloch, D. Maurer, and H. P. Brent, 2001. Neuroperception. Early visual experience and face processing, Nature, 410:890. Learmonth, A. E., N. S. Newcombe, and J. Huttenlocher, 2001. Toddlers’ use of metric information and landmarks to reorient, J. Exp. Child Psychol., 80:225–244. Lerner, Y., T. Hendler, D. Ben-Bashat, M. Harel, and R. Malach, 2001. A hierarchical axis of object processing stages in the human visual cortex, Cereb. Cortex, 114:287–297. Lerner, Y., T. Hendler, and R. Malach, 2002. Object-completion effects in the human lateral occipital complex, Cereb. Cortex, 12: 163–177. Maguire, E. A., N. Burgess, J. G. Donnett, R. S. J. Frackowiak, C. D. Frith, and J. O’Keefe, 1998. Knowing where and getting there: a human navigational network, Science, 280:921–924. Malach, R., J. B. Reppas, R. B. Benson, K. K. Kwong, H. Jiang, W. A. Kennedy, P. J. Ledden, T. J. Brady, B. R. Rosen, and R. B. H. Tootell, 1995. Object-related activity revealed by functional magnetic resonance imaging in human occipital cortex, Proc. Natl. Acad. Sci. USA, 92:8135–8138. Martin, A., and L. L. Chao, 2001. Semantic memory and the brain: structure and processes, Curr. Opin. Neurobiol., 112:194–201. McCarthy, G., A. Puce, J. C. Gore, and T. Allison, 1997. Facespecific processing in the human fusiform gyrus, J. Cogn. Neurosci., 9:605–610. Moscovitch, M., G. Winocur, and M. Behrmann, 1997. What is special about face recognition? Nineteen experiments on a person with visual object agnosia and dyslexia but normal face recognition, JOCN, 9:555–604. Naccache, L, and S. Dehaene, 2001. The priming method: imaging unconscious repetition priming reveals an abstract representation of number in the parietal lobes, Cereb. Cortex, 1110:966–974. Nunn, J. A., P. Postma, and R. Pearson, 2001. Developmental prosopagnosia: should it be taken at face value? Neurocase, 7:15–27. O’Keefe, J., and L. Nadel, 1978. The Hippocampus as a Cognitive Map, Oxford: Oxford University Press. Perrett, D., E. T. Rolls, and W. Caan, 1982. Visual neurons responsive to faces in the monkey temporal cortex, Exp. Brain Res., 47:329–342. Phillips, M. L., A. W. Young, C. Senior, M. Brammer, C. Andrew, A. J. Calder, E. T. Bullmore, D. I. Perret, D. Rowland, S. C. Williams, J. A. Gray, and A. S. David, 1997. A specific neural substrate for perceiving facial expressions of disgust, Nature, 389:495–498. Pierce, K., R. A. Muller, J. Ambrose, G. Allen, and E. Courchesne, 2001. Face processing occurs outside the fusiform “face area” in autism: evidence from functional MRI, Brain, 124(Pt 10):2059– 2073. Polk, T. A., and M. Farah, 1998. The neural development and organization of letter recognition: evidence from functional neuroimaging, computational modeling, and behavioral studies, Proc. Natl. Acad. Sci. USA, 95:847–852. Potter, M. C., 1976. Short-term conceptual memory for pictures, J. Exp. Psychol. Hum. Learning Memory, 5:509–522. Puce, A., T. Allison, M. Asgari, J. C. Gore, and G. McCarthy, 1996. Differential sensitivity of human visual cortex to faces, letter-

strings, and textures: a functional MRI study, J. Neurosci., 16: 5205–5215. Puce, A., T. Allison, J. C. Gore, and G. McCarthy, 1995. Face perception in extrastriate cortex studied by functional MRI, J. Neurophysiol., 74:1192–1199. Rossion, B., L. Dricot, A. Devolder, J. M. Bodart, M. Crommelinck, B. De Gelder, and R. Zoontjes, 2000. Hemispheric asymmetries for whole-based and part-based face processing in the human fusiform gyrus, J. Cogn. Neurosci., 12:793–802. Schultz, R. T., I. Gauthier, A. Klin, R. K. Fulbright, A. W. Anderson, F. Volkmar, P. Skudlarski, C. Lacadie, D. J. Cohen, and J. C. Gore, 2000. Abnormal ventral temporal cortical activity during face discrimination among individuals with autism and Asperger syndrome, Arch. Gen. Psychiatry, 57:331– 340. Sergent, J., S. Ohta, and B. MacDonald, 1992. Functional neuroanatomy of face and object processing: a positron emission tomography study, Brain, 115:15–36. Shah, N. J., J. C. Marshall, O. Zafiris, A. Schwab, K. Zilles, H. J. Markowitsch, and G. R. Fink, 2001. The neural correlates of person familiarity. A functional magnetic resonance imaging study with clinical implications, Brain, 124(Pt 4):804–815. Spiridon, M., and N. Kanwisher, 2002. How distributed is visual category information in human occipito-temporal cortex? An fMRI study, Neuron, 35:1157–1165. Tanaka, K., 1997. Mechanisms of visual object recognition: monkey and human studies, Curr. Opin. Neurobiol., 7:523–529. Tarr, M. J., and I. Gauthier, 2000. FFA: a flexible fusiform area for subordinate-level visual processing automatized by expertise, Nat. Neurosci., 3:764–769. Thorpe, S., D. Fize, and C. Marlot, 1996. Speed of processing in the human visual system, Nature, 381:520–522. Tong, F., K. Nakayama, J. T. Vaughan, and N. Kanwisher, 1998. Binocular rivalry and visual awareness in human extrastriate cortex, Neuron, 21:753–759. Tovee, M. J., E. T. Rolls, A. Treves, and R. P. Bellis, 1993. Information encoding and the responses of single neurons in the primate temporal visual cortex, J. Neurophysiol., 70:640–654. Tsao, D., Y. Sasaki, J. Mandeville, F. Leite, E. Pasztor, L. Wald, A. Dale, and G. Orban, 2001. fMRI reveals face-selective activity in awake behaving macaque, Soc. Neurosci. Abstr., Program #122.2. Vanduffel, W., D. Fize, J. B. Mandeville, K. Nelissen, P. Van Hecke, B. R. Rosen, R. B. Tootell, and G. A. Orban, 2001. Visual motion processing investigated using contrast agent-enhanced fMRI in awake behaving monkeys, Neuron, 32:565–577. Vuilluemier, P., R. N. Henson, J. Driver, and R. J. Dolan, 2002. Multiple levels of visual object constancy revealed by eventrelated fMRI of repetition priming, Nat. Neurosci., 5:491–499. Wada, Y., and T. Yamamoto, 2001. Selective impairment of facial recognition due to a haematoma restricted to the right fusiform and lateral occipital region, J. Neurol., Neurosurg., and Phychiatry, 71:254–257. Wagner, A. D., D. L. Schacter, M. Rotte, W. Koutstaal, A. Maril, A. M. Dale, B. R. Rosen, and R. L. Buckner, 1998. Building memories: remembering and forgetting of verbal experiences as predicted by brain activity, Science, 281:1188–1191. Xu, Y., and N. Kanwisher, 2001. What is the magnitude and theoretical significance of the FFA response to expert stimuli? Soc. Neurosci. Abstr., Prog. #417.9.

:      

1189

X MOTION, DEPTH, AND SPATIAL RELATIONS

80

Motion Cues in Insect Vision and Navigation MANDYAM SRINIVASAN AND SHAOWU ZHANG

A     evading a rapidly descending hand or orchestrating a flawless landing on the rim of a teacup would convince even the most skeptical observer that many insects are not only excellent fliers and navigators, but also possess visual systems that are fast, reliable, precise, and exquisitely sensitive to motion. It is no wonder, then, that there has been considerable interest in trying to fathom how insects detect, evaluate, and use motion cues in their daily lives. Early studies of the analysis of image motion by insects concentrated on the optomotor response (reviewed by Reichardt, 1969). An insect, flying tethered inside a striped drum, will tend to turn in the direction in which the drum is rotated. If the drum rotates clockwise, the insect will generate a yaw torque in the clockwise direction, and vice versa. This reaction helps the insect maintain a straight course by compensating for undesired deviations: a gust of wind that causes the insect to veer to the left, for example, would create rightward image motion on the eyes and cause the insect to generate a compensatory yaw to the right. Investigation of this optomotor response over several decades has provided valuable information on some of the characteristics of motion perception by the insect visual system (Borst and Egelhaaf, 1989; Buchner, 1984; Reichardt, 1969). More recent studies, carried out primarily with freely flying honeybees, have revealed a number of additional contexts in which image motion is analyzed to coordinate flight and to obtain a useful percept of the world. It appears, for example, that bees analyze motion cues in a variety of different ways for negotiating narrow gaps, estimating the distances to objects, avoiding obstacles, controlling flight speed, executing smooth landings, and gauging the distance traveled. Here we describe some of these strategies and attempt to elucidate the properties of the underlying motion-sensitive mechanisms. Unlike vertebrates, insects have immobile eyes with fixedfocus optics. Therefore, they cannot infer the distance of an object from the extent to which the directions of gaze must converge to view the object or by monitoring the refractive power that is required to bring the image of the object into focus on the retina. Furthermore, compared with human eyes, the eyes of insects are positioned much closer together and possess inferior spatial acuity. Therefore, even if an

insect possessed the neural apparatus required for binocular stereopsis, such a mechanism would be relatively imprecise and restricted to measuring ranges of only a few centimetres (Collett and Harkness, 1982; Horridge, 1987; Rossell, 1983; Srinivasan, 1993). Not surprisingly, insects have evolved alternative visual strategies for guiding locomotion and for seeing the world in three dimensions. Many of these strategies rely on using cues derived from the image motion that the animal experiences when it moves in its environment. Some of these cues are outlined below, and references to more complete accounts are provided.

Peering insects Over 100 years ago, Exner (1891), pondering the eyestalk movements of crabs, speculated that invertebrates might use image motion to estimate object range. However, the first clear evidence to support this conjecture did not arrive until the middle of the twentieth century, when Wallace (1959) made the astute observation that a locust sways its head from side to side before jumping onto a nearby object (Fig. 80.1A). Wallace hypothesised that this peering motion, typically 5 to 10 mm in amplitude, was a strategy for measuring object range. To test this hypothesis, he presented a locust with two objects subtending the same visual angle. One object was relatively small and was placed close to the locust, while the other was larger and situated farther away. He found that the locust, after peering, jumped almost invariably to the nearer object. In a further series of elegant experiments, recently confirmed more quantitatively by Sobel (1990), a target was oscillated from side to side, in synchrony with the insect’s peering movements. When the target was oscillated out of phase with the movement of the head, thereby increasing the speed and amplitude of the object’s image on the retina, the locust consistently underestimated the range of the target (Fig. 80.1C ). On the other hand, when the target was oscillated in phase with the head, it consistently overestimated the range (Fig. 80.1B). Thus, reduced image motion of the target caused the insect to overestimate the target’s range, while increased motion had the opposite effect. These findings demonstrated convincingly that the peering locust was estimating the range of the target in terms of the speed of the

1193

F 80.1. Experiments investigating how locusts measure the range of a target by peering, that is, moving the head from side to side. Range is estimated correctly when the target is stationary (A), overestimated when the target is moved in the same direction as the head (B), and underestimated when it is moved in the opposite direction (C). Thus, the range of the target is estimated in terms the motion of the target’s image during the peer. (Adapted from Sobel, 1990.)

image on the retina. It is now known that certain other insects, such as grasshoppers (Eriksson, 1980) and mantids (Horridge, 1986; Kral, 1998; Kral and Poteser, 1997; Poteser et al., 1998), also use peering to measure object range.

Flying insects Peering, however, is practical only when an insect is not locomoting. Are flying insects capable of gleaning range information from image motion, and if so, how do they accomplish this? Stable flight in a straight line would seem to be a prerequisite for extracting information on range (Horridge, 1987; Srinivasan, 1993). Research over the past 50 years has uncovered a number of different ways in which insects use image motion to stabilize flight control and to extract useful information about the environment. We shall begin by considering strategies for visual control and stabilization of flight and then examine the ways in which image motion is used to glean information about the structure of the environment, and about the insect’s movement within it. S F For insects, vision provides an important sensory input for the stabilization of flight. If an insect flying along a straight line is blown to the left by a gust of wind, the image on its frontal retina moves to the right. This causes the flight motor system to generate a corrective yaw torque, which brings the insect back on course (reviewed by Reichardt, 1969). Similar control mechanisms act to stabilize pitch and roll (e.g., Srinivasan, 1977). This optomotor response (Reichardt, 1969) has provided an excellent exper-

1194

, ,   

imental paradigm with which to probe the neural mechanisms underlying motion detection. Largely through studies of the optomotor response in flies, we now know that the direction of image movement is sensed by correlating the intensity variations registered by neighboring ommatidia, or facets, of the compound eye (reviewed by Reichardt, 1969). Research over the past 30 years has uncovered the existence of a number of motion-sensitive neurons with large visual fields, each responding preferentially to motion in a specific direction (Hausen, 1993; reviewed by Hausen and Egelhaaf, 1989) or to rotation of the fly about a specific axis (Krapp and Hengstenberg, 1996). These neurons are likely to play an important role in stabilizing flight and providing the fly with a visually kinesthetic sense. Their properties have been reviewed extensively (e.g., Egelhaaf and Borst, 1993; Hausen, 1993; Hausen and Egelhaaf, 1989), and we shall not repeat them here. H Hoverflies and certain species of bee display an impressive ability to hold a rigid position in midair, compensating almost perfectly for wind gusts and other disturbances. Kelber and Zeil (1997) investigated hovering in a species of stingless bee, Tetragonisca angustula. Guard bees of this species hover stably in watch near the entrance to their nest, protecting it from intruders. To investigate the visual stabilizing mechanisms, Kelber and Zeil acclimated the bees to the presence of a spiral pattern mounted on the vertical face of the hive, surrounding the entrance. When the spiral was briefly rotated to simulate expansion, the hovering guard bees darted away from the focus of apparent expansion;

when the spiral was rotated to simulate contraction, they moved toward the focus of contraction. These responses were always directed toward or away from the nest entrance, irrespective of the bee’s orientation and therefore irrespective of the region of the eye that experienced the experimentally imposed pattern of image motion. Clearly, then, these creatures were interpreting expansion and contraction of the image as unintended movements toward or away from the nest entrance and were compensating for them. N N G When a bee flies through a hole in a window, it tends to fly through its center, balancing the distances to the left and right boundaries of the opening. How does it gauge and balance the distances to the two rims? One possibility is that it does not measure distances at all, but simply balances the speeds of image motion on the two eyes as it flies through the opening. To investigate this possibility, Kirchner and Srinivasan (1989) trained bees to enter an apparatus that offered a reward of sugar solution at the end of a tunnel. Each side wall carried a pattern consisting of a vertical black-and-white grating (Fig. 80.2). The grating on one wall could be moved horizontally at any desired speed, either toward the reward or away from it. After the bees had received several rewards with the gratings stationary, they were filmed from above as they flew along the tunnel. When both gratings were stationary, the bees tended to fly along the midline of the tunnel, that is, equidistant from the two walls (Fig. 80.2A). But when one of the gratings was moved at a constant speed in the direction of the bees’ flight—thereby reducing the speed of retinal image motion on that eye relative to the other eye—the bees’ trajectories shifted toward the side of the moving grating (Fig. 80.2B). When the grating moved in a direction opposite to that of the bees’ flight—thereby increasing the speed of retinal image motion on that eye relative to the other—the bees’ trajectories shifted away from the side of the moving grating (Fig. 80.2C ). These findings demonstrate that when the walls were stationary, the bees maintained equidistance by balancing the speeds of the retinal images in the two eyes. A lower image speed on one eye evidently caused the bee to move closer to the wall seen by that eye. A higher image speed had the opposite effect. Were the bees really measuring and balancing image speeds on the two sides as they flew along the tunnel or were they simply balancing the contrast frequencies produced by the succession of dark and light bars of the gratings? This question was investigated by analyzing the flight trajectories of bees when the two walls carried gratings of different spatial periods. When the gratings were stationary, the trajectories were always equidistant from the two walls, even when the spatial frequencies of the gratings on the two sides—and therefore the contrast frequencies experienced

F 80.2. Experiment investigating how bees fly through the middle of a tunnel (the centering response). Bees are trained to fly through a tunnel 40 cm long, 12 cm wide, and 20 cm high to collect a reward placed at the far end. The flanking walls of the tunnel are lined with vertical black-and-white gratings with period of 5 cm. The flight trajectories of bees, as recorded by a video camera positioned above the tunnel, are shown in A to F. In each panel the shaded area represents the mean and standard deviation of the positions of the flight trajectories, analyzed from recordings of several hundred flights. The dark bars represent the black stripes of the patterns on the walls. The small arrow indicates the direction of bee flight and the large arrow the direction of pattern movement. When the patterns on the walls are stationary, bees tend to fly close to the midline of the tunnel (A, D). When the pattern on one of the walls is in motion, however, bees tend to fly closer to that wall if the pattern moves in the same direction as the bee (B, E ) and farther away from that wall if the pattern moves in the opposite direction (C, F ). These results indicate that bees balance the distances to the walls of the tunnel by balancing the speeds of image motion that are experienced by the two eyes, and that they are able to measure image speed rather independently of the spatial structure of the image. (Modified from Srinivasan et al., 1991.)

  :    

1195

by the two eyes—differed by a factor of as much as 4 (Fig. 80.2D). When one of the gratings was in motion, the trajectories shifted toward or away from the moving grating (as described above) according to whether the grating moved with or against the direction of the bees’ flight (Figs. 80.2E, 80.2F ). These results indicate that the bees were indeed balancing the speeds of the retinal images on the two eyes and not the contrast frequencies. The above findings are true irrespective of whether the gratings possess square-wave intensity profiles (with abrupt changes of intensity) or sinusoidal profiles (with gradual intensity changes) and irrespective of whether the contrasts of the gratings on the two sides are equal or considerably different (Srinivasan et al., 1991). Further experiments have revealed that when the velocities of the bee and the pattern are known, it is even possible to predict the position of a bee’s flight trajectory along the width of the tunnel, on the assumption that the bee balances the apparent angular velocities on either side of the tunnel (Srinivasan et al., 1991). These findings suggest that the bee’s visual system is capable of computing the apparent angular speed of a grating independently of its contrast and spatialfrequency content. Subsequent studies (Srinivasan and Zhang, 1997; Srinivasan et al., 1993) have investigated this centering response further by comparing its properties with those of the well-known optomotor response in an experimental setup which allows the two responses to be compared in the same individual under the same conditions. The results indicate that the centering response differs from the optomotor response in three respects. First, the centering response is sensitive primarily to the angular speed of the stimulus, regardless of its spatial structure. The optomotor response, on the other hand, is sensitive primarily to the temporal frequency of the stimulus; therefore, it confounds the angular velocity of a striped pattern with its spatial period. Second, the centering response is nondirectional, while the optomotor response is directionally selective. Third, the centering response is sensitive to higher temporal frequencies than is the optomotor response. Whereas the optomotor response exhibits a relatively low bandwidth (with half-magnitude points at 6 and 75 Hz), the centering response exhibits a relatively high bandwidth (with half-magnitude points at 3 Hz and well beyond 100 Hz). Thus, the motion-detecting processes underlying the centering response exhibit properties that are substantially different from those that mediate the wellknown optomotor response (Srinivasan and Zhang, 1997; Srinivasan et al., 1993). Models of movement-detecting mechanisms underlying the centering response are described in Srinivasan et al. (1999). Given that the role of the centering response is to ensure that the insect flies through the middle of a gap irrespective of the texture of the side walls, it is easy to see why this response is mediated by a movement-detecting system

1196

, ,   

which measures the angular speed of the image independently of its spatial structure. The movement-detecting system that subserves the optomotor response, on the other hand, does not need to measure image speed accurately: it merely needs to signal the direction of image motion reliably so that a corrective yaw of the appropriate polarity may be generated. C F S Do insects control the speed of their flight, and, if so, how? Work by David (1982) and by Srinivasan et al. (1996) suggests that flight speed is controlled by monitoring the velocity of the image of the environment. David (1982) observed fruitflies flying upstream in a wind tunnel, attracted by the odor of fermenting banana. The walls of the cylindrical wind tunnel were decorated with a helical black-and-white striped pattern so that rotation of the cylinder about its axis produced apparent movement of the pattern toward the front or the back. With this setup, the rotational speed of the cylinder (and hence the speed of the backward motion of the pattern) could be adjusted such that the fly was stationary (i.e., did not move along the axis of the tunnel). The apparent backward speed of the pattern then revealed the ground speed that the fly was “choosing” to maintain, as well as the angular velocity of the image of the pattern on the flies’ eyes. In this setup, fruitflies tended to hold the angular velocity of the image constant. Increasing or decreasing the speed of the pattern caused the flies to move backward or forward (respectively) along the tunnel at a rate such that the angular velocity of the image on the eye was always “clamped” at a fixed value. The flies also compensated for headwind in the tunnel, increasing or decreasing their thrust to maintain the same apparent ground speed (as indicated by the angular velocity of image motion on the eye). Experiments in which the angular period of the stripes was varied revealed that the flies were measuring (and holding constant) the angular velocity of the image on the eye, irrespective of the spatial structure of the image. Bees appear to use a similar strategy to regulate flight speed (Srinivasan et al., 1996). When a bee flies through a tapered tunnel, it decreases its flight speed as the tunnel narrows to keep the angular velocity of the image of the walls, as seen by the eye, constant at about 320 deg/sec (Fig. 80.3). This suggests that flight speed is controlled by monitoring and regulating the angular velocity of the image of the environment on the eye. (That is, if the width of the tunnel is doubled, the bee flies twice as fast.) On the other hand, a bee flying through a tunnel of uniform width does not change its speed when the spatial period of the stripes lining the walls is abruptly changed (Srinivasan et al., 1996). Thus, flight speed is regulated by a visual motiondetecting mechanism that measures the angular velocity of the image largely independently of its spatial structure. In

slow down to a safer speed when negotiating a narrow passage.

F 80.3. Experiment investigating visual control of flight speed. A, Bees are trained to fly through a tapered tunnel to collect a reward placed at the far end. The walls of the tunnel are lined with vertical black-and-white gratings with a period of 6 cm. B, A typical flight trajectory, as filmed from above by a video camera, where the bee’s position and orientation are shown every 50 msec. C, Mean and standard deviation of flight speeds measured at various positions along the tunnel (data from 18 flights). The Vshaped line shows the theoretically expected flight speed profile if the bees were to hold the angular velocity of the images of the walls constant at 320 deg/sec as they fly through the tunnel. The data indicate that bees control flight speed by holding constant the angular velocity of the image of the environment. (Adapted from Srinivasan et al., 1996.)

this respect, the speed-regulating system is similar to the centering system. However, it is not known whether the regulation of flight speed in bees is mediated by a directionally selective movement-detecting mechanism or a nondirectional one. An obvious advantage of controlling flight speed by regulating image speed is that the insect would automatically

E  D F It is well known that honeybees can navigate accurately and repeatedly to a food source. It is also established that bees communicate to their nestmates the distance and direction in which to fly to reach it through the famous waggle dance (von Frisch, 1993). But the cues by which bees gauge the distance flown to the goal have been a subject of controversy. A few years ago, Esch and Burns (1995, 1996) investigated distance measurement by enticing honeybees to find food at a feeder placed 70 m away from a hive in an open field and recording the perceived distance as signaled by the bees when they danced to recruit other nestmates in the hive. When the feeder was 70 m away, the bees signaled 70 m— the correct distance. But when the feeder was raised above the ground by attaching it to a helium balloon, the bees signaled a progressively shorter distance as the height of the balloon was increased. This occurred despite the fact that the balloon was now farther away from the hive! Esch and Burns explained this finding by proposing that the bees were gauging distance flown in terms of the motion of the image of the ground below, rather than, for example, through the energy consumed to reach the feeder. The higher the balloon, the lower the total amount of image motion that the bees experienced en route to the feeder. This hypothesis was examined by Srinivasan et al. (1996, 1997), who investigated the cues by which bees estimate and learn distances flown under controlled laboratory conditions. Bees were trained to enter a 3.2 m long tunnel and collect a reward of sugar solution at a feeder placed in the tunnel at a fixed distance from the entrance. The walls and floor of the tunnel were lined with black-and-white gratings perpendicular to the tunnel’s axis (Fig. 80.4A). During training, the position and orientation of the tunnel were changed frequently to prevent the bees from using any external landmarks to gauge their position relative to the tunnel entrance. The bees were then tested by recording their searching behavior in an identical fresh tunnel that carried no reward and was devoid of any scent cues. In the tests, these bees showed a clear ability to search for the reward at the correct distance, as indicated by the search distribution labeled by the squares in Figure 80.4B. How were the bees gauging the distance they had flown in the tunnel? Tests were carried out to examine the participation of a variety of potential cues, including energy consumption, time of flight, airspeed integration, and inertial navigation (Srinivasan et al., 1997). It turned out that the bees were estimating the distance flown by integrating, over time, the motion of the images of the walls on the eyes as they flew down the tunnel. The crucial experiment was one in which bees were trained and tested in conditions where

  :    

1197

F 80.4. A, Experiment investigating how honeybees gauge distance flown to a food source. Bees are trained to find a food reward placed at a distance of 1.7 m from the entrance of a 3.2 m long tunnel 22 cm wide and 20 cm high. The tunnel is lined with vertical black-and-white gratings with a period of 4 cm. B, When the trained bees are tested in a fresh tunnel with the reward absent, they search at the former location of the feeder, as shown by the bell-shaped search distributions. This is true irrespective of whether the period of the grating is 4 cm (as in the training, square symbols), 8 cm (triangles), or 2 cm (diamonds). The inverted triangle shows the former position of the reward, and the symbols below it depict the mean values of the search distributions in each case. Bees lose their ability to estimate the distance of the feeder when image-motion cues are removed by lining the tunnel with axial (rather than vertical) stripes (circles). These experiments and others (Srinivasan et al., 1997) demonstrate that (1) distance flown is estimated visually by integrating over time the image velocity that is experienced during the flight, and (2) the honeybee’s odometer measures image velocity independently of image structure. (Adapted from Srinivasan et al., 1997.)

1198

, ,   

image motion was eliminated or reduced by using axially oriented stripes on the walls and floor of the tunnel. The bees then showed no ability to gauge the distance traveled: in the tests, they searched uniformly over the entire length of the tunnel, showing no tendency to stop or turn at the former location of the reward (see the search distribution labeled by the circles in Fig. 80.4B). Trained bees tended to search for the feeder at the same position in the tunnel even if the period of the gratings lining the walls and floor was varied in the tests (search distributions labeled by triangles and diamonds in Fig. 80.4B). This indicates that the odometric system reads image velocity accurately over a fourfold variation in the spatial period of the grating. These results, considered together with those of Esch and Burns (1995, 1996), indicate that the bee’s “odometer” is driven by the image motion that is generated in the eyes during translatory flight. Evidently, bees use cues derived from image motion not only to stabilize flight and regulate its speed, but also to infer how far they have flown. We have seen above that the balloon experiment caused bees to underestimate the distance they had flown, because they experienced less image motion than they normally would while cruising to a natural food source. What happens when bees encounter the opposite situation, namely, one in which image motion cues are artificially exaggerated? Srinivasan et al. (2000a) explored this question by training bees to fly directly from their hive into a short, narrow tunnel that was placed very close to the hive entrance. The tunnel was 6.4 m long and 11 cm wide. A feeder was placed 6 m from the entrance. The walls and floor of the tunnel were lined with a random visual texture. The dances of bees returning from this feeder were video-filmed. Incredibly, these bees signaled a flight distance of about 200 m, despite the fact that they had flown only a small fraction of this distance. Evidently, the bees were grossly overestimating the distance they had flown in the tunnel, because the proximity of the walls and floor of the tunnel greatly magnified the image motion that they experienced, in comparison with what would normally occur when foraging outdoors (Esch et al., 2001). This experiment again drives home the point that image motion is the dominant cue that bees use to gauge how far they have traveled. E S L How does a bee execute a smooth touchdown on a surface? An approach that is perpendicular to the surface would generate strong looming (image expansion) cues which could, in principle, be used to decelerate flight at the appropriate moment. Indeed, work by Wagner (1982) and Borst and Bahde (1988) has shown that deceleration and extension of the legs in preparation for landing are triggered by movement-detecting mechanisms that sense the expansion of the image. Looming cues are weak, however, when a bee performs a grazing landing on

F 80.5. Typical results of an experiment investigating how bees make a grazing landing on a horizontal surface. A, B, Variation of forward flight speed (Vf ) with height (h) above the surface for two landing trajectories. C, D, Variation of descent speed (Vf ) with height (h) above the surface for two landing trajectories. The landing bee holds the angular velocity of the image of the ground

constant at 241 deg/sec in A and at 655 deg/sec in B, as calculated from the slopes of the linear regression lines. Also shown are the values of the correlation coefficient (r). Holding the image velocity of the ground constant during the approach automatically ensures that the landing speed is zero at touchdown. (Adapted from Srinivasan et al., 2000b.)

a surface. By grazing landings we mean landings whose trajectories are inclined to the surface at an angle that is considerably less than 45 degrees. In such landings, the motion of the image of the surface would be dominated by a strong translatory component in the front-to-back direction in the ventral visual field of the eye. To investigate how bees execute grazing landings, Srinivasan et al. (1996, 2000b, 2001) trained bees to collect a reward of sugar water on a textured horizontal surface. The reward was then removed, and the landings that the bees made on the surface in search of the food were videofilmed in three dimensions. Analysis of the landing trajectories revealed that the flight speed of the bee decreases steadily as it approaches the surface. In fact, the forward speed as well as the descent speed are approximately proportional to the height above the surface (Fig. 80.5), indicating that the bee is holding the angular velocity of the image of the surface approximately constant as the surface is approached. This strategy automatically ensures that both the forward and descent speeds are close to zero at touchdown. Thus, a smooth landing is achieved by an elegant and surprisingly simple process that

does not require explicit knowledge of the bee’s instantaneous speed or height (Srinivasan et al., 2000b). D O  D D The experiments described above show that bees stabilize flight, negotiate narrow passages, and orchestrate smooth landings by using what seem to be a series of simple, low-level visual reflexes. But they do not tell us whether flying bees see the world in three dimensions in the way we do. Do bees perceive the world as being composed of objects and surfaces at various ranges? While this is a difficult question—one that a philosopher might even declare unanswerable—one can at least ask whether bees can be trained to distinguish between objects at different distances. Lehrer et al. (1988) trained bees to fly over an artificial meadow and distinguish between artificial flowers at various heights. The training was carried out by associating a reward with a flower at a particular height. The sizes and positions of the flowers were varied randomly and frequently during the training. This ensured that the bees were trained to associate only the height of the flower (or, more accurately, the distance from the eye), and not its position, or angular subtense, with the reward. Using

  :    

1199

this approach—details of which are described in Srinivasan et al. (1989)—it was possible to train bees to choose either the highest flower, the lowest flower, or even one at an intermediate height. Clearly, then, the bees were able to distinguish flowers at different heights. Under the experimental conditions, the only cue that a bee could have used to gauge the height of each flower was the speed of the flower’s image as the bee flew over it: the taller the flower, the faster the motion of its image. Kirchner and Lengler (1994) extended this meadow experiment by training bees to distinguish the heights of artificial flowers that carried spiral patterns. Six flowers were presented at the same height, while a seventh was either higher (in one experiment) or lower (in another experiment). Bees trained in this way were tested with a constellation of three identical spiral-bearing flowers of the same height. One test flower was stationary, one was rotated to simulate expansion, and one was rotated to simulate contraction. Bees that had learned to find the higher flower in the training chose the “expanding” flower in the test, whereas bees that had learned to choose the lower flower in the training chose the “contracting” flower. For a bee flying above the flowers and approaching the edge of one of them, the expanding flower produced a higher image motion at its boundary than did the stationary one and was evidently interpreted to be the higher flower. The contracting flower, on the other hand, produced a lower image motion and was therefore taken to be the lower one. This experiment confirms the notion that image motion is an important cue in establishing the relative distances of objects. D O  B In all of the work described above, the objects that were being viewed were readily visible to the insects, since they presented a strong contrast—in luminance or color—against a structureless background. What happens if the luminance or color contrast is removed and replaced by motion contrast? To the human eye, a textured figure is invisible when it is presented motionless against a similarly textured background. But the figure pops out as soon as it is moved relative to the background. This type of relative motion, termed motion parallax, can be used to distinguish a nearby object from a remote background. Is an insect capable of distinguishing a textured figure from a similarly textured background purely on the basis of motion parallax? In a series of pioneering experiments, Reichardt and his colleagues in Tübingen showed that a fly is indeed capable of such figure-ground discrimination (Egelhaaf et al., 1988; Reichardt and Poggio, 1979). A tethered, flying fly will show no sign of detecting a textured figure when the figure oscillates in synchrony with a similarly textured background. But it will react to the figure by turning toward it when the figure moves incoherently with respect to the background.

1200

, ,   

F 80.6. Experiment investigating the ability of bees to use motion parallax cues to distinguish a figure from a similarly textured background. A, The apparatus presents a textured disc 6 cm in diameter positioned under a sheet of clear perspex at a height h cm above a similarly textured background (42 ¥ 30 cm; pixel size 5 ¥ 5 mm). The disc is shown as being brighter than the background only for clarity. B, Bars show percentages of landings occurring within the disc for various heights (h) of the disc above the background. The detectability of the disc is reduced as h decreases, reaching the random-hit level (horizontal line) when h = 0, that is, when there is no motion parallax. The inset shows a sample of the distribution of landings of trained bees on the perspex sheet when h = 5 cm. (Adapted from Srinivasan et al., 1990.)

In antipodean Canberra, this question was approached in a different way. Srinivasan et al. (1990) examined whether freely flying bees could be trained to find a textured figure when it was presented raised over a background of the same texture. The figure was a disc bearing a random black-andwhite Julesz texture (a texture composed of random black and white pixels, as shown in Fig. 80.6A). The disc was attached to the underside of a transparent perspex sheet which could be placed at any desired height above the background (Fig. 80.6A). It was found that bees could indeed be trained to find the figure and land on it, provided that the figure was raised at least 1 cm above the background (Fig. 80.6B). When the figure was placed directly on the background, the bees failed to find it (Srinivasan et al., 1990), demonstrating that the cue used to locate the figure is the relative motion between the images of the figure and the

background, caused by the bees’ own flight above the setup. Video films of the bees’ landings showed that when the disc was visible to the bees, they did not land at random on it: rather, they landed primarily near the boundary of the disc, facing the visual “cliff ” (Fig. 80.6B). These experiments showed that the boundary has special visual significance and that bees are capable of detecting it reliably. Kern et al. (1997) have shown, through behavioral experiments and modeling, that the detectability of such boundaries can be well accounted for by a neural network which compares image motion in spatially adjacent receptive fields. The ability to detect objects through the discontinuities in image motion that occur at the boundaries is likely to be important when an insect attempts to land on a leaf or a shrub. This is a situation where it may be difficult to distinguish individual leaves or establish which leaf is nearest, since cues based on contrast in luminance or color are weak. Visual problems of this nature are not restricted to insects. Over 130 years ago, Helmholtz (1866) speculated that humans might use cues derived from image motion in a similar way to distinguish individual trees in a dense forest.

Concluding remarks Insects are prime subjects for studying the ways in which cues based on image motion are extracted and used by visual systems. This is because these creatures, possessing poor or no stereopsis, literally need to move in order to see the world in three dimensions. We now know that insects use information on image motion in a variety of visually mediated functions and behaviors that extend well beyond the classically studied optomotor response. Flying insects exploit such cues to stabilize flight, regulate flight speed, negotiate narrow gaps, infer the ranges to objects, avoid obstacles, orchestrate smooth landings, distinguish objects from backgrounds, and monitor the distance traveled. The emerging picture shows that there are a number of motion-sensitive pathways in the insect visual system, each with a distinct set of properties and geared to a specific visual function. It would seem unlikely, however, that all of these systems (and other, still undiscovered ones) are operative all of the time. Obviously, the optomotor system would have to be switched off, or its corrective commands ignored, when the insect makes a voluntary turn or chases a target (Heisenberg and Wolf, 1993; Kirschfeld, 1997; Srinivasan and Bernard, 1977). There is also evidence, for example, that the optomotor system is at least partially nonfunctional when insects fly through narrow gaps (Srinivasan et al., 1993). One major challenge for the future, then, is to discover the conditions under which individual systems are called into play or ignored, and to understand the ways in which these systems interact to coordinate flight. Another challenge is to uncover the neural mechanisms that underlie these visual capacities.

Acknowledgments Some of the work described in this review was supported by the International Human Frontier Science Program (Grant RG-84/97), the U.S. Defense Advanced Research Projects Agency and the Office of Naval Research (Grant N0001499-1-0506), and the Australian-German Joint Research Cooperation Scheme. REFERENCES Borst, A., and S. Bahde, 1988. Visual information processing in the fly’s landing system, J. Comp. Physiol. A, 163:167–173. Borst, A., and M. Egelhaaf, 1989. Principles of visual motion detection, Trends Neurosci., 12:297–306. Buchner, E., 1984. Behavioral analysis of spatial vision in insects, in Photoreception and Vision in Invertebrates (M. A. Ali, ed.), New York: Plenum Press, pp. 561–621. Collett, T. S., and L. I. K. Harkness, 1982. Depth vision in animals, in Analysis of Visual Behavior (D. J. Ingle, M. A. Goodale, and R. J. W. Mansfield, eds.), Cambridge, MA: MIT Press, pp. 111–176. David, C. T., 1982. Compensation for height in the control of groundspeed by Drosophila in a new, “Barber’s Pole” wind tunnel, J. Comp. Physiol., 147:485–493. Egelhaaf, M., and A. Borst, 1993. Movement detection in arthropods, in Visual Motion and Its Role in the Stabilization of Gaze (F. A. Miles and J. Wallman, eds.), Amsterdam: Elsevier, pp. 203– 235. Egelhaaf, M., K. Hausen, W. Reichardt, and C. Wehrhahn, 1988. Visual course control in flies relies on neuronal computation of object and background motion, Trends Neurosci., 11:351–358. Eriksson, E. S., 1980. Movement parallax and distance perception in the grasshopper (Phaulacridium vittatum), J. Exp. Biol., 86: 337–340. Esch, H., and J. E. Burns, 1995. Honeybees use optic flow to measure the distance of a food source, Naturwissenschaften, 82: 38–40. Esch, H., and J. Burns, 1996. Distance estimation by foraging honeybees, J. Exp. Biol., 199:155–162. Esch, H. E., S. W. Zhang, J. Tautz, and M. V. Srinivasan, 2001. Honeybees share their world view with hive mates? Nature (Lond.), 411:581–583. Exner, S., 1891. The Physiology of the Compound Eyes of Insects and Crustaceans (R. C. Hardie, trans.), Berlin and Heidelberg, SpringerVerlag, pp. 130–131. Frisch, K. von, 1993. The Dance Language and Orientation of Bees, Cambridge, MA: Harvard University Press. Hausen, K., 1993. The decoding of retinal image flow in insects, in Visual Motion and Its Role in the Stabilization of Gaze (F. A. Miles and J. Wallman, eds.), Amsterdam: Elsevier, pp. 203–235. Hausen, K., and M. Egelhaaf, 1989. Neural mechanisms of visual course control in insects, in Facets of Vision (D. G. Stavenga and R. C. Hardie, eds.), Berlin and Heidelberg: Springer-Verlag, pp. 391–424. Heisenberg, M., and R. Wolf, 1993. The sensory-motor link in motion-dependent flight control of flies, in Visual Motion and Its Role in the Stabilization of Gaze (F. A. Miles and J. Wallman, eds.), Amsterdam: Elsevier, pp. 265–283. Helmholtz, H. von, 1866. Handbuch der physiologischen Optik, Hamburg: Voss Verlag ( J. P. C. Southall, trans., 1924; reprinted New York: Dover, 1962).

  :    

1201

Horridge, G. A., 1986. A theory of insect vision: velocity parallax, Proc. R. Soc. Lond. B, 229:13–27. Horridge, G. A., 1987. The evolution of visual processing and the construction of seeing systems, Proc. R. Soc. Lond. B, 230:279– 292. Kelber, A., and J. Zeil, 1997. Tetragonisca guard bees interpret expanding and contracting patterns as unintended displacement in space, J. Comp. Physiol. A, 181:257–265. Kern, R., M. Egelhaaf, and M. V. Srinivasan, 1997. Edge detection by landing honeybees: behavioural analysis and model simulations of the underlying mechanism, Vis. Res., 37:2103– 2117. Kirchner, W. H., and J. Lengler, 1994. Bees perceive illusionary distance information from rotating spirals, Naturwissenschaften, 81:42–43. Kirchner, W. H., and M. V. Srinivasan, 1989. Freely flying honeybees use image motion to estimate object distance, Naturwissenschaften, 76:281–282. Kirschfeld, K., 1997. Course control and tracking: orientation through image stabilization, in Orientation and Communication in Arthropods (M. Lehrer ed.), Basel: Birkhäuser Verlag, pp. 67– 93. Kral, K., 1998. Side-to-side head movements to obtain motion depth cues: a short review of research on the praying mantis, Behav. Processes, 43:71–77. Kral, K., and M. Poteser, 1997. Motion parallax as a source of distance information in locusts and mantids, J. Insect Behav., 10:145–163. Krapp, H. G., and R. Hengstenberg, 1996. Estimation of selfmotion by optic flow processing in single visual interneurons, Nature (Lond.), 384:463–466. Lehrer, M., M. V. Srinivasan, S. W. Zhang, and G. A. Horridge, 1988. Motion cues provide the bee’s visual world with a third dimension, Nature (Lond.), 332:356–357. Poteser, M., M.-A. Pabst, and K. Kral, 1998. Proprioceptive contribution to distance estimation by motion parallax in a praying mantid, J. Exp. Biol., 201:1483–1491. Reichardt, W., 1969. Movement perception in insects, in Processing of Optical Data by Organisms and by Machines (W. Reichardt ed.), New York: Academic Press, pp. 465–493. Reichardt, W., and T. Poggio, 1979. Figure-ground discrimination by relative movement in the visual system of the fly. Part I: experimental results, Biol. Cybern., 35:81–100. Rossell, S., 1983. Binocular stereopsis in an insect, Nature (Lond.), 302:821–822. Sobel, E. C., 1990. The locust’s use of motion parallax to measure distance, J. Comp. Physiol. A, 167:579–588.

1202

, ,   

Srinivasan, M. V., 1977. A visually-evoked roll response in the housefly: open-loop and closed-loop studies, J. Comp. Physiol., 119:1–14. Srinivasan, M. V., 1993. How insects infer range from visual motion, in Visual Motion and Its Role in the Stabilization of Gaze (F. A. Miles and J. Wallman, eds.), Amsterdam: Elsevier, pp. 139–156. Srinivasan, M. V., and G. D. Bernard, 1977. The pursuit response of the housefly and its interaction with the optomotor response, J. Comp. Physiol., 115:101–117. Srinivasan, M. V., M. Lehrer., and G. A. Horridge, 1990. Visual figure-ground discrimination in the honeybee: the role of motion parallax at boundaries, Proc. R. Soc. Lond. B, 238:331– 350. Srinivasan, M. V., M. Lehrer, W. Kirchner, and S. W. Zhang, 1991. Range perception through apparent image speed in freely-flying honeybees, Vis. Neurosci., 6:519–535. Srinivasan, M. V., M. Lehrer, S. W. Zhang, and G. A. Horridge, 1989. How honeybees measure their distance from objects of unknown size, J. Comp. Physiol. A, 165:605–613. Srinivasan, M. V., M. Poteser, and K. Kral, 1999. Motion detection in insect orentation and navigation, Vis. Res., 39:2749– 2766. Srinivasan, M. V., and S. W. Zhang, 1997. Visual control of honeybee flight, in Orientation and Communication in Arthropods (M. Lehrer ed.), Basel: Birkhäuser Verlag, pp. 67–93. Srinivasan, M. V., S. W. Zhang, M. Altwein, and J. Tautz, 2000a. Honeybee navigation: nature and calibration of the “odometer,” Science, 287:851–853. Srinivasan, M. V., S. W. Zhang, and N. Bidwell, 1997. Visually mediated odometry in honeybees, J. Exp. Biol., 200:2513–2522. Srinivasan, M. V., S. W. Zhang, and J. S. Chahl, 2001. Landing strategies in honeybees, and possible applications to autonomous airborne vehicles, Biol. Bull., 200:216–221. Srinivasan, M. V., S. W. Zhang, J. S. Chahl, E. Barth, and S. Venkatesh, 2000b. How honeybees make grazing landings on flat surfaces, Biol. Cybern., 83:171–183. Srinivasan, M. V., S. W. Zhang, and K. Chandrashekara, 1993. Evidence for two distinct movement-detecting mechanisms in insect vision, Naturwissenschaften, 80:38–41. Srinivasan, M. V., S. W. Zhang, M. Lehrer, and T. S. Collett, 1996. Honeybee navigation en route to the goal: visual flight control and odometry, J. Exp. Biol., 199:237–244. Wagner, H., 1982. Flow-field variables trigger landing in flies, Nature (Lond.), 297:147–148. Wallace, G. K., 1959. Visual scanning in the desert locust Schistocerca gregaria, Forskal, J. Exp. Biol., 36:512–525.

81

The Middle Temporal Area: Motion Processing and the Link to Perception KENNETH H. BRITTEN

T   area of the macaque monkey brain (MT, also known as V5) is one of the most studied parts of visual cortex. It has attracted great attention for very good reasons. By far the most important of these is that we have a very good idea of what it does—motion analysis. From the first description of the area (Dubner and Zeki, 1971), the remarkable preponderance of directional selectivity provided a good clue to function. Since then, the area has been the target of widely ranging experimental approaches: lesions, stimulation, anatomy, mapping, and imaging, to name but a few. The resulting large body of work has not overturned the primary conclusion that MT is a seminal area for motion analysis in cortex, but as one might expect, things have gotten more complicated. More complex properties have been elaborated, and new functions in addition to motion analysis have been suggested. The study of MT and motion is a good example of how scientific progress should work. Early descriptive data allowed the formulation of clear functional hypotheses. Tests of these hypotheses allowed the formulation of more elaborate, more correct models that incorporate the principles of the first generation of hypotheses. This is where we stand now: the more complete, more complex second-generation ideas drive current experimental work. In this chapter, I will first provide a brief overview of the classics, attempting, where possible, to fit these data into modern functional models of motion processing. Then I will review the body of work testing hypotheses relating activity in MT to perception. Cortical mechanisms of motion processing have been exhaustively reviewed (Albright and Stoner, 1995; Andersen, 1997; Battaglini et al., 1996; Duffy, 2000; Maunsell and Newsome, 1987; Orban, 1997; Parker and Newsome, 1998), and space does not allow this review to be encyclopedic. Instead, my goals are to capture current views on the function of this well-studied area and to illuminate the value of MT as a model system for studying the relationship between physiology and behavior.

Anatomy MT got its name from its anatomical location in the owl monkey, a New World primate (Allman and Kaas, 1971).

The inappropriateness of this name for its location in the macaque brain (between the occipital and parietal lobes; Fig. 81.1) leads to its alternative, more neutral name, V5. Nonetheless, I will use the original term, MT, throughout, though V5 is entirely equivalent. There is little dispute at present that a homologous area is present in many species of primates and even prosimians (Krubitzer and Kaas, 1990). The area is jointly defined by two anatomical features: dense myelination and direct reciprocal connections with area V1 (Ungerleider and Mishkin, 1979; Van Essen et al., 1981; Zeki, 1974b). Figure 81.1 also schematizes some of the more important cortical and subcortical connections of MT. In addition to its input from V1, MT receives ascending input from V2, V3, and the lateral subdivision of the pulvinar complex. MT is connected with a wide variety of other cortical areas in the superior temporal sulcus (FST, STP, MST), the parietal lobe (VIP, LIP, 7a), and the frontal lobe areas (area 46, FEF, SEF) and by descending connections to the brainstem (dorsolateral pontine nuclei, DTN, and NOT) and midbrain (superior colliculus). In addition, it has extensive connections with the cerebellum. This suite of connections places MT near the middle of a cortical hierarchy for motion processing sometimes called the motion system or motion pathway (Fig. 81.1). It starts in V1, where directionally selective neurons first appear, and heads toward posterior parietal cortex, where many of the structures participate in planning upcoming movements (Andersen et al., 1997). On this pathway, MT is the last area to have a clear retinotopy (Maunsell and Van Essen, 1987) and the first to be strongly connected to explicitly premotor structures. However, MT is not a bottleneck in this pathway by any means; V1 also connects to target areas in posterior parietal and frontal cortices by parallel routes that bypass MT. One of the hallmark features of MT’s anatomical organization is its retinotopy: it contains a fairly orderly map of contralateral visual space. The map varies from individual to individual, is often incomplete, and often contains spatial irregularities (Maunsell, 1986). The coordinates of the map are fairly consistent, with the fovea represented laterally and the vertical meridian representation running along the area boundaries. In addition to this overall retinotopy, MT

1203

FEF VIP

MST

FST

MT V3 V2 V1 Pll SC F 81.1. Summary of location and main connections of area MT. The left panel shows the superior temporal sulcus opened up to reveal the areas inside, including MT highlighted in yellow. The right panel shows the main connections of MT; subcortical structures are indicated by oval symbols. DTN, dorsal terminal nucleus; FEF, frontal eye fields; FST, fundus of the superior temporal sulcus;

MST, medial superior temporal; NOT, nucleus of the optic tract; Pll, pulvinar nucleus, pars lateralis; SC, superior colliculus; SEF, supplementary eye fields; STP, superior temporal polysensory; VIP, ventral intraparietal. (Left panel from Maunsell and Newsome, 1987, with permission.)

contains organized maps for several different physiological response properties, discussed below.

where N is the null direction response, P is the preferred direction response, and m is the maintained activity. The average value for this index in MT ranges from about 0.85 to 1.0 in different studies, indicating that strong directionality is the norm. Although different laboratories apply different criteria in using this index to call a cell directionally selective, the fraction is always a large majority (Albright, 1984; Lagae et al., 1993; Maunsell and Van Essen, 1983a; Zeki, 1978). Another clue to the importance of directionality in MT is that it is organized into a regular, columnar pattern (Albright et al., 1984). Direction is fairly consistent across cortical layers within a column, but changes systematically across the cortical surface (Figure 3), forming a fairly regular map of direction (see Fig. 81.3). While the map cannot be directly visualized in Old World primates owing to its location deep in a sulcus, in owl monkeys it lies exposed on the cortical surface. Optical imaging in this species has revealed a clear and consistent map (Geesaman et al., 1997). Direction columns in both species measure approximately onehalf millimeter in dimension, and transitions between them may either be gradual or abrupt, where preferred direction reverses along a single axis of motion. Equally important to the directional information carried by a directionally selective neuron, such as the one in Figure 81.2, is its precision, or directional bandwidth. Bandwidth describes the breadth of tuning of a single neuron and, by extension, the precision with which a neuronal population can encode the direction of stimulus movement. MT cells typically have fairly broad tuning for direction. Two related

Physiological response properties D Early reports on MT physiology emphasized the remarkable prevalence of directionally selective cells. The vast majority of MT cells are directionally selective, and most are strongly so. Figure 81.2 illustrates responses of a typical MT cell to a coherently moving random dot pattern, filling its receptive field. The polar plot relates the response of the cell (radial axis) to the direction of the stimulus (polar axis). This cell shows excitation in the preferred direction, as well as suppression in the opposite direction. (The maintained activity is indicated by the inner circle; it is about 17 impulses per second, a typical value.) The direction opposite the preferred is commonly referred to as the null direction and sometimes more precisely as the antipreferred direction. Responses in the null direction are not always suppressive; they range from strong suppression through no response to modest excitation (Albright, 1984; Britten et al., 1993; Maunsell and Van Essen, 1983a). Suppression by motion opposite the preferred direction (overt motion opponency) is widespread, though variable across cells, in MT; such responses are not typical of directionally selective cells at earlier stages on the motion pathway. Directionality along this best (preferred–null) axis is frequently characterized by the directionality index, DI: N -m DI = 1 P -m

1204

, ,   

a. Receptive field and stimulus

b. Single neuron response 90

receptive field

5 0

180

25 ips 270

5 F 81.2. Directionality of an MT neuron. a, Stimulus geometry indicating the stimulus arrangement used in the experiment. The dynamic random dots filled the RF of the neuron under study, and was moved in eight different directions at 45 degree intervals. b, Polar plot showing neuronal responses. The polar angle indicates

the stimulus direction, and the radial dimension indicates the response magnitude. The inner circle indicates the maintained activity of the cell. Where standard errors are not visible, they are smaller than the symbols.

100 to 120 degrees (Albright, 1984; Lagae et al., 1993; Maunsell and Van Essen, 1983a). This value depends to some extent on the stimulus; orientation selectivity modestly reduces the bandwidths for bar or grating stimuli.

Near

Far Zero

F 81.3. Schematic illustrating the map of direction and disparity in MT. Each box is a single column on the cortical surface (flattened for clarity). The white lines indicate the axis of the preferred direction of motion, and the colors indicate the stereo tuning of the region. (From DeAngelis and Newsome, 1999, with permission.) (See color plate 57).

measures are widely used to estimate bandwidth. One is the full width (or, frequently, half-width) at half-height, directly interpolated from the empirically measured tuning data. The other is the sigma parameter from Gaussian functions used to fit the directional tuning data. The sigma from a Gaussian fit is nearly identical to the half-width at half height. The best estimates from the literature give its average value as 50 to 60 degrees, or a full width at half-height of

S P A striking correlate of position on a cortical hierarchy is increasing receptive field (RF) size, and MT RFs match their middle position in this regard. If eccentricity is expressed as E, then MT cells’ RF diameters average approximately 0.8E (Maunsell and Van Essen, 1983a), about 10 times the diameter of a V1 cell of comparable eccentricity or 5 times that of a V2 cell. However, preferred spatial frequencies of MT cells do not scale with their larger RFs. These tend to peak near about 0.5 to 5 c/deg, not very different from their afferents, though they are somewhat more broadly tuned (J. A. Movshon, personal communication). This suggests that MT cells inherit their basic spatial tuning from their inputs. A large fraction (approximately half) of MT cells demonstrate spatial interactions on a much larger spatial scale: they possess antagonistic surrounds (Allman et al., 1985; Born, 2000; Raiguel et al., 1995). These surrounds are “silent” when stimulated alone but exert a profound modulatory effect on the response to stimulation in the RF center. The modulation is typically directional and maximally suppresses the response when motion is in the preferred direction. In the extreme, surround stimulation facilitates center responses when it is in the opposite direction. Such center-surround organization is optimal for the detection and discrimination of object motion when an object moves relative to the

:    

1205

background. Cells that lack such antagonistic surrounds tend to respond optimally to wide-field motion, such as might be produced by self-motion through the environment. Interestingly, cells with and without antagonistic surrounds tend to be anatomically separated in columns or clusters (Born and Tootell, 1992). These columns clearly have distinct functional roles, since microstimulation in the two types of regions produced opposite effects on smooth pursuit eye movements (Born et al., 2000). This strongly suggests a role for the surrounds in the segregation of moving objects from their backgrounds. Another interesting feature of the surrounds in MT is that they are often spatially heterogeneous rather than smooth and radially symmetric (Raiguel et al., 1995). This arrangement can be useful and is exploited in some models of high-level motion processing (Royden, 1997). T P Motion is as much about time as it is about space. The motion system in general is driven heavily by the magnocellular pathway (Nealey and Maunsell, 1994), which gives it characteristically rapid dynamics. Latencies in MT can be quite short, although the range is substantial. The minimum latency is as little as about 30 to 35 msec, and the median latency is approximately 90 msec (Heuer and Britten, 1999; Maunsell, 1986; Raiguel et al., 1999). Also characteristic of magnocellulardriven areas, MT cells respond to quite high temporal frequencies. Typically, they peak in the 3 to 10 Hz range, and most will have cut off by 30 to 50 Hz (J. A. Movshon, personal communication). MT cells typically reduce their responses substantially at lower temporal frequencies, and they show profound adaptation to sustained input. When presented with the sudden onset of continuous motion, MT cells give a directional transient response, typically about twice as large as their sustained response, though this varies considerably (Lisberger and Movshon, 1999). This initial transient is informative about the acceleration of a moving target and may be useful in guiding pursuit eye movements (Krauzlis and Lisberger, 1994). Adaptation over a longer time period (seconds to tens of seconds) in a cell’s preferred direction substantially reduces subsequent responses to any stimulus direction (Petersen et al., 1985; van Wezel and Britten, 2002). Another approach to investigating temporal properties of neurons is to analyze the distribution of spikes in a spike train. Because of its rapid kinetics and well-understood functional role, MT has been a fruitful target for such inquiry, and there is currently some debate about the statistics of MT spike trains. It is clear that MT rates can be modulated rapidly (see above) and that this rapid modulation captures substantial information about temporal variation in the stimulus (Bair and Koch, 1996; Buracas et al., 1998). What is less clear is the distribution of spikes under more sustained

1206

, ,   

stimulus conditions. Work at the Singer lab (Kreiter and Singer, 1996) has found very regular, stimulus-dependent peaks in MT spike-train autocorrelation functions. In certain models, this oscillatory behavior can carry substantial stimulus information (for review, see Maldonado et al., 1997). On the other hand, similar recordings from the Newsome laboratory showed substantial burstiness but little overt oscillatory behavior (Bair et al., 1992). S T Speed tuning of MT neurons is much less studied than is directionality, and is somewhat more complex than it appears to be. Several investigations of MT used simple bar, grating, or random-dot stimuli to characterize the phenomenology of MT speed preferences. These experiments revealed that MT neurons are typically bandpass tuned for speed: their responses peak at medium speeds and decline at either faster or slower speeds (Albright, 1984; Lagae et al., 1993; Maunsell and Van Essen, 1983a). Preferred speeds are typically 5 to 30 deg/sec, scaling with eccentricity such that neurons with more peripheral RFs prefer higher speeds (Maunsell and Van Essen, 1983a). However, this relationship is fairly loose, and at any eccentricity there is a substantial range of preferred speeds. These preferred speeds are noticeably higher than those of V1 cells at corresponding eccentricities, raising an interesting question: Does MT merely inherit its speed tuning from a subset of selected V1 (or V2 and V3) afferents or does it perform additional computations to alter the preferences imposed by its inputs? There is some evidence to support each view. Two approaches have been used to study the mechanisms underlying speed selectivity in MT, one based on distance and time and the other using a frequency-domain approach. One way to present stimuli rapidly at different locations and different delays is to present white noise, which contains elements randomly spaced in space and time. In Figure 81.4A, we see the results of such an analysis of spatial interactions within an MT RF. In this experiment, bars were rapidly flashed at randomly chosen locations across an MT cell’s RF. When two sequentially plotted points are located at an appropriate spatial and temporal interval for motion in the cell’s preferred direction and speed, facilitation is observed (“hot” colors). When the interval is in the opposite direction, suppression occurs (“cool” colors). Thus, one may characterize the scale of the spatial interactions that give the cell its preferred direction and speed. These interactions occur over relatively local distances, much smaller than the overall dimensions of the RF. Indeed, the dimensions over which such directional interactions occur closely approach the dimensions of the RFs of V1 cells, suggesting that MT cells inherit their directionality from that of their afferents. These results are consistent with an earlier study using sequentially flashed bars rather than noise (Newsome et al., 1986).

a. Spacial interactions

b. Spatiotemporal profiles (2 cells)

2

0

-2

8 32

32 Temporal Frequency (cyc/s)

Test location (deg)

128 64 32 16

facil.

4

128 64 32 16

4

4

16

16 2

2

8

8 1

1

4

4 .5

.5

2

2 .25

.25

1

1 .125 .5

.5

.25

.25

8

.125

supp.

-4 -4

-2 0 2 Reference location (deg)

4

.125 .25 .5 1 2 4 Spatial Frequency (cyc/deg)

.125 .25 .5 1 2 4 Spatial Frequency (cyc/deg)

F 81.4. Two views of underlying mechanisms in MT RFs. a shows two-bar interaction profiles of a single MT RF. In this experiment, a pair of bars was presented at a short interval at different locations in the RF. The location of the first bar is shown on the horizontal axis, and that of the second bar is shown on the vertical axis. If the test bar is slightly ahead of the reference bar (along the cell’s preferred-null axis), then the response to the test bar is facilitated (red colors). Conversely, if the offset is in the opposite direction, then the response is suppressed. The dimension of the red and blue profiles indicates the spatial extent of the interactions that lead to the cell’s directionality. b, Spatiotemporal response profile of two MT cells seen in the frequency domain. Each dot

represents the measured response to a single grating of a specified spatial and temporal frequency. The diameter of the dot indicates the response magnitude of the cell. The cell on the left has independent preferences for certain spatial and temporal frequencies, while for the cell on the right, the preferred temporal frequency shifts with the spatial frequency. This implies that the cell actually prefers a particular velocity (indicated by the lines paralleling the diagonal; in this case about 2 deg/sec) regardless of the spatial composition of the stimulus. (a from Livingstone et al., 2000, with permission; b modified from Priebe and Lisberger, 2003, with permission.) (See color plate 58.)

The other approach, using the frequency domain, is illustrated in Figure 81.4B. Spatial frequency is the inverse of spatial interval, and temporal frequency is comparably the inverse of flash rate. Moving sine-wave gratings have just one spatial and one temporal frequency, and speed naturally varies with the combination. This figure shows the responses of two single MT cells to different spatial and temporal frequency combinations. Diagonal lines indicate different constant speeds. The circles indicate the response magnitude of two MT cells. The cell on the left gives responses at certain combinations of spatial and temporal frequencies, but these are independent of each other (separable). The cell on the right, however, shows something different—it responds better to higher temporal frequencies if the spatial frequency of the stimulus is higher. This produces the diagonal trend in the response profile. This trend toward the diagonal is a phenomenon never seen in V1 and clearly reflects summation of specific inputs to form a representation that is genuinely speed tuned. Some trend toward the diagonal is found in more than half of MT cells (Perrone and Thiele, 2001), but true speed tuning is clearly a minority phenomenon (Movshon et al., 1988; Priebe and Lisberger 2003). As we will see again in the next section on directional integration, this neither-fish-nor-fowl pattern of results is characteristic of MT. The representation clearly has a novel and interesting feature not seen at earlier levels—in this case, speed

tuning. Yet, many of the cells in MT do not show the emergent property, or only show its partial development. M I  S Motions in the world are rarely discrete and uniform, as they are in the lab. Local motions from the individual contours of a single moving object are effortlessly unified into a single object motion vector. On the other hand, when multiple nearby objects move, their motions are perceptually well segregated. The dual phenomena of motion integration and segregation have perhaps received more attention from psychophysicists than any other aspect of motion processing (for review, see Braddick, 1993). They have also received some study, though considerably less, at the physiological level. Space does not allow an adequate treatment of this interesting and complex issue, but fortunately, many deeper treatments of the topic exist (Maunsell and Newsome, 1987; also see Chapter 82 in this volume). When an object with contours of many different orientations moves, each contour, locally, is seen moving in a direction perpendicular to its orientation—the famous aperture problem. A simple stimulus to explore the mechanism by which these local contour orientations are combined consists of a plaid formed from two gratings moving in different directions. Perceptually, this will often cohere into a single pattern moving in a direction consistent with both individual

:    

1207

grating (component) motions. In one of the most frequently cited MT experiments, Movshon and colleagues (1985) demonstrated that about a third of MT cells integrated across the individual contours of a plaid stimulus and responded to the same pattern direction seen by human observers. Another third responded to the individual contours, and the rest were intermediate. Thus, a continuum of motion integration appeared to be present in MT. Importantly, the same measurement was made in V1, and no cells in V1 performed this integration. Therefore, once again, MT appears to be solving a problem not solved at earlier stages of processing. More recently, additional work has elaborated on this point without calling the fundamental interpretation into question. MT is not unique in possessing neurons that can solve the aperture problem: similar observations have been made in other cortical and subcortical structures (Dumbrava et al., 2001; Gegenfurtner et al., 1997). Furthermore, pattern responses in MT appear to develop slowly over time—more slowly than the component responses (Pack et al., 2001)— and to depend on the anesthetic state of the animal (Pack and Born, 2001). Both of these observations are consistent with the idea that pattern responses in MT are not the result of a simple local calculation, but may depend on circuit dynamics, possibly including several cortical areas. However, the results are only suggestive at this point, and other interpretations are possible. The converse problem of motion segregation has also received some study; this interacts with the representation of stereoscopic depth (see the next section). Earlier studies indicated that multiple different motions within the RF of a single MT cell may interact, even if perceptually they remain well segregated. The best demonstration of this effect uses “transparent” moving dot patterns—two groups of dots moving in different directions. Perceptually, these remain fairly distinct (though not completely; e.g., see Marshak and Sekuler, 1979). However, if the two motions are aligned with a neuron’s preferred-null axis, the motion in the cell’s null direction profoundly suppresses the response to motion in the preferred direction (Snowden et al., 1991). This suppression remains present when the two groups of dots are spatially nonoverlapping (but both are within the cell’s RF). This result makes it appear as if MT is averaging multiple motions within its RF and not maintaining separate representations. However, in an insightful series of experiments Qian, Andersen, and colleagues showed that the interaction depends critically on the depth planes in which the motions are presented. If the dots forming the two planes are separated from each other in depth, then the interaction between the two disappear (Qian and Andersen, 1994). This points to the fundamental importance of stereoscopic depth in how MT processes motion, but as we shall soon see, it also raises interesting problems.

1208

, ,   

Perceptually, motions can be segregated over quite short distances (van Wezel et al., 1994), much smaller than the dimensions of MT RFs. Even though MT is clearly capable of segregating motions in different depth planes, there is no similar documentation of MT supporting fine-scale segregation in a single depth plane. This is potentially a motion analysis problem where the limiting information is represented elsewhere than in MT. V1 is a likely candidate because of its finer-grained representation. S D Motion in the real world is rarely confined to the frontoparallel plane, where experimenters tend to place their stimuli. Furthermore, stereoscopic depth, like motion parallax, is a profound cue to spatial relationships in one’s surroundings. MT and other areas on the motion pathway provide the bulk of the visual inputs to posterior parietal cortex, where space is king. Therefore, it comes as little surprise that MT cells are tuned to stereoscopic depth as well as to motion. Originally, this was interpreted as true tuning for the three-dimensional (3D) motion of objects (Zeki, 1974a), but there was actually a related but subtly different mechanism: cells in MT are independently tuned both for two-dimensional (2D) motion and for the depth plane of the stimulus (Bradley et al., 1998; Maunsell and Van Essen, 1983b; Qian and Andersen, 1994). Tuning for stereo depth in MT is not unlike that found in V1; cells tend to be tuned for near-zero disparities or else have openended tuning for near or far disparities. More recent experiments have demonstrated not only that MT cells are tuned for disparity, but also that MT contains an orderly topographic representation for stereo depth (DeAngelis and Newsome, 1999). Figure 81.3 shows a schematic depiction of how direction and stereo columns appear on the surface of MT. The oriented lines indicate the preferred axis of the direction columns, and the colored regions depict regions tuned for different stereoscopic depths. Interestingly, this map of depth is fractured; there are regions of stereo-tuned neurons interspersed with larger regions where the tuning is weak. These untuned regions still contain many individually selective cells; their tuning is scattered enough that they do not form a classic column. The regions of good tuning do show the hallmark of classic maps: progressive shifts of preferred disparity. Interestingly, there is no systematic relationship between the maps for disparity and the map for preferred direction. This again argues against the idea that MT is systematically organized into an orderly representation of 3D motion. However, as we shall see later, there is no doubt that the stereo signals are functional and contribute to the perception of the depth of moving objects. M  A Selective attention allows observers to perceive an attended object or region of space

more rapidly and accurately, at the cost of other objects. Despite considerable research effort over several decades, the physiological underpinnings remain poorly understood (for a review, see Desimone and Duncan, 1995). Physiological studies in many extrastriate areas reveal profound attentional effects, and area MT is no exception. Attention can be directed either to a specified spatial location or to a stimulus feature such as direction (color, orientation, and so on). The first experiments exploring attentional effects in MT used a design in which either spaceor feature-based attention could have been at work. In these experiments (Treue and Maunsell, 1996), two stimuli moved through the RF of an MT cell, and attention was directed to one of the two. When the attended target moved through the RF in the preferred direction, the response was usually enhanced compared to the response when the nonattended target moved in the preferred direction. The effects were quite large, changing the response by almost a factor of 2 on average. Much smaller effects were seen when the response to attention directed with the cell’s RF was compared with attention directed to a remote location. This result—attention exerting its strongest effects when two stimuli are within a single RF—appears to be typical of both dorsal and ventral stream areas. In these experiments, attention was triggered by a cue that was delivered well before the data were collected. In closely related experiments, Recanzone and Wurtz (2000) saw much smaller attentional effects. The biggest difference was one of time; in the latter study, the monkey was cued to a particular target less than about 300 msec before the responses were measured. The difference between the two sets of results suggests that attentional signals need substantial time to exert their influence on neuronal responses. Recent studies have allowed researchers to dissociate spatial and feature-based attentional modulation, and one may draw the tentative conclusion that directionally based attention is much more profound in its control of MT cell responses than is spatially directed attention. In one set of experiments, Seidemann and Newsome (1999) specifically cued a monkey to a particular spatial location within the RF. Such spatial attention produced only modest (10% to 15%, on average) modulation of the response. On the other hand, Treue and Martinez-Trujillo (1999) performed what was effectively the converse experiment: attention could be directed to either of two directions presented overlapping (transparently) within the RF. In this case, attention would profoundly affect the responses of the neuron, with magnitudes again approaching a factor of 2 in response amplitude. This response change, from related observations, appears to be a modulation of response gain, leaving the basic tuning of the neuron unchanged. In any case, attention clearly can profoundly modulate the responses of MT cells, and it appears that this modulation is based more

on the feature—direction—than on the spatial location of the stimulus.

The link to perception From its physiological properties, MT appears remarkably specialized for the analysis of motion. Motion perception has also been extensively studied, and well-formed models exist for how motion could be analyzed in the brain. Therefore, it is natural that this area was fertile ground for tests of whether, and how, MT activity might support motion perception. This line of inquiry has used both correlative approaches and direct perturbations in a variety of motion tasks. For the most part, the results of all this work support the notion that signals in MT are critically involved in motion perception. However, it is equally clear that MT cannot be tied to motion perception in a one-to-one manner. The correlation with perception clearly is not perfect, suggesting that other structures must also play key roles in motion perception. C  R P  P Correlation between neuronal properties and perceptual abilities is a powerful clue to function and forms one of the cornerstones of systems neuroscience. Correlation, of course, can mean any number of different things. I draw a distinction between two fundamentally distinct correlations: quantitative correlation of overall response properties with perceptual abilities and trial-by-trial correlation of neuronal responses with perceptual reports. This section only addresses the former, more abstract form of correlation. One good way to think about this approach is that it explores the sufficiency of the motion representation for the performance of motion tasks. One such study involved the perception of moving plaid patterns. How one perceives such plaids depends on the brightness of the nodes of the plaid where the dark stripes intersect. If the luminance of these intersections is consistent with a transparent interpretation (i.e., one semitransparent grating occluding another), then the plaid is perceived as two distinct gratings slipping across one another (Stoner and Albright, 1992). Otherwise, observers see a single plaid pattern moving as described above in the section “Motion Integration and Segregation.” This observation sets up a clean test of physiological correlates of perception, because a single objective parameter (the luminance of the grating intersections) reliably controls observers’ perception of motion. Therefore, one can ask whether MT cells show a similar phenomenon, and whether it depends on the luminance in a similar way. Interestingly, at the point at which human observers reported maximal transparency in the moving plaids, most MT pattern cells also tended to respond to the components of the grating rather than to the motion

:    

1209

of the plaid pattern as a whole. When the luminance of the intersections was adjusted to where the plaid pattern cohered, MT cells also tended to report the direction of the pattern rather than the components (reviewed in Albright and Stoner, 1995; Stoner and Albright, 1992). This correlation is compelling, but a variety of interpretive difficulties lurk nearby. Most notably, one is comparing two species: subjective reports from humans against monkey physiology. This, of course, could be the explanation for the minor mismatch noted above. Naturally, one would then wish to perform the same experiment in the same species or, even better, on the same individuals (as quantitative measurements of perception vary from individual to individual as well). However, it is difficult to get monkeys to honestly report their perception of genuinely ambiguous stimuli, because there is no objectively right or wrong answer upon which to base a reward. Fortunately, some reassurance comes from the pattern of eye movements made by monkeys. When faced with large moving displays, monkeys made small-amplitude tracking eye movements. When presented with plaids such as the ones used in the human psychophysical experiments, the monkeys’ eye movements tended to correlate with human perception of the direction of motion (Dobkins et al., 1998). These experiments have been extended using other stimulus manipulations (color, surrounding context) which influence the perception of motion direction (see Chapter 82 in this volume; see also Duncan et al., 2000). Across all of these manipulations, the signals in MT repeatedly demonstrate a correlation with the properties of human perception. Another study explored the sufficiency of neuronal signals in MT for the perception of motion direction in the presence of noise (Britten et al., 1992). In this experiment, apparent motion was created by pairing dots in space and time; the strength of the motion was controlled by the fraction of dots so paired. The task of the monkey was to discriminate the direction of motion between two opposite alternatives. Where the motion was strong, the task was easy, but it could be made arbitrarily difficult. In this way, one can measure the limits of performance of either single neurons or the whole animal. One of the strengths of this experiment was that the measurements avoided the problem of comparing across species: the perceptual and physiological measurements were made at the same time using the same stimuli. The comparison was made more acute in one additional way as well. The monkey was trained to good performance on the task for a wide range of conditions—different speeds, eccentricities, stimulus sizes, and so on. The cells were studied with the task parameters adjusted to optimize the directionality of each cell. This is tantamount to finding the best neurons in MT for each configuration of the task. Thus, the sensitivity that was measured was not of a random sample of MT cells, but ones for which the task was ideal.

1210

, ,   

The canonical result of this experiment is that MT cells are nearly as sensitive to weak motion signals as is the monkey itself. Thresholds for individual cells [estimated using receiver operating characteristic (ROC) analysis] were only about 20% higher than thresholds of the monkey for the same stimuli. Some cells were in fact significantly more sensitive than the animal. Many assumptions entered this analysis, and it is worthwhile to examine these in some detail. First, both cells and monkey were give 2 seconds to examine the motion stimulus. While the analysis integrated the cells’ signals perfectly over this relatively long duration, the monkey might not have been able to integrate as well. In fact, when the durations of the stimuli were shortened, both neuronal and behavioral sensitivity decreased as expected, but at somewhat different rates; the cells’ thresholds were much worse than those of the monkey at a stimulus duration of 250 msec. Although this measurement was not made simultaneously, as were the others, it still hints at circumstances under which the relationship between MT neuronal activity and motion perception can be dissociated. Further hints of this dissociation come from related experiments in which color is introduced into the stimulus. As this is discussed elsewhere in this volume (Chapter 82), I will mention the study only briefly. When the dots carrying the motion signal are colored so as to be distinct from the masking noise dots, they can be perceptually segmented, and thresholds fall dramatically (Croner and Albright, 1997). While MT signals are also improved, the perceptual improvement is much greater (Croner and Albright, 1999). The same approach has been applied to other stimulus dimensions as well. When monkeys are trained on a stereoscopic depth discrimination task, one can use nearly identical stimulus manipulations and analyses to estimate the sensitivity of neurons and of the animal (Uka and DeAngelis, 2001). Stimuli were again optimized to the neurons’ discriminative capacity, and the stereo signal indicating depth was degraded by the addition of noise. The nice thing about this experiment is that it is so similar to the motion experiments; the results can be directly compared across the two dimensions for direction and depth. In the depth experiment, a very similar result was obtained. Neurons in MT were again nearly identical in their sensitivity to the entire animal. Whether the same would be true of other dimensions is a fascinating question. At the very least, this finding suggests that MT does more than just analyze visual motion. R- C  P D Another form of correlation is potentially more revealing in probing whether sensory signals are actually used by the animal during a perceptual task. Several laboratories have used this general approach to probe the link between MT activity and perception. One of the first demonstrations of

such a correlation came from the work of Logothetis and Schall (1989). These authors used perceptually ambiguous stimuli caused by binocular rivalrous motion. When opposite directions of motion are presented to the two eyes, the subject perceives, alternately, each of the two directions. While monkeys were reporting their percepts of such rivalry, the responses of MT cells were monitored. Many MT cells were modulated such that their firing rates depended on which of the two directions was being reported by the monkey at the time. Unfortunately, of the cells that were modulated, approximately equal numbers were correlated with the percept (i.e., increased their firing rate when the monkey reported their preferred direction) or the opposite. There are several possible interpretations of this a tantalizing finding. One is that the monkey may not have been honest in reporting its percept; the stimulus was intrinsically ambiguous, and there is in principle no way to reward the animal objectively. As in the Stoner and Albright experiment described above, eye movements help; small tracking eye movements tended to correlate with the reported percept, lending credence to the monkeys’ reports of their experience. Rivalry is also, despite its usefulness in probing perception of ambiguous stimuli, a fairly nonphysiological stimulus. It is possible that the unpredictability of the modulation in MT relates to this fact. Another stimulus manipulation to provoke perceptual ambiguity is closer to normal visual experience. Picture a transparent cylinder, with dots attached to its surface, rotating on its long axis. Under normal viewing conditions, with stereoscopic depth contributing, it is correctly and unambiguously viewed as rotating in a single direction. But if you remove the stereo depth from the image in the lab, it still appears vividly 3D (termed structure-from-motion or the kinetic depth effect), but the direction in which it rotates is ambiguous. Stare at it for a while, blink your eyes, and it will reverse in direction, with a different moving surface now seen as being in front. Two studies have explored correlates of this bistable percept in MT (Andersen et al., 1997; Dodd et al., 2001), and have found striking evidence that MT activity predicts the percept as it is occurring on a trial-by-trial basis. Dodd et al. quantified their results with a metric referred to as the choice probability—the probability with which an observer could infer what decision the monkey was about to make based only on the discharge of the neuron. This probability averaged a very significant 0.67 (0.5 would be chance). Thus, MT cells carry a distinct signature of the monkey’s upcoming decision in their firing rates. A very similar observation was made by the Newsome group in the context of the previously described direction discrimination task (Britten et al., 1996), where the monkey is discriminating between opposite directions of motion in the presence of masking noise. In this task, the alternatives can be made completely uncertain (pure noise), forcing the

monkey to guess. Even in this case, neuronal discharge predicts the monkey’s decision to some degree. It should be noted that in this task, the percept is not clearly bistable, as it is in the cylinder task. Instead, there is weak, inconsistent, and unclear motion with no net direction. Interestingly, in this case, the average value of the choice probability was smaller (0.56) on average than that reported by Dodd et al. Thus, it appears that the value of this correlation depends on the task the anima is performing. Further evidence for task dependence comes from the work of the Hoffmann group (Thiele and Hoffman, 1996). This experiment used variable contrast gratings in the context of a four-alternative discrimination task. Furthermore, the monkey reported its decision using a lever response rather than an eye movement. In this experiment, while there was a slight trend toward correlation between neuronal responses in MT and the monkeys’ choices, it was not statistically significant. However, responses in two higher-order motion areas (MST and the superior temporal polysensory area) did show significant correlation. Overall, the frequent presence of a trial-by-trial signature of directional decisions in neuronal firing rates is very revealing. Unquestionably, this indicates the intimate relationship between activity in the area and perceptually guided behavior. Two distinct mechanisms, however, are consistent with this family of observations. First, the correlation could reflect causation: when the cells are more vigorously reporting the presence of their preferred stimulus, the monkey is more likely to report it. Alternately, “top-down” influences might be in effect: when the monkey is biased toward or attending to a particular alternative on the task, this both raises the firing rates of the neurons and raises the chance of the decision. But either mechanism is consistent with a close quantitative relationship between neuronal activity and choice. L  T F Lesions are frequently used to test the necessity of a specific structure for some function, and it is only natural that this approach has been brought to bear on the link between MT and the perception of motion. Necessity can be a bit of a slippery concept; it is rarely black and white in a system as complex as the cortical visual system. This is evident in the pattern of results emerging from a number of lesion studies on MT. Lesions to MT and the surrounding cortical areas have been tested in a large number of labs, and the results are quite consistent across studies. Lesions restricted entirely to MT substantially elevate thresholds for the discrimination of opposed directions masked by noise (Newsome and Paré, 1986), the discrimination of small differences in speed (Orban et al., 1995; Pasternak and Merigan, 1994), and the discrimination of small differences in direction (Lauwers et al., 2000). Furthermore, these effects seem selective for motion where this has been tested. MT lesions did not affect

:    

1211

b. Stimulation results

a. Task setup

1 Proportion Preferred Decisions

Receptive field Pref target

Stimulus

0.8 0.6 0.4 stimulated control

0.2

Null target

0 10 deg

Fixation Point

-20

-10 0 10 Coherence (%)

20

30

F 81.5. Results of an MT microstimulation experiment. The monkey is judging the direction of a random-dot stimulus filling the RF of a multiunit cluster in MT, as indicated in a. The coherence of the dots varies across a range spanning the psychophysical threshold, and on half of the trials microstimulation (200 Hz, 10 microamperes biphasic) is applied to the electrode.

b, Psychophysical results of the experiment showing the large shift in performance induced by the microstimulation. For all stimuli, microstimulation caused an increase in the proportion of preferreddirection decisions, showing that the current injection biased the monkey’s performance.

contrast thresholds (Newsome and Paré, 1986) or the detection of color or texture differences (Schiller, 1993). Thus, these studies suggest a selective necessity of MT for motionrelated tasks. It is important to note that the necessity is not absolute: function will recover substantially given time after the lesion is placed. In the study of Newsome and Paré, thresholds recovered substantially or completely (for the more restricted lesions) after as little as 10 days. This recovery takes much longer and is less complete for larger lesions, and furthermore depends on the exact type of motion stimulus tested (Pasternak and Merigan, 1994; Rudolph and Pasternak, 1999). Interestingly, in most studies where recovery was incomplete, there was substantial involvement of the adjacent area, MST. This suggests that together the two areas are much more necessary for the normal discrimination of motion than is MT by itself. This is generally consistent with the anatomy of the motion pathway: parallel routes to the parietal lobe exist, but all pass through either MT or MST. However, it should be emphasized that even after complete removal of MT and MST, monkeys are not completely motion-blind. Other cortical areas clearly must contribute to the perception of motion. Of course, another possibility for the recovery of function after more restricted lesions is reorganization of the representation in MT itself. Yamasaki and Wurtz (1991) explored reorganization in MT following lesions, and found the representation in the areas of the MT retinotopic map outside the lesion to be quite stable. From this, they concluded that the recovery from lesions was largely produced by mechanisms outside MT itself.

M  T F Although lesions are a basic method for establishing the necessity of a structure, they do have some interpretive difficulties. Notably, they are subject to false negatives when redundant structures can fill in for the removed one. They also can be subject to false positives, because lesion effects can be indirect instead of direct. An alternative means of testing the involvement of a structure with a specified function, timehonored in the analysis of motor systems, is restricted activation. Stimulation is in some sense the converse of a lesion: one activates a particular circuit to see if the expected change in behavior ensues under one’s functional hypothesis. This approach has been revealing in testing the function of MT in perception, and has provided the strongest case yet of this structure’s multiple perceptual roles. MT is organized in a columnar fashion, as is clear from the schematic in Figure 81.4, and this enables the application of intracortical microstimulation. If a single column carries a particular directional signal—say, leftward motion—then activation of this column should lead to an increased perception of leftward motion and more left choices in a left versus right discrimination task. Such a prototypical result is illustrated in Figure 81.5 from the work of Salzman and Newsome (Salzman et al., 1992). In this case, the stimulated performance curve is shifted upward (or leftward) from the control curve, indicating a large and statistically reliable increase in the proportion of choices in favor of the preferred direction of neurons at the stimulation location. This result, which was typical of the experiment, indicates that MT is directly and causally involved in the monkey’s directional decisions. Another advantage of this

1212

, ,   

approach over the lesion approach is that one also gains some understanding of the nature of the involvement of the neurons in the decision. Parametric manipulations of this basic experiment have shown that in order to get a perceptual effect, the stimulus the monkey is judging must overlie the RF of the column of neurons being activated, and that using larger currents damages the directional specificity of the behavioral effects (Murasugi et al., 1993). Microstimulation also has very good temporal specificity, as it can be applied at precise times within a trial relative to the visual stimulus and the monkey’s behavioral choice. Interestingly, the time window in which microstimulation produces reliable effects appears to be task-dependent. In the original version of the task, the monkey is given one stimulus and then has all the information it will ever get to make its decision. In this case, microstimulation is effective only when it is temporally coincident with the visual stimulus; if it occurs either earlier or later, it is largely ineffective. However, if one changes the task to a delayed match-tosample version, microstimulation becomes effective even when no stimulus is present in the interval between the sample and the match stimuli (Bisley and Pasternak, 2000). Although no one has used microstimulation to test what modalities MT participates in, after the manner of the Schiller lesion experiment, we now have evidence from this approach that MT represents more than just motion. Because MT has columns representing stereoscopic depth as well as direction of motion, it is natural to test the role of MT in the judgment of stereoscopic depth as well. When MT is tested in this way, the perceptual effects of microstimulation on depth judgments are just as large and systematic as they are for judgments of motion direction (DeAngelis et al., 1998). This indicates that MT clearly contributes to perception of multiple stimulus dimensions. The involvement of MT in stereoscopic depth perception results makes sense for other reasons as well. Stereoscopic depth is similar to motion with respect to the nature of the computations involved, and both are powerful cues to spatial relationships in the world.

Concluding remarks MT is the paragon of specialization in extrastriate cortex. No other area is as homogeneous in its response properties, nor have many areas received such depth of study. We know a lot about the mechanisms underlying MT responses, as well as the functional role of these responses in motion perception. A good case can be made—unusual in a complex network like extrastriate cortex—for both the necessity and the sufficiency of MT for motion perception. Yet, neither its necessity nor its sufficiency is complete from what we know. Clearly, MT does not work alone. The results of lesion experiments, as well as what we know of the anatomy of the

motion pathway, show that MT acts in concert with many other areas to support the perception of motion. Furthermore, MT carries information about visual features other than motion, such as stereo depth cues. Sufficiency is an even more slippery concept: MT certainly has motion signals sufficient to guide perception in some cases. But no cortical structure can be sufficient in the strict sense of the word: perception must depend on the integrity of upstream and downstream areas, and probably on parallel areas carrying similar information. We can account for many qualitative and even quantitative features of motion perception using known aspects of physiology in MT. No one now seriously questions the linking hypothesis that MT is an integral part of how we see moving things. The bad news is that our knowledge of this structure is still woefully inadequate in many regards. There are striking deficits in our knowledge of function at both at the descriptive and the mechanistic levels. To start with, motion psychophysics is complex, with many phenomena beyond simple direction or speed discrimination. Such perceptual phenomena as long-range apparent motion and second- or third-order motion are seeking good physiological correlates. Distinctions drawn between these phenomena by psychophysicists might reflect different neuronal substrates (e.g., Wilson and Ferrera, 1992), and this is subject to direct experimental test. Additionally, recent results on stereopsis suggest that the role of MT outside of motion might be wider than was previously thought. Stereo, like motion, is a good cue to the spatial structure of the scene, so one can easily imagine further experiments testing MT and other structures in the motion system as providing more general information on 2D and 3D space. When it comes to questions of underlying physiological mechanisms, we are even more profoundly ignorant. There are good models largely consistent with the current experimental data but their predictions have not really been tested. To some extent, this is because the level of question makes the tests difficult. To cite just one of the many possible examples, consider velocity selectivity. A simple summation model such as the one proposed by Simoncelli and Heeger (1998) is consistent with the presence of truly velocity-selective cells, and furthermore suggests that the property of velocity selectivity emerges as a consequence of convergence of the appropriate inputs from V1 or other lower-order areas. Testing this hypothesis involves tracing patterns of cell-tocell connectivity across multiple areas—a heroic experiment with current tools. Nonetheless, techniques that have been successfully applied to similar circuit-level questions in striate cortex could provide a wealth of valuable data to test and constrain present and future mechanistic models of MT processing. Looking in the other direction, the field is well poised to form and test hypotheses regarding how information is read

:    

1213

out from MT and used in making decisions or guiding movements. Such work is well underway in several labs, and the next decade should be an exciting time in pursuing this level of question. Similarly, the question of how attention modulates sensory processing in MT is ripe for mechanistic study. We know that top-down influences have a profound effect on MT, but how this happens is still an question. Again, this is an issue that is best addressed at the circuit level. Overall, then, we see that the past 25 years have been exciting ones in the study of the cortical mechanisms of perception, and MT is one of the poster children of how such questions can be addressed. Our ideas about its role in perception have evolved and will presumably continue to evolve. The next 25 years should be even more exciting. REFERENCES Albright, T. D., 1984. Direction and orientation selectivity of neurons in visual area MT of the macaque, J. Neurophysiol., 52:1106–1130. Albright, T. D., R. Desimone, and C. G. Gross, 1984. Columnar organization of directionally selective cells in visual area MT of macaques, J. Neurophysiol., 51:16–31. Albright, T. D., and G. R. Stoner, 1995. Visual motion perception, Proc. Nat. Acad. Sci. USA, 92:2433–2440. Allman, J. M., and J. H. Kaas, 1971. A representation of the visual field in the caudal third of the middle temporal gyrus of the owl monkey (Aotus trivirgatus), Brain Res., 31:85–105. Allman, J. M., F. Meizin, and E. McGuinness, 1985. Direction and velocity-specific responses from beyond the classical receptive field in the middle temporal visual area (MT), Perception, 14:105–126. Andersen, R. A., 1997. Neural mechanisms of visual motion perception in primates, Neuron, 18:865–872. Andersen, R. A., L. H. Snyder, D. C. Bradley, and J. Xing, 1997. Multimodal representation of space in the posterior parietal cortex and its use in planning movements, Annu. Rev. Neurosci., 20:303–330. Bair, W., and C. Koch, 1996. Temporal precision of spike trains in extrastriate cortex of the behaving macaque monkey, Neural Comput., 8:1185–1202. Bair, W., C. Koch, W. Newsome, and K. Britten, 1992. Power spectrum analysis of MT neurons from awake monkey, Soc. Neurosci. Abstr., 18:12. Battaglini, P. P., C. Galletti, and P. Fattori, 1996. Cortical mechanisms for visual perception of object motion and position in space, Behav. Brain Res., 76:143–154. Bisley, J. W., and T. Pasternak, 2000. The multiple roles of visual cortical areas MT/MST in remembering the direction of visual motion, Cereb. Cortex, 10:1053–1065. Born, R. T., 2000. Center-surround interactions in the middle temporal visual area of the owl monkey, J. Neurophysiol., 84:2658–2669. Born, R. T., J. M. Groh, R. Zhao, and S. J. Lukasewycz, 2000. Segregation of object and background motion in visual area MT: effects of microstimulation on eye movements, Neuron, 26:725–734. Born, R. T., and R. B. Tootell, 1992. Segregation of global and local motion processing in primate middle temporal visual area, Nature, 357:497–499.

1214

, ,   

Braddick, O., 1993. Segmentation versus integration in visual motion processing, Trends Neurosci., 16:263–268. Bradley, D. C., G. C. Chang, and R. A. Andersen, 1998. Encoding of three-dimensional structure-from-motion by primate area MT neurons, Nature, 392:714–717. Britten, K. H., W. T. Newsome, M. N. Shadlen, S. Celebrini, and J. A. Movshon, 1996. A relationship between behavioral choice and the visual responses of neurons in macaque MT, Vis. Neurosci., 13:87–100. Britten, K. H., M. N. Shadlen, W. T. Newsome, and J. A. Movshon, 1992. The analysis of visual motion: a comparison of neuronal and psychophysical performance, J. Neurosci., 12:4745–4765. Britten, K. H., M. N. Shadlen, W. T. Newsome, and J. A. Movshon, 1993. Responses of neurons in macaque MT to stochastic motion signals, Vis. Neurosci., 10:1157–1169. Buracas, G. T., A. M. Zador, M. R. DeWeese, and T. D. Albright, 1998. Efficient discrimination of temporal patterns by motionsensitive neurons in primate visual cortex, Neuron, 20:959– 969. Croner, L. J., and T. D. Albright, 1997. Image segmentation enhances discrimination of motion in visual noise, Vis. Res., 37:1415–1427. Croner, L. J., and T. D. Albright, 1999. Segmentation by color influences responses of motion-sensitive neurons in the cortical middle temporal visual area., J. Neurosci., 19:3935–3951. DeAngelis, G. C., B. G. Cumming, and W. T. Newsome, 1998. Cortical area MT and the perception of stereoscopic depth, Nature, 394:677–680. DeAngelis, G. C., and W. T. Newsome, 1999. Organization of disparity-selective neurons in macaque area MT, J. Neurosci., 19:1398–1415. Desimone, R., and J. Duncan, 1995, Neural mechanisms of selective attention, Ann. Rev. Neurosci., 18:193–222. Dobkins, K. R., G. R. Stoner, and T. D. Albright, 1998. Perceptual, oculomotor, and neural responses to moving color plaids, Perception, 27:681–709. Dodd, J. V., K. Krug, B. G. Cumming, and A. J. Parker, 2001. Perceptually bistable 3-D figures evoke high choice probabilities in cortical area MT, J. Neurosci., 21:4809–4821. Dubner, R., and S. M. Zeki, 1971. Response properties and receptive fields of cells in an anatomically defined region of the superior temporal sulcus, Brain Res., 35:528–532. Duffy, C. J., 2000. Optic flow analysis for self-movement perception, Int. Rev. Neurobiol., 44:199–218. Dumbrava, D., J. Faubert, and C. Casanova, 2001. Global motion integration in the cat’s lateral posterior-pulvinar complex, Eur. J. Neurosci., 13:2218–2226. Duncan, R. O., T. D. Albright, and G. R. Stoner, 2000. Occlusion and the interpretation of visual motion: perceptual and neuronal effects of context, J. Neurosci., 20:5885–5897. Geesaman, B., R. Born, R. Andersen, and R. Tootell, 1997. Maps of complex motion selectivity in the superior temporal cortex of the alert macaque monkey: a double-label 2-deoxyglucose study, Cereb. Cortex, 7:749–757. Gegenfurtner, K. R., D. C. Kiper, and J. B. Levitt, 1997. Functional properties of neurons in macaque area V3, J. Neurophysiol., 77:1906–1923. Heuer, H. W., and K. H. Britten, 1999. Response normalization by middle temporal neurons. II: effects of contrast, J. Neurophysiol., 88:3398–3408. Krauzlis, R., and S. Lisberger, 1994. Temporal properties of visual motion signals for the initiation of smooth pursuit eye movements in monkeys, J. Neurophysiol., 72:150–162.

Kreiter, A. K., and W. Singer, 1996. Stimulus dependent synchronization of neuronal responses in the visual cortex of the awake macaque monkey, J. Neurosci., 16:798–828. Krubitzer, L. A., and J. H. Kaas, 1990. Cortical connections of MT in four species of primates: areal, modular, and retinotopic patterns, Vis. Neurosci., 5:165–204. Lagae, L., S. Raiguel, and G. A. Orban, 1993. Speed and direction selectivity of macaque middle temporal neurons, J. Neurophysiol., 69:19–39. Lauwers, K., R. Saunders, R. Vogels, E. Vandenbussche, and G. A. Orban, 2000. Impairment in motion discrimination tasks is unrelated to amount of damage to superior temporal sulcus motion areas, J. Comp. Neurol., 420:539–557. Lisberger, S. G., and J. A. Movshon, 1999. Visual motion analysis for pursuit eye movements in area MT of macaque monkeys, J. Neurosci., 19:2224–2246. Livingstone, M. S., C. C. Pack, and R. T. Born, 2001. Twodimensional substructure of MT receptive fields, Neuron, 30:781– 793. Logothetis, N. K., and J. D. Schall, 1989. Neuronal correlates of subjective visual perception, Science, 245:761–763. Maldonado, P. E., I. Gödecke, C. M. Gray, and T. Bonhoeffer, 1997. Orientation selectivity in pinwheel centers in cat striate cortex, Science, 276:1551–1555. Marshak, W., and S. Sekuler, 1979. Mutual repulsion between moving visual targets, Science, 205:1399–1401. Maunsell, J. H. R., 1986. Physiological evidence for two visual systems, in Matters of Intelligence (L. Vaina ed.), Dordrecht: Reidel. Maunsell, J. H. R., and W. T. Newsome, 1987. Visual processing in monkey extrastriate cortex, Annu. Rev. Neurosci., 10:363–401. Maunsell, J. H. R., and D. C. Van Essen, 1983a. Functional properties of neurons in the middle temporal visual area (MT) of the macaque monkey: I. Selectivity for stimulus direction, speed and orientation, J. Neurophysiol., 49:1127–1147. Maunsell, J. H. R., and D. C. Van Essen, 1983b. Functional properties of neurons in the middle temporal visual area (MT) of the macaque monkey: II. Binocular interactions and the sensitivity to binocular disparity, J. Neurophysiol., 49:1148–1167. Maunsell, J. H. R., and D. C. Van Essen, 1987. Topographic organization of the middle temporal visual area in the macaque monkey: representational biases and the relationship to callosal connections and myeloarchitectonic boundaries, J. Comp. Neurol., 266:535–555. Movshon, J. A., E. H. Adelson, M. S. Gizzi, and W. T. Newsome, 1985. The analysis of moving visual patterns, in Study Group on Pattern Recognition Mechanisms (C. Chagas, R. Gattass, and C. Gross, eds.), Vatican City: Pontifica Academia Scientiarum, pp. 117–151. Movshon, J. A., W. T. Newsome, M. S. Gizzi, and J. B. Levitt, 1988. Spatio-temporal tuning and speed sensitivity in macaque visual cortical neurons, Invest. Ophthalmol. Vis. Sci. Suppl., 29:327. Murasugi, C. M., C. D. Salzman, and W. T. Newsome, 1993. Microstimulation in visual area MT: effects of varying pulse amplitude and frequency, J. Neurosci., 13:1719–1729. Nealey, T., and J. Maunsell, 1994. Magnocellular and parvocellular contributions to the responses of neurons in macaque striate cortex, J. Neurosci., 14:2069–2079. Newsome, W. T., A. Mikami, and R. H. Wurtz, 1986. Motion selectivity in macaque visual cortex. III. Psychophysics and physiology of apparent motion, J. Neurophysiol., 55:1340–1351. Newsome, W. T., and E. B. Paré, 1986. MT lesions impair discrimination of direction in a stochastic motion display, Soc. Neurosci. Abstr., 12:1183.

Orban, G. A., 1997. Visual processing in macaque area MT/V5 and its satellites (MSTd and MSTv), in Cerebral Cortex (K. S. Rockland, J. H. Kaas, and A. Peters, eds.), New York: Plenum, pp. 359–434. Orban, G. A., R. C. Saunders, and E. Vandenbussche, 1995. Lesions of the superior temporal cortical motion areas impair speed discrimination in the macaque monkey, Eur. J. Neurosci., 7:2261–2276. Pack, C. C., V. K. Berezovskii, and R. T. Born, 2001. Dynamic properties of neurons in cortical area MT in alert and anaesthetized macaque monkeys, Nature, 414:905–908. Pack, C. C., and R. T. Born, 2001. Temporal dynamics of a neural solution to the aperture problem in visual area MT of macaque brain, Nature, 409:1040–1042. Parker, A. J., and W. T. Newsome, 1998. Sense and the single neuron: probing the physiology of perception, Annu. Rev. Neurosci., 21:227–277. Pasternak, T., and W. H. Merigan, 1994. Motion perception following lesions of the superior temporal sulcus in the monkey, Cereb. Cortex, 4:247–259. Perrone, J. A., and A. Thiele, 2001. Speed skills: measuring the visual speed analyzing properties of primate MT neurons, Nat. Neurosci., 4:526–532. Petersen, S. E., J. F. Baker, and J. M. Allman, 1985. Directionspecific adaptation in area MT of the owl monkey, Brain Res., 346:146–150. Priebe, N. J., M. M. Churchland, and S. G. Lisberger, 2003. Reconstruction of target speed for the guidance of smooth pursuit eye movements, J. Neurosci., in press. Qian, N., and R. A. Andersen, 1994. Transparent motion perception as detection of unbalanced motion signals. II. Physiology, J. Neurosci., 14:7367–7380. Raiguel, S., M. M. Van Hulle, D. K. Xiao, V. L. Marcar, and G. A. Orban, 1995. Shape and spatial distribution of receptive fields and antagonistic motion surrounds in the middle temporal area (V5) of the macaque, Eur. J. Neurosci., 7:2064–2082. Raiguel, S. E., D. K. Xiao, V. L. Marcar, and G. A. Orban, 1999. Response latency of macaque area MT/V5 neurons and its relationship to stimulus parameters, J. Neurophysiol., 82:1944– 1956. Recanzone, G. H., and R. H. Wurtz, 2000. Effects of attention on MT and MST neuronal activity during pursuit initiation, J. Neurophysiol., 83:777–790. Royden, C. S., 1997. Mathematical analysis of motion-opponent mechanisms used in the determination of heading and depth, J. Opt. Soc. Am. A, 14:2128–2143. Rudolph, K., and T. Pasternak, 1999. Transient and permanent deficits in motion perception after lesions of cortical areas MT and MST in the macaque monkey, Cereb. Cortex, 9:90–100. Salzman, C. D., C. M. Murasugi, K. H. Britten, and W. T. Newsome, 1992. Microstimulation in visual area MT: effects on direction discrimination performance, J. Neurosci., 12:2331–2355. Schiller, P., 1993. The effects of V4 and middle temporal (MT) area lesions on visual performance in the rhesus monkey, Vis. Neurosci., 10:717–746. Seidemann, E., and W. T. Newsome, 1999. Effect of spatial attention on the responses of area MT neurons, J. Neurophysiol., 81:1783–1794. Simoncelli, E. P., and D. J. Heeger, 1998. A model of neuronal responses in visual area MT, Vis. Res., 38:743–761. Snowden, R. J., S. Treue, R. G. Erickson, and R. A. Andersen, 1991. The response of area MT and V1 neurons to transparent motion, J. Neurosci., 11:2768–2785.

:    

1215

Stoner, G. R., and T. D. Albright, 1992. Neural correlates of perceptual motion coherence, Nature, 358:412–414. Thiele, A., and K.-P. Hoffman, 1996. Neuronal activity in MST and STPp, but not MT changes systematically with stimulusindependent decisions, NeuroReport, 7:971–976. Treue, S., and J. C. Martínez-Trujillo, 1999. Feature-based attention influences motion processing gain in macaque visual cortex, Nature, 399:575–579. Treue, S., and J. H. R. Maunsell, 1996. Attentional modulation of visual motion processing in cortical areas MT and MST, Nature, 382:539–541. Uka, T., and G. C. DeAngelis, 2003. Contribution of middle temporal area to coarse depth discrimination: comparison of neuronal and psychophysical sensitivity, J. Neurosci., 23:3515–3530. Ungerleider, L. G., and M. Mishkin, 1979. The striate projection in the superior temporal sulcus of Macaca mulatta: location and topographic organization, J. Comp. Neurol., 188:347– 366. Van Essen, D. C., J. H. R. Maunsell, and J. L. Bixby, 1981. The middle temporal visual area in the macaque: myeloarchitecture, connections, functional properties and topographic representation, J. Comp. Neurol., 199:293–326.

1216

, ,   

van Wezel, R. J. A., and K. H. Britten, 2002. Motion Adaptation in area MT, J. Neurophysiol., 88:3469–3476. van Wezel, R. J. A., F. A. J. Verstraten, R. E. Fredericksen, and W. A. van de Grind, 1994. Spatial integration in coherent motion detection and in the movement aftereffect, Perception, 23:1189– 1195. Wilson, H. R., and V. P. Ferrera, 1992. A psychophysically motivated model for two-dimensional motion perception, Vis. Neurosci., 9:79–97. Yamasaki, D. S., and R. H. Wurtz, 1991. Recovery of function after lesions in the superior temporal sulcus in the monkey, J. Neurophysiol., 66:651–673. Zeki, S. M., 1974a. Cells responding to changing image size and disparity in the cortex of the rhesus monkey, J. Physiol., 242:827–841. Zeki, S. M., 1974b. Functional organization of a visual area in the posterior bank of the superior temporal sulcus of the rhesus monkey, J. Physiol., 236:549–573. Zeki, S. M., 1978. Uniformity and diversity of structure and function in rhesus monkey prestriate visual cortex, J. Physiol., 277: 273–290.

82

Merging Processing Streams: Color Cues for Motion Detection and Interpretation KAREN R. DOBKINS AND THOMAS D. ALBRIGHT

T    a visual motion processor is that of detecting the continuity of image features as they are displaced in time and space. This process, referred to as the motion correspondence problem, involves “matching” features at one moment in time with the same objects, displaced in space, at a later moment in time. Although there exist several different types of image features that might be used for this matching process, here we focus on the role of color since this aspect of vision provides one of the most reliable means for discerning object boundaries. Despite the appeal of utilitarian arguments, the degree to which the primate motion system makes use of object color has been an issue of long-standing debate in vision science. Indeed, it has been suggested that color information should exert little or no influence on motion detection, a notion that sprang from evidence for parallel processing of color versus motion in the primate visual system. Contrary to this once popular viewpoint, several new lines of evidence suggest that the motion system uses color information in ways that are both substantial and highly functional. Such findings, in turn, have inspired efforts to identify the mechanisms by which color information reaches motion processing areas of the brain. In this chapter, we begin by presenting psychophysical evidence for the use of color in human motion perception. We then address evidence for neural correlates of these perceptual phenomena. In particular, we focus on a region of primate extrastriate cortex, known as the middle temporal visual area (area MT), which is thought to play a key role in motion perception. We then discuss the possible sources of input to area MT that may underlie the ability of neurons in this area to signal motion of chromatically defined patterns. To this end, we provide an overview of the three main retinogeniculate pathways of the primate visual system—parvocellular, koniocellular, and magnocellular—whose color responses are thought to be selective for red/green, blue/yellow, and luminance modulation, respectively. Finally, we describe the results of several neurophysiological experiments carried out using macaque monkeys as subjects, which address the extent to which these different pathways might contribute to chromatic responses observed in area MT. Note that parallels drawn between human psychophysical data and

macaque neural data are justified in light of the known similarities of visual system organization and function between these two primates (e.g., De Valois et al., 1974; Golomb et al., 1985; Newsome et al., 1989; Newsome and Paré, 1988).

Definitions of color channels Before reviewing the psychophysical evidence for chromatic input to motion processing, it is necessary to review briefly the different dimensions of color vision. Theories of color vision posit the existence of three postreceptoral channels, which are derived from the sums and differences of the three cone types in the eye. The achromatic or luminance channel signals a weighted sum of long-wavelength-selective (L) and mediumwavelength-selective (M) cones (i.e., L + M, with some debate regarding the contribution of short-wavelength-selective (S) cones; e.g., Boynton et al., 1985; Eisner and Macleod, 1980; Stockman et al., 1991). Two chromatic channels signal weighted sums and differences of the cones. The red/green chromatic channel signals differences between L and M cones (i.e., L - M or M - L). The blue/yellow chromatic channel signals differences between S-cones and the sum of L and M cones (i.e., S(L + M)). These three channels are referred to as cardinal channels, and the stimuli that isolate them are referred to as the cardinal axes of three-dimensional color space (Fig. 82.1). Note that stimuli varying along either the red/green or blue/yellow axis (or any axis within the plane of the two) do not vary in luminance; for this reason, stimuli modulated within this plane are referred to as equiluminant.

Psychophysical evidence for color input to motion processing Over the past several decades, the reliance of motion detectors on signals arising within the luminance channel has been firmly established (see Nakayama, 1985, for review). By comparison, the use of red/green and blue/yellow chromatic information for motion processing has been far more controversial. Psychophysical studies probing the extent of chromatic influences on motion perception are numerous, and the results have been variously interpreted (see Dobkins and Albright, 1998; Gegenfurtner and Hawken, 1996, for

1217

F 82.1. Three color channels. The luminance channel signals a weighted sum of long-wavelength-selective (L) and mediumwavelength-selective (M) cones, that is, L + M. The red/green chromatic channel signals differences between L and M cones, that is, L - M. The blue/yellow chromatic channel signals differences between short-wavelength-selective (S) cones and the sum of L and M cones, that is, S - (L + M). (Reprinted from Neuron, vol. 25, Dobkins, K. R., Moving Colors in the Lime Light, Pages 15–18, Copyright 2000 with permission from Elsevier Science.) (See color plate 59.)

detailed reviews). One of the most direct ways in which researchers have investigated chromatic input to motion processing is to simply determine whether the direction of a chromatically defined moving stimulus can be discriminated. For red/green stimuli, studies employing sinusoidal gratings (e.g., Cavanagh and Anstis, 1991; Cavanagh et al., 1984; Cropper and Derrington, 1994, 1996; Derrington and Henning, 1993; Dobkins and Albright, 1993; Dobkins and Teller, 1996; Gegenfurtner and Hawken, 1995; Hawken et al., 1994; Lindsey and Teller, 1990; Metha et al., 1994; Mullen and Boulton, 1992; Palmer et al., 1993), random-dot kinematograms (Cavanagh et al., 1985; Morgan and Cleary, 1992; Morgan and Ingle, 1994; Simpson, 1990; Troscianko, 1987; but cf. Ramachandran and Gregory, 1978), and periodic dot displays (Agonie and Gorea, 1993; Dobkins and Albright, 1990; Gorea and Papathomas, 1989; Gorea et al., 1992; Green, 1989; Papathomas et al., 1991) have consistently shown that direction discrimination persists for chromatic stimuli, albeit often less reliably than for stimuli that contain luminance contrast. Far fewer studies have been conducted using stimuli that vary along the blue/yellow dimension of color. Similar to the case for red/green stimuli, motion of blue/yellow gratings can be discerned, yet the

1218

, ,   

motion typically appears relatively impoverished compared to moving stimuli containing luminance contrast (Cavanagh and Anstis, 1991; Cavanagh et al., 1984; Dougherty et al., 1999; Gegenfurtner and Hawken, 1995; Lee and Stromeyer, 1989; Moreland, 1982). In addition to the above-mentioned experiments, several studies have attempted to quantify precisely the degree to which chromatic information contributes to motion perception. One approach has been to employ a motion nulling paradigm in which a heterochromatic (red/green or blue/yellow) grating moving in one direction is superimposed upon an achromatic (i.e., luminance) grating moving in the opposite direction. This approach allows the sensitivity of motion detectors to chromatic contrast to be quantified and expressed relative to a benchmark of sensitivity for luminance contrast. Specifically, the amount of luminance contrast in the achromatic grating required to “null” the motion of the heterochromatic grating identifies the equivalent luminance contrast (EqLC) rendered by the chromatic contrast in the heterochromatic grating. For red/green gratings, EqLC has been shown to vary from 4% to 12%, depending on the spatial and temporal frequency of the stimulus (Cavanagh and Anstis, 1991; Teller and Palmer, 1996; Thiele et al., 2002; and see Agonie and Gorea, 1993, for similar values obtained using periodic dot displays). For blue/yellow gratings, EqLC values are a bit smaller, on the order of 2% to 5% (Cavanagh and Anstis, 1991; Teller et al., 1997). In sum, there exists a substantial amount of psychophysical evidence for both red/green and blue/yellow chromatic input to motion processing.

Neural correlates of color-based motion perception Attempts to explore the neural substrates of chromatic contributions to motion perception have focused on the properties of cells in area MT of monkey extrastriate visual cortex. Area MT is recognized as a key component of the neural substrate for motion perception, with the vast majority of MT neurons exhibiting a high degree of selectivity for direction of motion (see Albright, 1993, for a review). Neurons in MT do not exhibit selectivity for object color (Albright, 1984; Baker et al., 1981; Maunsell and Van Essen, 1983b; Van Essen et al., 1981; Zeki, 1974). The absence of color selectivity in MT neurons has been heralded as evidence for the segregation of color and motion processing pathways. Only recently has attention been given to the possibility that directionally selective neurons can use information about object color to encode direction while possessing no selectivity for color per se. In support of this possibility, several studies have shown that neurons in MT are able to signal the direction of moving red/green and blue/yellow gratings, although responses are significantly weaker than, for moving luminance gratings (Dobkins and Albright, 1990, 1994; Gegenfurtner et al., 1994; Saito et al., 1989; Seidemann

et al., 1999; Thiele et al., 1999; and see Ffytche et al., 1995; Tootell et al., 1995; Wandell et al., 1999, for similar results obtained using brain imaging techniques in humans). These MT results thus mirror the impoverished, but not altogether absent, sensitivity to chromatic motion observed perceptually.

Sources of chromatic (red/green and blue/yellow) input to area MT We next address the avenues by which area MT may have access to chromatic information. To this end, we turn to a discussion of the color selectivities of the three main retinogeniculate pathways of the primate visual system— parvocellular (P), koniocellular (K), and magnocellular (M)—and then discuss the potential for each pathway to reach motion processing regions such as MT. Details of the anatomy and physiology of these three pathways can be found in several previous reviews (e.g., Dobkins, 2000; Dobkins and Albright, 1998; Hendry and Reid, 2000; Kaplan et al., 1990; Merigan and Maunsell, 1993). To begin, the names of these pathways are based on their anatomical distinction in the lateral geniculate nucleus (LGN) of the thalamus. Four LGN layers contain densely packed, small (parvo or P ) cells, and two contain more sparsely placed, large (magno or M ) cells. Just below each M and P layer exist layers (six in total) of extremely small dustlike cells, winning them the name konio or K cells. (Note that in the past, the K layers were referred to as interlaminar or intercalated.) The anatomical projections of these three pathways have been shown to remain entirely separate from the level of the retinal ganglion cells in the eye to primary visual cortex (area V1). Specifically, the P pathway originates within the midget retinal ganglion cells, which project selectively to the P layers of the LGN. The P layers, in turn, project selectively to layer 4Cb of V1. In a parallel fashion, the M pathway originates within parasol ganglion cells of the retina, which project selectively to the M layers of the LGN. The M layers, in turn, send their projections to layer 4Ca of area V1. The K pathway (or a portion thereof) originates within blue-ON bistratified ganglion cells of the retina, which project selectively to the middle two K layers of the LGN (i.e., layers K3 and K4). These K layers, in turn, send their projections to the blobs in the superficial layers of V1 (as well as sending sparse projections directly to extrastriate visual areas such as MT; see below). C R  P, K,  M C The color selectivities of cells within the P, K, and M pathways are thought to map roughly onto the red/green, blue/yellow, and luminance dimensions of color, respectively (see Chapter 30 for a discussion of other characteristics of the cells in these pathways). These differential color selectivities arise from the

different manner in which cone types feed into the three pathways. Specifically, midget ganglion cells of the P pathway receive opponent signals from L and M cones (i.e., L - M or M - L), and thus respond preferentially to modulation along the red/green dimension (e.g., De Monasterio and Gouras, 1975; De Valois et al., 1966; Derrington et al., 1984; Gouras and Zrenner, 1979, 1981; Reid and Shapley, 1992; Wiesel and Hubel, 1966). For example, a given P cell may be activated when stimulated by red light yet inhibited when stimulated by green light (or vice versa). Blue-ON bistratified cells of the K pathway receive excitatory input from Scones and inhibitory input from L and M cones, and thus respond best to modulation along the blue/yellow dimension (e.g., Calkins, 2001; Calkins et al., 1998; Dacey and Lee, 1994). Specifically, they are excited by blue light and inhibited by yellow light. There also appears to exist another class of ganglion cells, which are inhibited by blue light and excited by yellow light (e.g., Herr et al., 2003; Klug et al., 1993; and see Shinomori et al., 1999, for related psychophysical evidence); however, these cells are extremely scarce and may not be part of the K pathway. In sum, cells of both the P and K pathways exhibit selectivity for the sign of chromatic contrast (i.e., they are excited by one color and inhibited by another), a property that allows them to signal chromatic identity. Parasol ganglion cells of the M pathway receive additive (i.e., L + M) cone signals and thus respond best to luminance modulation (e.g., De Monasterio and Gouras, 1975; Derrington et al., 1984; Wiesel and Hubel, 1966). Unlike cells of the P and K pathways, M cells do not exhibit selectivity for stimulus chromaticity. Despite their lack of chromatic selectivity, however, the majority of cells within the M pathway can signal chromatic (red/green) contrast in the following manner. First, they respond to borders defined by red/green contrast and to temporal changes between red and green, without regard for the sign of chromatic contrast (Dacey, 1996; Derrington et al., 1984; Gouras and Eggers, 1983; Hubel and Livingstone, 1990; Kaiser et al., 1990; Kruger, 1979; Lee et al., 1988, 1989a, 1989b, 1989c; Logothetis et al., 1990; Schiller and Colby, 1983; Shapley and Kaplan, 1989; Valberg et al., 1992). A second way in which M cells can signal chromatic contrast arises from the fact that the red/green luminance balance point—the contrast for which red and green phases of a stimulus elicit responses of equal magnitude—varies across the population of M cells. This variability ensures that, even at equiluminance, M cells as a population can never be truly silenced by heterochromatic stimuli (Logothetis et al., 1990). Whether M cells also respond to modulation along the blue/yellow dimension has been an issue of debate. Although early reports suggested negligible S-cone input to M cells (e.g., Dacey and Lee, 1994; Lee et al., 1988), more recently, substantial S-cone input has been reported (Calkins, 2001;

  :    

1219

Chatterjee and Callaway, 2002). In particular, Chatterjee and Callaway found that S-cones contribute roughly 10% to the responsivity of M cells, a percentage that reflects the proportion of S-cones in the retina. They further found that all three cone types provide input of the same sign (i.e., L + M + S). This indicates that, similar to the case for red/green stimuli, M cells can respond to blue/yellow contrast, yet are not selective for the sign of chromatic variation. In sum, the responses of M cells to both red/green and blue/yellow chromatic contrast can be said to be unsigned, that is, M cells provide information about chromatic contrast while not conveying information about chromatic identity per se. This unsigned chromatic signal can potentially allow for chromatic motion correspondence observed psychophysically and in area MT, an issue we return to later in this chapter. P, K,  M P P  MT Although it was originally believed that only cells of the M pathway projected to MT, anatomical evidence now suggests that the P and K pathways also provide at least minimal input. The projections from the M pathway are the most clear and well studied. Specifically, projections from magnocellularrecepient layer 4Ca of V1 project to layer 4B (Livingstone and Hubel, 1987), which in turn sends both direct and indirect (via area V2) projections to MT (DeYoe and Van Essen, 1985; Lund et al., 1975; Maunsell and Van Essen, 1983a; Shipp and Zeki, 1985). Direct evidence for P and K pathway input to area MT is far less abundant. In one study using an in vitro photostimulation technique, Sawatari and Callaway (1996) reported that many layer 4B neurons receive strong input from both magnocellular-recipient layer 4Ca neurons and parvocellular-recipient layer 4Cb neurons. Because layer 4B projects directly to area MT, such findings suggest that MT may receive a substantial amount of P-cell input. Most recently, Hendry and Reid (2000) reported direct (yet sparse) connections between K layers in the LGN and area MT (which bypass area V1 altogether). In addition to these anatomical studies, other experiments have used neurophysiological methods to measure the amount of functional M and P pathway input to MT. In these studies, stimulus-evoked activity in MT was measured while specific LGN laminae were simultaneously inactivated. The results of these studies revealed large decrements in MT responsivity during magnocellular LGN inactivation but only small decrements in responsivity during parvocellular LGN inactivation (Maunsell et al., 1990). Based on these results, it appears that MT receives predominantly M pathway-driven input but a small amount of P pathwaydriven input. Note that because their relevance was underappreciated at the time these experiments were conducted, inputs from the K pathway have not been considered.

1220

, ,   

Relative contributions of P, K, and M pathways to chromatic motion processing With the knowledge that all three pathways (P, K, and M) send projections to area MT, there is the potential for each to contribute to the chromatic motion responses observed in this area. Several neurophysiological experiments have investigated the nature of chromatic motion responses in MT as a way of revealing the different pathways’ contributions. The resulting evidence for M, K, and P pathway inputs are reviewed in sequence, below. E   M-P C Two main studies have provided evidence that red/green chromatic responses originating in M cells may be sufficient to account for chromatic motion responses observed in area MT. In the first study, Dobkins and Albright (1994) employed a novel stimulus designed to test whether chromatic motion detection in MT is based on unsigned chromatic border information (reflective of signals carried within the M pathway) or chromatic identity per se (reflective of signals carried within the P pathway). The stimulus consisted of red/green heterochromatic sine-wave gratings that undergo repetitive chromatic contrast reversal with each spatial displacement (Fig. 82.2A). Under these conditions, motion correspondence based on unsigned chromatic borders is placed in direct opposition to correspondence based on conservation of the chromatic sign. If motion detectors in MT use M pathwaydriven input, they should respond best to motion in the direction of the nearest chromatic border, regardless of chromatic sign (in the chromatically “unsigned” direction; solid arrow in Fig. 82.2A), since spatial proximity is itself a potent cue for motion correspondence (e.g., Ullman, 1980). If, on the other hand, MT neurons use P pathway-driven input, they should respond best to motion in the direction that preserves chromatic sign (in the chromatically “signed” direction; dashed arrow in Fig. 82.2A). Representative data from one MT neuron tested with this stimulus are shown in Figure 82.2B. Data were obtained for both the signed (S) and unsigned (U) cues moving in the neuron’s preferred direction at each of eight different luminance contrasts between the red and green stripes of the grating. These results show that for a small range of luminance contrasts near equiluminance, selectivity for direction of motion [as quantified by an index of directionality: DI = (U - S)/(U + S)] was such that the neuron responded best to motion of the unsigned chromatic cue (DI > 0). Away from equiluminance, the neuron was more sensitive to motion in the signed direction (DI < 0). This latter effect reflects a preference to preserve the sign of luminance contrast when it is also present in the stimulus (see Dobkins and Albright, 1994, Fig. 5). As outlined above, the larger responses to unsigned versus signed motion near equiluminance suggest that the

F 82.2. A, Heterochromatic gratings that undergo reversal of the sign of chromatic contrast coincident with each spatial displacement. The position in space of a one-dimensional slice of the grating is shown for four successive moments in time. Motion of the proximal unsigned chromatic border is rightward (solid arrow), while motion of the signed chromatic cue is leftward (dashed arrow). B, Data from one MT neuron presented with the moving contrastreversed stimulus. Luminance contrast between the red and green phases of the grating was varied across eight different levels ranging in equal (5.4%) intervals from -24.2% (red more luminous) to +13.9% (green more luminous). Per-stimulus-time-histograms obtained in response to movement of the unsigned (U) and signed

(S) cues in the neuron’s preferred direction are shown at the bottom for each red/green luminance contrast level tested (S/S = spikes/sec). Direction indices (DI = (U - S)/(U + S)), computed from responses elicited by the motion of unsigned and signed cues, are plotted as a function of luminance contrast (above). In line with the prediction based on M-cell responses (see text), near equiluminance (i.e., 0% luminance contrast), the neuron responded best to motion in the unsigned direction (i.e., DIs were positive). Away from equiluminance, DIs became negative, indicating that under these conditions, the neuron was more sensitive to motion in the direction that preserved the sign of both chromatic and luminance contrast. (See color plate 60.)

ability of MT neurons to signal the motion of chromatically defined stimuli can be accounted for by unsigned chromatic responses arising within the M pathway. In addition, the MT responses to heterochromatic gratings seen in Figure 82.2B are strikingly similar to those obtained in corresponding human psychophysical experiments (Dobkins and Albright, 1993), suggesting that MT provides the neural substrate for chromatic motion processing revealed perceptually. In a second series of experiments, Thiele et al. (1999, 2001) adopted an EqLC paradigm (after Cavanagh and Anstis, 1991) in order to measure the strength of chromatic input to motion detectors in MT. As described earlier in this

chapter, the stimulus employed in this paradigm consisted of two sine-wave gratings—one heterochromatic (red/green), the other achromatic (yellow/black)—superimposed and moving in opposite directions (Fig. 82.3A). The amount of luminance contrast in the achromatic grating required to null the motion of the heterochromatic grating identifies the EqLC rendered by the chromatic contrast in the heterochromatic stimulus. In one experiment (Thiele et al., 2001), both MT neural responses and psychophysical data were obtained simultaneously from macaque monkeys, allowing for direct comparisons between psychophysical and neural estimates of EqLC. Representative data from one

  :    

1221

14 12 10 8 6 4 2 0 –2 –30

neuronal psychophysical mean EqLC (%)

mean EqLC (%)

C

–20

–10

0

10

20

heterochromatic luminance contrast (%)

F 82.3. A, Space-time plot of the EqLC stimulus. This stimulus consists of two superimposed sinusoidal gratings—one achromatic (yellow/black), the other heterochromatic (red/green) —moving in opposite directions. The amount of luminance contrast in the achromatic grating required to null the motion of the heterochromatic grating identifies the EqLC rendered by the chromatic contrast in the heterochromatic grating. B, Representative neuronal and perceptual responses to the stimulus. In this example, the heterochromatic component was set to be equiluminant for the monkey. Gray per-stimulus-time-histograms (PSTHs) illustrate activity recorded when the heterochromatic component moved in the MT neuron’s preferred direction (and thus, the achromatic component moved simultaneously in the antipreferred direction). Black-outlined PSTHs illustrate activity for the opposite directional polarity. Means and standard errors of neuronal responses (sp/s) are plotted as a function of achromatic luminance contrast (above, left). Neuronal EqLC was determined from intersections of curve fits to these data points. Corresponding psychophysical data are also plotted (above, right). Shown is the proportion of decisions in

1222

, ,   

30

14 12 10 8 6 4 2 0 –2 –30

–20

–10

0

10

20

30

heterochromatic luminance contrast (%)

favor of motion in the achromatic direction as a function of luminance contrast in achromatic grating. Psychophysical EqLC was determined from the null point (i.e., the point at which the perceived direction of motion was equally likely in favor of either stimulus component). C, Mean neuronal (black lines) and psychophysical ( grey lines) EqLC values for two monkeys are plotted as a function of luminance contrast in the heterochromatic grating (0% = equiluminance, +25% = green more luminous, -25% = red more luminous). These results reveal a strong correpondence between neuronal and psychophysical estimates of EqLC, providing evidence that MT underlies chromatic motion processing revealed perceptually. In addition, in line with the M-cell model (see text), a marked decrease in EqLC was observed for heterochromatic gratings possessing 25% luminance contrast. (Reprinted from Neuron, vol. 32, Thiele, Dobkins and Albright, Neural Correlates of Chromatic Motion Perception, Pages 251–358, Copyright 2001 with permission from Elsevier Science.) (See color plate 61.)

testing session are presented in Figure 82.3B. Shown are simultaneously collected neural and psychophysical data for a condition in which the heterochromatic grating was set to be equiluminant for the monkey. These data reveal a strong correspondence between neural EqLC (11.6%) and psychophysical EqLC (10.7%), providing further evidence that MT underlies chromatic motion processing revealed perceptually. In order to determine whether chromatic input to MT could be accounted for by chromatic responses originating in M cells, neural and psychophysical EqLC values obtained for equiluminant stimuli were compared to those obtained when the heterochromatic grating contained ±25% luminance contrast (+25% = green more luminous than red, -25% = red more luminous than green). Based on a model of the known scatter of red/green balance points across M cells, Cavanagh and Anstis (1991) demonstrated that EqLC should be greatest at equiluminance and should decline rapidly as luminance contrast is added to the heterochromatic grating (see also Thiele et al., 1999). That is, the effectiveness of the red/green portion of the heterochromatic grating is expected to diminish when luminance contrast is added to the grating. By contrast, a model based on chromatically selective P-cell responses predicts that EqLC should remain constant as luminance contrast is added to the heterochromatic grating, thus indicating an unvarying contribution of red/green contrast to motion processing with increases in luminance contrast. In Figure 82.3C, mean neural and psychophysical EqLC values obtained for two monkeys are plotted as a function of luminance contrast in the heterochromatic grating. In line with the M-cell model, a marked decrease in EqLC was observed for heterochromatic gratings possessing 25% luminance contrast. Such findings lend further support to the notion that chromatic motion responses in MT can be accounted for by M-cell input. Interestingly, this pattern of results observed in monkey subjects is quite different from that obtained in human observers. In humans, EqLC is found to be invariant with increases in heterochromatic luminance contrast (Cavanagh and Anstis, 1991; Thiele et al., 2002). Such results conform to the P-cell model, leaving open the possibility that, in humans, there exists a relatively greater P-cell contribution to MT and/or that areas outside MT might contribute to chromatic motion processing. E  K P I? The above-described experiments employed red/green stimuli and were aimed at revealing M versus P pathway contributions to chromatic motion processing in MT. Other recent neurophysiological studies in macaque monkeys have addressed the potential contribution of K pathway input by employing blue/yellow stimuli that isolate activity in S-cones (Seidemann et al., 1999). The results of these studies, shown in Figure 82.4,

F 82.4. Mean responses and standard errors for a sample of MT neurons are plotted as a function of contrast, separately for luminance and blue/yellow (i.e., S-cone-isolating) stimuli. The response to blue/yellow stimuli may potentially result from either K- or M-pathway input to MT (see text). (Data from Seidemann, Poirson, Wandell and Newsome, Neuron, vol. 24, Color Signals in Area MT of the Macaque Monkey, Pages 911–917, Copyright 1999 with permission from Elsevier Science.)

reveal that blue/yellow stimuli elicit directionally selective responses in MT, although the contrast sensitivity of MT neurons is much lower for blue/yellow, compared to achromatic, moving gratings (see Wandell et al., 1999, for similar results obtained using functional magnetic resonance imaging in human subjects). This sensitivity to blue/yellow modulation may arise from the direct (yet sparse) connections known to exist between K layers in the LGN and area MT (Hendry and Reid, 2000). Alternatively, or in addition to this possibility, blue/yellow information might reach area MT via the M pathway, since recent evidence suggests that cells of the M pathway receive significant S-cone input (Calkins, 2001; Chatterjee and Callaway, 2002). Certainly, more extensive studies will be required to unequivocally establish the nature of blue/yellow input to MT. E   P-P C While much of both the psychophysical and neurophysiological work has focused on the use of chromatic contrast as a cue for motion correspondence, other studies have demonstrated the influence of chromatic information on the integration versus segmentation of motion signals. Results from two different studies investigating this aspect of motion processing suggest that Ppathway input may underlie chromatic modulation of motion responses in area MT. In the first study, Dobkins et al. (1998) used moving plaid patterns (consisting of two superimposed component gratings whose motion directions differ from one another) to investigate whether motion integration in MT is influenced by chromatic information. In a previous psychophysical study, Kooi et al. (1992) showed

  :    

1223

that when both component gratings that make up the plaid pattern are either red-bright/green-dark or greenbright/red-dark (referred to as symmetric plaids), the plaid is perceived to move coherently. Conversely, when one grating is red-bright/green-dark and the other is green-bright/reddark (referred to as asymmetric plaids), the components appear to slide noncoherently across one another. Thus, depending on the color similarity of the component gratings, either pattern or component motion dominates motion perception (see also Cropper et al., 1996; Dobkins et al., 1992; Farell, 1995; Kooi and De Valois, 1992; Krauskopf and Farell, 1990; Krauskopf et al., 1996, for similar psychophysical results obtained with moving color plaids). To reveal the neural basis of this perceptual phenomenon, Dobkins et al. (1998) measured responses in MT neurons elicited by symmetric versus asymmetric plaid patterns. In these experiments, they used a modified version of a plaid stimulus in which only one of the two component gratings moved, while the other remained stationary. In contrast to conventional plaids, this plaid design yields a single motion percept for both coherent and noncoherent plaid conditions (Fig. 82.5A). Example data from one MT neuron tested with symmetric plaids (perceived to move coherently) and asymmetric plaids (perceived to move noncoherently) are shown in Figure 82.5B. For each of these two plaid types, responses were obtained for both pattern motion and component motion presented in the neuron’s preferred direction. When the plaid pattern was symmetric, the neuron responded better to pattern than to component motion. Conversely, when the plaid pattern was asymmetric, the neuron responded better to component motion. The responses of this neuron were thus remarkably similar to those observed in psychophysical experiments; that is, pattern motion dominated under symmetric conditions, while component motion dominated under asymmetric conditions. This effect of color cues on motion integration in MT addresses the origin of the chromatic signals. It is highly unlikely that this modulatory effect could arise from signals originating within the M pathway, since the cells of this pathway lack chromatic selectivity and thus should not respond differentially to the red-bright/green-dark versus the green-bright/red-dark component gratings that make up the plaid stimuli. This is especially true given that the luminance contrast of the component gratings in these experiments was high enough (i.e., roughly 20%) to saturate the responses of M cells (Kaplan and Shapley, 1986). Thus, despite known variabity in red/green balance points across the M-cell population (as described earlier), all M cells are expected to respond equally (and maximally) to the two component grating types. By contrast, the chromatic selectivity of P cells allows them to respond differentially to redbright/green-dark versus green-bright/red-dark component gratings. For example, a red-on P cell (which is excited by red

1224

, ,   

F 82.5. A, Color plaid stimuli in which one component moves while the other remains stationary. Under these conditions, both noncoherent and coherent motion are associated with a single motion percept. When the plaids are symmetric (top panel)—that is, both components are green-bright/red-dark—coherent pattern motion is perceived (in this example, moving upward). When the plaids are asymmetric (bottom panel)—that is, one green-bright/ red-dark and one red-bright/green-dark component—noncoherent component motion is perceived (in this example, moving up to the right). B, Data from one MT neuron presented with color plaid stimuli. Responses are shown for both pattern motion (black bars) and component motion (white bars) presented in the neuron’s preferred direction. When the plaid pattern was symmetric, the neuron responded better to pattern than to component motion. Conversely, when the plaid pattern was asymmetric, the neuron responded better to component motion. This influence of color information on motion integration suggests the existence of functional chromatic P pathway input to area MT (see text). (See color plate 62.)

light and inhibited by green light) is expected to respond more strongly to a red-bright/green-dark grating than to a green-bright/red-dark grating. A green-on P cell is expected to respond in a complementary fashion. Because P cells can distinguish the different component gratings, their input to MT has the potential to exert a modulatory influence on the integration of chromatic motion signals. Thus, in addition to the previously described evidence for an M-cell contribution to chromatic motion processing in MT, these color plaid

F 82.6. A, Homochromatic and heterochromatic stochastic motion stimuli. The stimulus consisted of a random array of small bright dots moving against a dark background. A varying proportion of the dots moved in the same direction, and thus constituted a motion signal, while the rest moved randomly and constituted motion noise. The proportion of signal dots, expressed as a percentage of the total and termed the correlation, is shown for three different levels of correlation: 0%, 50%, and 100%. In the homochromatic condition (upper panel), all of the dots had the same color. In the heterochromatic condition (bottom panel), the signal dots were red (open circles) while the noise dots were green (filled circles) or vice versa. B, Simultaneously obtained behavioral and neuronal performance functions obtained in one experiment using the

random-dot stimuli. Behavioral functions are shown at the top, and the corresponding neuronal functions obtained at the same time are shown below. Homochromatic, open triangles and dashed lines; heterochromatic, closed circles and solid lines. A large (~10-fold), statistically significant decrease in the behavioral heterochromatic threshold (thresholds: homochromatic = 5.0%, heterochromatic = 0.7%) was accompanied by a large (~10-fold), statistically significant decrease in the neuronal heterochromatic threshold (thresholds: homochromatic = 23.8%, heterochromatic = 2.4%). (Reprinted from Neuron, vol. 24, Croner and Albright, Seeing The Big Picture: Integration of Image Cues in the Primate Visual System, Pages 777–789, Copyright 1999 with permission from Elsevier Science.) (See color plate 63.)

experiments suggest that there also exists functional P-cell chromatic input to area MT. In another set of experiments, Croner and Albright (1997, 1999) used a stochastic motion stimulus (after Newsome and Paré, 1988; Williams and Sekuler, 1984) to investigate the influence of color on motion segmentation both psychophysically and in area MT. This stimulus consisted of a random array of small bright dots moving against a dark background. A varying proportion of the dots moved in the same direction and thus constituted a motion signal, while the rest moved randomly and constituted motion noise. In the conventional configuration, all of the dots were of the same color (Fig. 82.6A, upper panel, “homochromatic”). When subjects reported the perceived direction of motion in these stimuli, their performance improved as the signal strength (the proportion of dots moving in a correlated fashion) increased (e.g., Newsome and Paré, 1988). To investigate the effects of color on motion segmentation, Croner and Albright simply made the signal and noise dots differ in color (Fig. 82.6A, lower panel, “heterochromatic”). When humans and monkeys were tested on these stimuli, their performance on the heterochomatic

stimuli was signifcantly better than their performance on the homochromatic stimuli (Fig. 82.6B; and see Croner and Albright, 1997), indicating that motion signals can be segmented—and their detection thus facilitated—on the basis of chromatic identity information. To examine the neural basis of this effect, Croner and Albright (1999) recorded responses of MT neurons to the stimuli depicted in Figure 82.6A while monkeys performed a motion discrimination task. Approximately 20% of the MT neurons studied showed a significant improvement in their ability to discriminate direction when the signal and noise dots were of different colors. For many neurons (e.g., Fig. 82.6B), this change in neuronal discriminability closely paralled changes in perceptual discriminability observed on the same trials. Thus, a subpopulation of MT neurons is strongly influenced by chromatic identity information, just as are observers’ perceptual judgments. What kind of input is required to mediate this chromatic contribution to MT responses? As for the chromatic plaid studies described above, the dots in the stimuli diagrammed in Figure 82.6A were of high luminance contrast relative to

  :    

1225

background. Since the responses of cells within the M pathway are not chromatically selective and because they saturate under these high-contrast conditions, it is unlikely that the relevant input comes from residual chromatic sensitivity of neurons in this pathway. Rather, chromatically selective infomation such as that carried by the P pathway must be involved. More specifically, Croner and Albright (1997) proposed that contextual cues for feature grouping lead to adjustments of the relative gain of the sensory signals reaching or passing through area MT. These gain adjustments, accordingly, enable motion signals bearing a common property (e.g., the color red) to be processed selectively, with minimal disruption by dynamic noise.

Concluding remarks In conclusion, there now exists substantial evidence that all three color dimensions—luminance, red/green, and blue/yellow—influence motion processing revealed perceptually and in area MT. This color information appears to reach MT via input from the three subcortical pathways— magnocellular, parvocellular, and koniocellular, respectively. Moreover, contrary to previous views on the relationship between color and motion processing, the evidence reviewed here indicates that color augments motion processing in profound and functionally specific ways. We anticipate that future studies will address the neuronal circuitry and synaptic interactions that give rise to these functions.

Acknowledgments We thank Greg Horwitz and Bart Krekelberg for thoughtful discussion and comments on the manuscript. T.D.A. is an Investigator of the Howard Hughes Medical Institute. REFERENCES Agonie, C., and A. Gorea, 1993. Equivalent luminance contrast of red-green drifting stimuli: dependency on luminance-color interactions and on the psychophysical task, J. Opt. Soc. Am. A, 10:1341–1352. Albright, T. D., 1984. Direction and orientation selectivity of neurons in visual area MT of the macaque, J. Neurophysiol., 52:1106–1130. Albright, T. D., 1993. Cortical processing of visual motion, in Visual Motion and Its Use in the Stabilization of Gaze ( J. Wallman and F. A. Miles, eds.), Amsterdam: Elsevier, pp. 177–201. Baker, J. F., S. E. Petersen, W. T. Newsome, and J. M. Allman, 1981. Visual response properties of neurons in four extrastriate visual areas of the owl monkey (Aotus trivirgatus): a quantitative comparison of medial, dorsomedial, dorsolateral, and middle temporal areas, J. Neurophysiol., 45:397–416. Boynton, R. M., R. T. Eskew, and C. X. Olson, 1985. Blue cones contribute to border distinctness, Vis. Res., 25:1349–1352. Calkins, D. J., 2001. Seeing with S cones, Prog. Retin. Eye Res., 20: 255–287.

1226

, ,   

Calkins, D. J., Y. Tsukamoto, and P. Sterling, 1998. Microcircuitry and mosaic of a blue-yellow ganglion cell in the primate retina, J. Neurosci., 18:3373–3385. Cavanagh, P., and S. Anstis, 1991. The contribution of color to motion in normal and color-deficient observers, Vis. Res., 31:2109–2148. Cavanagh, P., J. Boeglin, and O. E. Favreau, 1985. Perception of motion in equiluminous kinematograms, Perception, 14:151–162. Cavanagh, P., C. W. Tyler, and O. E. Favreau, 1984. Perceived velocity of moving chromatic gratings, J. Opt. Soc. Am. A, 1:893–899. Chatterjee, S., and E. M. Callaway, 2002. S cone contributions to the magnocellular visual pathway in macaque monkey, Neuron, 35:1135–1146. Croner, L. J., and T. D. Albright, 1997. Image segmentation enhances discrimination of motion in visual noise, Vis. Res., 37:1415–1427. Croner, L. J., and T. D. Albright, 1999. Segmentation by color influences responses of motion-sensitive neurons in the cortical middle temporal visual area, J. Neurosci., 19:3935–3951. Cropper, S. J., and A. M. Derrington, 1994. Motion of chromatic stimuli: first-order or second-order? Vis. Res., 34:49–58. Cropper, S. J., and A. M. Derrington, 1996. Rapid colour-specific detection of motion in human vision, Nature, 379:72–74. Cropper, S. J., K. T. Mullen, and D. R. Badcock, 1996. Motion coherence across different chromatic axes, Vis. Res., 36:2475– 2488. Dacey, D. M., 1996. Circuitry for color coding in the primate retina, Proc. Natl. Acad. Sci. USA, 93:582–588. Dacey, D. M., and B. B. Lee, 1994. The “blue-on” opponent pathway in primate retina originates from a distinct bistratified ganglion cell type, Nature, 367:731–735. De Monasterio, F. M., and P. Gouras, 1975. Functional properties of ganglion cells of the rhesus monkey retina, J. Physiol. (Lond.), 251:167–195. De Valois, R. L., I. Abramov, and G. H. Jacobs, 1966. Analysis of response patterns of LGN cells, J. Opt. Soc. Am., 56:966–977. De Valois, R. L., H. C. Morgan, M. C. Polson, W. R. Mead, and E. M. Hull, 1974. Psychophysical studies of monkey vision. I. Macaque luminosity and color vision tests, Vis. Res., 14:53–67. Derrington, A. M., and G. B. Henning, 1993. Detecting and discriminating the direction of motion of luminance and colour gratings, Vis. Res., 33:799–811. Derrington, A. M., J. Krauskopf, and P. Lennie, 1984. Chromatic mechanisms in lateral geniculate nucleus of macaque, J. Physiol. (Lond.), 357:241–265. DeYoe, E. A., and D. C. Van Essen, 1985. Segregation of efferent connections and receptive field properties in visual area V2 of the macaque, Nature, 317:58–61. Dobkins, K. R., 2000. Moving colors in the lime light, Neuron, 25:15–18. Dobkins, K. R., and T. D. Albright, 1990. Color facilitates motion in visual area MT, Soc. Neurosci. Abstr., 16:1220. Dobkins, K. R., and T. D. Albright, 1993. What happens if it changes color when it moves?: psychophysical experiments on the nature of chromatic input to motion detectors, Vis. Res., 33:1019–1036. Dobkins, K. R., and T. D. Albright, 1994. What happens if it changes color when it moves?: the nature of chromatic input to macaque visual area MT, J. Neurosci., 14:4854–4870. Dobkins, K. R., and T. D. Albright, 1998. The influence of chromatic information on visual motion processing in the primate visual system, in High-Level Motion Processing—Computational,

Neurobiological and Psychophysical Perspectives (T. Watanabe ed.), Cambridge, MA: MIT Press, pp. 53–94. Dobkins, K. R., G. R. Stoner, and T. D. Albright, 1992. Oculomotor responses to perceptually coherent and non-coherent plaids, Soc. Neurosci. Abstr., 18:1034. Dobkins, K. R., G. R. Stoner, and T. D. Albright, 1998. Perceptual, oculomotor and neural responses to moving color plaids, Perception, 27:681–709. Dobkins, K. R., and D. Y. Teller, 1996. Infant motion : detection (M : D) ratios for chromatic-defined and luminance-defined moving stimuli, Vis. Res., 36:3293–3310. Dougherty, R. F., W. A. Press, and B. A. Wandell, 1999. Perceived speed of colored stimuli, Neuron, 24:893–899. Eisner, A., and D. I. Macleod, 1980. Blue-sensitive cones do not contribute to luminance, J. Opt. Soc. Am. A, 70:121–123. Farell, B., 1995. Spatial structure and the perceived motion of objects of different colors, in Early Vision and Beyond (T. Papathomas, C. Chubb, A. Gorea, and E. Kowler, eds.), Cambridge, MA: MIT Press, pp. 121–131. Ffytche, D. H., B. D. Skidmore, and S. Zeki, 1995. Motion-fromhue activates area V5 of human visual cortex, Proc. R. Soc. Lond. B Biol. Sci., 260:353–358. Gegenfurtner, K. R., and M. J. Hawken, 1995. Temporal and chromatic properties of motion mechanisms, Vis. Res., 35:1547–1563. Gegenfurtner, K. R., and M. J. Hawken, 1996. Interaction of motion and color in the visual pathways, Trends Neurosci., 19:394–401. Gegenfurtner, K. R., D. C. Kiper, J. M. Beusmans, M. Carandini, Q. Zaidi, and J. A. Movshon, 1994. Chromatic properties of neurons in macaque MT, Vis. Neurosci., 11:455–466. Golomb, B., R. A. Andersen, K. Nakayama, D. I. MacLeod, and A. Wong, 1985. Visual thresholds for shearing motion in monkey and man, Vis. Res., 25:813–820. Gorea, A., J. Lorenceau, J. D. Bagot, and T. V. Papathomas, 1992. Sensitivity to colour- and to orientation-carried motion respectively improves and deteriorates under equiluminant background conditions, Spatial Vis., 6:285–302. Gorea, A., and T. V. Papathomas, 1989. Motion processing by chromatic and achromatic visual pathways, J. Opt. Soc. Am. A, 6:590–602. Gouras, P., and H. Eggers, 1983. Responses of primate retinal ganglion cells to moving spectral contrast, Vis. Res., 23:1175–1182. Gouras, P., and E. Zrenner, 1979. Enhancement of luminance flicker by color-opponent mechanisms, Science, 205:587–589. Gouras, P., and E. Zrenner, 1981. Color coding in primate retina, Vis. Res., 21:1591–1598. Green, M., 1989. Color correspondence in apparent motion, Percept. Psychophys., 45:15–20. Hawken, M. J., K. R. Gegenfurtner, and C. Tang, 1994. Contrast dependence of colour and luminance motion mechanisms in human vision, Nature, 367:268–270. Hendry, S. H. C., and R. C. Reid, 2000. The koniocellular pathway in primate vision, Annu. Rev. Neurosci., 23:127–153. Herr, S., K. Klug, P. Sterling, and S. Schein, 2003. Inner S-cone bipolar cells provide all of the central elements for S cones in macaque retina. J. Comp. Neurol. (in press). Hubel, D. H., and M. S. Livingstone, 1990. Color and contrast sensitivity in the lateral geniculate body and primary visual cortex of the macaque monkey, J. Neurosci., 10:2223–2237. Kaiser, P. K., B. B. Lee, P. R. Martin, and A. Valberg, 1990. The physiological basis of the minimally distinct border demonstrated in the ganglion cells of the macaque retina, J. Physiol. (Lond.), 422:153–183.

Kaplan, E., B. B. Lee, and R. M. Shapley, 1990. New views of primate retinal function, in Progress in Retinal Research (N. N. Osborne and G. J. Chader, eds.), New York: Pergamon Press, pp. 273–336. Kaplan, E., and R. M. Shapley, 1986. The primate retina contains two types of ganglion cells, with high and low contrast sensitivity, Proc. Natl. Acad. Sci. USA, 83:2755–2757. Klug, K., Y. Tsukamoto, P. Sterling, and S. J. Schein, 1993. Blue cones OFF-midget ganglion cells in macaque, Invest. Ophthalmol. Vis. Sci. Suppl., 34:986. Kooi, F. L., and K. K. De Valois, 1992. The role of color in the motion system, Vis. Res., 32:657–668. Kooi, F. L., K. K. De Valois, E. Switkes, and D. H. Grosof, 1992. Higher-order factors influencing the perception of sliding and coherence of a plaid, Perception, 21:583–598. Krauskopf, J., and B. Farell, 1990. Influence of colour on the perception of coherent motion, Nature, 348:328–331. Krauskopf, J., H. J. Wu, and B. Farell, 1996. Coherence, cardinal directions and higher-order mechanisms, Vis. Res., 36:1235– 1245. Kruger, J. K., 1979. Responses to wavelength contrast in the afferent visual systems of the cat and the rhesus monkey, Vis. Res., 19:1351–1358. Lee, B. B., P. R. Martin, and A. Valberg, 1988. The physiological basis of heterochromatic flicker photometry demonstrated in the ganglion cells of the macaque retina, J. Physiol. (Lond.), 404:323–347. Lee, B. B., P. R. Martin, and A. Valberg, 1989a. Amplitude and phase of responses of macaque retinal ganglion cells to flickering stimuli, J. Physiol. (Lond.), 414:245–263. Lee, B. B., P. R. Martin, and A. Valberg, 1989b. Nonlinear summation of M- and L-cone inputs to phasic retinal ganglion cells of the macaque, J. Neurosci., 9:1433–1442. Lee, B. B., P. R. Martin, and A. Valberg, 1989c. Sensitivity of macaque retinal ganglion cells to chromatic and luminance flicker, J. Physiol. (Lond.), 414:223–243. Lee, J., and C. F. D. Stromeyer, 1989. Contribution of human shortwave cones to luminance and motion detection, J. Physiol. (Lond.), 413:563–593. Lindsey, D. T., and D. Y. Teller, 1990. Motion at isoluminance: discrimination/detection ratios for moving isoluminant gratings, Vis. Res., 30:1751–1761. Livingstone, M. S., and D. H. Hubel, 1987. Connections between layer 4B of area 17 and the thick cytochrome oxidase stripes of area 18 in the squirrel monkey, J. Neurosci., 7:3371– 3377. Logothetis, N. K., P. H. Schiller, E. R. Charles, and A. C. Hurlbert, 1990. Perceptual deficits and the activity of the coloropponent and broad-band pathways at isoluminance, Science, 247:214–217. Lund, J. S., R. D. Lund, A. E. Hendrickson, A. H. Bunt, and A. F. Fuchs, 1975. The origin of efferent pathways from the primary visual cortex, area 17, of the macaque monkey as shown by retrograde transport of horseradish peroxidase, J. Comp. Neurol., 164:287–303. Maunsell, J. H., T. A. Nealey, and D. D. DePriest, 1990. Magnocellular and parvocellular contributions to responses in the middle temporal visual area (MT) of the macaque monkey, J. Neurosci., 10:3323–3334. Maunsell, J. H., and D. C. Van Essen, 1983a. The connections of the middle temporal visual area (MT) and their relationship to a cortical hierarchy in the macaque monkey, J. Neurosci., 3:2563– 2586.

  :    

1227

Maunsell, J. H., and D. C. Van Essen, 1983b. Functional properties of neurons in middle temporal visual area of the macaque monkey. I. Selectivity for stimulus direction, speed, and orientation, J. Neurophysiol., 49:1127–1147. Merigan, W. H., and J. H. Maunsell, 1993. How parallel are the primate visual pathways? Annu. Rev. Neurosci., 16:369–402. Metha, A. B., A. J. Vingrys, and D. R. Badcock, 1994. Detection and discrimination of moving stimuli: the effects of color, luminance, and eccentricity, J. Opt. Soc. Am. A, 11:1697– 1709. Moreland, J. D., 1982. Spectral sensitivity measured by motion photometry, Doc. Ophthalmol. Proc. Ser., 33:61–66. Morgan, M. J., and R. Cleary, 1992. Effects of colour substitutions upon motion detection in spatially random patterns, Vis. Res., 32:815–821. Morgan, M. J., and G. Ingle, 1994. What direction of motion do we see if luminance but not colour contrast is reversed during displacement? Psychophysical evidence for a signed-colour input to motion detection, Vis. Res., 34:2527–2535. Mullen, K. T., and J. C. Boulton, 1992. Absence of smooth motion perception in color vision, Vis. Res., 32:483–488. Nakayama, K., 1985. Biological image motion processing: a review, Vis. Res., 25:625–660. Newsome, W. T., K. H. Britten, and J. A. Movshon, 1989. Neuronal correlates of a perceptual decision, Nature, 341:52– 54. Newsome, W. T., and E. B. Paré, 1988. A selective impairment of motion perception following lesions of the middle temporal visual area (MT), J. Neurosci., 8:2201–2211. Palmer, J., L. A. Mobley, and D. Y. Teller, 1993. Motion at isoluminance: discrimination/detection ratios and the summation of luminance and chromatic signals, J. Opt. Soc. Am. A, 10:1353– 1362. Papathomas, T. V., A. Gorea, and B. Julesz, 1991. Two carriers for motion perception: color and luminance, Vis. Res., 31:1883– 1892. Ramachandran, V. S., and R. L. Gregory, 1978. Does colour provide an input to human motion perception? Nature, 275:55– 56. Reid, R. C., and R. M. Shapley, 1992. Spatial structure of cone inputs to receptive fields in primate lateral geniculate nucleus, Nature, 356:716–718. Saito, H., K. Tanaka, H. Isono, M. Yasuda, and A. Mikami, 1989. Directionally selective response of cells in the middle temporal area (MT) of the macaque monkey to the movement of equiluminous opponent color stimuli, Exp. Brain Res., 75:1– 14. Sawatari, A., and E. M. Callaway, 1996. Convergence of magnoand parvocellular pathways in layer 4B of macaque primary visual cortex, Nature, 380:442–446. Schiller, P. H., and C. L. Colby, 1983. The responses of single cells in the lateral geniculate nucleus of the rhesus monkey to color and luminance contrast, Vis. Res., 23:1631–1641. Seidemann, E., A. B. Poirson, B. A. Wandell, and W. T. Newsome, 1999. Color signals in area MT of the macaque monkey, Neuron, 24:911–917.

1228

, ,   

Shapley, R., and E. Kaplan, 1989. Responses of magnocellular LGN neurons and M retinal ganglion cells to drifting heterochromatic gratings, Invest. Ophthalmol. Vis. Sci., 30:323. Shinomori, K., L. Spillmann, and J. S. Werner, 1999. S-cone signals to temporal OFF-channels: asymmetrical connections to postreceptoral chromatic mechanisms, Vis. Res., 39:39–49. Shipp, S., and S. Zeki, 1985. Segregation of pathways leading from area V2 to areas V4 and V5 of macaque monkey visual cortex, Nature, 315:322–325. Simpson, W. A., 1990. The use of different features by the matching process in short-range motion, Vis. Res., 30:1421–1428. Stockman, A., D. I. MacLeod, and D. D. DePriest, 1991. The temporal properties of the human short-wave photoreceptors and their associated pathways, Vis. Res., 31:189–208. Teller, D. Y., T. E. Brooks, and J. Palmer, 1997. Infant color vision: moving tritan stimuli do not elicit directionally appropriate eye movements in 2- and 4-month-olds, Vis. Res., 37:899–911. Teller, D. Y., and J. Palmer, 1996. Infant color vision: motion nulls for red/green vs luminance-modulated stimuli in infants and adults, Vis. Res., 36:955–974. Thiele, A., A. A. Rezec, and K. R. Dobkins, 2002. Chromatic input to motion processing in the absence of attention, Vis. Res., 42:1359–1401. Thiele, A., K. R. Dobkins, and T. D. Albright, 1999. The contribution of color to motion processing in MT, J. Neurosci., 19:6571–6587. Thiele, A., K. R. Dobkins, and T. D. Albright, 2001. Neural correlates of chromatic motion processing, Neuron, 32:1–20. Tootell, R. B., J. B. Reppas, K. K. Kwong, R. Malach, R. T. Born, T. J. Brady, B. R. Rosen, and J. W. Belliveau, 1995. Functional analysis of human MT and related visual cortical areas using magnetic resonance imaging, J. Neurosci., 15:3215–3230. Troscianko, T., 1987. Perception of random-dot symmetry and apparent movement at and near isoluminance, Vis. Res., 27:547–554. Ullman, S., 1980. The effect of similarity between line segments on the correspondence strength in apparent motion, Perception, 9:617–626. Valberg, A., B. B. Lee, P. K. Kaiser, and J. Kremers, 1992. Responses of macaque ganglion cells to movement of chromatic borders, J. Physiol. (Lond.), 458:579–602. Van Essen, D. C., J. H. Maunsell, and J. L. Bixby, 1981. The middle temporal visual area in the macaque: myeloarchitecture, connections, functional properties and topographic organization, J. Comp. Neurol., 199:293–326. Wandell, B. A., A. B. Poirson, W. T. Newsome, H. A. Baseler, G. M. Boynton, A. Huk, S. Gandhi, and L. T. Sharpe, 1999. Color signals in human motion-selective cortex, Neuron, 24:901–909. Wiesel, T. N., and D. H. Hubel, 1966. Spatial and chromatic interactions in the lateral geniculate body of the rhesus monkey, J. Neurophysiol., 29:1115–1156. Williams, D. W., and R. Sekuler, 1984. Coherent global motion percepts from stochastic local motions, Vis. Res., 24:55–62. Zeki, S. M., 1974. Functional organization of a visual area in the posterior bank of the superior temporal sulcus of the rhesus monkey, J. Physiol. (Lond.), 236:549–573.

83

Functional Mapping of Motion Regions GUY A. ORBAN AND WIM VANDUFFEL

M    know about motion processing comes from the many elegant single-cell studies in the nonhuman primate. This progress is reviewed in other chapters. Functional brain imaging, initially limited to positron emission tomography (PET), now includes functional magnetic resonance imaging (fMRI) as a neuroscientific tool for the study of motion regions. These mapping techniques have the advantage of being noninvasive and therefore are applicable to humans. They also provide a much wider-scale view of the system: one studies functional regions in the whole brain rather than recording single neurons in a given brain region. Therefore, it is not surprising that during the past 15 years, with the advent of PET (Fox et al., 1986) and especially with fMRI (Belliveau et al., 1991; Kwong et al., 1992), there has been considerable progress in the understanding of the human visual system, particularly its motion processing component. Since functional imaging records the global activity of large groups of neurons indirectly, via a vascular response, the interpretation of these human measurements would have been impossible without the background of single-cell results. Progress was hampered by the fact that the two techniques were applied to different species, since it is generally not feasible to record single cells in humans and fMRI had not been performed in monkey. This situation has changed recently with the introduction of controlled fMRI experiments in the monkey (Logothetis et al., 1999) and especially in the awake monkey (Nakahara et al., 2002; Vanduffel et al., 2001). Indeed, the two species can now be compared using the same technique, paving the way for resolving longstanding debates concerning homologies. Furthermore, within the same species, single-cell recordings and fMRI can now be compared (Logothetis et al., 2001), providing insight into the origin of the fMRI signal. Additional comparisons with other physiological techniques such as optical imaging or double-label deoxyglucose have also become possible. Two main paradigms have been used in human functional imaging, following the strategies established by single-cell studies. In one the subject is passive, fixating a point on the display, and activity evoked by different stimuli presented in the background is compared (Lueck et al., 1989). This provides information about the preattentive processing of motion information, an important initial stage. We might not be here reading this chapter without this capability, as we would probably not have survived previous unexpected

traffic situations. In the second paradigm, the subject is active and has his or her attention drawn to the moving stimuli, usually by being instructed to perform some task with them (Corbetta et al., 1991). Since the cortex can process only motion occurring on the retina, eye movements will interfere with the interpretation of responses to moving stimuli. In initial imaging studies no eye movements were measured, but these measurements, either electro-oculographic recordings (Dupont et al., 1994) or infrared-based measurements (Tootell et al., 1997), were gradually introduced and are now standard.

The human motion complex of occipitotemporal cortex Because of a lack of sensitivity in PET studies, or because of the use of surface coils in fMRI that restrict the part of the brain explored, the initial imaging studies concentrated on a motion-sensitive region, located in the temporo-parietooccipital junction. Zeki et al. (1991) compared the brain activity when subjects passively viewed a random-dot pattern that was stationary to one that was moving in one of eight possible directions. One extrastriate region in the temporo-parieto-occipital cortex stood out by being significantly more active for moving than for stationary stimuli. Zeki et al. (1991) proposed that this region was the human homolog of monkey MT/V5. It should be noted that this result, like those of most imaging studies, was obtained by statistical analysis: the human motion complex of occipitotemporal cortex (hMT/V5) appeared as a set of voxels in which the difference in activity between motion and control was statistically significant. This has two implications. First, much depends on the control condition, and here Zeki et al. (1991) introduced the static pattern as a control condition. This allows one to dissociate the effect of the stimulus pattern (which is removed) from its movement (which remains). However, by comparing just two closely matched conditions, one can conclude only that there is a relative difference in activity. By adding a further, lower-order condition, an empty fixation condition, as introduced by Tootell et al. (1995b) (see below), one can ascertain whether motion and stationary conditions differ in activation from some baseline level. One can then infer that the neuronal population, the activity of which is reflected by the MR signals, increases its activity above spontaneous activity, that is, shows an excitatory response.

1229

F 83.1. The cerebral hemispheres from four subjects, showing the hMT/V5+ of each as defined by PET activation experiments, superimposed on the individual’s own MRI (rendered brain). Each subject occupies a row: the two hemispheres are shown as viewed at rotations of 50 and 90 degrees from the occipital pole. The Statistical Parametric Map (SPM) image is edited to leave only hMT/V5+. (From Watson et al., 1993.) (See color plate 64.)

Monkey fMRI has confirmed that this assertion indeed is valid for hMT/V5+ (see below). One does not know, however, to what extent this increased neuronal activity of the hMT/V5+ population reflects activity of inhibitory or excitatory neurons. The situation is analogous for single-cell studies, although some firing pattern criteria have been proposed to recognize inhibitory interneurons. Second, the statistical decision depends on the threshold chosen and the volume of search. The accepted standard is p < .05, but this can be applied to a region of interest (ROI) or to the whole brain. The ROI approach is the most sensitive, but it depends on the preliminary localizer scan to include all relevant voxels. Searching over the whole brain, on the other hand, requires correction for multiple comparisons and can be applied to a single subject or to a group of subjects. For a group analysis, one can use either a fixedeffect analysis (as in the Zeki et al., 1991, study), which allows one to draw conclusions only about the subjects tested, or a random effects analysis (Holmes and Friston, 1998), which allows inferences to be made about the general population. In a subsequent study from the same laboratory (Watson et al., 1993) using a more sensitive camera, hMT/V5 was

1230

, ,   

studied in single subjects as well as across the group. This allowed the investigators to demonstrate the variability of its anatomical localization in the different subjects, at least when using a fixed reference such as the Talairach coordinates (Fig. 83.1). The authors noted that hMT/V5+ was closely associated with the ascending branch of the inferior temporal sulcus (ITS), a finding which has been amply confirmed by subsequent studies (Dumoulin et al., 2000; Tootell et al., 1995b). They noted that this region corresponds to a cortical field that is heavily myelinated from birth. A third PET study from our group provided the first independent confirmation of these observations with control of eye movements (Dupont et al., 1994). The final identification of hMT/V5 was provided by Tootell et al. (1995b) using the greater spatial (5 to 10 mm rather than 15 to 20 mm in PET) and temporal resolution of fMRI. This technique not only has better resolution, but also allows a more complete functional investigation of a cortical region. Many tests can be performed on the same subject, because no radioactive tracer has to be injected. Tootell et al. provided evidence that hMT/V5 was not only motion sensitive, more so than V1 (Fig. 83.2), but also that

F 83.2. Time courses of the MRI signal amplitude from V1 (A) and hMT/V5+ (B) of the same subject, sampled every 2 seconds during a 6-minute scan (average of two presentations). (From Tootell et al., 1995b.)

it displayed three properties observed in monkey MT/V5: it was very sensitive to luminance contrast, it responded only weakly to equiluminant moving color stimuli, and it responded to ipsilateral stimulation. Because of the many functional similarities to monkey MT/V5, because of its localization with respect to retinotopic regions (Fig. 83.3), and owing to its distinct histological properties (Tootell and Taylor, 1995; see also Watson et al., 1993), this human cortical region was accepted as a distinct human cortical area. Work in the monkey has led to the consensus that several criteria need to be met for a region of cortex to be considered a separate area: retinotopic organization, connectivity, architectonics, and functional properties. Given the difficulty of obtaining detailed histological data in humans (but see Zilles et al., 1995), few visual cortical regions have achieved the status of distinct areas. hMT/V5, together with early regions V1, V2, V3, and V3A, is generally accepted as a separate cortical area.

However, even the identification of the motion-sensitive region in the ITS as the human homolog of MT/V5 may be premature. Monkey MT/V5 is joined by several satellites (Fig. 83.3) which are themselves selective for motion (Desimone and Ungerleider, 1986; for review see Orban, 1997). Thus, the human motion-sensitive region in the ITS may well correspond to the entire complex, including MT/V5 and its satellites, as suggested by DeYoe et al. (1996). This view has become generally accepted, and the region is referred to as hMT/V5+. Several authors have attempted to subdivide the complex using known properties of MST neurons: large ipsilateral overlap (Desimone and Ungerleider, 1986; Raiguel et al., 1997), responsiveness to optic flow patterns (Duffy and Wurtz, 1991; Lagae et al., 1994; Saito et al., 1986), and response during pursuit (Komatsu and Wurtz, 1988). hMT/V5+ responds to ipsilateral stimulation (Brandt et al., 2000; Ffytche et al., 2000; Tootell et al., 1998; Tootell et al., 1995b). Morrone et al. (2000) suggested that the parts of the hMT/V5+ complex that respond to translation and optic flow components are distinct entities. The part responsive to optic flow, presumed to be the homolog of MSTd, is located more ventrally and appears only when the flow stimuli are changing in time. This localization is in agreement with the results of an earlier passive study of optic flow by de Jong et al. (1994) and with these of a study from our group (Peuskens et al., 2001a) investigating the neural correlates of heading judgments based on optical flow. Both studies observed activity in a ventral satellite of hMT/V5+ related to expansion/ contraction. On the other hand, Dukelow et al. (2001), using ipsilateral overlap and pursuit in the dark, reached a different conclusion: according to these authors, MSTd and MSTl are located anterior to MT/V5 proper in the human complex. This contradiction illustrates the difficulty of relating human imaging to monkey single-cell properties. The recent study of Vanduffel et al. (2001) in the awake monkey makes this even more clear. Up to now, the hMT/V5 complex was believed to include homologs of the subregions of MST, in addition to MT/V5. Yet, when using the translating random-dot pattern typically used in human studies (Sunaert et al., 1999) in monkeys, Vanduffel et al. (2001) observed motion sensitivity in MT/V5, of course, as well as in MSTv and in FST, but not in MSTd (Figs. 83.3, 83.4). The advent of monkey fMRI has made it possible to identify the satellites in the human complex properly. One should scan the human complex at high resolution and test stimuli that differentiate the various parts of the motion complex in monkey fMRI. The aggregation of several functional regions into a single-motion complex may also explain some of the variability in its localization (Watson et al., 1993). As expected from single-cell studies, monkey MT/V5 was activated (above baseline) by both stationary and moving

  :  

1231

F 83.3. Location of hMT/V5+ with respect to the early retinotopic regions (A) and to the motion response of hV3A (B) on a flattened cortical surface from a single hemisphere. TOS, transverse occipital sulcus; IPS, intraparietal sulcus. C, D, Location of MT/V5 and satellites: schematic location of MT/V5, MSTv, MSTd, and FST in STS (modified from Orban, 1997) and actual

activation of MT/V5, MSTv, and FST by moving random dots in monkey M4 (Vanduffel et al., unpublished). In panel C, P and C indicate peripheral and central visual field representation; scale bar = 5 mm. In panel D, stimuli were restricted to the central 7 degrees of the visual field, corresponding to the central representations in panel C. (A and B from Tootell et al., 1997.) (See color plate 65.)

stimuli, but more so by motion. This activation translates into MR signals of opposite polarity in the standard BOLD (blood oxygen level dependent) fMRI and in contrastenhanced fMRI, using MION (monocrystalline iron oxide nanoparticle) as the contrast agent (Fig. 83.4). One should note that the definition of hMT/V5+ in humans reflects a difference in the activity level of the

hMT/V5+ population for the two types of stimuli. This is very different from the criterion used in single-cell studies, where regions are considered to be motion selective when they have large proportions of direction-selective cells. Even in monkey fMRI, which shows that the statistical definition of a motion-sensitive region indeed applies to MT/V5, this definition does not reflect the direction selectivity of the

1232

, ,   

range. Single-cell studies (Lagae et al., 1993; Orban et al., 1986) have suggested that speed tuning and direction selectivity tend to co-occur in neuronal populations. This may explain why Vanduffel et al. (2001) did observe a correlation, albeit a weak one, between motion sensitivity in the fMRI and the proportion of direction-selective neurons in the single-cell studies. Attempts have been made to use motion opponency as a direct indication of direction selectivity (Heeger et al., 1999), but this reflects the mutual inhibition between neurons tuned to opposite directions rather than direction selectivity as such. An alternative is to use unidirectional adaptation, as done by Tolias et al. (2001) in the anesthetized monkey. Because the reversal of direction after adaptation was not compared directly and statistically to the control event (phase shift in the same direction), this study remained inconclusive. Furthermore, Pack et al. (2001) have recently shown that responses of MT/V5 neurons to moving plaids are dependent on anesthesia.

Other motion-sensitive regions Although hMT/V5+ is the most sensitive motion-responsive region, being active in both hemispheres of all subjects tested so far (over 100), many other regions are also responsive to motion.

F 83.4. A, Comparison of BOLD and MION MR signals. Percent signal change in left and right MT/V5 (monkey M1) with respect to the no-stimulus (gray) condition for the three stimulus conditions: no stimulus, stationary, and moving random dots. Average of multiple time series with different order of conditions. Vertical lines indicate the standard error of the mean. Notice reversal of the MR signal sign in MION. B, SPMs of the two hemispheres of M3 on the coronal section trough caudal superior temporal sulcus (STS) for the comparison of moving versus stationary dots. Voxel size 2 ¥ 2 ¥ 2 mm. Lateral arrows, MT/V5; medial arrows, floor of STS: MSTv. (From Vanduffel et al., 2001.) (See color plate 66.)

underlying neuronal population. In fact, it probably reflects the speed tuning of the population. Because of the small eye movements that occur during fixation, even a stationary stimulus produces extremely slow speeds on the retina (on the order of 0.1 to 0.2 deg/sec) (Stavenski et al., 1975), and most MT/V5 neurons are much less sensitive to these slow speeds than to the 4 to 6 deg/sec speed used in the motion condition of the fMRI studies (Cheng et al., 1994; Churchland and Lisberger, 2001; Lagae et al., 1993; Mikami et al., 1986). This view has received support from the study of Chawla et al. (1999a), who reported that motion MR responses over hMT/V5+ and hV3A displayed an inverted U shape when speed was manipulated in the 1 to 32 deg/sec

P V C In the initial studies of Zeki et al. (1991) and Watson et al. (1993), primary visual cortex exhibited motion selectivity. Unilateral activation of V1 was also reported by Dupont et al. (1994). On the other hand, Tootell et al. (1995b) reported no difference in V1 MR responses between static and radially moving dots. Dupont et al. (1997), Goebel et al. (1998), and Dieterich et al. (1998) reported at least unilateral activation of V1. Sunaert et al. (1999) observed significant motion activation in V1 of half of the hemispheres studied. V1 activation in the monkey is also inconsistent (Vanduffel et al., 2001; Fig. 83.5), but this simply means that the activation of V1 by moving and stationary stimuli is not significantly different. The reason for the weakness of V1 activation may be the speed sensitivity of central V1 neurons more than the reduced proportion of direction-selective neurons (Hawken et al., 1988; Orban et al., 1986). Most neurons in the central representation of V1 are low pass for speed (Orban et al., 1986). In the anesthetized, paralyzed preparation, they respond well to very slow stimuli, corresponding to stationary stimuli in the awake subject, and this response decreases once speeds reach 3 to 10 deg/sec. Thus, for many of the V1 neurons in an awake subject, the stimulus moving at 4 to 6 deg/sec may be as effective as a stationary one. This explanation receives support from the interaction between size and motion sensitivity observed by Sunaert et al. (1999) in V1. Only the 14 degree wide stimuli produced consistent motion activation

  :  

1233

F 83.5. SPMs for moving versus stationary dots displayed on the partially unfolded hemispheres of monkey M3. The main sulci are indicated. (Adapted from Vanduffel et al., 2001.) (See color plate 67.)

in V1. More peripheral V1 neurons are less sensitive to slow speeds (Orban et al., 1986), being thus less active during fixation of static stimuli.

humans and monkeys. This was confirmed by the fMRI study of Vanduffel et al. (2001), who reported that monkey V3A displays no motion sensitivity.

V3A Motion sensitivity in a part of the cuneus was noted by Watson et al. (1993) and Dupont et al. (1994). Watson et al. had suggested that this activation may correspond to V3, which in the monkey contains a fair proportion of direction-selective neurons (Gegenfurtner et al., 1997), but Tootell et al. (1995b) demonstrated that retinotopically defined V3 in humans exhibits little motion sensitivity. The cuneal motion-sensitive region was shown by Tootell et al. (1997) to correspond to retinotopically defined V3A and to be located near the transverse occipital sulcus (Fig. 83.2). This activation has since been observed by a number of authors (Ahlfors et al., 1999; Braddick et al., 2001; Chawla et al., 1999a; Cornette et al., 1998a; Goebel et al., 1998; Rees et al., 2000; Sunaert et al., 1999), although only Goebel et al. (1998) explicitly mapped the retinotopic regions. This suggests that there is a species difference here between

P C The initial studies of Watson et al. (1993), Dupont et al. (1994), and Tootell et al. (1995b) had noted a medial posterior parietal activation by moving compared to stationary stimuli. These parietal motion-sensitive regions were described in detail by Sunaert et al. (1999) using the resolution of fMRI applied to the whole brain. They described two regions in the occipital part of the intraparietal sulcus (IPS): one more ventral (VIPS) and one more dorsal at the junction with the parieto-occipital (PO) sulcus (POIPS). Similar regions were observed by Shulman et al. (1999) and by Goebel et al. (1998). More dorsally along the parietal part of the IPS (Fig. 83.6), Sunaert et al. (1999) distinguished an anterior region (DIPSA), also observed by Dupont et al. (1997) and Braddick et al. (2001), and a pair of more posterior regions that were difficult to distinguish from one another (DIPSM/L). This latter motion activation

1234

, ,   

F 83.6. SPMs of two human subjects (A and B) corresponding to contrast motion minus stationary status superimposed on selected transversal sections. 3, DIPSA; 5, FEF; 9, DIPSM; 16, DIPSL; 13, cingulate motion-sensitive region. (See color plate 68.)

was observed by Dieterich et al. (1998). Strong activation of most of these parietal regions was observed in an attentive tracking task compared to a bouncing-ball control condition by Culham et al. (1998). They identified more or less segregated activation in regions roughly corresponding to VIPS, DIPSM/L, and DIPSA as described by Sunaert et al. (1999). The last region may correspond to what Bremmer et al. (2001) identified as the homolog of monkey VIP. Bremmer et al. used the multimodality of monkey VIP to trace the homologous region in humans. However, any multimodal region may correspond to the site identified by Bremmer et al., particularly since no specific motion contrast was used in the tactile and auditory modalities. Cross-modal activation in a visuoauditory speed discrimination task was observed more posteriorly by Lewis et al. (2000) in a region that seems to correspond to DIPSM/L. The same region was also engaged by a purely visual speed discrimination, as was a region close to VIPS. In contrast to the rather extensive motion activation of IPS in humans, only one motion-sensitive region, probably

corresponding to VIP, was observed in the monkey fMRI by Vanduffel et al. (2001). This may suggest that the parietal cortex of humans and monkeys exhibits differences in motion sensitivity perhaps related to that noted for V3A. In addition to the lateral parietal regions, there are indications that very large moving stimuli activate the PO cortex, possibly the homolog of monkey V6 (Cheng et al., 1995; Previc et al., 2000). The PO cortex is also responsive to motion reversals when their visibility is modulated (Cornette et al., 1998b). V C Watson et al. (1993) and Dupont et al. (1997), using PET, observed a motion-sensitive region in lingual cortex. This was confirmed in later fMRI studies by Sunaert et al. (1999), Rees et al. (2000), and Braddick et al. (2001). In keeping with Watson et al.’s early suggestion, Vanduffel et al. (2001) observed motion activation in ventral V2 and/or VP (Fig. 83.5). Occasionally, motion-related responses have been reported in fusiform cortex (Dupont et al., 1994; Rees et al., 2000).

  :  

1235

Finally, motion responses have also been reported (Rees et al., 2000; Sunaert et al., 1999) in the cortex located behind hMT/V5+, which has been referred to as the kinetic occipital (KO) region (see below). O R Motion sensitivity of the frontal eye fields (FEF) has been reported in humans (Culham et al., 1998; Orban et al., 1999; Sunaert et al., 1999; Fig. 83.6) as well as in the monkey (Vanduffel et al., 2001; Fig. 83.5). FEF receives strong projections from MT/V5 and its satellites (Schall et al., 1995), and FEF in humans has been shown to be activated during visual pursuit (Petit and Haxby, 1999; Petit et al., 1997). Motion sensitivity has also been observed in the lateral sulcus at the retroinsular level (Braddick et al., 2001; Dupont et al., 1994; Sunaert et al., 1999). Sunaert et al. (1999) have suggested that this region is the homolog of the visual region bordering parietoinsular vestibular cortex (Grüsser et al., 1990). Vanduffel et al. (2001) observed motion sensitivity in this region in one of the their two animals (Fig. 83.5). Finally, in humans, motion sensitivity has been reported in the superior temporal sulcus (STS) (Ahlfors et al., 1999; Braddick et al., 2001; Sunaert et al., 1999) in a region responding to facial movements (Puce et al., 1998) as well as in cingulate cortex (Braddick et al., 2001; Cornette et al., 1998a; Sunaert et al., 1999; Fig. 83.6).

Types of moving stimuli and stimulus parameters S P A variety of stimulus patterns have been used to investigate motion sensitivity: Zeki et al. (1991) and most studies from the London group used translating random-dot patterns, as did Cornette et al. (1998a). We (Dupont et al., 1994, 1997; Sunaert et al., 1999; Van Oostende et al., 1997) have used translating random-textured patterns with small dots and 50% density, also used by Braddick et al. (2001). Tootell et al. (1997) and Tootell et al. (1995b) have used radially moving dots (i.e., alternating contraction and expansion), as did Goebel et al. (1998) and de Jong et al. (1994) (only expansion). Tootell et al. (1995b) also utilized moving gratings, as did Dupont et al. (1997, 2000). There has been no systematic study comparing motion responses for these different patterns. Similarly, stimuli of various sizes have been used, often dictated by the MRI setting. Most groups have used stimuli about 30 degrees in diameter (Goebel et al., 1998; Tootell et al., 1995b; Zeki et al., 1991). We have routinely used smaller stimuli, the standard being 7 degrees for humans (Sunaert et al., 1999) and 14 degrees for monkeys (Vanduffel et al., 2001). However within the 3 to 14 degree range, diameter had little effect on motion sensitivity except in V1 (see above). Very large stimuli (80 to 100 degrees in diameter) were used by Cheng et al. (1995) and Previc et al. (2000).

1236

, ,   

F 83.7. Time course of the MION MR signal from MT/V5 of a monkey subject for different durations of the motion epochs (indicated by color). (From Vanduffel et al., unpublished.) (See color plate 69.)

Previc et al. suggested that motion stimuli occupying a wide field of view are processed in the medial occipital pathways extending up to the PO sulcus, while smaller motion stimuli (less than 50 degrees in diameter) engage only lateral occipital pathways including hV3A and hMT/V5+. S P Other stimulus parameters have hardly been explored, except with regard to speed and contrast, as mentioned before. Cornette et al. (1998b) noted that reducing the number of motion axes from four, as used by Zeki et al. (1991) and our group, to one reduced the hMT/V5+ activation by 25%. A possible explanation of this decrease is adaptation by the use of a single axis (Tolias et al., 2001). We tested the effect of epoch duration on MT/V5 activation in the monkey (Fig. 83.7). Epochs shorter than 15 seconds produced smaller activation in contrast agent– enhanced fMRI (using MION). We know that MT/V5 neurons respond to motion presentations as short as 10 msec (Orban, 1997). Hence the limitation on epoch duration arises at the vascular transduction stage of the fMRI, that is, within the changes in cerebral blood circulation occurring as a consequence of neuronal activity increase and which constitute the signal captured by the fMRI. In MIONenhanced fMRI, these changes are essentially changes in cerebral blood volume (Leite et al., 2002). I M  D N While incoherently moving dots (dots moving in random directions but at the same speed) activate hMT/V5+ more than coherently moving dots (all dots moving at the same speed and in the same direction) in passive subjects (McKeefry et al., 1997; see also Previc et al., 2000), the opposite has been reported under conditions of opposed motion discrimination (Rees

et al., 2000). The MR motion response over hMT/V5+ increased linearly with coherence in the latter study, mimicking, according to Rees et al., the behavior of MT/V5 neurons (Britten et al., 1992). Similar increases were noted over KO and hV3A. In a dynamic noise stimulus or flicker, obtained by showing the motion frames in random order, both direction and speed are random, while incoherently moving dots still have the same speed and exhibit only random directions. Tootell et al. (1995b) reported that hMT/V5+ responds to flicker, albeit less than to moving dots. This was elaborated by Sunaert et al. (1999), who observed that while V1 responded very well to flicker (in fact, better than to motion), hV3A and especially hMT/V5+ responded less to flicker than to motion (Fig. 83.8), a finding in agreement with McCarthy et al. (1995) and Braddick et al. (2001). The difference in flicker versus motion response in V1 and hMT/V5+ closely matches the parallel observation by Heeger et al. (1999) that motion opponency occurs in hMT/V5+ but not in V1. Sunaert et al. (1999) observed that the flicker responses were abolished only at the level of parietal cortex (see also Orban et al., 1999). Braddick et al. (2001) confirmed that parietal cortex is significantly more activated by motion than by flicker, as are STS and lateral sulcus motion regions. Preliminary results indicate that the same holds true in the monkey (Fig. 83.8). The decrease in flicker response over hMT/V5+ compared to V1 matches the properties of MT/V5 neurons which are less responsive to flicker than are their, direction-selective V1 counterparts (Qian and Andersen, 1994). This has been attributed to the increased mutual inhibition between neurons tuned to opposite directions (Heeger et al., 1999; Qian and Andersen, 1994). Thus, flicker suggests a distinction between lower-order motion regions such as hMT/V5+, hV3A, and lingual cortex and higher-order ones such as intraparietal sulcus, STS, lateral sulcus, and FEF.

to find any anatomical segregation between first-order and higher-order motion mechanisms (Dumoulin et al., 2001; Dupont et al., 2000; Somers et al., 1999). In the same vein, attentive tracking, in which subjects mentally follow one of many moving stimuli, and which has been presented as a higher-order motion mechanism (Cavanagh, 1992), activates many of the same regions as those ascribed to moving random dots or gratings (Culham et al., 1998; see above). A related question is the extent to which the motionresponsive regions are activated by motion when that motion is defined by other attributes such as color. Tootell et al. (1995b) have shown that at equiluminance, moving gratings evoked little response over hMT/V5+, far less than luminance gratings. Ffytche et al. (1995) reported that no motionfrom-hue responses (luminance motion signals were removed by a flicker masking strategy) could be observed, except over V1/V2 and over hMT/V5+ (only on the right side). In particular, no response was seen over V4 (the color region in the terminology of that group). Not only do motion signals arising from L and M cones reach hMT/V5+, but signals from S cones also do so in humans (Wandell et al., 1999) as well as in monkey (Seidemann et al., 1999). In the same vein, Seghier et al. (2000) reported that moving illusory contours induced by coherent rotation of pacman figures, compared to incoherent rotation that did not, evoked the percept of a moving Kanisza square, activated V1/V2, hMT/V5+, and the KO/LOS (lateral occipital sulcus) region. It was unclear whether these activations reflected the perception of the illusory contour or its motion. The view that seems to emerge is that all moving stimuli are processed in a single-motion processing pathway (but see Phinney and DeYoe, 2001, for stereo-defined motion and Previc et al., 2000, above), which may diverge farther up and reach multiple end stations (e.g., in STS, parietal cortex, and ventral cortex).

N-D S Both random-dot patterns and gratings are luminance-defined stimuli, whose motion can be detected by a modified Fourier mechanism or energy detector. According to a number of psychophysical studies (Cavanagh, 1992; Chubb and Sperling, 1988), other stimuli can be detected only by higher-order mechanisms, although it is unclear how many higher-order mechanisms operate in the human visual system. Smith et al. (1998) have argued that second-order motion activates V3 and VP more than first-order motion, while both types of motion produce equal activation in hMT/V5+. It is difficult to derive any conclusion from this study, as the only comparisons made were between higher-order motion and either first-order motion or higher-order stationary stimuli. Subsequent studies, reported only in abstract form, have failed

Initially, it was believed that there should be a close link between hMT/V5+ activity and the perception of motion, although the responsiveness to flicker noted by Tootell et al. (1995b) was an early indication of the oversimplification of this view. The prolonged activity of hMT/V5+ during the motion aftereffect (waterfall illusion) and the similarity in time course of the two effects suggested a link between perception and hMT/V5+ activity, although Tootell et al. (1995a) cautioned that the aftereffect may also depend on areas upstream from hMT/V5+. The link was further stressed by the observation that the hMT/V5+ activity rebounded after adaptation only when a stimulus was visible, just like the perceptual effect (Culham et al., 1999), or when the stationary stimulus eliciting the aftereffect was in exactly the same

Perception of motion and hMT/V5+ activity

  :  

1237

F 83.8. A–D, Activity profiles plotting adjusted MR signal change with respect to the stationary dots conditions for four conditions: moving (UNI), stationary (STA), and flickering at 15 and 6 Hz (FLI15 and FLI6) dots in four cortical regions (local maximum). Average of three human subjects. E, Flicker reduction (UNI-

1238

, ,   

FLI6/STA) plotted as a function of cortical area in human fMRI (Sunaert et al. 1999, but V1 sampled over same retinotopic parts as monkey data), monkey fMRI (bold) and for monkey single cells (Qian and Andersen, 1994). (A–D, from Sunaert et al., 1999; E from Vanduffel et al., unpublished.)

position as the adaptation stimulus (He et al., 1998). Huk et al. (2001) have recently challenged these studies (see also Hautzel et al., 2001; Taylor et al., 2000) and have noted that they were performed without attentional control. When Huk et al. controlled attention, the difference in activity evoked by a stationary stimulus after unidirectional or bidirectional adaptation vanished. The authors went on to show that directional adaptation occurs in hMT/V5+ by comparing responses to motion in the same or different directions to responses to the direction of adaptation while still controlling attention with a speed discrimination task. They observed large direction-selective adaptation in hMT/V5+ and in hV3A but not in early retinotopic areas (V1, V2, and V3). Using a different stimulus, Enigma, in which many subjects see illusory motion, Zeki et al. (1995) tried to establish a relationship between hMT/V5+ activity and perception of motion even where no physical motion was present. The authors observed activity in or near hMT/V5+ during illusory motion control compared to a static control (obtained by a slight modification of the Enigma). The fact that the activity was slightly more inferior and anterior was interpreted as reflecting an origin in one of the satellites of MT/V5, perhaps MSTd. The authors also stress that only hMT/V5+ was active in the human brain during the illusory perception, but this may have simply reflected the ineffectiveness of the stimulus. It is difficult if not impossible to derive any conclusion from negative results in imaging. Similar reasoning underlies the experiment by Kourtzi and Kanwisher (2000) showing that static images which imply motion, such as a picture of a diver, activate hMT/V5+ more than static images which do not imply motion. In two independent experiments, images of people as well as of animals and scenes were used, indicating the generality of the result. This small effect (a 2% increase compared to a 1.5% increase from the fixation baseline) was significant because only hMT/V5+, identified by a localizer scan, was tested, so that no correction for multiple comparisons was required. The technique of using the localizer is vulnerable to omissions, and Kourtzi and Kanwisher indicated that the effect extended to regions surrounding hMT/V5+. Senior et al. (2000), using similar stimuli, reported maximum activation in a region located behind hMT/V5+. Using somewhat different static stimuli portraying gestures, Peigneux et al. (2000) reported activation of the middle temporal gyrus extending from the STS region to hMT/V5+. Thus, it seems that there is no simple relationship between hMT/V5+ activity and perception of motion. On the one hand, it is now well documented that that hMT/V5+ can be active when no motion is perceived, such as during equiluminance and during flicker. On the other hand, conditions in which no motion is perceived can evoke the same level of

activity as those in which motion is perceived, such as after uni- and bidirectional adaptation, once attention is controlled. Finally, stimuli implying motion may have a wider representation than MT/V5+ itself. This does not mean that hMT/V5+ activity is not important for the perception of motion: perception may arise from differences in activity between subpopulations of MT/V5 neurons or from activity in areas upstream to hMT/V5+. Finally, it must be noted that MT/V5 has been implicated in functions other than motion perception, such as control of eye visual pursuit (Churchland and Lisberger, 2001; Lagae et al., 1993; Newsome et al., 1985), stereopsis (DeAngelis et al., 1998), and extraction of three-dimensional structure from motion (Xiao et al., 1997), and that activity over hMT/V5+ has been observed in dimming detection tasks (Claeys et al., 2001), which may relate to the flicker sensitivity and the high contrast sensitivity documented for hMT/V5+. Thus, the view that is emerging is that MT/V5, rather than being a unique “motion center,” acts in concert with different sets of other regions to fulfill a wider variety of behavioral functions than motion perception alone.

Attention to motion G A The effect of attention to motion in general was assessed by O’Craven et al. (1997). The clearest experiment was their second, in which subjects were asked to pay attention to the back dots, which were either moving among white stationary dots, stationary among white moving dots, or stationary without white dots. This attention manipulation allowed these investigators to disentangle the effect of motion, as such (and the effect of dot density) from the attention to motion, which amounted to about 30% of the sensory motion response. One should note, however, that since the subjects did not perform any task, we do not know how well they allocated their attention to the dots intended. Büchel et al. (1998) used a different strategy: they trained subjects to track increasingly small, brief changes in the speed of a continuously moving stimulus. During the scanning there were no actual changes, but the subjects still reported seeing some, and the attention effect was demonstrated behaviorally on the motion aftereffect. The attention condition compared to the no-attention condition showed increased activity in hMT/V5+ (or slightly ventral to it) and hV3A, but also in V1/V2, an activation not observed by O’Craven et al. (1997). One should note, however, that the attention of the subject was drawn to the change in speed, not to motion in general. Büchel et al. also reported attention effects in parietal cortex in loci close to DIPSM/L and DIPSA (see above) and in FEF. O’Craven et al. could not observe these changes because of the limitations of a surface coil. Chawla et al. (1999b) compared attention-to-motion to attention-to-color in an event-related

  :  

1239

F 83.9. SPM for contrasting discrimination of the opposite direction of a moving grating to dimming detection. Average of nine human subjects (PET). The four main activations sites are indicated. (From Dupont et al., unpublished.) (See color plate 70.)

design and observed an increase in the hMT/V5+ response during attention-to-motion. The converse effect, the drawing away of attention from motion, has been observed by Rees et al. (1997). The authors showed that attention to central visual stimulus substantially reduced the motion activity of hMT/V5. Rees et al. presented words in the center of the display and asked subjects to detect bisyllabic words compared to a low-load condition in which they detected uppercase words. We made similar observations in a control experiment for our initial monkey fMRI experiments, in which fixation was controlled by requiring the monkey to detect orientation changes of a very small, light bar. We measured the effect of a 32% reduction in the hMT/V5+ motion activation as the result of attention to this central bar (Sunaert et al., unpublished). Rees et al. (2001) subsequently showed that a similar task requiring attention to auditorily presented words had little effect on motion activation in hMT/V5+, hV3A, and KO. A  D  M Rather than asking subjects to attend to a moving versus stationary stimulus, one can draw the subjects’ attention to a given aspect of motion, thus manipulating featural attention. This is usually done with a task requiring the subject to judge some aspect of motion (e.g., speed or direction). The manipulation of featural attention (Corbetta et al., 1991) should not be confused with task manipulation, in which the feature remains constant but the task performed with the attribute varies (Dupont et al., 1993; Fias et al., 2002; Orban et al., 1997). As O’Craven et al. (1997) noted, the use of a discrimination task does not allow for a pure attention manipulation, since either featural attention or decision factors can explain the differences in activation between the discrimination task and the control task. However, a conjunction analysis might provide that distinction (Peuskens et al., 2001a). Figure 83.9 shows one of the most simple discrimination tasks, that of opposed motion discrimination, which has

1240

, ,   

been studied extensively in single-cell studies by Newsome and coworkers (Britten et al., 1992, Newsome and Paré, 1988). Compared to a dimming detection control task, discrimination of opposite directions of motion activates hMT/V5+ and hV3A but also parietal and premotor cortex. The parietal activation, and a fortiori the premotor one, may well reflect decision-related processes, according to singlecell studies (Shadlen and Newsome, 1996). In contrast, the activation of hMT/V5+ and hV3A most likely reflects featural attention. While the discrimination of opposite directions of motion does activate hMT/V5+, fine discriminations of direction, compared to the same dimming control task, fails to activate this area (Cornette et al., 1998a) but activates hV3A and parietal cortex. It is worth noting that the computations involved in these two tasks are very different. In the case of opposite direction of motion, direction-selective cells tuned to those directions are required (Britten et al., 1992). For fine discrimination, it may well be that the slope of the direction-tuning curve is important if one extrapolates from orientation discrimination (Schoups et al., 2001). For these fine discriminations, directional (tuned to a single direction) and nondirectional (tuned to opposite directions) cells then become equally useful. Still related to direction processing is the heading task, in which the spatial distribution of directions contains the information about the direction of self-motion. This task, again with reference to the same dimming control task, activates hMT/V5+, as well as parietal and premotor cortex, but not hV3A (Peuskens et al., 2001a). Here the single-cell studies (Duffy and Wurtz, 1991; Lagae et al., 1994; Saito et al., 1986) have shown that neurons selective to expansion occur in MSTd, the homolog of which is included in the hMT/V5+ complex. Finally, expectations about the direction of motion elicited by a stationary cue (arrow) activate hM5/V5+ and also parietal motion-sensitive regions along the IPS (VIPS, DIPSM/L, and DIPSA; Shulman et al., 1999).

A  S The aspect of motion which has received most attention is speed of motion. In their seminal study, Corbetta et al. (1991) compared same-different judgements of speed to a divided attention condition. They observed a strong activation over what is now known as hV3A, but also a weaker one over hMT/V5+. In a subsequent study, Beauchamp et al. (1997) compared hMT/V5+ activation by an identical stimulus, an annulus of coherently moving dots, under two attention conditions. Subjects either compared speeds in the two halves of the annulus or compared the colors in these halves, resulting in a 35% reduction of hMT/V5+ activity when attention was drawn away from the speed. Huk and Heeger (2000) again reported a 10% increase in activation of hMT/V5+, but not of V1 or hV3A, when comparing a speed discrimination to a contrast discrimination. In contrast, we (Orban et al., 1998; Sunaert et al., 2000b) observed an activation of hV3A (and V3) but not of hMT/V5+ when speed identification and a successive speed discrimination task were compared to a dimming control task. This result does not depend on the psychophysical performance level (Sunaert et al., 2000b). It remains unclear which of the small differences in stimulus and task explain the differences between these studies. It should be noted that in our last study (Sunaert et al., 2000b), we also observed a very small increase (5%) in the activity of hMT/V5+ in speed discrimination versus dimming detection. The frequently quoted observation that speed discrimination is impaired in monkeys after lesioning that includes MT/V5 (Orban et al., 1995b) simply indicates that MT/V5 is critical to speed discrimination, not that its activity level needs to be influenced by attention to speed. Finally, it is worthwhile to mention another peculiarity of speed discriminations. While all discriminations involving direction of motion engage parietal regions, this is not the case for speed discriminations (Sunaert et al., 2000a; but see Lewis et al., 2000).

Other behavioral functions of motion-sensitive regions C  M Up to now, we have concentrated on the use of motion processing to judge motion in the outside world. Motion processing, however, has a much wider role (Nakayama, 1985). One additional function is the control of eye movements. It has been shown that optokinetic nystagmus, elicited by a rotating drum compared to fixating a stationary drum, activates hMT/V5+ but also V1 and FEF (Dieterich et al., 1998). Similarly, activation of hMT/V5+ has been observed during pursuit eye movements (Barton et al., 1996; O’Driscoll et al., 1998). In both cases, the effects of visual stimulation and of eye movements were difficult to distinguish. Finally, it has been shown that the FEF contains two subregions, the more lateral and inferior

of which is activated by pursuit rather than by saccades (Petit and Haxby, 1999; Petit et al., 1997). It has been noted that hMT/V5+ is active when subjects view hand movements (Decety et al., 1994), even when this is compared to random motion (Bonda et al., 1996). Clearly, the role of motion regions in the control of eye and body movements needs more work, but the immobilization of the subject in the scanner, required for quality imaging, renders those studies more difficult. K B Motion processing can also be used to extract shape information from motion displays, both a two-dimensional (2D) (flat) shape that is extracted from the discontinuities in the velocity distribution and a threedimensional (3D) shape derived from the speed gradients. Differences in motion direction produce the percept of kinetic boundaries, which are perceptually as sharp as luminance-defined boundaries. In comparing orientation discrimination using kinetic gratings rather than luminancedefined gratings, we (Orban et al., 1995a) observed an activation in a region posterior to hMT/V5+ rather than in hMT/V5+ itself. Subsequent studies (Dupont et al., 1997; Van Oostende et al., 1997) with passive subjects viewing kinetic gratings compared to luminance gratings, uniform motion (coherently translating dots), and transparent motion confirmed that this region, which we now refer to as the kinetic occipital (KO) region, is specifically involved in the processing of kinetic contours. It is located close to the lateral occipital sulcus, between hMT/V5+ anteriorly and hV3A and V3 posteriorly (Fig. 83.10A). In this region, activation can be obtained by comparing scenes and objects to uniform textures (Malach et al., 1995), but this “lateral occipital” activation extends well beyond the KO region. Subsequent studies (Grill-Spector et al., 1998) have shown that LO is indeed activated by both luminance- and motion-defined objects. Using larger stimuli with radial kinetic contours, Reppas et al. (1997) failed to observe hMT/V5+ activation, as was the case in most of the subjects in our studies, but they did observe activation in early retinotopic regions rather than in KO. The activation of cortex representing the peripheral field observed by Reppas et al. may have been due to the fixed position of boundaries which, when compared to uniform motion, may induce differences in activity in neurons with surrounds ( Jones et al., 2001). Failure to observe any activation over KO may have been due to the lack of resolvable contours in the center of the display, since KO responded to small kinetic gratings in our studies. A later study from the same laboratory using our stimuli did indeed observe KO activation along with hMT/V5+ activation (Tootell and Hadjikani, 2001). It is worth noting that in all these studies the kinetic boundaries were stationary. This is very different from another type of

  :  

1241

form-from-motion where the whole envelop (contour) moves. This has been shown to activate hMT/V5+ (Wang et al., 1999). T-D S  M Relatively little work has been done on the extraction of 3D shape from motion. Comparing 3D dynamic displays, in which 3D objects (random lines) are perceived, to 2D dynamic displays, in which flat objects are perceived, we (Orban et al., 1999) observed activation, across rigid and nonrigid conditions, of hMT/V5+, a lateral occipital region, and a series of regions along the IPS, as well as, in at least half of the subjects, lingual and fusiform cortex. Control experiments revealed that this activation was due neither to the differences in lower- and higher-order motion characteristics as such nor to attention. The activation of hMT/V5+ is in agreement with single-cell studies showing that MT neurons are selective for the direction of speed gradients, which correspond to the direction of tilt in depth (Xiao et al., 1997; see also Bradley et al., 1998). Using a different stimulus (a half-3D sphere created from random dots) and a different control (incoherently moving dots), Paradis et al. (2000) obtained quite a different result: activation of ventral lingual cortex and of a PO region. The reasons for this difference are unclear. On the one hand, their control condition is known to activate hMT/V5+ strongly; on the other hand, moving dots may not be as efficient for producing 3D motion activation as random lines (Sunaert et al., 2000a). B M A related phenomenon is biological motion, in which a set of moving dots create the impression of a moving human figure ( Johannson, 1973). This stimulus compared to a scrambled control activates the STS motion region (Bonda et al., 1996; Grossman et al., 2000). This same STS region has been shown to be more strongly activated by facial movements than by simple, coherent motion of random dots (Puce et al., 1998). How specific that region is for biological motion compared to other complex motion displays is presently under investigation (Grèzes et al., 2001; Grossman and Blake, 2001; Peuskens et al., 2001b).

Conclusions F 83.10. A, Location of the KO (red diamond) region compared to hMT/V5+ (green), V3A (blue), V3 dorsal (orange), and LO (yellow) diamonds. B, Nodes of the 3D from motion processing network on rendered brain; red, nodes from group and singlesubject analysis; yellow, single-subject analysis; black circles, bilateral nodes; 1, hMT/V5+; 2, DIPSA; 3, lateral occipital sulcus; 5, POIPS; 6, TRIPS ( junction transverse occipital and intraparietal sulci); 11, DIPSL; 12, VIPS; 14, lingual node; 15, fusiform node. (A from Van Oostende et al., 1997; B from Orban et al., 1999.) (See color plate 71.)

1242

, ,   

Many regions in human and monkey cortex are motion sensitive, the most prominent of these being the MT/V5 motion complex. This diversity is likely to reflect the different computations that can be performed on the retinal motion signals rather than distinct pathways for different types of motion stimuli. The different computations performed on the motion signals relate both to the many aspects of motion perception and to the many other behavioral functions of motion processing. Indeed, the most important function of the visual motion regions, like that of the visual system in general, is not to analyze the retinal motion signals,

and indirectly motion in the outside world, but to adapt the results of this analysis to the needs of other brain systems that have to make use of the visual information. The advent of fMRI in the awake monkey holds the promise of rapid progress in understanding the function of the cerebral motion networks, which can now be studied at both the population and the single-neuron level REFERENCES Ahlfors, S. P., G. V. Simpson, A. M. Dale, J. W. Belliveau, A. K. Liu, A. Korvenoja, J. Virtanen, M. Huotilainen, R. B. Tootell, H. J. Aronen, and R. J. Ilmoniemi, 1999. Spatiotemporal activity of a cortical network for processing visual motion revealed by MEG and fMRI, J. Neurophysiol., 82:2545–2555. Barton, J. J. S., T. Simpson, E. Kiriakopoulos, C. Stewart, A. Crawley, B. Guthrie, M. Wood, and D. Mikulis, 1996. Functional MRI of lateral occipitotemporal cortex during pursuit and motion perception, Ann. Neurol., 40:387–398. Beauchamp, M. S., R. W. Cox, and E. A. DeYoe, 1997. Graded effects of spatial and featural attention on human area MT and associated motion processing areas, J. Neurophysiol., 78:516– 520. Belliveau, J. W., D. N. Kennedy, R. C. McKInstry, B. R. Buchbinder, R. M. Weiskoff, M. S. Cohen, J. M. Vevea, T. J. Brady, and B. R. Rosen, 1991. Functional mapping of the human visual cortex by magnetic resonance imaging, Science, 254:716–719. Bonda, E., M. Petrides, D. Ostry, and A. Evans, 1996. Specific involvement of human parietal systems and the amygdala in the perception of biological motion, J. Neurosci., 16:3737–3744. Braddick, O. J., J. M. O’Brien, J. Wattam-Bell, J. Atkinson, T. Hartley, and R. Turner, 2001. Brain areas sensitive to coherent visual motion, Perception, 30:61–72. Bradley, D. C., G. C. Chang, and R. A. Andersen, 1998. Encoding of three-dimensional structure-from-motion by primate MT neurons, Nature, 392:714–717. Brandt, T., T. Stephan, S. Bense, T. A. Yousry, and M. Dieterich, 2000. Hemifield visual motion stimulation: an example of interhemispheric crosstalk, NeuroReport, 11:2803–2809. Bremmer, F., A. Schlack, N. J. Shah, O. Zafiris, M. Kubischik, K. Hoffmann, K. Zilles, and G. R. Fink, 2001. Polymodal motion processing in posterior parietal and premotor cortex: a human fMRI study strongly implies equivalencies between humans and monkeys, Neuron, 29:287–296. Britten, K. H., M. N. Shadlen, W. T Newsome, and J. A. Movshon, 1992. The analysis of visual motion: a comparison of neuronal and psychophysical performance, J. Neurosci., 12:4745–4763. Büchel, C., O. Josephs, G. Rees, R. Turner, C. D. Frith, and K. J. Friston, 1998. The functional anatomy of attention to visual motion. A functional MRI study, Brain, 121:1281–1294. Cavanagh, P., 1992. Attention-based motion perception, Science, 257:1563–1565. Chawla, D., C. Buechel, R. Edwards, A. Howseman, O. Josephs, J. Ashburner, and K. J. Friston, 1999a. Speed-dependent responses in V5: a replication study, NeuroImage, 9:508–515. Chawla, D., G. Rees, and K. J. Friston, 1999b. The physiological basis of attentional modulation in extrastriate visual areas, Nat. Neurosci., 2:671–676. Cheng, K., H. Fujita, I. Kanno, S. Miura, and K. Tanaka, 1995. Human cortical regions activated by wide-field visual motion: a H215O PET study, J. Neurophysiol., 74:413–427.

Cheng, K., T. Hasegawa, K. S. Saleem, and K. Tanaka, 1994. Comparison of neuronal selectivity for stimulus speed, length, and contrast in the prestriate areas V4 and MT of the macaque monkey, J. Neurophysiol., 71:2269–2280. Chubb, C., and G. Sperling, 1988. Drift-balanced random stimuli: a general basis for studying non-Fourier motion perception, J. Opt. Soc. Am. A, 5:1986–2006. Churchland, M. M., and S. G. Lisberger, 2001. Shifts in the population response in the middle temporal visual area parallel perceptual and motor illusions produced by apparent motion, J. Neurosci., 21:9387–9402. Claeys, K., P. Dupont, S. Sunaert, P. Van Hecke, E. De Schutter, and G. A. Orban, 2001. Human brain activations in a color discrimination task, Soc. Neurosci. Abstr., 27:286.17. Corbetta, M., F. M. Miezin, S. Dobmeyer, G. L. Shulman, and S. E. Petersen, 1991. Selective and divided attention during visual discriminations of shape, color, and speed: functional anatomy by positron emission tomography, J. Neurosci., 11:2383–2402. Cornette, L., P. Dupont, A. Rosier, S. Sunaert, P. Van Hecke, J. Michiels, L. Mortelmans, and G. A. Orban, 1998a. Human brain regions involved in direction discrimination, J. Neurophysiol., 79:2749–2765. Cornette, L., P. Dupont, W. Spileers, S. Sunaert, J. Michiels, P. Van Hecke, L. Mortelmans, and G. A. Orban, 1998b. Human cerebral activity evoked by motion reversal and motion onset, Brain, 121:143–157. Culham, J. C., S. A. Brandt, P. Cavanagh, N. G. Kanwisher, A. M. Dale, and R. B. H. Tootell, 1998. Cortical fMRI activation produced by attentive tracking of moving targets, J. Neurophysiol., 80:2657–2670. Culham, J. C., S. P. Dukelow, T. Vilis, F. A. Hassard, J. S. Gati, R. S. Menon, and M. A. Goodale, 1999. Recovery of fMRI activation in motion area MT following storage of the motion aftereffect, J. Neurophysiol., 81:388–393. DeAngelis, G. C., B. G. Cumming, and W. T. Newsome, 1998. Cortical area MT and the perception of stereoscopic depth, Nature, 394:677–680. Decety, J., D. Perani, M. Jeannerod, V. Bettinardi, B. Tadary, R. Woods, J. C. Mazziotta, and F. Fazio, 1994. Mapping motor representations with positron emission tomography, Nature, 371:600–602. de Jong, B. M., S. Shipp, B. Skidmore, R. S. J. Frackowiak, and S. Zeki, 1994. The cerebral activity related to the visual perception of forward motion in depth, Brain, 117:1039–1054. Desimone, R., and L. G. Ungerleider, 1986. Multiple visual areas in the caudal superior temporal sulcus of the macaque, J. Comp. Neurol., 248:164–189. DeYoe, E. A., G. J. Carman, P. Bandettini, S. Glickman, J. Wieser, R. Cox, D. Miller, and J. Neitz, 1996. Mapping striate and extrastriate visual areas in human cerebral cortex, Proc. Natl. Acad. Sci. USA, 93:2382–2386. Dieterich, M., S. F. Bucher, K. C. Seelos, and T. Brandt, 1998. Horizontal or vertical optokinetic stimulation activates visual motion-sensitive, ocular motor and vestibular cortex areas with right hemispheric dominance. An fMRI study, Brain, 121:1479– 1495. Duffy, C. J., and R. H.Wurtz, 1991. Sensitivity of MST neurons to optic flow stimuli: I. A continuum of response selectivity to largefield stimuli, J. Neurophysiol., 65:1329–1345. Dukelow, S. P., J. F. X. DeSouza, J. C. Culham, A. V. van den Berg, R. S. Menon, and T. Vilis, 2001. Distinguishing subregions of the human MT+ complex using visual fields and pursuit eye movements, J. Neurophysiol., 86:1991–2000.

  :  

1243

Dumoulin, S. O., R. G. Bittar, N. J. Kabani, C. L. Baker, Jr, G. Le Goualher, G. Bruce Pike, and A. C. Evans, 2000. A new anatomical landmark for reliable identification of human area V5/MT: a quantitative analysis of sulcal patterning, Cereb. Cortex, 10:454–463. Dumoulin, S. O., R. F. Hess, C. L. Baker, Jr., and A. C. Evans, 2001. fMRI responses to first and second-order motion, Invest. Ophthalmol. Vis. Sci., 42:S322. Dupont, P., B. De Bruyn, R. Vandenberghe, A. Rosier, J. Michiels, G. Marchal, L. Mortelmans, and G. A. Orban, 1997. The kinetic occipital region in human visual cortex, Cereb. Cortex, 7:283–292. Dupont, P., G. A. Orban, B. De Bruyn, A. Verbruggen, and L. Mortelmans, 1994. Many areas in the human brain respond to visual motion, J. Neurophysiol., 72:1420–1424. Dupont, P., G. A. Orban, R. Vogels, G. Bormans, J. Nuyts, C. Schiepers, M. De Roo, and L. Mortelmans, 1993. Different perceptual tasks performed with the same visual stimulus attribute activate different regions of the human brain: a positron emission tomography study, Proc. Natl. Acad. Sci. USA, 90:10927– 10931. Dupont, P., G. Sáry, and G. A. Oran, 2000. Cerebral regions processing first and second order motion: a PET study, Soc. Neurosci. Abstr., 26:2086. Ffytche, D. H., A. Howseman, R. Edwards, D. R. Sandeman, and S. Zeki, 2000. Human area V5 and motion in the ipsilateral visual field, Eur. J. Neurosci., 12:3015–3025. Ffytche, D. H., B. D. Skidmore, and S. Zeki, 1995. Motion-fromhue activites area V5 of the human visual cortex, Proc. R. Soc. Lond. B, 260:353–358. Fias, W., P. Dupont, B. Reynvoet, and G. A. Orban, 2002. The quantitative nature of a visual task differentiates between ventral and dorsal stream, J. Cogn. Neurosci., 14:646–658. Fox, P. T., M. A. Mintun, M. E. Raichle, F. M. Miezin, J. M. Allman, and D. C. Van Essen, 1986. Mapping human visual cortex with positron emission tomography, Nature, 323:806–809. Gegenfurtner, K. R., D. C. Kiper, and J. B. Levitt, 1997. Functional properties of neurons in macaque area V3, J. Neurophysiol., 77:1906–1923. Goebel, R., D. Khorram-Sefat, L. Muckli, H. Hacker, and W. Singer, 1998. The constructive nature of vision: direct evidence from functional magnetic resonance imaging studies of apparent motion and motion imagery, Eur. J. Neurosci., 10:1563–1573. Grèzes, J., P. Fonlupt, B. Bertenthal, C. Delon-Martin, C. Segebarth, and J. Decety, 2001. Does perception of biological motion rely on specific brain regions? NeuroImage, 13:775–785. Grill-Spector, K., T. Kushnir, S. Edelman, Y. Itzchak, and R. Malach, 1998. Cue-invariant activation in object-related areas of the human occipital lobe, Neuron, 21:191–202. Grossman, E. D., and R. Blake, 2001. Brain activity evoked by inverted and imagined biological motion, Vis. Res., 41:1475–1482. Grossman, E. D., M. Donnelly, R. Price, D. Pickens, V. Morgan, G. Neighbor, and R. Blake, 2000. Brain areas involved in perception of biological motion, J. Cogn. Neurosci., 12:711–720. Grüsser, O.-J., M. Pause, and U. Schreiter, 1990. Vestibular neurons in the parieto-insular cortex of monkeys (Macaca fascicularis): visual and neck receptor responses, J. Physiol., 430:559–583. Hautzel, H., J. G. Taylor, B. J. Krause, N. Schmitz, L. Tellmann, K. Zimmons, N. J. Shah, H. Herzog, and H.-W. Muller-Gartner, 2001. The motion aftereffect: more than area MT/V5?: evidence from 15O-butanol PET studies, Brain Res., 892:281–292. Hawken, M. J., A. Parker, and J. S. Lund, 1988. Laminar organization of contrast sensitivity of direction selective cells in

1244

, ,   

the striate cortex of the old world monkey, J. Neurosci., 8:3541– 3548. He, S., E. R. Cohen, and X. Hu, 1998. Close correlation between activity in brain area MT/V5 and the perception of a visual motion aftereffect, Curr. Biol., 8:1215–1218. Heeger, D. J., G. M. Boynton, J. B. Demb, E. Seidemann, and W. T. Newsome, 1999. Motion opponency in visual cortex, J. Neurosci., 19:7162–7174. Holmes, A. P., and K. J. Friston, 1998. Generalisability, random effects and population inference. NeuroImage, 7:S754. Huk, A. C., and D. J. Heeger, 2000. Task-related modulation of visual cortex, J. Neurophysiol., 83:3525–3536. Huk, A. C., D. Ress, and D. J. Heeger, 2001. Neuronal basis of the motion aftereffect reconsidered, Neuron, 32:161–172. Johannson, G., 1973. Visual perception of biological motion and a model for its analysis, Percept. Psychophys., 14:201–211. Jones, H. E., K. L. Grieve, W. Wang, and A. M. Sillito, 2001. Surround suppression in primate V1, J. Neurophysiol., 86:2011– 2028. Komatsu, H., and R. H. Wurtz, 1988. Relation of cortical areas MT and MST to pusuit eye movements. I. Localization and visual properties of neurons, J. Neurophysiol., 60:580–603. Kourtzi, Z., and N. Kanwisher, 2000. Activation in human MT/MST by static images with implied motion, J. Cogn. Neurosci., 12:48–55. Kwong, K. K., J. W. Belliveau, D. A. Chesler, I. E. Goldberg, R. M. Weiskoff, B. P. Poncelet, D. N. Kennedy, B. E. Hoppel, M. S. Cohen, R. Turner, B. Rosen, and T.J. Brady, 1992. Dynamic magnetic resonance imaging of human brain activity during primary sensory stimulation, Proc. Natl. Acad. Sci. USA, 89:5675–5679. Lagae, L., H. Maes, S. Raiguel, D. Xiao, and G. A. Orban, 1994. Responses of macaque STS neurons to optic flow components: a comparison of areas MT and MST, J. Neurophysiol., 71:1597–1626. Lagae, L., S. Raiguel, and G. A. Orban, 1993. Speed and direction selectivity of macaque middle temporal neurons, J. Neurophysiol., 69:19–39. Leite, F. P., D. Tsao, W. Vanduffel, D. Fize, Y. Sasaki, L. L. Wald, A. M. Dale, K. K. Kwong, G. A. Orban, B. R. Rosen, R. B. H. Tootell, and J. B. Mandeville, 2002. Repeated fMRI using iron oxide contrast agent in awake, behaving macaques at 3 Tesla, NeuroImage, 16:283–294. Lewis, J. W., M. S. Beauchamp, and E. A. DeYoe, 2000. A comparison of visual and auditory motion processing in human cerebral cortex, Cereb. Cortex, 10:873–888. Logothetis, N. K., H. Guggenberger, S. Peled, and J. Pauls. 1999. Functional imaging of the monkey brain, Nat. Neurosci., 2:555–562. Logothetis, N. K., J. Pauls, M. Augath, T. Trinath, and A. Oeltermann, 2001. Neurophysiological investigation of the basis of the fMRI signal, Nature, 412:150–157. Lueck, C. J., S. Zeki, K. J. Friston, M.-P. Deiber, P. Cope, V. J. Cunningham, A. A Lammertsma, C. Kennard, and R. S. J. Frackowiak, 1989. The colour centre in the cerebral cortex of man, Nature, 340:386–389. Malach, R., J. B. Reppas, R. R. Benson, K. K. Kwong, H. Jiang, W. A. Kennedy, P. J. Ledden, T. J. Brady, B. R. Rosen, and R. B. H. Tootell, 1995. Object-related activity revealed by functional magnetic resonance imaging in human occipital cortex, Proc. Natl. Acad. Sci. USA, 92:8135–8139. McCarthy, G., M. Spicer, A. Adrignolo, M. Luby, J. Gore, and T. Allison, 1995. Brain activation associated with visual motion

studied by functional magnetic resonance imaging in humans, Hum. Brain Mapping, 2:234–243. McKeefry, D. J., J. D. G. Watson, R. S. J. Frackowiak, K. Fong, and S. Zeki, 1997. The activity in human areas V1/V2, V3, and V5 during the perception of coherent and incoherent motion, NeuroImage, 5:1–12. Mikami, A., W. T. Newsome, and R. H. Wurtz, 1986. Motion selectivity in macaque visual cortex: I. Mechanisms of direction and speed selectivity in extrastriate area MT, J. Neurophysiol., 55:1308–1327. Morrone, M. C., M. Tosetti, D. Montanaro, A. Fiorentini, G. Cioni, and D. C. Burr, 2000. A cortical area that responds specifically to optic flow, revealed by fMRI, Nat. Neurosci., 3:1322–1328. Nakayama, K., 1985. Biological image motion processing: a review, Vis. Res., 25:625–660. Nakahara, K., T. Hayashi, S. Konishi, and Y. Miyashita, 2002. Functional MRI of macaque monkeys performing a cognitive set-shifting task, Science, 295:1532–1536. Newsome, W. T., and E. B. Paré, 1988. A selective impairment of motion perception following lesions of the middle temporal visual area (MT), J. Neurosci., 8:2201–2211. Newsome, W. T., R. H. Wurtz, M. R. Dursteler, and A. Mikami, 1985. Deficits in visual motion processing following ibotenic acid lesions of the middle temporal vosial area of the macaque monke, J. Neurosci., 5:825–840. O’Craven, K. M., B. R. Rosen, K. K. Kwong, A. Treisman, and R. L. Savoy, 1997. Voluntary attention modulates fMRI activity in human MT-MST, Neuron, 18:591–598. O’Driscoll, G. A., S. M. Strakowski, N. M. Alpert, S. W. Matthysse, S. L. Rauch, D. L. Levy, and P. S. Holzman, 1998. Differences in cerebral activation during smooth pursuit and saccadic eye movements using positron-emission tomography, Biol. Psychiatry, 44:685–689. Orban, G. A., 1997. Visual processing in macaque area MT/V5 and its satellites (MSTd and MSTv), in Cerebral Cortex, vol. 12, Extrastriate Cortex in Primates (K. S. Rockland, J. H. Kaas, and A. Peters, eds.), New York: Plenum Press, pp. 359–434. Orban, G. A., P. Dupont, B. De Bruyn, R. Vandenberghe, A. Rosier, and L. Mortelmans, 1998. Human brain activity related to speed discrimination tasks, Exp. Brain Res., 122:9–22. Orban, G. A., P. Dupont, B. De Bruyn, R. Vogels, R. Vandenberghe, and L. Mortelmans, 1995a. A motion area in human visual cortex, Proc. Natl. Acad. Sci. USA, 92:993–997. Orban, G. A., P. Dupont, R. Vogels, G. Bormans, and L. Mortelmans, 1997. Human brain activity related to orientation discrimination tasks, Eur. J. Neurosci., 9:246–259. Orban, G. A., H. Kennedy, and J. Bullier, 1986. Velocity sensitivity and direction selectivity of neurons in areas V1 and V2 of the monkey: influence of eccentricity, J. Neurophysiol., 56:462–480. Orban, G. A., S. Sunaert, J. T. Todd, P. Van Hecke, and G. Marchal, 1999. Human cortical regions involved in extracting depth from motion, Neuron, 24:929–940. Orban, G. A., R. Saunders, and E. Vandenbussche, 1995b. Lesions of the superior temporal cortical motion areas impair speed discrimination in the macaque monkey, Eur. J. Neurosci., 7:2261–2276. Pack, C. C., V. K. Berezovskii, and R. T. Born, 2001. Dynamic properties of neurons in cortical area MT in alert and anaesthetized macaque monkeys, Nature, 414:905–908. Paradis, A. L., V. Cornilleau-Peres, J. Droulez, P. F. Van De Moortele, E. Lobel, A. Berthoz, D. Le Bihan, and J. B. Poline,

2000. Visual perception of motion and 3-D structure from motion: an fMRI study, Cereb. Cortex, 10:772–783. Peigneux, P., E. Salmon, M. van der Linden, G. Garraux, J. Aerts, G. Delfiore, C. Degueldre, A. Luxen, G. Orban, and G. Franck, 2000. The role of lateral occipitotemporal junction and area MT/V5 in the visual analysis of upper-limb postures, NeuroImage, 11:644–655. Petit, L., V. P. Clark, J. Ingeholm, and J. V. Haxby, 1997. Dissociation of saccade-related and pursuit-related activation in human frontal eye fields as revealed by fMRI, J. Neurophysiol., 77:3386–3390. Petit, L., and J. V. Haxby, 1999. Functional anatomy of pursuit eye movements in humans as revealed by fMRI, J. Neurophysiol., 82:463–471. Peuskens, H., S. Sunaert, P. Dupont, P. Van Hecke, and G. A. Orban, 2001a. Human brain regions involved in heading estimation, J. Neurosci., 21:2451–2461. Peuskens, H., S. Sunaert, J. Vanrie, K. Verfaillie, P. Van Hecke, and G. A. Orban, 2001b. Disentangling brain areas activated by biological motion, Soc. Neurosci. Abstr., 27:165.35. Phinney, R. E., and E. A., DeYoe, 2001. fMRI investigation of purely stereoscopic (cyclopean) motion processing, Soc. Neurosci. Abstr., 27:680.10. Previc, F. H., M. Liotti, C. Blakemore, J. Beer, and P. Fox, 2000. Functional imaging of brain areas involved in the processing of coherent and incoherent wide field-of-view visual motion, Exp. Brain Res., 131:393–405. Puce, A., T. Allison, S. Bentin, J. C. Gore, and G. McCarthy, 1998. Temporal cortex activation in humans viewing eye and mouth movements, J. Neurosci., 18:2188–2199. Qian, N., and R. A. Andersen, 1994. Transparent motionperception as detection of unbalanced motion signals, J. Neurosci., 14:7367–7380. Raiguel, S., M. M. Van Hulle, D.-K. Xiao, V. L. Marcar, L. Lagae, and G. A. Orban, 1997. Size and shape of receptive fields in the medial superior temporal area (MST) of the macaque, NeuroReport, 8:2803–2808. Rees, G., K. Friston, and C. Koch, 2000. A direct quantitative relationship between the functional properties of human and macaque V5, Nat. Neurosci., 3:716–723. Rees, G., C. D. Frith, and N. Lavie, 1997. Modulating irrelevant motion perception by varying attentional load in an unrelated task, Science, 278:1616–1619. Rees, G., C. Frith, and N. Lavie, 2001. Processing of irrelevant visual motion during performance of an auditory attention task, Neuropsychologia, 39:937–949. Reppas, J. B., S. Niyogi, A. M. Dale, M. I. Sereno, and R. B. Tootell, 1997. Representation of motion boundaries in retinotopic human visual cortical areas, Nature, 388:175–179. Saito, H., M. Yukie, K. Tanaka, K. Hikosaka, Y. Fukada, and E. Iwai, 1986. Integration of direction signals of image motion in the superior temporal sulcus of the macaque monkey, J. Neurosci., 6:145–157. Schall, J. D., A. Morel, D. J. King, and J. Bullier, 1995. Topography of visual cortex connections with frontal eye field in macaque, J. Neurosci., 15:4464–4487. Schoups, A., R. Vogels, N. Qian, and G. A. Orban, 2001. Practising orientation identification improves orientation coding in V1 neurons, Nature, 412:549–553. Seghier, M., M. Dojat, C. Delon-Martin, C. Rubin, J. Warnking, C. Segebarth, and J. Bullier, 2000. Moving illusory contours activate primary visual cortex: an fMRI study, Cereb. Cortex, 10:663–670.

  :  

1245

Seidemann, E., A. B. Poirson, B. A. Wandell, and W. T. Newsome, 1999. Color signals in area MT of the macaque monkey, Neuron, 24:911–917. Senior, C., J. Barnes, V. Giampietro, A. Simmons, E. T. Bullmore, M. Brammer, and A. S. David, 2000. The functional neuroanatomy of implicit-motion perception or representational momentum, Curr. Biol., 10:16–22. Shadlen, M. N., and W. T. Newsome, 1996. Motion percpetion: seeing and deciding, Proc. Natl. Acad. Sci. USA, 93:628–633. Shulman, G. L., J. M. Ollinger, E. Akbudak, T. E. Conturo, A. Z. Snyder, S. E. Petersen, and M. Corbetta, 1999. Areas involved in encoding and applying directional expectations to moving objects, J. Neurosci., 19:9480–9496. Smith, A. T., M. W. Greenlee, K. D. Singh, F. M. Kraemer, and J. Hennig, 1998. The processing of first- and second-order motion in human visual cortex assessed by functional magnetic resonance imaging (fMRI), J. Neurosci., 18:3816–3830. Somers, D. C., A. E. Seiffert, A. M. Dale, and R. H. Tootell, 1999. fMRI investigations of motion aftereffects with 1st- and 2ndorder stimuli, Invest. Ophthalmol. Vis. Sci., 40:S199. Stavenski, A. A., D. A. Robinson, R. M. Steinman, and G. T. Timberlake, 1975. Miniature eye movements of fixation in rhesus monkey, Vis. Res., 15:1269–1273. Sunaert, S., G. Fesl, J. Todd, A. Rosier, P. Van Hecke, and G. A. Orban, 2000a. Human cortical regions involved in extracting 3D structure from motion depend on stimulus type and transparency, Soc. Neurosci. (Abs.), 26:1583. Sunaert, S., P. Van Hecke, and G. Marchal, 1999. Motion-responsive regions of the human brain, Exp. Brain Res., 127:355–370. Sunaert, S., P. Van Hecke, G. Marchal, and G. A. Orban, 2000b. Attention to speed of motion, speed discrimination, and task difficulty: an fMRI study, NeuroImage, 11:612–623. Taylor, J. G., N. Schmitz, K. Ziemons, M.-L. Grosse-Ruyken, O. Gruber, H.-W. Mueller-Gaertner, and N. J. Shah, 2000. The network of brain areas involved in the motion aftereffect, NeuroImage, 11:257–270. Tolias, A. S., S. M. Smirnakis, M. A. Augath, T. Trinath, and N. K. Logothetis, 2001. Motion processing in the macaque: revisited with functional magnetic resonance imaging, J. Neurosci., 21:8594–8601. Tootell, R. B. H., and N. Hadjikhani, 2001. Where is “dorsal V4” in human visual cortex? Retinotopic, topographic and functional evidence, Cereb. Cortex, 11:298–311. Tootell, R. B. H., J. D. Mendola, N. K. Hadjikhani, P. J. Ledden, A. K. Liu, J. B. Reppas, M. I. Sereno, and A. M. Dale, 1997. Functional analysis of V3A and related areas in human visual cortex, J. Neurosci., 17:7060–7078. Tootell, R. B. H., J. D. Mendola, N. K. Hadjikhani, A. K. Liu, and A. M. Dale, 1998. The representation of the ipsilateral visual

1246

, ,   

field in human cerebral cortex, Proc. Natl. Acad. Sci. USA, 95:818–824. Tootell, R. B. H., J. B. Reppas, A. M. Dale, R. B. Look, M. I. Sereno, R. Malach, T. J. Brady, and B. R. Rosen, 1995a. Visual motion aftereffect in human cortical area MT revealed by functional magnetic resonance imaging, Nature, 375:139–141. Tootell, R. B. H., J. B. Reppas, K. K. Kwong, R. Malach, R. T. Born, T. J. Brady, B. R. Rosen, and J. W. Belliveau, 1995b. Functional analysis of human MT and related visual cortical areas using magnetic resonance imaging, J. Neurosci., 15:3215–3230. Tootell, R. B. H., and J. B. Taylor, 1995. Anatomical evidence for MT and additional cortical visual areas in humans, Cereb. Cortex, 5:39–55. Van Oostende, S., S. Sunaert, P. Van Hecke, G. Marchal, and G. A. Orban, 1997. The kinetic occipital (KO) region in man: an fMRI study, Cereb. Cortex, 7:690–701. Vanduffel, W., D. Fize, J. B. Mandeville, K. Nelissen, P. Van Hecke, B. R. Rosen, R. B. H. Tootell, and G. A. Orban, 2001. Visual motion processing investigated using contrast-agent enhanced fMRI in awake behaving monkeys, Neuron, 32:565–577. Wandell, B. A., A. B. Poirson, W. T. Newsome, H. A. Baseler, G. M. Boynton, A. Huk, S. Gandhi, and L. T. Sharpe, 1999. Color signals in human motion-selective cortex, Neuron, 24:901– 909. Wang, J., T. Zhou, M. Qiu, A. Du, K. Cai, Z. Wang, C. Zhou, M. Meng, Y. Zhuo, S. Fan, and L. Chen, 1999. Relationship between ventral stream for object vision and dorsal stream for spatial vision: an fMRI + ERP study, Hum. Brain Mapping, 8:170–181. Watson, J. D. G., R. Myers, R. S. J. Frackowiak, J. V. Hajnal, R. P. Woods, J. C. Mazziotta, S. Shipp, and S. Zeki, 1993. Area V5 of the human brain: evidence from a combined study using positron emission tomography and magnetic resonance imaging, Cereb. Cortex, 3:79–94. Xiao, D.-K., V. L. Marcar, S. E. Raiguel, and G. A. Orban, 1997. Selectivity of macaque MT/V5 neurons for surface orientation in depth specified by motion, Eur. J. Neurosci., 9:956–964. Zeki, S., J. D. G. Watson, and R. S. J. Frackowiak, 1995. Going beyond the information given: the relation of illusory visual motion to brain activity, Proc. R. Soc. Lond. B, 252:215–222. Zeki, S., J. D. G. Watson, C. J. Lueck, K. J. Friston, C. Kennard, and R. S. J. Frackowiak, 1991. A direct demonstration of functional specialization in human visual cortex, J. Neurosci., 11:641–649. Zilles, K., G. Schlaug, M. Matelli, G. Luppino, A. Schleicher, M. Qu, A. Dabringhaus, R. Seitz, and P. E. Roland, 1995. Mapping of human and macaque sensorimotor areas by integrating architectonic, transmitter receptor, MRI and PET data, J. Anat., 187:515–537.

84

Optic Flow WILLIAM H. WARREN

OPTIC FLOW    of motion present at the eye of a moving observer. Such flow patterns contain information about self-motion, moving objects, and the threedimensional (3D) layout of the environment, and could potentially be exploited to control locomotion. The term was first used by Gibson (1950) to generalize Helmholtz’s notion of motion parallax from single objects to the continuous environment surrounding the observer. He developed the concept during World War II while working on methods for pilot testing and training, after he concluded that the classical depth cues were inadequate to explain a pilot’s practical ability to land an airplane. But little research was done on the subject until the 1980s, when its relevance to robot and vehicle guidance was recognized and computer animation made it amenable to study. Optic flow is a key example of Gibson’s (1979) ecological approach to perception and action. This functionalist approach emphasizes that the task of vision is to guide successful behavior in the natural environment, and stresses the importance of higher-order optical information that specifies complex properties of that environment and its relationship to the observer. The nervous system is viewed as providing causal support for the detection of informational variables and the modulation of action variables. Optic flow offers a case study of a higher-order variable that has been formally analyzed and empirically investigated in some detail. In what follows, we will trace optic flow through the cycle of perception and action. The past 20 years have seen intensive research on the topic, with parallel developments in the domains of human perception, primate neurophysiology, and computational vision. This chapter aims to integrate these perspectives to address both mechanistic questions about how optic flow patterns are extracted by the visual cortex and functional questions about how this information is used to perceive selfmotion and control behavior. (For related reviews see Lappe et al., 1999; Warren, 1998; regarding structure from motion, see Todd, 1995.)

The optic flow field Optic flow is typically represented as an instantaneous velocity field in which each vector corresponds to the optical motion of a point in the environment, as in Figure 84.1A. It is immediately apparent that this flow field has a radial struc-

ture, with a focus of expansion (FOE) lying in the direction of self-motion. When one views the moving display, it is also apparent that the environment is a planar surface receding in depth. The radial pattern of vector directions depends solely on the observer’s direction of translation, or heading, and is independent of the 3D structure. The magnitude of each vector, on the other hand, depends on both heading and depth and decreases quickly with distance. The radial flow pattern thus specifies one’s current heading, whether or not the local FOE itself is visible. This was Gibson’s fundamental hypothesis about the perception of self-motion from optic flow, and was the starting point for both psychophysical and neurophysiological experiments. Although the velocity field description is compatible with the motion selectivity of cortical areas V1 and MT, it does not represent higher-order temporal components of the optic flow such as acceleration, or the trajectories of points over time. This appears to be a reasonable approximation, for the visual system is relatively insensitive to acceleration and relies primarily on the first-order flow to determine heading (Paolini et al., 2000; Warren et al., 1991a). O R However, the detection of optic flow by a moving eye is complicated by the fact that the eye can also rotate (Gibson, 1950, pp. 124–127). If the observer simply translates on a straight path, the flow pattern on the retina is radial (Fig. 84.1A). This is called the translational component of retinal flow, and recovery of heading from it is straightforward. A rotation of the observer, such as a pursuit eye or head movement, merely displaces the image on the retina, producing the rotational component of retinal flow.1 Specifically, pitch or yaw of the eye create patterns of vertical or horizontal lamellar flow (Fig. 84.1B), and roll about the line of sight creates a pattern of rotary flow. But if the eye is simultaneously translating and rotating, which commonly occurs when one fixates a point in the world during locomotion, the retinal flow is the vector sum of these two components (Fig. 84.1C). The resulting flow field is more complex, without a qualitative feature corresponding to

1

To keep these distinctions clear, I will consistently use translation and rotation to describe observer movement and lamellar and rotary to describe flow patterns.

1247

T

(a)

R

(b)

T+R

(c)

F 84.1. Retinal flow field for a ground plane with rotation about a diagonal axis. a, Translational component produced by observer translation toward the x: radial flow from the heading point. b, Rotational component produced by eye rotation downward and to the right: lamellar flow upward and to the left. c, Flow field

produced by the sum of a and b due to heading toward the x while rotating to track the o on the ground plane. The singularity in the field is now at the fixation point. Note the motion parallax (differences in vector direction and length) at different distances across the surface.

one’s heading; indeed, the singularity in the flow field is now at the fixation point. Thus, to determine heading, the visual system must somehow analyze the translational and rotational components and recover the direction of self-motion. This has come to be known as the rotation problem. Fortunately, the retinal flow in a 3D scene contains sufficient information to solve this problem in principle. Specifically, motion parallax between points at different depths corresponds to observer translation, whereas common lamellar motion across the visual field corresponds to observer rotation.

heading. A further problem is that the velocity field does not specify one’s path over time—for instance, whether one is traveling on a straight or a curved path. In fact, the same flow field can be generated by a straight path together with eye rotation or by a circular path of self-motion. A particularly troublesome case appears in Figure 84.2, in which translation plus rotation about a vertical axis produces the same velocity field as a circular path on the ground plane. When presented with such flow displays, observers often report seeing a curved path of self-motion rather than a straight path plus rotation. How, then, can one determine whether one is traveling on a straight or a curved path? We will call this the path problem.

T P  S- Even if the rotation problem can be solved, however, it only yields one’s instantaneous

1248

, ,   

F 84.2. Retinal flow field for a ground plane with rotation about a vertical axis. a, Translational component produced by observer translation toward the x. b, Rotational component produced by eye rotation to the right. c, Flow field produced by the sum of a and b due to translating toward the x while rotating to track the o on the post, which is at eye level. An identical velocity field is generated by travel on a circular path to the right, with instantaneous heading along the tangent toward the x.

Before tackling these issues, let’s consider the detection of optic flow patterns.

Detecting optic flow From a mechanistic point of view, the basic question is how optic flow is detected by the visual system. It now appears that optic flow patterns are extracted in two steps, by first detecting local motion and then integrating it spatially in units sensitive to patterns of motion. This can be demonstrated psychophysically using random-dot displays in which the proportion of “signal” dots moving in a coherent flow pattern is varied relative to “noise” dots that jump to random positions on each frame. As the area of coherent motion in

the display increases, the direction of motion can be reported at a lower signal-to-noise ratio, providing evidence of spatial summation (Burr et al., 1998). Such coherence thresholds reveal summation for radial, rotary, and lamellar flow patterns over large visual angles up to 36 to 72 degrees. The data are closely predicted by an ideal integrator that pools local motion signals in units sensitive to pattern motion with large receptive fields. F- U  MSTd Such findings converge with results from single-cell recordings in the dorsal region of the medial superior temporal area (MSTd) of macaque visual cortex (see Chapters 83 and 85, this volume). Saito et al. (1986) reported cells selective for expansion or con-

:  

1249

traction, clockwise or counterclockwise rotary flow, or directional lamellar flow, and proposed that they could be constructed by integrating input from velocity-selective cells in area MT. The MSTd cells are characterized by large receptive fields 15 to 65 degrees in diameter, distributed over the visual field to eccentricities of 50 degrees, with a higher concentration closer to the fovea (Raiguel et al., 1997). These cells exhibit stronger responses to larger stimulus patterns, consistent with spatial integration of local velocity signals (Duffy and Wurtz, 1991b; Tanaka and Saito, 1989). About half of MSTd cells are selective for radial or rotary flow at various locations in their receptive fields, a property known as position invariance. Rigorous tests of position invariance ensure that a cell’s response is not simply due to a coincidental match between the stimulus and a local region of the receptive field (Lagae et al., 1994). This finding is consistent with an architecture in which clusters of velocityselective MT cells, each having a radial or rotary arrangement at different retinal loci, converge on one MSTd cell. Such radial and rotary cells are also broadly tuned to the overall speed of motion, so that they may form a distributed representation of mean flow speed (Orban et al., 1995; Tanaka and Saito, 1989). There are also indications of sensitivity to speed gradients within the flow pattern, with some cells preferring an increasing gradient from center to periphery and others preferring a decreasing gradient (Duffy and Wurtz, 1997). However, the gradients tested were quite extreme, and MSTd cells are relatively insensitive to the fine speed gradients needed to distinguish 3D shape. Eliminating the pattern of vector directions has a far greater impact on a cell’s response than eliminating the speed gradient (Tanaka et al., 1989). MST projects to higher areas in the dorsal visual pathway, including parietal area 7a. Cells in this area are more narrowly tuned to radial, rotary, spiral, and lamellar flow patterns (Siegel and Read, 1997), indicating that the processing of optic flow continues beyond MSTd. D  T? There has been a great deal of interest in the hypothesis that specialized channels in MSTd form a basis set for the decomposition of optic flow. Koenderink and van Doorn (1975, 1981; Longuet-Higgins and Prazdny, 1980) showed that any optic flow field can be analyzed locally into lamellar motion plus three elementary components: divergence (div) or rate of local expansion, which together with surface slant can be used to estimate heading; curl or rate of local rotary flow; and deformation (def ) or rate of shear along two orthogonal axes, which is specific to surface shape. This insight inspired a number of studies on sensitivity to radial, rotary, and lamellar flow in MSTd. However, there is often confusion between divergence and radial flow. Formally, div is a measure of the local rate of expansion, not the radial motion pattern or the FOE.

1250

, ,   

The balance of psychophysical research does not support decomposition into div, curl, def, and lamellar motion. For example, when the div is removed from an expansion pattern, resulting in radial flow with decelerating dot motion outward from the FOE, the detection thresholds for expansion in noise are unaffected (Kappers et al., 1996). Such results are contrary to a specialized divergence detector. Although the maximum of divergence can be located in visual displays (Regan and Beverly, 1982), there is no evidence that it is actually used to judge heading (Warren et al., 1988). Similar conclusions have emerged from single-cell recordings in MSTd. When cells selective for radial or rotary flow are presented with a spiral flow pattern containing both div and curl components, the response is lower than with the equivalent radial or rotary pattern alone (Orban et al., 1992), indicating that the spiral flow is not decomposed. A majority of cells are not tuned to a single elementary component but respond to two or three individual components (Duffy and Wurtz, 1991a) or to combinations of radial and rotary flow (Graziano et al., 1994). There thus appears to be a continuum of cells selective not only for radial and rotary flow but also for intermediate spiral flow patterns. Taken together, the evidence indicates that MSTd cells do not decompose optic flow into elementary components, but rather act as templates or filters for complex flow patterns. The advantage of a template mechanism is that, rather than simply recoding the information in the flow field, its response can signal the presence of meaningful patterns of stimulation.

Perception of translational heading From a functional point of view, the primary question about optic flow is whether people can in fact perceive their selfmotion from optic flow patterns. Cutting et al. (1992) estimated that a heading accuracy of about 1 to 3 degrees is required to guide ordinary locomotion such as running and skiing. When observers view a random-dot display of radial flow and judge whether they are heading to the left or right of a probe, they are highly accurate, with heading thresholds as low as 0.5 degree (Warren, 1976; Warren et al., 1988). Translational heading can thus be perceived from radial flow with sufficient accuracy to control locomotion. S I Heading judgments also reveal spatial integration, similar to simple coherence thresholds. In principle, a heading estimate could be obtained by triangulating just two motion vectors in a radial flow pattern. However, such a mechanism would be vulnerable to noise in local motion sensors, leading to large heading errors (Koenderink and van Doorn, 1987). Since each vector provides an estimate of the heading, pooling them allows this

redundancy in the flow pattern to reduce triangulation error. Consistent with such pooling, heading thresholds decrease – as dots are added to a radial flow display, following a 1/÷N rule, up to an asymptote with about 30 dots in a 40 ¥ 32 degree display (Warren et al., 1988). Heading judgments are also highly robust to flow field noise created by perturbing vector directions (Warren et al., 1991a). Both of these properties are reproduced by a simple neural model of translational heading in which local motion signals are pooled in large-field radial flow templates (Hatsopoulos and Warren, 1991), consistent with the spatial integration observed in MSTd. F S  S G In line with the fact that the radial structure of flow is independent of depth, heading thresholds are similar in different 3D environments, including a ground plane, a frontal plane, and a cloud of dots. Judgments even remain accurate when speed gradients are removed by randomizing the magnitudes of local vectors, preserving their directions, but they become impossible when vector directions are randomized, preserving their magnitudes (Warren et al., 1991a). This result confirms the primary importance of the pattern of vector directions and is strikingly consistent with MSTd responses (Tanaka et al., 1989). However, heading judgments are influenced by large speed gradients. If vector magnitudes on one side of the FOE are greatly increased, perceived heading is biased toward the opposite side, and subjects report travel on a curved path (Dyre and Andersen, 1996). This is plausible because, in a homogeneous environment, the pattern of flow speeds is correlated with the path of travel. On a straight path, vectors tend to increase symmetrically from the FOE (Fig. 84.2A), whereas on a curved path, vectors on the outside of the path tend to be greater than those on the inside (Fig. 84.2C ). H   P  M O Most optic flow analyses presume a rigid scene, yet we locomote successfully in a dynamic environment with independently moving objects. A moving object can significantly alter the radial flow pattern, creating a local region of inconsistent motion. To cope with this, the visual system might segment the scene and make separate estimates of self-motion from the background flow and of object motion from the discrepant flow (Hildreth, 1992). A simpler solution would be to estimate the FOE by pooling all motion vectors, consistent with spatial integration in MSTd, although this would lead to predictable heading errors. In fact, heading judgments exhibit just such errors when the object crosses the FOE (Warren and Saunders, 1995). If the object is moving in depth, it creates a piece of a radial flow pattern with a secondary FOEo. This is enough to bias perceived heading opposite the direction of object motion by

a few degrees toward the FOEo. Such an effect is consistent with spatial integration of all motion to locate the FOE. However, if the object is moving in the frontal plane, it creates a piece of lamellar flow, which biases perceived heading in the same direction as object motion (Royden and Hildreth, 1996). This effect is similar to an illusory shift in the FOE that occurs when a lamellar flow pattern is transparently superimposed on a radial flow pattern (Duffy and Wurtz, 1993) and could be a consequence of using lamellar flow to estimate observer rotation (Lappe and Duffy, 1999). On the other hand, both effects might also be explained as a result of pooling the motion parallax between the object and the background together with the local dot motions, which could be a consequence of using motion parallax to recover heading. These results lead to the surprising conclusion that moving objects are not segmented in the course of perceiving heading, and suggest that self-motion relative to the environment may be determined by a task-specific mechanism that responds to any flow within its receptive field. MSTd cells might provide such a mechanism, for they respond selectively to a given flow pattern whether it is carried by a large field of random dots or the local boundary of a square (Geesaman and Andersen, 1996). This suggests that MSTd detects flow patterns produced by relative motion between the observer and the environment, without differentiating local object motion. I S-M D  MSTd? Selectivity for large-field radial, rotary, and lamellar flow patterns and their combinations makes MSTd a likely candidate for extracting information about self-motion. Such global motion patterns are generated by observer translation, roll, and pitch/yaw with respect to the environment. As we have seen, there are a number of commonalities between the characteristics of MSTd cells and heading judgments: (1) large receptive fields suitable for global flow, (2) spatial integration, (3) dominance of vector directions but also (4) responses to large speed gradients, (5) speed tuning that could code the velocity of observer translation or rotation, and (6) failure to differentiate local object motion and global flow. At first blush, the position invariance of MSTd cells appears inconsistent with this hypothesis, for heading detection by single cells implies a preferred locus for the FOE in the receptive field. However, this objection confuses response selectivity with response amplitude. Duffy and Wurtz (1995) found that 90% of expansion cells do have a preferred FOE and exhibit a graded response as the focus is shifted away from this position, which is well fit by a Gaussian tuning curve (Raiguel et al., 1997). A population of such cells is in principle sufficient for a precise distributed coding of heading direction. A second issue is that MSTd cells do not appear to form a contiguous retinotopic map of heading

:  

1251

direction. Nevertheless, similarly tuned cells tend to be clustered together in approximate cortical columns (Britten, 1998), which is not incompatible with distributed coding. The most direct evidence that MSTd is involved in heading is that microstimulation in a cluster of cells biases radial flow localization judgments in the direction preferred by nearby cells in two-thirds of cases (Britten and von Wezel, 1998). The conclusion is reinforced by the finding that vestibular stimulation produced by physically translating the animal modulates the response amplitude and even the direction preference to radial flow in MSTd cells (Duffy, 1998). This strongly implies that MSTd is a site that integrates sensory stimulation about self-motion. In humans, brain imaging studies indicate that a possible homolog of MSTd, an inferior satellite of MT/V5, is selective for radial and rotary flow patterns (Morrone et al., 2000). Using an active judgment task, Peuskens et al. (2001) found that attending to heading per se enhanced specific activity in this MT/V5 satellite, as well as in a posterior region of the dorsal intraparital sulcus (a possible homolog of area 7a) and in dorsal premotor cortex. These results suggest that heading is detected in a pathway that is functionally similar to primate areas MSTd and 7a and directly related to motor behavior.

The rotation problem Determining heading from radial flow during pure translation is relatively straightforward. But when the observer also makes a pursuit rotation of the eye or head, a component of lamellar flow is added, radically altering the retinal flow field (Fig. 84.2C ). How, then, might the instantaneous direction of heading be determined during rotation? There are two general approaches to the rotation problem. E T Extraretinal theories propose that internal signals about the rotational velocity of the eye and head, possibly including efferent, proprioceptive, and vestibular signals, are used to estimate the rotational component of self-motion (Banks et al., 1996; Royden et al., 1994). The rotational component can then be subtracted from the retinal flow pattern in order to recover the translational component and hence the instantaneous direction of heading. If any rotation remains in the flow pattern, it can be attributed to a curved path of self-motion. Note that the resulting heading estimate is in retinal coordinates, or an oculocentric frame of reference. Thus, extraretinal signals about the position of the eye and head may also be needed to transform the heading into head-centric and body-centric reference frames. A possible mechanism for extraretinal theories is dynamic tuning of receptive fields in MSTd. During a pursuit eye movement, the preferred FOE tuning of most expansion

1252

, ,   

cells actually shifts on the retina in the direction of rotation, partially compensating for pursuit (Bradley et al., 1996). However, the mean shift is only half of what is needed to compensate fully for the eye movement. Some cells show no shift at all, but rather a modulation in the amplitude of their response, consistent with the notion of a gain field (Shenoy et al., 1999). On the other hand, Page and Duffy (1999) reported that a computed population vector fully compensates for pursuit rotation and might be “read out” by a higher area. Area 7a contains cells that are narrowly tuned to radial flow and show gain modulation by eye position (Read and Siegel, 1997), potentially converting an oculocentric heading estimate into a head-centric frame. Physiological results would thus seem to support an extraretinal approach. However, Shenoy et al. (1999) found large receptive field shifts in response to the flow pattern alone, even when the eye was stationary. This suggests that partial compensation can occur on the basis of the retinal flow itself. Moreover, these experiments only tested planar (non-3D) flow patterns, and compensation might be greater with motion parallax in the display. This leads us to the second approach. R F T Gibson argued that the optical information available in natural environments is sufficient for visual perception, and was skeptical of the need for extraretinal signals. Retinal flow theories propose that heading can be determined from the pattern of retinal flow alone (Warren and Hannon, 1990). This is theoretically possible because observer rotation simply adds a common lamellar component to the flow field, such that the differential motion due to translation remains invariant. Numerous computational models have formally shown that instantaneous heading can be recovered from the velocity field during rotation (see Hildreth and Royden, 1998; Warren, 1998, for reviews). As before, this heading estimate is in an oculo-centric frame of reference, so extraretinal position signals would seem to be necessary to convert it to headcentric and body-centric reference frames to recover one’s absolute heading in space. But such coordinate transformations could be bypassed by determining one’s object-relative heading, the direction of translation with respect to objects that are also registered in retinal coordinates. This is physiologically plausible because target position and optic flow analyses converge at least by area 7a. There are three main classes of retinal flow theories. One class first estimates observer rotation from the lamellar flow, then subtracts the rotational component from the retinal flow pattern to recover the translational component (Perrone, 1992). In principle, the rotational component could be measured by integrating retinal velocities independently about three orthogonal axes (Koenderink, 1986). It is thus plausible that MSTd cells selective for horizontal,

vertical, and rotary flow patterns detect observer rotation; indeed, the preferred direction of pursuit eye movements in these cells is opposite the preferred direction of lamellar flow. Because the motion of more distant points is increasingly dominated by the rotational component (Fig. 84.2), depth information such as binocular disparity could also contribute to estimating the rotational component (van den Berg and Brenner, 1994). A second class of theories determines the heading directly from motion parallax, which is also known as relative or differential motion (Longuet-Higgins and Prazdny, 1980; Rieger and Lawton, 1985). The relative motion between two points at different distances along a line of sight can be described by a difference vector. Remarkably, the set of difference vectors for a 3D scene forms a radial pattern centered on the heading. The rotation problem can thus be solved quite elegantly if there is sufficient depth structure in the environment. Moreover, relative motion and common motion could be extracted in parallel to decompose observer translation and rotation. In theory, differential motion could be detected by an array of antagonistic center-surround units, with either circularly symmetric or bilateral center-surrounds (Royden, 1997). A majority of cells in area MT actually possesses such a bilateral opponent-motion organization, is sensitive to speed differences between center and surround, and is segregated in separate columns from classic MT velocity units (Born and Tootell, 1992; Xiao et al., 1997). Given that MT projects to MSTd, these two motion pathways could detect radial patterns of ordinary flow vectors and difference vectors in parallel, allowing the visual system to extract heading in both planar and 3D environments (Warren, 1998). Consistent with this idea, the responses of MSTd cells are enhanced by the addition of motion parallax in a radial flow display during fixation, and even more so during pursuit movements (Upadhyay et al., 2000). Thus, motion parallax signals contribute to activity in MSTd, especially during eye rotation, a finding that deserves further investigation. A third class of theories is based on templates for the set of flow patterns produced by all possible combinations of observer translation and rotation (Lappe and Rauschecker, 1993; Perrone and Stone, 1994). The space of flow patterns is constrained by restricting rotations to pursuit of stationary points in the environment. Perrone and Stone (1998) showed that their model templates reproduce many properties of MSTd cells, including selectivity for radial, lamellar, and spiral flow, position invariance, and a preferred FOE. P H  R The psychophysical evidence on the rotation problem is mixed, and the issue remains controversial. In these experiments, displays are manipulated to dissociate retinal and extraretinal contributions. In the actual rotation condition, a radial flow pattern is

F 84.3. Retinal flow field for a frontal plane with rotation about a diagonal axis. The observer is translating toward the x while rotating to track the o on the plane downward and to the right. This produces a pseudo-FOE at the fixation point. Because the distance of the surface from the eye changes very little, only a little motion parallax is defined across a large display.

presented on the screen (Fig. 84.1A) together with a moving fixation point that induces a pursuit eye movement; thus, extraretinal signals correspond to the rotation. In the simulated rotation condition, the flow pattern on the screen simulates the effect of forward translation plus a pursuit rotation (Fig. 84.1C) while the fixation point remains stationary, so extraretinal signals correspond to zero rotation. The flow pattern on the retina is thus the same in both conditions, while the extraretinal signal is manipulated. If judgments are equally accurate in the simulated and actual conditions, it implies that heading can be perceived from the retinal flow alone—even with conflicting extraretinal signals. On the other hand, if judgments are markedly less accurate in the simulated condition, it implies an extraretinal contribution. Warren and Hannon (1988, 1990) reported that heading judgments with a random-dot ground plane (Fig. 84.1C ) or 3D cloud were comparable in the two conditions, yielding heading thresholds below 1.5 degrees. On the other hand, with a frontal plane, which contains almost no motion parallax (Fig. 84.3), performance was at chance in the simulated condition but remained accurate in the actual condition. This pattern of results is consistent with retinal flow theories and confirms the importance of motion parallax. Observers even report an illusory eye rotation in the simulated condition corresponding to motion of the (stationary) fixation point away from the heading. But good performance with a frontal plane during actual eye rotation also indicates that extraretinal signals can contribute to heading judgments. However, in these experiments the mean rotation rate over a trial was less than 1 deg/sec. At higher simulated rota-

:  

1253

tion rates (1 to 5 deg/sec),2 heading judgments were subsequently found to be highly inaccurate (Banks et al., 1996; Royden et al., 1994). Constant errors rose to 15 degrees in the direction of simulated rotation but remained close to zero during actual eye rotation. This pattern of results is consistent with the extraretinal theory for high rotation rates. At the same time, other research supported a retinal flow theory (van den Berg, 1993; Wang and Cutting, 1999). Binocular disparity appeared to make heading judgments more reliable during simulated rotation (van den Berg and Breener, 1994), although Ehrlich et al. (1998) failed to replicate this effect. Royden (1994) pointed out that observers frequently report a curved path of self-motion in the simulated rotation condition, and that heading errors in the direction of simulated rotation are consistent with the perception of a curved path. This might be expected from the ambiguity of the velocity field, for the same instantaneous field can be produced by a straight path plus rotation or by a circular path of self-motion with radius r (Fig. 84.2C ), which are related by the ratio of translation and rotation speeds (r = T/R). Because extraretinal signals indicate that rotation is zero in the simulated condition, this could lead to the perception of a curved path. It is important to distinguish the concepts of heading and path, for although they are identical for a straight path, the instantaneous heading direction is tangent to a curved path. It is apparently difficult for observers to judge their instantaneous heading per se, for even when instructed to do so, they tend to report their perceived path. This leaves open the possibility that the oculocentric heading is accurately recovered during simulated rotation, but because the body-centric heading changes over time (literally drifting across the screen), subjects report a curved path. One response to this dilemma was to try to elicit reports of instantaneous oculocentric heading by other means. When instructed to base heading judgments on the illusory motion of the fixation point, observers are more accurate than with path judgments of the same displays (van den Berg, 1996). This suggests that the visual system implicitly recovers heading during simulated rotation. When asked to judge the direction in which they are skidding while traveling on a circular path, observers’ heading judgments also improve (Stone and Perrone, 1997), indicating that the instantaneous heading direction can be estimated. Another response to the dilemma is to investigate the conditions for accurate path perception, since that is what observers tend to report.

The path problem Observers can judge their prospective path of self-motion very accurately on a straight path, a circular path (Warren et al., 1991b), and even an elliptical path whose curvature changes with time (Kim and Turvey, 1998) from displays that simulate the view through the windshield of a turning car.3 A circular path can be parameterized by the instantaneous heading direction (Tˆ ), which is tangent to the path at the observation point, and the curvature of the path (k = 1/r). Path curvature is equal to the rate of change in heading with respect to distance traveled (k = dTˆ/ds), or with respect to time, scaled by translation speed (k = (dTˆ/dt)/T ). A higherorder path can be considered piecewise as a series of osculating circles, each of which is tangent to the path and defines a local path curvature. We do not know how the visual system recovers curved paths of self-motion. The most straightforward solution is to determine the current body-centric heading and its rate of change, as just described. But contrary to this approach, path perception is quite accurate for windshield-like displays in which the body-centric heading doesn’t change. This suggests a second solution: recovering the circular path directly from the flow field. A circular path on a ground plane generates a curved velocity field (Fig. 84.2C) that is stationary over time and uniquely corresponds to path curvature (for a constant eye height). The locomotor flow line that passes under the observer’s feet specifies the oculocentric path (Lee and Lishman, 1977), and its location with respect to objects in the scene specifies the object-relative path. For higher-order paths, the flow field curvature changes over time with the path curvature. A third solution is to determine curvature from the ratio of rotation to translation speeds (k = Rp/T ), where Rp is the rotation attributed to path curvature after compensating for eye and head rotation (Ehrlich et al., 1998). It is possible that spiral cells in MSTd code the path curvature from ground plane flow. It does not appear that extended paths are recovered by individual cells, for they respond similarly to a given flow pattern regardless of the one that precedes or follows it in a continuous sequence of stimulation (Paolini et al., 2000). Thus, single MSTd cells do not code for second-order temporal properties such as acceleration or changes in the flow pattern, but they could code the instantaneous path curvature. R  P P How, then, might one determine whether the path of self-motion is straight or curved from retinal flow alone? Li and Warren (2000) pro-

2

A rotation rate of 1 deg/sec would be produced by fixating the ground plane 10 m ahead while walking at 1 m/sec and a 5 deg/sec rotation by fixating 4 m ahead.

1254

, ,   

3

Specifically, the line of sight is along the tangent to the path, such that the instantaneous heading remains fixed on the screen.

indicate that eye rotation is zero; hence, the rotational component of flow must be due to a curved path. But over time, the retinal flow specifies that object-relative heading is constant, so the observer must be traveling on a straight path through the depicted environment. If the visual system makes use of both retinal and extraretinal signals, it is not surprising that the data are inconsistent. But with distinct reference objects in the display, object-relative paths tend to dominate.

heading A

difference vectors (T)

x B

heading fixation

retinal flow (T+R)

A

x

B

F 84.4. Object-relative heading. Bottom: Sketch of retinal flow in a complex scene. The observer is heading toward x while rotating rightward to track o. Top: Difference vectors from motion parallax specify the instantaneous heading. The current objectrelative heading is a few degrees to the left of object B. If the path is straight, the heading will remain fixed in the scene over time; if the path is curved to the right, the heading point will shift rightward with respect to object B.

posed that the visual system updates the object-relative heading over time. Specifically, the visual system continually extracts the instantaneous heading with respect to objects in the scene, presumably on the basis of motion parallax (Fig. 84.4). The sequence of such headings over time traces out the observer’s path through the environment. For example, if the path is straight, then the heading remains fixed in the scene, whereas if the path is curved, the heading shifts relative to objects over time. This solution to the path problem requires sufficiently dense motion parallax for an accurate heading estimate and distinct reference objects in the scene that can be tracked over time. However, most previous research used random dots, which are necessarily sparse and not easily tracked. In contrast, when displays contain dense texture-mapped scenes with distinct objects, path judgments are reasonably accurate, with mean errors below 4 degrees at simulated rotation rates up to 7 deg/sec (Cutting et al., 1997; Li and Warren, 2000). This is also the case for active steering during simulated rotation (Li and Warren, 2002). Object-relative heading thus allows the visual system to determine the path of self-motion through the environment on the basis of retinal flow, even when it is in conflict with extraretinal signals. The mixed results for heading judgments are probably a consequence of the cue conflict for path perception in the simulated rotation condition. Extraretinal signals

R  E S The preceding evidence indicates that both retinal flow and extraretinal signals can contribute to heading and path perception. This raises the question of how retinal and extraretinal signals are related. Recent results suggest that an extraretinal estimate of rotation is not simply subtracted from retinal flow; rather, the two interact nonlinearly (Crowell and Andersen, 2001; van den Berg et al., 2001). When real and simulated pursuit are in the same direction, extraretinal signals exert a partial influence, but when they are in opposite directions, there is little extraretinal influence. Moreover, if the retinal flow corresponds to a 3D scene (i.e., contains motion parallax or perspective), eye rotation is determined from the common lamellar flow, providing accurate pursuit compensation. Extraretinal signals merely indicate whether or not the eye is rotating, and gate the interpretation of lamellar flow as being due to a pursuit rotation or a curved path. This is consistent with the transparent motion illusion of Duffy and Wurtz (1993), which suggests that lamellar flow is used to estimate rotation when motion parallax is present. On the other hand, if the retinal flow is planar (not 3D), the rotation rate is quantitatively estimated from extraretinal signals, but with a gain of only 50%. This is consistent with the partial compensation for eye movements observed in MSTd cells with planar flow patterns (Bradley et al., 1996). Furthermore, it appears that extraretinal signals begin to make a contribution half a second after the onset of flow (Grigo and Lappe, 1999). With simulated rotation displays shorter than 500 msec, heading toward a large frontal plane (Fig. 84.3) is judged accurately based on minimal motion parallax, whereas with longer displays heading is perceived erroneously at the fixation point. Given that fixations normally last for 300 to 500 msec, the results imply that under ordinary conditions the visual system may rely primarily on retinal flow. Two neural models of MSTd incorporate extraretinal influences. Beintema and van den Berg (1998) assume flow templates with Gaussian tuning for combinations of radial and lamellar flow, similar to those of Perrone and Stone (1994). The model uses an extraretinal velocity signal to modulate the gain of a subset of units tuned to the corresponding rotation rate. Lappe (1998) assumes flow templates with sigmoidal tuning and uses an extraretinal velocity signal

:  

1255

Object

-

Retinal position

Co Retinal flow

Motion parallax

dt

-

To

+

path

Lamellar motion Extra-retinal signals

velocity position

Rp

R gate

Object-relative heading

Tr

eye

Co

+

dt

Cr

Object-relative path

Tb

Absolute heading

Cb

dt

Absolute path

-

Re

F 84.5. A heuristic summary of the flow of information in the visual system for recovering heading and path of self-motion. T, observer translation; R, total rotation; Re, rotation attributed to eye and head pursuit; Rp, rotation attributed to path curvature; C, circular path. Subscripts o, b, and r indicate quantities in oculo-centric, body-centric, and object-relative reference frames, respectively.

to shift the tuning of individual cells so that the population response compensates fully for rotation. Given that a majority of expansion cells appear to have Gaussian-like tuning, and some evidence of gain modulation in MSTd, the first model would seem to be more plausible; on the other hand, the second model captures the observation that partial shifts in individual cells can yield full compensation in a population vector. However, both models assume that extraretinal signals provide an accurate measure of rotation rate, rather than gating the interpretation of retinal flow, and neither incorporates object-relative heading. Given the current state of knowledge, one must remain agnostic about such models, but they can help to guide further tests of MSTd. A heuristic interpretation of the flow of information in the visual system, based on the results we have surveyed, appears in Figure 84.5. The lower part of the diagram illustrates the recovery of absolute heading and the upper part the recovery of object-relative heading. The evidence is generally consistent with the view that oculocentric translation (To) is determined from motion parallax and total rotation (R) from lamellar flow, either via parallel MT-MSTd pathways for differential and common motion or via compound templates in MSTd. The role of extraretinal velocity signals is largely limited to gating the total rotation R to eye and head pursuit or to path curvature (Rp), from which the instantaneous circular path (Co) is determined. In the former case, they may provide a quantitative estimate of eye and head rotation (Re), which is subtracted from R to estimate Rp when motion parallax is absent or fixation exceeds a half-second. To recover the absolute heading and path, extraretinal position information is needed to convert from oculocentric to bodycentric frames. In contrast, objectrelative heading (Tr) is determined by merging the oculocentric heading (To) with an object’s retinal position, bypassing the coordinate transformation. Similarly, to recover the object-relative path (Cr), the path determined from the curved flow field (Co) is merged

1256

, ,   

with the object’s position, or object-relative heading may be updated over time. Normally, the absolute and relative solutions are congruent, but during simulated rotation they are in conflict, and the object-relative solution tends to dominate.

Controlling locomotion with optic flow We have now established that humans can perceive selfmotion from optic flow under a variety of conditions and that specialized neural pathways exist to extract this information. Despite these findings, it is not a foregone conclusion that optic flow is actually used to control locomotor behavior, for other strategies are also available. To locomote toward a stationary goal, an obvious strategy is simply to walk in the egocentric direction of the target, without using optic flow at all. For example, one could center the goal at the midline of the body and walk forward. Alternatively, optic flow from the scene could be used to steer toward a goal, using relations that are all within the visual domain. For example, one could walk so as to cancel the error between the FOE and the goal (Gibson, 1950) or between the perceived heading and the goal. These hypotheses can be dissociated by using wedge prisms, which deflect the visual direction of the FOE from the direction of walking (Rushton et al., 1998). Thus, if participants guide walking by placing the deflecting FOE on the deflecting target, the “virtual” heading error between the FOE and the target will still be canceled, leading them to walk a straight path to the goal. But if they walk in the direction of the displaced image of the target, the virtual heading error will equal the prism deflection, and they will trace out a curved path to the goal. This is precisely what is observed with prisms in an open field, where there is minimal optic flow from fine grass texture, demonstrating that people can use egocentric direction to guide walking. On the other hand, we used a virtual reality lab to displace the FOE from

the direction of walking in a similar manner and manipulated the textured surface area in the display (Warren et al., 2001). In this setup, paths are significantly straighter and virtual heading error is reduced to near zero as optic flow and motion parallax are added to the display, consistent with an optic flow strategy. Similar improvements have also been reported with prisms as visual structure is added (Harris and Carre, 2001; Wood et al., 2000). Thus, both optic flow and egocentric direction contribute to the visual control of walking. Such redundancy provides robust locomotor control under a variety of environmental conditions. In contrast, when walking to intercept a moving target, people rely solely on the visual direction of the target. Specifically, they walk with a direction and speed that keeps the bearing angle of the target constant, much like a sailor colliding with another boat (Fajen and Warren, 2002). This makes sense because radial flow from the whole scene is no longer informative about one’s path toward a moving target. We recently developed a dynamic theory of steering and obstacle avoidance that is based on object-relative heading with respect to goals and obstacles (Fajen et al., in press). Specifically, a goal acts like an attractor of heading, whose strength increases with its angle from the current heading and decreases with distance. In contrast, obstacles act like repellors of heading, whose strength decreases with both angle and distance. The resultant of these forces determines the current walking direction at a constant speed. The model closely fits human behavior for walking toward a goal, detouring around an obstacle, intercepting a moving target, avoiding a moving obstacle, and even predicts routes through an array of obstacles. In principle, such a model could be extended to include interactions among these basic behaviors, accounting for paths of locomotion in a complex dynamic environment.

Conclusion We have followed optic flow through the cycle of perception and action, describing how information is generated by motion through the environment, detected to determine selfmotion, and used to control behavior. Yet important questions remain unanswered. How, exactly, are translation and rotation determined from motion parallax and lamellar flow—in parallel pathways or compound templates? Can these processes account for the way moving objects bias perceived heading? How are curved paths of self-motion recovered and coded in this pathway? Precisely how do retinal and extraretinal signals interact? Are there distinct cortical representations of oculo-, head-, and body-centric heading or of object-relative heading? Is locomotion guided by perceived heading per se or by some other aspect of optic flow? And how is flow used in concert with proprioceptive and vestibular information to control locomotor behavior?

REFERENCES Banks, M. S., S. M. Ehrlich, B. T. Backus, and J. A. Crowell, 1996. Estimating heading during real and simulated eye movements, Vis. Res., 36:431–443. Beintema, J. A., and A. V. van den Berg, 1998. Heading detection using motion templates and eye velocity gain fields, Vis. Res., 38:2155–2179. Born, R. T., and R. B. H. Tootell, 1992. Segregation of global and local motion processing in primate middle temporal visual area, Nature, 357:497–499. Bradley, D. C., M. Maxwell, R. A. Andersen, M. S. Banks, and K. V. Shenoy, 1996. Mechanisms of heading perception in primate visual cortex, Science, 273:1544–1547. Britten, K. H., 1998. Clustering of response selectivity in the medial superior temporal area of extrastriate cortex of the macaque monkey, Vis. Neurosci., 15:553–558. Britten, K. H., and R. J. A. von Wezel, 1998. Electrical microstimulation of cortical area MST biases heading perception in monkeys, Nat. Neurosci., 1:59–63. Burr, D. C., M. C. Morrone, and L. M. Vaina, 1998. Large receptive fields for optic flow detection in humans, Vis. Res., 38:1731– 1743. Crowell, J. A., and R. A. Andersen, 2001. Pursuit compensation during self-motion, Perception, 30:1465–1488. Cutting, J. E., K. Springer, P. A. Braren, and S. H. Johnson, 1992. Wayfinding on foot from information in retinal, not optical, flow, J. Exp. Psychol. Gen., 121:41–72. Cutting, J. E., P. M. Vishton, M. Flückiger, B. Baumberger, and J. D. Gerndt, 1997. Heading and path information from retinal flow in naturalistic environments, Percept. Psychophys., 59:426–441. Duffy, C. J., 1998. MST neurons respond to optic flow and translational movement, J. Neurophysiol., 80:1816–1827. Duffy, C. J., and R. H. Wurtz, 1991a. Sensitivity of MST neurons to optic flow stimuli. I. A continuum of response selectivity to large-field stimuli, J. Neurophysiol., 65:1329–1345. Duffy, C. J., and R. H. Wurtz, 1991b. Sensitivity of MST neurons to optic flow stimuli. II. Mechanisms of response selectivity revealed by small-field stimuli, J. Neurophysiol., 65:1346–1359. Duffy, C. J., and R. H. Wurtz, 1993. An illusory transformation of optic flow fields, Vis. Res., 33:1481–1490. Duffy, C. J., and R. H. Wurtz, 1995. Response of monkey MST neurons to optic flow stimuli with shifted centers of motion, J. Neurosci., 15:5192–5208. Duffy, C. J., and R. H. Wurtz, 1997. Medial superior temporal area neurons respond to speed patterns in optic flow, J. Neurosci., 17:2839–2851. Dyre, B. P., and G. J. Andersen, 1996. Image velocity magnitudes and perception of heading, J. Exp. Psychol. Hum. Percept. Perform., 23:546–565. Ehrlich, S. M., D. M. Beck, J. A. Crowell, T. C. A. Freeman, and M. S. Banks, 1998. Depth information and perceived self-motion during simulated gaze rotations, Vis. Res., 38:3129–3145. Fajen, B. R., and W. H. Warren, 2002. Intercepting a moving target on foot. Submitted for publication. Fajen, B. R., and W. H. Warren, 2003. Behavioral dynamics of steering, obstacle avoidance, and route selection, J. Exp. Psychol. Hum. Percept. Perform., 29:343–362. Geesaman, B. J., and R. A. Andersen, 1996. The analysis of complex motion patterns by form/cue invariant MSTd neurons, J. Neurosci., 16:4716–4732. Gibson, J. J., 1950. Perception of the Visual World, Boston: Houghton Mifflin.

:  

1257

Gibson, J. J., 1979. The Ecological Approach to Visual Perception, Boston: Houghton Mifflin. Graziano, M. S. A., R. A. Andersen, and R. J. Snowden, 1994. Tuning of MST neurons to spiral motions, J. Neurosci., 14:54– 67. Grigo, A., and M. Lappe, 1999. Dynamical use of different sources of information in heading judgments from retinal flow, J. Opt. Soc. Am. A, 16:2079–2091. Harris, M. G., and G. Carre, 2001. Is optic flow used to guide walking while wearing a displacing prism? Perception, 30:811–818. Hatsopoulos, N. G., and W. H. Warren, 1991. Visual navigation with a neural network, Neural Networks, 4:303–317. Hildreth, E., 1992. Recovering heading for visually-guided navigation, Vis. Res., 32:1177–1192. Hildreth, E. C., and C. S. Royden, 1998. Computing observer motion from optical flow, in High-Level Motion Processing (T. Watanabe ed.), Cambridge, MA: MIT Press, 269–293. Kappers, A. M. L., S. F. te Pas, and J. J. Koenderink, 1996. Detection of divergence in optical flow fields, J. Opt. Soc. Am. A, 13:227–235. Kim, N.-G., and M. T. Turvey, 1998. Visually perceiving heading on circular and elliptical paths, J. Exp. Psychol. Hum. Percept. Perform., 24:1690–1704. Koenderink, J. J., 1986. Optic flow, Vis. Res., 26:161–180. Koenderink, J. J., and A. J. van Doorn, 1975. Invariant properties of the motion parallax field due to the movement of rigid bodies relative to an observer, Opt. Acta, 22:737–791. Koenderink, J. J., and A. J. van Doorn, 1981. Exterospecific component of the motion parallax field, J. Opt. Soc. Am., 71:953–957. Koenderink, J. J., and A. J. van Doorn, 1987. Facts on optic flow, Biol. Cybern., 56:247–254. Lagae, L., H. Maes, S. Raiguel, D.-K. Xiao, and G. A. Orban, 1994. Responses of macaque STS neurons to optic flow components: a comparison of areas MT and MST, J. Neurophysiol., 71:1597–1626. Lappe, M., 1998. A model of the combination of optic flow and extraretinal eye movement signals in primate extrastriate visual cortex: neural model of self-motion from optic flow and extraretinal cues, Neural Networks, 11:397–414. Lappe, M., F. Bremmer, and A. V. van den Berg, 1999. Perception of self-motion from visual flow, Trends Cogn. Sci., 3:329–336. Lappe, M., and C. J. Duffy, 1999. Optic flow illusion and single neuron behaviour reconciled by a population model, Eur. J. Neurosci., 11:2323–2331. Lappe, M., and J. P. Rauschecker, 1993. A neural network for the processing of optic flow from ego-motion in man and higher mammals, Neural Comput., 5:374–391. Lee, D. N., and R. Lishman, 1977. Visual control of locomotion, Scand. J. Psychol., 18:224–230. Li, L., and W. H. Warren, 2000. Perception of heading during rotation: sufficiency of dense motion parallax and reference objects, Vis. Res., 40:3873–3894. Li, L., and W. H. Warren, 2002. Retinal flow is sufficient for steering during simulated rotation, Psychol. Sci., 13:485–491. Longuet-Higgins, H. C., and K. Prazdny, 1980. The interpretation of a moving retinal image, Proc. R. Soc. Lond. B, 208:385–397. Morrone, M. C., M. Tosetti, D. Montanaro, A. Fiorentini, G. Cioni, and D. C. Burr, 2000. A cortical area that responds specifically to optic flow, revealed by fMRI, Nat. Neurosci., 3: 1322–1328. Orban, G. A., L. Lagae, S. Raiguel, D. Xiao, and H. Maes, 1995. The speed tuning of middle superior temporal (MST) cell responses to optic flow components, Perception, 24:269–285.

1258

, ,   

Orban, G. A., L. Lagae, A. Verri, S. Raiguel, D. Xiao, H. Maes, and V. Torre, 1992. First-order analysis of optical flow in monkey brain, Proc. Natl. Acad. Sci. USA, 89:2595–2599. Page, W. K., and C. J. Duffy, 1999. MST neuronal responses to heading direction during pursuit eye movements, J. Neurophysiol., 81:596–610. Paolini, M., C. Distler, F. Bremmer, M. Lappe, and K. P. Hoffman, 2000. Responses to continuously changing optic flow in area MST, J. Neurophysiol., 84:730–743. Perrone, J. A., 1992. Model for the computation of self-motion in biological systems, J. Opt. Soc. Am. A, 9:177–194. Perrone, J. A., and L. S. Stone, 1994. A model of self-motion estimation within primate extrastriate visual cortex, Vis. Res., 34:2917–2938. Perrone, J. A., and L. S. Stone, 1998. Emulating the visual receptive-field properties of MST neurons with a template model of heading estimation, J. Neurosci., 18:5958–5975. Peuskens, H., S. Sunaert, P. Dupont, P. V. Hecke, and G. A. Orban, 2001. Human brain regions involved in heading estimation, J. Neurosci., 21:2451–2461. Raiguel, S., M. M. van Hulle, D.-K. Xiao, V. L. Marcar, L. Lagae, and G. A. Orban, 1997. Size and shape of receptive fields in the medial superior temporal area (MST) of the macaque, NeuroReport, 8:2803–2808. Read, H. L., and R. M. Siegel, 1997. Modulation of responses to optic flow in area 7a by retinotopic and oculomotor cues in monkey, Cereb. Cortex, 7:647–661. Regan, D., and K. I. Beverly, 1982. How do we avoid confounding the direction we are looking and the direction we are moving? Science, 215:194–196. Rieger, J. H., and D. T. Lawton, 1985. Processing differential image motion, J. Opt. Soc. Am. A, 2:354–360. Royden, C. S., 1994. Analysis of misperceived observer motion during simulated eye rotations, Vis. Res., 34:3215–3222. Royden, C. S., 1997. Mathematical analysis of motion-opponent mechanisms used in the determination of heading and depth, J. Opt. Soc. Am. A, 14:2128–2143. Royden, C. S., J. A. Crowell, and M. S. Banks, 1994. Estimating heading during eye movements, Vis. Res., 34:3197–3214. Royden, C. S., and E. C. Hildreth, 1996. Human heading judgments in the presence of moving objects, Percept. Psychophys., 58:836–856. Rushton, S. K., J. M. Harris, M. Lloyd, and J. P. Wann, 1998. Guidance of locomotion on foot uses perceived target location rather than optic flow, Curr. Biol., 8:1191–1194. Saito, H., M. Yukie, K. Tanaka, K. Hikosaka, Y. Fukada, and E. Iwai, 1986. Integration of direction signals of image motion in the superior temporal sulcus of the macaque monkey, J. Neurosci., 6:145–157. Shenoy, K. V., D. C. Bradley, and R. A. Andersen, 1999. Influence of gaze rotation on the visual response of primate MSTd neurons, J. Neurophysiol., 81:2764–2786. Siegel, R. M., and H. L. Read, 1997. Analysis of optic flow in the monkey parietal area 7a, Cereb. Cortex, 7:327–346. Stone, L. S., and J. A. Perrone, 1997. Human heading estimation during visually simulated curvilinear motion, Vis. Res., 37:573–590. Tanaka, K., Y. Fukada, and H. Saito, 1989. Underlying mechanisms of the response specificity of expansion/contraction and rotation cells in the dorsal part of the medial superior temporal area of the macaque monkey, J. Neurophysiol., 62:642–656. Tanaka, K., and H. Saito, 1989. Analysis of motion of the visual field by direction, expansion/contraction, and rotation cells clus-

tered in the dorsal part of the medial superior temporal area of the macaque monkey, J. Neurophysiol., 62:626–641. Todd, J. T., 1995. The visual perception of three-dimensional structure from motion, in Perception of Space and Motion, 2nd ed. (W. Epstein and S. Rogers, eds.), San Diego, CA: Academic Press, pp. 201–226. Upadhyay, U. D., W. K. Page, and C. J. Duffy, 2000. MST responses to pursuit across optic flow with motion parallax, J. Neurophysiol., 84:818–826. van den Berg, A. V., 1993. Perception of heading, Nature, 365: 497–498. van den Berg, A. V., 1996. Judgements of heading, Vis. Res., 36: 2337–2350. van den Berg, A. V., J. A. Beintema, and M. A. Frens, 2001. Heading and path percepts from visual flow and eye pursuit signals, Vis. Res., 41:3467–3486. van den Berg, A. V., and E. Brenner, 1994. Why two eyes are better than one for judgments of heading, Nature, 371:700–702. Wang, R. F., and J. E. Cutting, 1999. Where we go with a little good information, Psychol. Sci., 10:71–75. Warren, R., 1976. The perception of egomotion, J. Exp. Psychol. Hum. Percept. Perform., 2:448–456. Warren, W. H., 1998. The state of flow, in High-Level Motion Processing (T. Watanabe ed.), Cambridge, MA: MIT Press, pp. 315–358.

Warren, W. H., A. W. Blackwell, K. J. Kurtz, N. G. Hatsopoulos, and M. L. Kalish, 1991a. On the sufficiency of the velocity field for perception of heading, Biol. Cybern., 65:311–320. Warren, W. H., and D. J. Hannon, 1988. Direction of self-motion is perceived from optical flow, Nature, 336:162–163. Warren, W. H., and D. J. Hannon, 1990. Eye movements and optical flow, J. Opt. Soc. Am. A, 7:160–169. Warren, W. H., B. A. Kay, W. D. Zosh, A. P. Duchon, and S. Sahuc, 2001. Optic flow is used to control human walking, Nat. Neurosci., 4:213–216. Warren, W. H., D. R. Mestre, A. W. Blackwell, and M. W. Morris, 1991b. Perception of circular heading from optical flow, J. Exp. Psychol. Hum. Percept. Perform., 17:28–43. Warren, W. H., M. W. Morris, and M. Kalish, 1988. Perception of translational heading from optical flow, J. Exp. Psychol. Hum. Percept. Perform., 14:646–660. Warren, W. H., and J. A. Saunders, 1995. Perception of heading in the presence of moving objects, Perception, 24:315–331. Wood, R. M., M. A. Harvey, C. E. Young, A. Beedie, and T. Wilson, 2000. Weighting to go with the flow? Curr. Biol., 10:R545R546. Xiao, D.-K., S. Raiguel, V. Marcar, and G. A. Orban, 1997. The spatial distribution of the antagonistic surround of MT/V5 neurons, Cereb. Cortex, 7:662–677.

:  

1259

85

The Cortical Analysis of Optic Flow CHARLES J. DUFFY

G’ (1950) description of optic flow shifted visual motion studies from the perspective of a stationary observer viewing moving objects to that of a moving observer navigating past stationary objects. He emphasized that optic flow is an important cue to heading direction and the threedimensional layout of the visual environment. Optic flow is analyzed in the context of other selfmovement cues. Proprioceptive and vestibular sensations accompany all self-movement. Kineoceptive sensation and motor signals emerge during active self-movement. These cues are always available, but people moving with their eyes open in a lighted environment are greatly guided by vision. Visual self-movement cues include optic flow and changes in the position, size, and shape of objects. There is no absolute boundary between optic flow and object motion, but the distinction highlights their interdependence: optic flow analysis can proceed without the burden of recognizing all of the objects in the flow field, and optic flow can assist in object recognition when objects create inconsistencies in the flow field. This review of the cortical analysis of optic flow focuses on single-neuron studies in monkeys. Relevant studies of cat and human cortex are included for comparison but are not represented comprehensively. Behavioral and psychophysical studies are treated separately in this volume (Chapter 84). Informative studies of optic flow analysis in insects and birds (e.g., Krapp and Hengstenberg, 1996; Wylie et al., 1999) are omitted because their links to cortical motion processing are beyond the scope of this chapter.

Linking dorsal association cortex and spatial vision The visual functions of parietotemporal cortex were revealed in electrical stimulation and lesion studies (Ferrier, 1876). These findings were overshadowed by the mapping of occipital visuotopy (Holmes, 1918; Inouye, 1909). Interest in extrastriate vision was sustained by experimental evidence of visual function after striate removal (Kluver, 1936) and by clinical evidence of differential parietal and temporal involvement in spatial and object vision (Kleist, 1935). Updated approaches revealed visual evoked electrical activity in extrastriate cortex (Doty et al., 1964) and striate projections into these areas (Cragg, 1969; Zeki, 1969). Lesion analysis put these findings in the context of dorsal extrastriate specialization for spatial perception and con-

1260

trasted it with ventral extrastriate specialization for object identification (Ungerleider and Brody, 1977; Ungerleider and Mishkin, 1982). Lesion studies suggested that the colliculo-pulvino-parietal visual pathway (Humphrey and Weiskrantz, 1967; Schneider, 1969) supports behavioral responses to visual motion (Doty, 1973; Pasik et al., 1969). Thus, converging dorsal geniculostriate and colliculopulvino-parietal pathways combine to serve vision for spatial orientation (Trevarthan, 1968). The behavioral relevance of this system was seen in patients with striate lesions causing blindsight (Sanders et al., 1974) and extrastriate lesions causing spatial deficits (Botez, 1975). Posterior parietal neurons were found to combine visuosensory and visuomotor signals in a manner that suggested involvement in extrapersonal space perception (Mountcastle, 1976). These findings linked visual motion processing, visuospatial perception, and dorsal extrastriate cortex in a way that provided the theoretical foundation of efforts to understand the cortical analysis of optic flow.

Input to dorsal extrastriate areas Converging projections from striate cortex to the posterior bank of the superior temporal sulcus (STS) are seen by anterograde fiber degeneration (Garey, 1968; Spatz et al., 1970; Zeki, 1971) and radiolabeled amino acid transport (Ungerleider and Mishkin, 1979). Combined anatomical and physiological studies identify a densely myelinated projection from striate cortex to the STS’s middle temporal area (MT) with a distinct boundary at its lateral border with V4, an emphasis on movement responsiveness ( Van Essen et al., 1981), and a visuotopic map of the contralateral hemifield (Maunsell and Van Essen, 1987). This projection extends from MT on the posterior bank of the STS to the floor and anterior bank of the STS, with central visual field more prominent posteriorly and peripheral vision more prominent anteriorly (Ungerleider and Desimone, 1986b). MT is connected reciprocally to adjacent areas in the STS: dorsally to the medial superior temporal area (MST) and along the floor to the fundal superior temporal area (FST) (Desimone and Ungerleider, 1986; Kaas and Morel, 1993; Ungerleider and Desimone, 1986a). STS cortex also receives input from prestriate areas V2, V3, and V4 in the lunate sulcus (Baizer et al., 1991), projects ipsilaterally into posterior parietal cortex (Neal et al., 1988), and creates

patchy callosal connections ( Van Essen et al., 1981). The STS’s posterior parietal targets receive convergent somatosensory ( Jones and Powell, 1970), frontal, and limbic (Mesulam et al., 1977) input. Horse-radish peroxidase (HRP) retrograde tracing and anterograde radio-amino acid tracing show reciprocal connections between the STS and the medial pulvinar thalamus. There are additional projections from pulvinar oralis, suprageniculate, and limitans thalamic nuclei to the inferior parietal lobule (IPL) ( Yeterian and Pandya, 1985, 1989, 1991). These findings support the notion of a gradient of geniculostriate and colliculopulvinar projections into dorsal extrastriate cortex. The geniculostriate projections are more prominent in the STS, whereas the colliculopulvinar projections are more prominent in posterior parietal cortex, where they are joined by a wide variety of other reciprocal connections that create a dense network of convergence (Elston and Rosa, 1997). The net effect is a highly interconnected, distributed system for visuospatial processing in this region (Felleman and Van Essen, 1991; Lewis and Van Essen, 2000).

Posterior dorsal stream motion processing MT is the most posterior of the dorsal motion processing areas and the first motion-devoted area in that anteriorly directed stream. MT is roughly synonymous with the densely myelinated zone created by striate cortical projections to the posterior bank of the STS. Its neurons are selective for motion direction, but not object shape, with the columnar organization of direction preferences in a map of the contralateral visual field (Allman and Kaas, 1971; Dubner and Zeki, 1971; Zeki, 1974) MT has large receptive fields (~10 degrees square), with medial MT having larger peripheral receptive fields (MTp) and lateral MT having smaller foveal receptive fields (MTf ) (Ungerleider and Desimone, 1986a, 1986b). Direction, speed, and orientation tuning (Maunsell and Van Essen, 1983a) focus MT’s activity on combinations of those properties (Albright et al., 1984; Mikami et al., 1986). Some MT neurons have opposite preferred directions in different parts of their receptive fields that may interact with distance cues to serve figure-ground discrimination (Allman et al., 1985; Maunsell and Van Essen, 1983b). These neurons are clustered in columns (Born and Tootell, 1992), with some showing center-surround directional organization and others showing asymmetrical arrangements (Xiao et al., 1997). Antagonistic receptive field neurons project preferentially to dorsal MST (MSTd), whereas synergistic receptive fields project preferentially to lateral MST (MSTl) and FST (Berezovskii and Born, 2000). Anterior to MT is ventrolateral MST (MSTv or MSTl), with a mix of large and small receptive field neurons

(Komatsu and Wurtz, 1988a; Saito et al., 1986). An anterior subzone has small, peripheral receptive fields that do not include the fixation point (Tanaka et al., 1993). MSTl neurons show size- and speed-dependent direction selectivity. Small moving-dot patterns (40 degrees square) ( Yin and Mountcastle, 1977) that respond to moving stimuli regardless of their color, orientation, or shape (Robinson et al., 1978). These neurons are sensitive to the radial pattern of motion around the fixation point, preferring object motion either toward the fixation point or away from the fixation point. This opponent vector organization is accompanied by foveal sparing such that these neurons do not respond to stimuli at the fixation point (Motter and Mountcastle, 1981). Opponent vector neurons also possess

:     

1261

axial direction preferences and broadly tuned speed sensitivity, potentially enhancing their role in self-movement analysis (Motter et al., 1987; Steinmetz et al., 1987). Neurons within the STS were first approached by the gradual encroachment of recording studies, first from the more anterior area 7a and then from the more posterior area MT. Recordings of visual tracking neurons in area 7a extended on to the anterior bank of the STS and revealed a combination of visual and pursuit responses. These neurons were activated by apparent target movement induced by the opposite direction of surround movement (the Dunker illusion) (Sakata et al., 1978). The preferred pursuit direction of these neurons was either the same as or the opposite of their preferred visual motion direction, suggesting that they receive both retinal (visual) and extraretinal (ocular proprioceptive or oculomotor corollary) input (Sakata et al., 1983). This parietal transition zone in the STS also contains visual neurons that respond best to either object movement in the frontoparallel plane, object movement in depth (approaching or receding), or object rotation (clockwise or counterclockwise). This combination of response properties suggests that these neurons represent the complex movement of real objects in the environment (Sakata et al., 1985, 1986). The complex direction selectivities of anterior STS neurons focused interest on MSTd. MSTd neurons were divided into three categories: 30% preferred small-object motion stimuli, 30% preferred large-field pattern motion stimuli, and the remainder were equally responsive to both (Tanaka et al., 1986). The large-field neurons were further classified by their direction selectivities: direction-selective neurons (~65%) responded best to dot patterns moving in the frontoparallel plane. Size neurons (~20%) responded best when the boundaries of an object moved to simulate symmetrical expansion or contraction. Rotation neurons (~15%) responded best when the pattern rotated clockwise or counterclockwise. The size and rotation stimuli showed position invariance, preferring the same type of movement at different locations in the receptive field such that simple local directionality did not readily explain their responses. These properties defined the DSR (direction, size, rotation) region, which is probably the same as MSTd (Saito et al., 1986).

Optic flow responses in MSTd MSTd neuronal DSR responses were further characterized as preferring very large stimuli (circular diameters >40 degrees), with stronger responses to the movement of random-dot patterns than to small moving objects. Their direction selectivities are linked to the combination of local directionalities distributed throughout the stimuli, with less

1262

, ,   

critical contributions from the size and speed of visible elements in the pattern. These findings were the foundation of the notion that MSTd pattern motion responses were the product of an assembled pattern of MT-like local motion responses (Tanaka and Saito, 1989). MSTd neurons were subsequently described as showing a continuum of selectivities for large (90 degrees square), planar, circular, and radial patterns of optic flow. This continuum was divided into three groups based on the statistical significance of responses to each type of optic flow: single-component neurons responded to one type, occurring as planar (10%), radial (9%), and circular (4%) neurons. Double-component neurons responded to two types with planocircular (17%) and planoradial (17%) neurons. Triple-component neurons responded to all three types as planocirculoradial (29%) neurons (Duffy and Wurtz, 1991a). These classes of neurons differed with respect to the position invariance of their optic flow selectivities, that is, their ability to maintain the same optic flow preferences when tested with small optic flow stimuli presented at different locations in the visual field. Some neurons show the same optic flow preferences throughout their receptive fields, whereas others show different optic flow preferences at different sites (Duffy and Wurtz, 1991b; Graziano et al., 1994; Lappe, 1996; Fig. 85.1). The site of greatest responsiveness to various optic flow stimuli also differed across a neuron’s receptive field. One type of flow can have its greatest activation at one site, while another type of optic flow can have its greatest activation at a different site (Lagae et al., 1994). Triple-component neurons show the least position invariance with different optic flow preferences at different locations in their receptive fields, consistent with their receptive fields being composed of local, planar directional subfields. In contrast, single-component neurons show the greatest position invariance with the same optic flow preferences at different locations in their receptive fields. Local directional subfields, or a mosaic of locally directional subfields, do not readily explain these responses. However, stronger inhibition in single-component neurons might resolve this conflict by suggesting that MSTd receptive fields are composed of partially overlapping, excitatory and inhibitory, planar directional subfields. From this perspective, stronger inhibitory subfields create the more selective singlecomponent neurons, whereas weaker inhibitory subfields create the less selective triple-component neurons (Duffy and Wurtz, 1991b). STS neurons were characterized by Orban et al. (1992) according to their selectivities for expansion/contraction, rotation, and deformation created by directional shear, noting that many were also responsive to planar motion. Neurons in areas MT and MST all responded to com-

Heading selectivity in MSTd

A

B

F 85.1. The complex receptive field organization of an MSTd neuron. Directional responses were tested at 25 positions across the central 100 degrees square of the visual field. A, Color contour map of the amplitude of the strongest responses across sites showing great excitatory effects at two separate sites in the right hemifield. Insets show maps of responses to the eight different directions tested. B, Vector plots of net directions of responses across sites in the same neuron showing a variable pattern of rightward motion selectivity. (From Logan and Duffy, 1998.) (See color Plate 72)

binations of planar motion and optic flow. However, tests of position invariance showed that MT neurons were all position sensitive in a manner consistent with simple planar directional receptive fields, in this regard, like the triple component neurons of adjacent area MSTd (Lagae et al., 1994).

MSTd receptive field sizes average 33 degrees square, with substantial extension into the ipsilateral hemifield and no clear relationship between receptive field size and eccentricity (Raiguel et al., 1997). These large receptive fields are well suited to the task of estimating the heading of selfmovement based on the large patterns of optic flow. Evidence for heading-selective neuronal activation in MSTd was obtained by comparing responses to various optic flow stimuli. These neurons were tested with optic flow simulating nine directions of forward self-movement as outward radial motion with different foci of expansion (FOEs). MSTd neurons respond selectively to particular FOE stimuli. The mapping of heading preferences revealed three patterns: (1) preferences for peripheral FOEs in one radial segment of the visual field, (2) preferences for FOEs at or near the fixation point, and (3) preferences for FOEs around, but not at, the fixation point. These FOE preferences might detect eccentric headings, gaze-aligned headings, and subtle deviations from gaze-aligned headings, respectively (Fig. 85.2; Duffy and Wurtz, 1995). MSTd’s heading selectivity has been thought to depend on local directional subfields and to match the predictions of neural network models of MST based on MT-like small, planar-directional receptive fields. These projections provide a weighted sampling of the optic flow field arranged to implement an algorithmic solution to heading determination (Heeger and Jepson, 1990). The MST-like layer shows sigmoidal relationships between FOE location and response amplitude that might support a population representation of heading (Lappe and Rauschecker, 1993). An alternative neural network model of MST heading detection relies on directional template matching (Perrone and Stone, 1994). In this model, MT-like direction- and speed-tuned visual field subunits create a mosaic of sensors across the visual field. A subset of these direction-speed sensors project to each MST-like neuron to impart stimulus selectivity for optic flow. This model’s units showed Gaussian distributions of FOE selectivity and replicated optic flow position invariance (Perrone and Stone, 1998). A three-layered neural network might parse observer and object motion (Zemel and Sejnowski, 1995). The model can train on optic flow to modify the weights of network connections (Zhang et al., 1993) and implement an algorithmic approach to developing optic flow-selective unit responses (Nagel, 1987). The input layer has MT-like local directional units that project to an MST-like middle layer that then projects to an output layer for heading representation. This model responds to optic flow stimuli from observer movement and to small patterns created by object motion (Zemel et al., 1998).

:     

1263

F 85.2. The continuum of optic flow selectivity in MSTd neurons. Icons along the top indicate the patterns of dot motion in the twelve 100 degree square optic flow stimuli used in this study. A–C, Spike density histograms of the firing rates of three neurons during the presentation of the optic flow stimuli. The vertical bar indicates the 75 spikes per second firing rate; horizontal bars indicate the 1 second stimulus duration. A, The responses of a single-

component radial neuron preferring outward radial optic flow. B, The responses of a double-component planoradial neuron preferring leftward planar and outward radial optic flow. C, The responses of a triple-component planocirculoradial neuron preferring rightward planar, clockwise circular, and outward radial optic flow. (From Duffy and Wurtz, 1995.)

Another model might be derived by extending the overlapping gradient’s view of MSTd’s optic flow responses (Duffy and Wurtz, 1991b) to create a hierarchical processing model. Same-direction MT units covering a large part of the visual field could converge to create positiondependent triple-component neurons. Two triple-component neurons with different location and direction preferences could combine as the excitatory and inhibitory inputs to double-component planocircular or planoradial neurons with planar responses from the excitatory subfield and their radial or circular selectivity from directional inhibition. Adding another inhibitory directional input could take advantage of the orthogonality of circular and radial motion to create single-component responses.

nization has been found; a clustering of similar optic flow selectivities has been observed in some penetrations but not others (Duffy and Wurtz, 1991b). Such inconsistency might reflect the different angles at which penetrations can cross MSTd along the curved path of the STS. Accounting for this curvature reveals some tendency for alternating clusters of expansion and rotation neurons across cortex (Lagae et al., 1994). Functional clustering reinforces this conclusion, with nearby neurons having more similar properties and distant neurons having dissimilar properties (Britten and van Wezel, 1998). Radioactive isotope labeling reveals the distribution of expansion and rotation responsiveness. H3- and C14-glucose administered during visual stimulation reveals an interdigitation of expansion-and rotation-dominated columns on the posterior bank and floor of the STS in the region of areas MT and MSTl. This interdigitation does not extend into MSTd on the anterior bank of the STS. It is not clear if the interdigitated labeling in MT and MSTl represents selective responsiveness to the patterns of optic flow or to the local planar motion components of the stimuli (Geesaman et al., 1997).

The functional organization of MSTd The functional organization of MSTd neuronal receptive fields has been explored by mapping direction selectivity across the central 100 degrees of the visual field. The resulting spatial-directional receptive field maps reveal several distinct subfields within each receptive field. The preferred directions of motion vary across subfields, with different subfields showing different proportions of excitatory and inhibitory responses. These subfields are differentially activated by different patterns of optic flow (Fig. 85.3; Logan and Duffy, 1998). The columnar organization of MSTd has been examined by comparing neuronal response properties recorded along microelectrode penetration tracks. No clear visuotopic orga-

1264

, ,   

MSTd neuronal response mechanisms Planar direction selectivity supports MSTd’s responses to radial, circular, and spiral stimuli. MSTd neuronal heading specificity can be mimicked by overlapping radial or circular optic flow with planar motion stimuli. In less selective triple-component planocirculoradial neurons, the underly-

A

A

B

B

F 85.3. Polar plots of directional responses of two optic flow neurons in MSTd. Arrows are scaled to the amplitude of the neuronal response to the corresponding optic flow stimulus. Responses to four planar motion stimuli are represented by the horizontal and vertical limbs, responses to inward and outward radial stimuli are represented by the oblique limbs, and responses to the clockwise and counterclockwise circular stimuli are represented by arcs. The large frames indicate the directional responses to 100 degree square

stimul; the small frames indicate the directional responses to the set of nine 33 degree square stimuli. A, Responses of a positiondependent triple-component neuron that showed substantially different optic flow selectivities at different locations. B, Responses of a position-invariant single-component inward radial-selective neuron that showed stable optic flow selectivities at different locations. (From Duffy and Wurtz, 1991b.)

ing planar response mechanism’s relationship to local motion in the pattern stimuli is obvious. In more selective single-component circular or radial neurons, the underlying planar response mechanism is obscured by inhibitory interactions (Fig. 85.4). This does not mean that MSTd encodes heading by decomposing optic flow into its planar and other components. It only shows that MSTd derives heading from the available neural mechanisms of planar directional, excitatory, and inhibitory responsiveness (Duffy and Wurtz, 1997c). The link between planar motion and optic flow responses in MSTd is supported by the form/cue invariance of their directional selectivity. Neuronal stimulus preferences evoked by random-dot optic flow are similar to the preferences evoked by a variety of discrete visual object motion stimuli. This similarity extends to preferences for radial, circular, and spiral stimuli but not to the strength of the responses or tuning width in that spiral stimulus space (Geesaman and Andersen, 1996). These results are consistent with a fundamental role for planar motion sensitivity: spiral stimulus space reflects local planar direction around 360 degrees, and both optic flow and moving objects present planar motion.

Further characterization of object motion responses in MSTd has focused on sensitivity to the direction and the three-dimensional orientation of the rotating object. These sensitivities are position invariant across changes in the object’s position in the visual field. Speed gradients in these stimuli are critical cues for deriving structure-from-motion percepts, and MSTd neuronal responses reflect sensitivity to the same stimulus parameters (Sugihara et al., 2002). These responses appear to be a consequence of MSTd’s direction and speed sensitivity, and obscure the functional distinction between MSTd and MSTl with respect to the analysis of self-movement and object motion. MSTd neuronal motion selectivity varies during prolonged stimulation. Phasic responses in the first 200 msec of stimulation are less selective than those seen thereafter, but they are not entirely nonselective, their amplitude differing substantially with different optic flow stimuli. Later, tonic responses are more selective for certain optic flow patterns, with the degree of selectivity varying across neurons and continued activity measured for up to 15 seconds of optic flow stimulation. Thus, these responses may continue to represent heading during sustained observer motion (Duffy and Wurtz, 1997b).

:     

1265

F 85.4. Patterns of FOE selectivity in three MSTd neurons. A, Arrows depict the patterns of dot motion in the 33 optic flow stimuli presented with FOEs at 10 degree intervals in the central visual field. B–D, Response amplitude indicated by the size of the oval at the position corresponding to that of the evoking

stimulus. B, A neuron that preferred down-left eccentric FOEs. C, A neuron that preferred central FOEs. D, A neuron that preferred central FOEs but not the centered FOE. (From Duffy and Wurtz, 1995.)

As with object motion responses, speed supplements direction as a primary stimulus parameter in MSTd’s responses to optic flow. Speed tuning for complex optic flow patterns and planar motion most commonly show optimal responses over a range of speeds from 5 to 50 deg/sec. Slower or faster stimuli evoke successively smaller responses, with individual neurons having idiosyncratic speed-response profiles. Speed and direction tuning interact to create composite responsiveness to particular stimuli, leading to the suggestion that MSTd combines these signals to encode optic flow (Orban et al., 1995). The notion that speed tuning might refine optic flow selectivity is supported by MSTd neuronal response sensitivity to the gradient of speeds in optic flow. Naturalistic optic flow contains speed gradients that reflect the three-dimensional layout of the visual environment, with distant features moving more slowly and nearer features more rapidly. MSTd neurons are sensitive to these speed gradients in optic

flow, so that some neurons respond best to stimuli simulating more distant features in the center of the visual field and nearer features in the periphery, as seen when moving down a corridor. Other neurons respond best to the reverse pattern, as seen when approaching an object in front of a distant background. MSTd neurons are also affected by the overlap of speed gradients that create motion parallax effects, with some neurons preferring greater simulated depth of field in optic flow stimuli (Duffy and Wurtz, 1997a). The great majority of MSTd neurons (89%) are sensitive to binocular disparity, with 46% preferring crossed disparities (near), 38% preferring uncrossed disparities (far), and a few being tuned to a particular disparity. Almost half (40%) of the neurons reverse their preferred direction from crossed to uncrossed disparities. This does not depend on vergence angle, so that disparity relative to the plane of fixation affects responses across a range of absolute distances. This might

1266

, ,   

contribute to foreground-background discrimination as a moving observer fixates on a stationary point (Roy et al., 1992). Together, these findings demonstrate that MSTd neurons are well suited to the analysis of optic flow during observer self-movement. Their response properties support sensitivity to the heading of self-movement, the structure of the environment, and the movement of discrete objects in the environment. Crossing the STS from MT to MSTl to MSTd, there is increasing complexity of visual motion responsiveness to both optic flow and object motion.

Temporal cortical responses to optic flow Neuronal responses to optic flow are seen in STS cortex anterior to MSTd. About 1 cm ventroanteriorly along the STS is the superior temporal polysensory area (STP). Neurons in STP are diverse: 40% are exclusively visual, ~20% also respond to auditory stimuli, ~20% also respond to somatosensory stimuli, and ~20% respond to all three modalities. Their visual receptive fields are large, often including the entire visual field. Most are insensitive to the shape of visual stimuli, but 30%, such as neurons in inferotemporal cortex, are shape selective and prefer faces or facial features. Most are direction selective, with some preferring movement in depth and others preferring centrifugal or centripetal movement relative to the fixation point, such as opponent vector organization in posterior parietal cortex (Bruce et al., 1981). The functional dichotomy of STP neuronal responsiveness extends to object motion responses that can be divided into two groups: those that are more selective for particular objects and those that are more selective for particular directions of movement. STP neurons that are more selective for particular objects respond only to the independent movement of those objects. The volitional movement of objects under the mechanical control of the animal’s hand evokes only weak responses (Hietanen and Perrett, 1993, 1996a). The visual cues that define the object are critical to these responses, with some responding only to luminance-defined objects and other responding only to motion-defined objects (Anderson and Siegel, 1998). STP neurons that are more selective for the direction of object motion most often prefer object motion along the cardinal axes of extrapersonal space: horizontal, vertical, and depth (Oram et al., 1993). These responses exhibit a bodyindependent motion discrimination such as the bodyindependent responses of object form neurons. That is, they respond to object movement when the animal is stationary, not when the animal is moving in a manner that might account for the apparent relative motion of the object (Hietanen and Perrett, 1996b).

The influence of relative movement on object responses suggests that these neurons might also respond to selfmovement cues in optic flow. Optic flow responses are observed in 65% of STP neurons, half responding nonselectively to a variety of optic flow stimuli and half responding selectively to certain optic flow stimuli. More than half of these neurons respond to patterned optic flow but not to planar translational motion, whereas the remainder prefer planar motion. Responses to centered FOE stimuli comprise the majority of optic flow responses, although they too are accompanied mainly by translational movement responses (Anderson and Siegel, 1999). Thus, STP neurons appear to intermix object form, object movement, and optic flow responses. The prevalence of radial expansion optic flow preferences, and the suppression of object motion responses by congruent background motion, suggest that this area may mediate the use of optic flow to detect the independent movement of animate objects seen during observer self-movement.

Parietal cortical responses to optic flow Neuronal responses to optic flow occur in several areas in posterior parietal cortex (PPC). Neurons in the transition zone between MSTd and area 7a, in the upper-anterior bank of the STS and in the adjacent convexity cortex, have large receptive fields, often more than 90 degrees in diameter. Some respond to object movement in the frontoparallel or depth planes (Sakata et al., 1985) and others are activated object rotation, with weaker responses to planar motion or shear (the combination of two locally opposite directions). Many neurons prefer rotation in the depth plane and some respond to Ames window illusory rotation, suggesting a highlevel link to object motion processing (Sakata et al., 1994). Object motion responses in area 7a show opponent vector organization with preferences for movement toward or away from the fixation point, as well as superimposed preferences for a particular motion direction (Motter et al., 1987; Steinmetz et al., 1987). However, this radial organization of direction selectivity does not predict their responses to optic flow. Their optic flow responses can be selective for particular directions of radial or circular motion, or they can respond nonselectively to multiple directions of radial or circular motion (Siegel and Read, 1997). In addition, these neurons are sensitive to the speed of motion in optic flow, with interactions between speed tuning and optic flow selectivity, so neurons that respond to more than one type of optic flow may have different preferred speeds for those different optic flow stimuli (Phinney and Siegal, 2000). Neurons in the floor of the intraparietal sulcus (IPS), in the ventral intraparietal area ( VIP), have large visual receptive fields that respond selectively to optic flow. These

:     

1267

responses commonly combine radial or circular and planar motion preferences. The planar direction preferences are consistent across their receptive fields, suggesting that local planar motion preferences do not create templates favoring that neuron’s preferred radial or circular stimuli (Schaafsma and Duysens, 1996). The dorsal-medial extension of PPC includes area PEc, the caudal posterior parietal region. Neurons in this region are responsive to somatosensory stimuli and are thought to be involved in motor planning for reaching (Lacquaniti et al., 1995). They also have large visual receptive fields with robust direction selectivity. Most show simple directional responses to object motion, but some show opponent vector organization, preferring motion either inward to or outward from the fixation point. They also respond selectively to optic flow stimuli with different FOEs, with mixed object motion and optic flow responses similar to what has been observed elsewhere in the parietal lobe (Raffi et al., 2002; Squatrito et al., 2001). In sum, optic flow responses can be elicited from a number of areas around the occipito-parieto-temporal junction. In all of these areas there is a range of preferences for optic flow stimuli simulating visual motion seen during observer self-movement. These optic flow responses are accompanied by selective responses to object motion stimuli, with both types of stimuli eliciting position invariance and speed sensitivity.

Optic flow responses in cat cortex The dorsal association cortex of the cat includes extrastriate visual processing areas (Clare and Bishop, 1954) that receive input from the geniculo- and colliculocortical systems. The posteromedial (PMLS) and posterolateral (PLLS) regions in the lateral sulcus contain object motion direction-selective visual neurons with large binocular receptive fields that are broadly tuned for speed, with faster speeds preferred in more peripheral receptive fields. Most PLLS neurons and many PMLS neurons have receptive field locations and direction selectivities that create a preference for motion directed away from the fixation point, with fewer cells preferring motion toward the fixation point (von Grunau and Frost, 1983). Radial-selective neurons are combined with circular-selective neurons juxtaposing orthogonal directions in a potential substrate for optic flow analysis (Rauschecker et al., 1987). Clare-Bishop neurons also respond to object motion: onethird prefer approaching stimuli, one-third prefer translational motion, and half as many prefer receding stimuli, with the remainder being nonresponsive or nonselectively responsive. These selectivities are partly attributable to size effects and partly to binocular distance cues. The posterior lateral sulcus shows more translational motion selectivity, and ante-

1268

, ,   

rior areas show more approach/recession selectivity and size effects (Toyama et al., 1990). Anterior but not posterior lateral sulcus neurons show a centrifugal bias, as the preponderance of preferred object motion is directed away from the fixation point. In the posterior region, directional preferences are mainly orthogonal to a line from the receptive field center to the fixation point. This is consistent with a separation of radial and circular motion analysis, suggesting a large-scale organization of optic flow responses in cat lateral sulcus that has not been seen across the monkey STS (Sherk et al., 1995). This large-scale organization may relate to differences in cortical projections, with the posterior predominance of corticotectal neurons and the anterior predominance of corticostriatal neurons. Corticotectal neurons have smaller receptive fields, slower speed preferences, centrifugal object motion, and a preference for contracting optic flow. Corticostriatal neurons have larger receptive fields, faster speed preferences, an absence of centrifugal bias, and a preference for expanding optic flow (Niida et al., 1997). Alternatively, regional differences in these responses may relate to receptive field size: posterior PMLS neurons with smaller, central receptive fields that prefer slower motion are less influenced by element size and speed gradients in optic flow. Anterior PMLS neurons with larger, peripheral receptive fields that prefer faster motion are more responsive to naturalistic element size and speed gradients (Brosseau-Lachaine et al., 2001). The relationship between centrifugal bias and optic flow analysis was tested by raising kittens in darkness, except for timed exposure to either expanding or contracting optic flow stimuli presented either by controlled translational movement or by video simulation. Neither the direction nor the type of stimulation altered the centrifugal bias for object motion, although these animals did have more nonresponsive and nondirectional neurons. Thus, dark rearing altered the neuronal responses, but restricted exposure to certain patterns of optic flow did not change their object motion direction selectivity (Brenner and Rauschecker, 1990). Lateral sulcus neurons prefer naturalistic optic flow stimuli as seen moving across a textured ground plane. Seventy percent of the neurons respond to optic flow, and most have stronger responses when the visual elements have distanceproportioned sizes and accelerate with simulated proximity. There is a preponderance of neurons that prefer optic flow stimuli simulating forward observer movement with mainly downward motion, compared to backward observer movement simulated with mainly upward motion (Kim et al., 1997). The optic flow selectivities of these neurons are not related to their object motion responses, with most neurons showing larger responses to the optic flow but no relationship between the preferred optic flow and object motion directions (Mulligan et al., 1997).

Radial, circular, and translational directionality differ between PMLS and PLLS. About 70% of the neurons in both areas respond to more than one type of optic flow, with an additional ~10% being unresponsive in both areas. The remaining neurons prefer one type of motion, more commonly favoring radial motion over circular or translational motion. In both areas, the nonselective neurons respond more strongly to object motion, whereas the selective neurons respond more strongly to optic flow (Li et al., 2000). In sum, cat lateral sulcus neurons show many of the same response properties as monkey STS neurons. In cats, there may be greater regional separation of different response specializations and greater directional selectivity than in monkeys. Our understanding of optic flow analysis is still too incomplete to address the issue of whether cats and monkeys use the same neuronal processing mechanisms for optic flow analysis.

Optic flow evoked activity in human cortex Neuroimaging has identified optic flow–responsive areas of human cerebral cortex. H2O15 positron emission tomography (PET scanning) shows regional changes in cerebral blood flow (rCBF ) with optic flow stimulation. Activation by random-dot motion is subtracted from that evoked by optic flow simulating observer movement across the ground plane to create a map of optic flow–selective rCBF that is matched to magnetic resonance images ( MRI) used to identify cortical landmarks. Optic flow specific activation is localized to the right dorsal precuneus, right superior parietal lobule, and bilateral fusiform gyri, suggesting some right hemisphere lateralization of optic flow analysis in humans. The activated areas are roughly homologous with monkey areas V3, 7a, and TEO, respectively, but the homolog of MT/MST was not activated in this study (de Jong et al., 1994). Complementary views of optic flow activation in humans come from combining H2O15 rCBF PET imaging with blood oxygen level dependent (BOLD) functional MRI (fMRI). PET comparisons of rCBF during optic flow and stationary dot displays show multifocal optic flow specific activation in bilateral cuneus (V2, V3a), left-to-right MT/MST, bilateral dorsal IPS, bilateral dorsal premotor area, and right cerebellum. The presentation of similar stimuli during fMRI studies confirmed optic flow activation in bilateral MT/MST. Combining these data sets yielded an impression that the right MT/MST complex and the right dorsal IPS areas were most selectively responsive to optic flow (Peuskens et al., 2001). BOLD-based fMRI has been used to distinguish between the human cortical homologs of monkey areas MT and MST. In these studies, activation of MT’s purely contralateral receptive fields was compared to activation of MST’s typically bilateral receptive fields. Full-field optic flow

showed activated sites selected as corresponding to human MT/MST, and also at sites in the IPS, parieto-occipital sulcus, and calcarine sulci. Contralateral hemifield stimulation activated only a more posterior segment of the MT/MST complex, with a more anterior segment activated by either contralateral or ipsilateral stimulation and considered to be homologous to monkey MSTd. Pursuit eye movements tracking a visual target also activated MT+, whereas pursuit of the imagined image of the subject’s finger moving in darkness, eye movement without retinal slip, activated an anterolateral area that was considered homologous to monkey MSTl (Dukelow et al., 2001). Neuroimaging shows the same dual roles of dorsal extrastriate cortex for optic flow and object motion analysis in humans that have been seen in single-neuron studies of monkeys. MRI shows selective activation of MT+ with intact objects versus the scrambled images of those objects. In addition, there is activation by both the movement and the shape of the objects, with synergistic interactions between movement and shape. Thus, MT+ may serve optic flow analysis and also contribute to the analysis of objects defined by movement or other cues (Kourtzi et al., 2002).

Eye position influences optic flow responses The responses of posterior parietal visual fixation neurons are influenced by fixation position in the frontoparallel and depth planes (Sakata et al., 1980). Flashing a stationary stimulus at a constant receptive field location evokes different responses at different gaze angles. During active fixation, 61% of area 7a neurons show a substantial enhancement of visual responses at some gaze angles with no change in resting activity. In contrast, only 10% of the neurons show gaze angle effects during casual fixations between spontaneous saccades (Andersen and Mountcastle, 1983). As a result, additive models of visual gaze interactions have been rejected in favor of a multiplicative model that describes the area of preferred gaze as a gain field (Andersen and Braunstein, 1985). Most fixation-responsive neurons (>90%) in areas 7a and MSTd show gaze angle gain fields that can be characterized as a three-dimensional function relating horizontal and vertical eye position to response amplitude. Usually, this function describes a plane, but in ~15% of the neurons this function peaks and falls off within the visual field (Squatrito and Maioli, 1996). Gaze angle interacts with visual and pursuit signals in MSTd, where 34% of pursuit neurons and 10% of visual motion neurons also show gaze angle effects (Squatrito and Maioli, 1997). Gaze angle modulation of neuronal responses extends across cortical areas and functional response properties. Most neurons in areas MT (61%) and MST (82%) show gaze

:     

1269

angle effects that alter their fixation, pursuit, and visual responses without changing their direction selectivity. Gaze angle effects occur in fewer 7a (42%) and lateral intraparietal (LIP) (39%) neurons, but again, these effects alter responses to fixation, pursuit, and visual motion stimulus. In all of these areas, gaze angle response functions can be fit by a plane. In individual neurons, the same linear function could be applied to fixation, pursuit, and visual responses. This suggests that gaze angle modulates neuronal responses through a mechanism that is shared across response properties (Bremmer et al., 1997a; Bremmer et al., 1997b). In area 7a, responses to optic flow depend on gaze angle, with more neurons (44%) showing good gaze angle fits to a linear response function than to a nonlinear function (28%). This is complicated by the observation that the function fitting the gaze angle effect often depends on the visual stimulus. This might be attributable to optic flow selectivity causing the response functions for different stimuli to extend over different ranges of response amplitude. Thus, stimulusdependent differences in gaze angle functions might reflect nonlinear interactions with visual responses (Read and Siegel, 1997).

Pursuit eye movements and cortical motion processing STS cortex participates in pursuit initiation and maintenance. Focal ibotenic acid lesions in MT create a retinotopic impairment of pursuit initiation in all directions, with saccade errors to moving, but not stationary, stimuli suggesting impaired motion analysis for eye movement control (Newsome et al., 1985). Foveal MT lesions cause pursuit maintenance deficits with movement toward the side of the lesion (Dursteler et al., 1987). Injections anywhere in MST cause pan-directional pursuit initiation deficits in the contralateral hemifield. Lesions in MSTl also cause directional pursuit maintenance deficits with target movement toward the side of the lesion whether the movement began in the contralateral or the ipsilateral hemifield (Dursteler and Wurtz, 1988). All of these lesion effects are temporary. The duration of MT lesion effects is proportional to the size of the lesion. When MT and most of MST is lesioned, recovery remains incomplete after 7 months, regardless of visual experience ( Yamasaki and Wurtz, 1991). Pursuit targets presented with feedback-controlled stabilization of the retinal image, or with blinking-off of the target during its movement, differentiate pursuit activation from retinal versus nonretinal input (e.g., efference copy of the movement command). Pursuit-related activity in MTf lapses during stabilization or blinking, suggesting a retinal origin. Pursuit-related activity in MSTd persists during stabilization and blinking, suggesting a nonretinal origin. Some MSTl neurons lapse and others persist. The combination of

1270

, ,   

retinal and nonretinal pursuit signals in MSTd suggests that these neurons sum retinal slip and eye velocity signals to drive pursuit (Newsome et al., 1988). Support for the retinal origin of V4 and MT pursuit responses, and for the nonretinal origin of MST pursuit responses, comes from comparisons of responses to retinal motion induced by object or ocular movement. V4 and MT neurons show the same responses to retinal motion regardless of whether it is the result of object or ocular movement, whereas MST neurons respond differently to the same retinal motion resulting from object motion or ocular movement (Erickson and Thier, 1991). This view is supported by studies in which monkeys are trained to pursue an illusory target, the illusory intersection of incomplete lines in a regular figure. MT neurons that respond during the pursuit of a real object do not respond during pursuit of an illusory target. In contrast, some MST neurons respond during pursuit of real and illusory targets. This subset of MST pursuit-activated neurons do not stop their pursuit activity during a blinking-off of the target, seemingly identifying the subpopulation that accesses a nonretinal pursuit signal (Ilg and Thier, 1997). MT neurons prefer the same direction during pursuit and small spot movement. MSTd neurons reverse their visual motion direction preferences with stimulus size, typically across changes in stimulus dimensions from 20 ¥ 20 to 30 ¥ 30 degrees. The preferred pursuit direction is the same as that preferred with small visual stimuli and the opposite of that preferred with large visual stimuli. The largest responses are evoked by slow pursuit (8 degrees). Both sustained and transient stereopsis can be stimulated by firstorder (luminance) and second-order (contrast) forms of spatial information. Transient stereopsis relies more heavily on second-order information than does sustained stereopsis. The transient system is more broadly tuned than the sustained system to differences between the texture (carrier) properties of the stimulus presented to the two eyes (i.e.,

1304

, ,   

spatial frequency, orientation, contrast, and contrast polarity) (Edwards et al., 1999; Schor et al., 1998). Vivid stereoscopic depth can be perceived with brief stimuli, even when the detailed structure of the two retinal images differs markedly. Contrast envelope size (Schor et al., 2001) and, to a greater extent, temporal synchrony of the two eyes’ stimuli (Cogan et al., 1995) are the primary means for selecting matched binocular inputs for transient stereopsis. Second-order stimuli work well in concert with first-order stimuli to assist the computation of binocular matches in a coarse-to-fine strategy. Second-order information from the contrast-defined edges provides coarse information to guide matches of small disparities in the luminance-defined texture by the fine system (Wilcox and Hess, 1997). This is not a function reserved exclusively for second-order stimuli because coarse first-order luminance information also guides disparity matches (Wilson et al., 1991a). This coarse-to-fine strategy utilizing contrast and luminance information is effective because spatial information represented by contrast and luminance is usually highly correlated in images formed in natural environments.

Stereo acuity Stereopsis is the sense of relative depth between two or more features. It is stimulated by differences between absolute disparities subtended by these features (Westheimer, 1979a). Absolute disparity of an object quantifies its retinal disparity relative to the horopter. It is the difference in the locations of its retinal images in the two eyes from corresponding retinal points. Stereopsis is stimulated by differences between several absolute disparities, that is, relative disparity. Relative disparity describes the disparity difference between features. Relative disparity thresholds for stereopsis are lowest when targets are imaged on or near the horopter. Stereo sensitivity to relative disparity varies dramatically with the distance of these targets from the horopter or plane of fixation (Fig. 87.4). The average disparity of targets from the fixation plane is described as their depth pedestal. Stero-depth discrimination thresholds, measured as a function of the depth pedestal, describe a depth discrimination threshold function that is summarized by a Weber fraction (stereo threshold/depth pedestal). The Weber fraction for stereopsis indicates that the noise or variability of the absolute disparities subtended by the targets is less than 5% over a range of disparity pedestals up to 0.5 degree (Badcock and Schor, 1985). Judgments of depth beyond this range of pedestals are clouded by the appearance of diplopia.

Spatial pooling and uncertainty Stereo sensitivity can either be enhanced or reduced by nearby targets. The threshold for detecting depth corruga-

F 87.4. Threshold for depth discrimination between a test stimulus and a comparison stimulus as a function of the disparity offset (pedestal) of the comparison stimulus. Each curve shows the results using bandpass-filtered bar stimuli whose center spatial frequencies range from 0.15 to 9.6 c/deg. Stereo depth is most sensitive for targets located on the horopter, and threshold increases exponentially with disparity pedestal.

tions of a surface such as the folds in a hanging curtain decreases with depth-modulation frequency (reciprocal of spacing between the folds), up to 0.3 c/deg where the threshold is lowest (Tyler, 1975). At depth-modulation frequencies lower than 0.3 c/deg, the threshold for stereopsis is elevated and appears to be limited by a disparity gradient (i.e., a minimal rate of change in the depth/degree of target separation). At depth-modulation frequencies higher than 0.3 c/deg, the stereo threshold is elevated as a result of depth averaging of adjacent targets. Similar effects are seen with separations between point stimuli for depth (Westheimer, 1986; Westheimer and McKee, 1979). When a few isolated targets are viewed foveally, changes in the binocular disparity of one target produce suprathreshold depth changes or depth biases in other targets when their separation is less than 4 arc min. This depth attraction illustrates a pooling of disparity signals (Parker and Yang, 1989).

When targets are separated by more than 4 to 6 arc min, the depth bias is in the opposite direction and features appear to repel one another in depth. Attraction and repulsion also occur with cyclopean targets (i.e., targets that can only be revealed by stereo depth, such as clumps of tree foliage) (Stevenson et al., 1991), showing that they are not based simply on positional effects at the monocular level. The enhancement of depth differences by repulsion might be considered a depth-contrast phenomenon that is analogous to Mach bands in the luminance domain (enhanced perceived contrast of edges) that are thought to result from lateral inhibitory interactions. Depth distortions that are analogous to Mach bands have been demonstrated with horizontal disparity variations between vertically displaced contours (Lunn and Morgan, 1996), demonstrating analogous spatial interactions in the horizontal disparity domain and the luminance contrast domain.

: 

1305

F 87.5. Position and phase encoding of disparity: two possible schemes of disparity encoding are shown. Position coding: a, Solid curves show the receptive field profile of left and right eye inputs to a binocular cell. The abscissa for each curve gives retinal position, and the ordinate gives the cells sensitivity to luminance. b, The disparity tuning of the binocular cell that is sensitive to posi-

tional disparity. Disparity coding: c, Solid curves show the receptive field profile of left and right eye inputs to a binocular cell. The abscissa for each curve gives retinal phase, and the ordinate gives the cell’s sensitivity to luminance. d, The disparity tuning of the binocular cell that is sensitive to phase disparity.

Frame of reference

range of spatial frequencies (narrowband luminance stimulus) (1.75 octaves) such as those produced by a difference of two Gaussians, the stereo threshold becomes lower as spatial frequencies increase up to 2.5 c/deg (Schor et al., 1984a; Smallman and MacLeod, 1994). When expressed as a phase disparity, stereo threshold is constant over this range of spatial frequencies. At spatial frequencies higher than 2.5 c/deg, stereo threshold has a constant position disparity. The constant position disparity at high spatial frequencies contradicts theories of disparity sensitivity based upon phase sensitivity within spatial channels. In addition to phase sensitivity, there may be other factors that limit disparity resolution, such as position sensitivity. Given the presence of both a phase and a position limit, the threshold is set by the limit that is higher. For low spatial frequencies, the constant phase limit would be higher than the position limit, and the converse would be true for high spatial frequencies. The higher limit would determine the threshold.

Typically, stereo acuity is measured while looking straight ahead with respect to a flat (frontoparallel) frame of reference rather than the curved horopter. The frontoparallel frame of reference is an implicit or familiar frame of reference. Its pattern of disparities is usually symmetrical with respect to the gaze normal direction and the horopter. However, the frontoparallel plane is somewhat arbitrary since it contains disparities that do not lie on the zerodisparity surface of the horopter. A slanted plane can also be used as an arbitrary explicit frame of reference to make relative depth judgments. Indeed, depth discrimination off the plane of fixation is improved when the targets are presented within a reference surface of dots that have the same mean disparity as the pedestal (Andrews et al., 2001). However, with a frontoparallel reference plane, stereo thresholds increase with the distance in depth (disparity pedestal) of targets from the plane of fixation (Badcock and Schor, 1985). These observations suggest that the frame of reference for relative depth judgments is most effective when it is well defined by visual stimuli.

Disparity processing by early spatial filters Stereo acuity depends upon spatial scale (range of spatial frequencies). The dependence of stereopsis on luminance spatial frequency suggests that disparity is processed early in the visual system by linear channels that are tuned to discrete bands of spatial frequency. When tested with a limited

1306

, ,   

Disparity coding metrics (phase vs. position) Physiologically, binocular receptive fields have two fundamental organizational properties (Fig. 87.5). The receptive fields that represent binocular disparity can be offset from a point of zero retinal correspondence (position disparity). They can also have zero positional disparity combined with a quadrature organization in which there is a 90 degree phase shift of the areas of excitation and inhibition in one eye’s receptive field compared to the other (phase disparity) (DeAngelis et al., 1995). The position limit at high spatial

frequencies could result from positional jitter in all binocular receptive field sizes. Jitter would have a large influence on the disparity sensitivity of small receptive fields but only a minor influence on the sensitivity of large receptive fields. The same jitter is large relative to the phase coding ability of small (high-frequency) receptive fields. Jitter is responsible for the breakdown of the size-disparity correlation. At high spatial frequencies, stereo threshold is limited by positional uncertainty that causes the threshold to remain constant above 2.5 c/deg. For further discussion of neural mechanisms underlying stereopsis, see Chapter 48. Other models of stereopsis assume that disparity processing is based solely upon positional information in spatial channels tuned to frequencies above 2.5 c/deg (Kontsevich and Tyler, 1994). In this model, elevated stereo thresholds at low spatial frequencies (below 2.5 c/deg) result from reduced effective contrast of low spatial frequencies passed by the lowest binocular spatial frequency channel that is tuned to 2.5 c/deg. The bandpass stereo tuning characteristics observed with interocular differences in low spatial frequencies (2 degrees) during steady fixation (Enright, 1991; McKee et al., 1990). Thresholds increase proportionately with target eccentricity (Berry, 1948; McKee et al., 1990). Retinal position uncertainty is believed to increase with retinal eccentricity (Levi et al., 1984), and this source of noise is thought to account for the increase of stereo threshold with target separation (McKee et al., 1990). Thresholds of both simultaneous and sequential stereopsis at 1.5 degrees of target separation are equal (approximately 0.5 to 1.0 arc min) (Enright, 1991). Simultaneous stereopsis continues to improve at smaller separations down to a lower limit of a few seconds of arc; however, sequential stereopsis does not improve at separations of less than 2 degrees (McKee et al., 1990). With small target separations, the sequential stereo threshold is three to four times higher than for a single simultaneous view of the same target (Enright, 1991; Kumar and Glaser, 1994; McKee et al., 1990; Westheimer, 1979a). Eye position uncertainty, in the form of unregistered vergence fluctuations, is believed to account for the elevated threshold for sequential stereopsis (without eye movements) (McKee et al., 1990). Vergence noise would not impair simultaneous stereopsis because vergence jitter is common to binocular images seen simultaneously, and it is canceled in a presumed differencing process (Westheimer, 1979a).

Sensorimotor interactions Saccadic gaze shifts in sequential stereopsis improve depth discrimination between widely spaced targets by imaging them near the fovea and horopter, where disparity resolution is most acute. Stereo sensitivity to depth differences between widely separated targets (>5 degrees) can be improved by as much as 50% with sequential gaze shifts that image the two targets sequentially onto the fovea compared to the same test without gaze shifts (Enright, 1991; Ogle, 1956; Wright, 1951). This improvement is thought to be due primarily to

the reduction of retinal position uncertainty when targets are imaged on the fovea (McKee et al., 1990). Depth comparisons between targets in different locations are represented in head-centric coordinates, and stereodepth perception depends upon target location relative to the head as well as horizontal retinal image disparity. In particular, vergence information is essential for estimating target distance from the head when vertical disparity information is sparse or absent (Brenner and van Damme, 1998; Wright, 1951). In the extreme case, vergence aligns the eyes precisely during fixation so that absolute retinal image disparity is reduced to zero. Yet depth differences between the sequentially fixated targets that subtend zero disparity can still be discriminated because efferent signals that align the eyes can be used to estimate target distance from the head (Brenner and van Damme 1998). Stereo threshold is limited by the uncertainty with which eye position is sensed (Backus et al., 1999), by the duration or dwell time of each fixation, and by time delays between sequential views (Kumar and Glaser, 1994) that are related to the dynamics of saccadic eye movements (Bahill et al., 1975). Even small to moderate saccades take 25 to 50 msec (Bahill et al., 1975), and this introduces an inter-stimulus interval (ISI) during which information about the first target depth may be lost or corrupted. The ISI could be extended beyond the duration of the saccade by saccadic suppression (reduced visual sensitivity during saccades) that occurs while changes in visual direction are updated to correspond to abrupt changes in eye position (Matin, 1986). The minimum dwell time for fixating in a given direction of gaze is approximately the latency for a saccadic eye movement, or about 200 msec. The simultaneous stereo threshold is elevated by over fivefold when target duration is reduced from 1 second to about 7.5 msec (Harwerth and Rawlings, 1977). Thresholds for sequential stereopsis are reduced as the number of alternate saccadic fixations increases up to four to eight saccades (two to four cycles of alternate fixation) (Enright, 1991). Sequential stereo thresholds also become lower during the first four to six cycles of alternating sequential target presentations to the fovea without saccadic eye movements (Kumar and Glaser, 1994). Repeated fixations allow averaging to reduce the uncertainty of retinal disparity cues and target location sensed with eye position cues. In addition, the memory of sequential views could be refreshed by repeated gaze shifts (Hayhoe et al., 1998). Thresholds become elevated as stimulus onset asynchrony (SOA) increases (Kumar and Glaser, 1994). Increasing stimulus duration of sequential stimuli elevates the stereo threshold by lengthening the time between the onsets of the two stimuli (i.e., increase of SOA) (Kumar and Glaser, 1994). The improvement of sequential stereopsis appears to depend on the number of times the stimulus appears and the SOA, rather than the total presentation time. It is more

: 

1309

important to have short than long time intervals between sequential stereo stimuli.

Temporal masking Temporal masking (Alpern, 1953) also elevates thresholds for sequential stereopsis (Butler and Westheimer, 1978; Tyler and Foley, 1974). Saccadic gaze shifts produce time delays between successively fixated targets. The transient onset of the second stimulus could mask the appearance of the first stimulus (backward masking) or vice versa, especially when the two stimuli are large textured surfaces (Kahneman, 1968). Backward masking has been shown to influence sensitivity to apparent motion with random-dot patterns (Braddick, 1973). An example of backward masking in stereopsis was reported by Butler and Westheimer (1978). They observed that the stereo threshold was elevated by adjacent contours presented 100 msec after the onset of the stereo test stimulus. The spatiotemporal masking is greatest when the mask and stereo stimulus have the same disparity. Masking is reduced by half when the disparity of the mask differs from that of the stimulus by only 15 sec arc.

Calibration errors The magnitude of perceived depth difference between two targets may differ in simultaneous and sequential stereopsis. Calibration errors for retinal and extraretinal cues for stereo depth could have different emphasis or weights in these two forms of stereopsis. Retinal and extraretinal cues for stereo depth, including horizontal disparity, retinal eccentricity, and eye position signals, often have calibration errors that could produce perceived depth and slant biases (i.e., constant errors). Biases in horizontal disparity are related to uniform and nonuniform magnification effects for perceived direction, as described by Ogle (1962), that cause planar surfaces to appear curved and slanted. Biases in perceived visual direction are described by Ogle (1962) as partition errors. Perceived directions, stimulated by equal physical horizontal retinal eccentricities from the fovea (nasal vs. temporal), appear to have unequal perceived magnitudes (unequal oculocentric directions) in the nasal and temporal halves of the visual field (Kundt and Munsterberg asymmetries). These errors in perceived eccentricity of targets imaged on the two hemiretinas could cause directional biases of head-centric depth and slant estimates based on retinal eccentricity and gaze azimuth (Ø) (see equation 2). Eye movement signal biases are related to minor muscle paresis or weakness within the oculomotor system that can cause over- or underestimation errors of sensed eye position that vary with direction of gaze. When foveal and peripheral targets are viewed simultaneously without eye movements, retinal position and disparity biases contribute to constant errors of depth and slant

1310

, ,   

estimates. When separated targets are fixated foveally with sequential gaze shifts, eye position biases will also contribute to constant head-centric errors of depth and slant estimates. Thus, the perceived depth difference between two widely spaced targets might appear different, depending on whether they were fixated sequentially with the fovea or viewed simultaneously at two different retinal locations.

Conclusions Space perception is based on several sources of information that are estimated by the visual system. The perception of direction, distance, surface shape and orientation, and object volume can be obtained from monocular sources of information including motion parallax, texture gradients, perspective cues, and stimulus overlap (partial occlusion), and from the binocular source of retinal image disparity (stereopsis). The subject of stereopsis has been used to investigate spatial resolution limits of the visual system, to learn how 3D spatial information is represented in the brain, and to understand how perceptual systems reduce information content to a manageable level and also reduce information ambiguity of stimuli by employing ecological inferences as well as computational strategies. Disparity by itself is an ambiguous depth cue since it is in retinal coordinates. To yield a depth percept, it must be mapped into head-centric coordinates using information about direction and distance to interpret relative depth magnitude and to reference depth in head-centric coordinates. Currently, a concentrated effort is being made to determine how this mapping occurs and the sensory and motor sources of information that are used to obtain information about distance and direction. Ultimately, these investigations will lead to an understanding of how we build up a 3D perceptual representation of space with sequential views such that our percept of space extends beyond any momentary field of view. REFERENCES Alpern, M., 1953. Metacontrast, J. Opt. Soc. Am., 43:648–657. Andrews, T. J., A. Glennester, and A. J. Parker, 2001. Stereoacuity thresholds in the presence of a reference surface, Vis. Res., 41:3051–3061. Backus, B. T., M. S. Banks, R. van Ee, and J. A. Crowell, 1999. Horizontal and vertical disparity, eye position, and stereoscopic slant perception, Vis. Res., 39:1143–1170. Banks, M. S., R. van Ee, and B. T. Backus, 1997. The computation of binocular visual direction: a re-examination of Mansfield and Legge (1997), Vis. Res., 37:1605–1610. Badcock, D. R., and C. M. Schor, 1985. Depth-increment detection function for individual spatial channels, J. Opt. Soc. Am. A, 2(7):1211–1215. Bahill, A. T., M. R. Clark, and L. Start, 1975. The main sequence, a tool for studying human eye movements, Math. Biosci., 24:191–204.

Berry, R. N., 1948. Quantitative relations among vernier, real depth and stereoscopic depth acuities, J. Exp. Psychol., 38:708–821. Braddick, O., 1973. The masking of apparent motion in randomdot patterns, Vis. Res., 13:355–369. Brenner, E., and W. J. M. van Damme, 1998. Judging distance from ocular convergence, Vis. Res., 38:493–498. Butler, T. W., and G. Westheimer, 1978. Interference with stereoscopic acuity: spatial, temporal and disparity tuning, Vis. Res., 18:1387–1392. Cogan, A. I., L. L. Konstevich, A. J. Lomakin, D. L. Halpern, and R. Blake, 1995. Binocular disparity processing with oppositecontrast stimuli, Perception, 24:33–47. DeAngelis, G. C., I. Ohzawa, and R. D. Freeman, 1995. Neuronal mechanisms underlying stereopsis: how do simple cells in the visual cortex encode binocular disparity? Perception, 24:3–31. Edwards, M., D. R. Pope, and C. M. Schor, 1999. Orientation tuning of the transient-stereopsis system, Vis. Res., 39:2717– 2727. Enright, J. T., 1991. Exploring the third dimension with eye movements: better than stereopsis, Vis. Res., 31:1549–1562. Erkelens, C. J., and R. van Ee, 1997. Capture of the visual direction of monocular objects by adjacent binocular objects, Vis. Res., 37:1193–1196. Frisby, J. P., and J. E. W. Mayhew, 1980. Spatial frequency tuned channels: implications for structure and function from psychophysical and computational studies of stereopsis, Philos. Trans. R. Soc. Lond., 290:95–116. Garding, J., J. Porrill, J. E. Mayhew, and J. P. Frisby, 1995. Stereopsis, vertical disparity and relief transformations, Vis. Res., 35:703–722. Gillam, B. J., and B. Lawergren, 1983. The induced effect, vertical disparity and stereoscopic theory, Percept. Psychophys., 34:121–130. Harwerth, R. S., and S. C. Rawlings, 1977. Viewing time and stereoscopic threshold with random-dot stereograms, Am. J. Optom. Physiol. Opt., 54:452–457. Hayhoe, M. M., D. G. Bensinger, and D. Ballard, 1998. Task constraints in visual working memory, Vis. Res., 38:125–137. Helmholtz, H. von., 1909. Handbuch der Physiologischen Optik (third German edition 1962; English translation by J. P. C. Southall, trans.) Hess, R. F., and L. M. Wilcox, 1994. Linear and non-linear filtering in stereopsis, Vis. Res., 34:2431–3438. Howard, I. P., 1982. Human Visual Orientation, Chichester, UK: Wiley. Kahneman, D., 1968. Method, findings, and theory in the study of visual masking, Psychol. Bull., 70:404–425. Kontsevich, L. L., and C. W. Tyler, 1994. Analysis of stereo thresholds for stimuli below 2.5 c/deg, Vis. Res., 34:2317–2329. Kumar, T., and D. A. Glaser, 1994. Some temporal aspects of stereoacuity, Vis. Res., 34:913–925. Levi, D. M., A. Kleins, and P. Actsebaomo, 1984. Detection and discrimination of the direction of motion in central and peripheral vision of normal and amblyopic observers, Vis. Res., 24: 789–800. Liu, L., S. B. Stevenson, and C. M. Schor, 1994. A polar coordinate system for describing binocular disparity, Vis. Res., 34:1205–1222. Lunn, P. D., and M. J. Morgan, 1996. The analogy between stereo depth and brightness, Perception, 24:901–904. Mann, V. A., A. Hein, and R. Diamond, 1979. Localization of targets by strabismic subjects: contrasting patterns in constant and alternating suppressors, Percept. Psychophys., 25:29–34. Marr, D., and T. Poggio, 1976. Cooperative computation of stereo disparity, Science, 194:283–287.

Matin, L., 1986. Visual localization and eye movements, in Boff, K. R., Handbook of Perception and Human performance, vol. 1: Sensory Processes and Perception (K. R. Boff, L. Kaufman, and J. P. Thomas, eds.), New York: Wiley-Interscience, chapter 20, pp. 1–40. McKee, S. P., L. Welch, D. G. Taylor, and S. F. Bowne, 1990. Finding the common bond: stereoacuity and the other hyperacuities, Vis. Res., 30:879–891. Mitchison, G. J., and S. P. McKee, 1987. The resolution of ambiguous stereoscopic matches by interpolation, Vis. Res., 27:285– 294. Müller, J., 1843. Elements of Physiology (W. Baly, trans.), London: Tayler and Walton. Ogle, K. N., 1956. Stereoscopic acuity and the role of convergence, J. Opt. Soc. Am., 46:269–273. Ogle, K. N., 1962. Spatial localization through binocular vision, in The Eye, vol. 4 (H. Davson ed.), New York: Academic Press, pp. 271–324. Papathomas, T. V., and B. Julesz, 1989. Stereoscopic illusion based on the proximity principle, Perception, 18:589–594. Parker, A. J., and Y. Yang, 1989. Spatial properties of disparity pooling in human stereo vision, Vis. Res., 29:1525–1538. Pizlo, Z., 2001. Perception viewed as an inverse problem, Vis. Res., 41:3145–3161. Rogers, B. J., and M. F. Bradshaw, 1995. Disparity scaling and the perception of frontoparallel surfaces, Perception, 24:155–179. Schor, C. M., and D. Badcock, 1985. A comparison of stereo and vernier acuity within spatial channels as a function of distance from fixation, Vis. Res., 25:1113–1119. Schor, C. M., M. Edwards, and D. Pope, 1998. Spatial-frequency tuning of the transient-stereopsis system, Vis. Res., 38:3057– 3068. Schor, C. M., M. Edwards, and M. Sato, 2001. Envelope size tuning for stereo-depth perception of small and large disparities, Vis. Res., 41:2555–2567. Schor, C. M., and C. W. Tyler, 1981. Spatio-temporal properties of Panum’s fusional area, Vis. Res., 21:683–692. Schor, C. M., I. C. Wood, and J. Ogawa, 1984a. Binocular sensory fusion is limited by spatial resolution, Vis. Res., 24:661–665. Schor, C. M., I. C. Wood, and J. Ogawa, 1984b. Spatial tuning of static and dynamic local stereopsis, Vis. Res., 24:573–578. Smallman, H. S., and D. I. MacLeod, 1994. Size-disparity correlation in stereopsis at contrast threshold, J. Opt. Soc. Am. A, 11: 2169–2183. Stevenson, S. B., and L. K. Cormack, 2000. A contrast paradox in stereopsis, motion detection, and vernier acuity, Vis. Res., 40:2881–2884. Stevenson, S., L. Cormack, and C. M. Schor, 1989. Hyperacuity, superresolution, and gap resolution in human stereopsis, Vis. Res., 29:1597–1605. Stevenson, S. B., L. K. Cormack, and C. M. Schor, 1991. Depth attraction and repulsion in random dot stereograms, Vis. Res., 31:805–813. Tyler, C. W., 1975. Spatial organization of binocular disparity sensitivity, Vis. Res., 15:583–590. Tyler, C. W., and J. M. Foley, 1974. Stereomovement suppression for transient disparity changes, Perception, 3:287–296. Westheimer, G., 1979a. Cooperative neural processes involved in stereoscopic acuity, Exp. Brain Res., 36:585–597. Westheimer, G., 1979b. The spatial sense of the eye, Invest. Ophthalmol. Vis. Sci., 18:893–912. Westheimer, G., 1986. Spatial interaction in the domain of disparity signals in human stereoscopic vision, J. Physiol., 370: 619–629.

: 

1311

Westheimer, G., and S. P. McKee, 1979. What prior uniocular processing is necessary for stereopsis? Invest. Ophthalmol. Vis. Sci., 18:614–621. Wilcox, L. M., and R. F. Hess, 1995. Dmax for stereopsis depends on size, not spatial frequency content, Vis. Res., 35:1061–1069. Wilcox, L. M., and R. F. Hess, 1997. Scale selection for secondorder (nonlinear) stereopsis, Vis. Res., 37:2981–2992. Wilson, H. R., R. Blake, and D. L. Halpern, 1991a. Coarse spatial scales constrain the range of binocular fusion on fine scales, J. Opt. Soc. Am., 8:229–236.

1312

, ,   

Wilson, H. R., V. P. Ferrera, and C. Yo, 1991b. A psychophysically motivated model for two-dimensional motion perception, Vis. Neurosci., 9:79–97. Wright, W. D., 1951. The role of convergence in stereoscopic vision, Proc. Phys. Soc. Lond. B., 64:289–297. Zhang, Z., M. Edwards, and C. M. Schor, 2001. Spatial interactions minimize relative disparity between adjacent surfaces, Vis. Res., 41:2295–3007.

88

Binocular Rivalry RANDOLPH BLAKE

W    through two laterally separated eyes, and yet we experience a single, stable visual world. The brain, in other words, manages to blend, or fuse, the two monocular images in a way that belies any hint of their dual origins, and it does so in a way that extracts stereoscopic information about the three-dimensional (3D) layout of objects in the environment. Several other chapters in this volume (Chapters 48, 49, and 87) cover material on binocular single vision and stereopsis. The neural operations responsible for binocular single vision and stereopsis, however, are labile in the sense that they can be disrupted when dissimilar monocular stimuli are imaged on corresponding retinal locations. Under these conditions, the eyes transmit contradictory information to the brain. Faced with this contradiction, the brain lapses into an unstable state characterized by alternating periods of monocular dominance that persist as long as the eyes receive discordant stimuli—the phenomenon termed binocular rivalry. This chapter introduces the phenomenon, describes major characteristics of rivalry including conditions under which it does and does not occur, summarizes what is known about the neural bases of rivalry, and concludes with a section on the role of rivalry in normal vision. Because the literature on rivalry is so large, a comprehensive review is beyond the scope of this chapter. Interested readers are referred to any of several recent reviews of this literature (Blake, 2001; Fox, 1991; Howard and Rogers, 1995). For a very thorough, upto-date bibliography on rivalry, take a look at Dr. Robert O’Shea’s reference list on the World Wide Web at http://psy.otago.ac.nz/r_oshea/br_bibliography.html. Finally, readers are invited to visit the author’s web page, which provides demonstrations of binocular rivalry and some of its major characteristics: http://www.psy. vanderbilt.edu/faculty/blake/rivalry/BR.html.

Binocular rivalry’s characteristics E R You can experience rivalry for yourself using each of the three pairs of pictures in Figure 88.1. You will need to follow the instructions in the caption to achieve the conditions for rivalry, and when you do, you will notice several of rivalry’s hallmark characteristics. First, note that your visual experience is highly dynamic, with one figure dominating perception for several seconds, only to become phenomenally suppressed in favor of the competing

figure. Sometimes you may see bits and pieces of both figures, creating an impression of a dynamic patchwork. But often you’ll see only one figure or the other for several seconds at a time, and neither figure will stay exclusively visible indefinitely (unless you have a strongly dominant eye). Note, too, that you cannot hold one eye’s view dominant at will, even when one of the competing figures is a potentially interesting picture such as a female face; the alternations in dominance and suppression evidently are obligatory and may stem from neural adaptation of the currently dominant image (Lehky, 1988; Matsuoka, 1984; Mueller, 1990). You can, however, rescue one eye’s figure from suppression prematurely by introducing an abrupt “transient” in the suppressed figure. In the laboratory, this can be accomplished by abruptly incrementing the contrast of a suppressed target (Blake et al., 1990; Wilson et al., 2001) or by introducing motion into a previously stationary, suppressed target (Walker and Powell, 1979). To demonstrate how these transient events can disrupt suppression, view rivalry between a pair of the rival figures in Figure 88.1. When one of the targets becomes suppressed, simply flick your finger in front of that figure; this maneuver will immediately restore the figure to dominance. If you were to repeatedly break suppression of that figure in this way—artificially forcing it to remain dominant—you would also find that it would stay dominant for shorter and shorter periods of time (Blake et al., 1990). When considering binocular rivalry, it is important not to confuse it with Troxler’s effect, the spontaneous fading of a visual figure that can occur when maintaining strict visual fixation. Troxler’s effect does not require discordant stimulation to the two eyes, and it occurs readily in peripheral parts of the visual field and can be observed in central vision with targets of a few degrees in visual angle or larger. Troxler’s effect is usually attributed to local retinal adaptation, whereas rivalry almost certainly arises from central neural events. It is important to distinguish Troxler’s effect from rivalry suppression when studying the disappearance of rival targets viewed in the periphery (Levelt, 1965). W T R Binocular rivalry can be instigated by differences between left- and right-eye views along almost any stimulus dimension, including contour orientation (Wade, 1974), spatial frequency (Fahle, 1982), and motion velocity (Breese, 1899; van de Grind et al., 2001).

1313

F 88.1. Provided here are three pairs of rival targets, each of which illustrates the alternations in dominance and suppression characterizing binocular rivalry. To achieve dichoptic stimulation (i.e., presentation of each figure to separate eyes), you must view these pairs either by crossing your eyes or by diverging your eyes, so that the fovea of each eye is seeing one image or the other. (Readers unable to “free-fuse” the targets may see anaglyphic examples of rivalry presented on the author’s web page: http://www.psy.vanderbilt.edu/faculty/blake/rivalry/BR.html.) While viewing the upper pair of rival targets—faces of two differ-

1314

, ,   

ent people—see if you can hold one face perceptually dominant indefinitely. Most people viewing rival targets find this to be impossible. The middle pair of rival targets were used by Wilson et al. (2001) to study dominance waves experienced as one portion of a previously suppressed figure emerges into dominance, quickly spreading to encompass the rest of the figure. To see these waves, fixate the central “bull’s-eye” portion of the figure and see what happens as the radial grating assumes dominance. The bottom pair of rival targets are computer-generated images created by David Bloom and used here with permission of the artist. (See color plate 74.)

F 88.2. When one views binocularly a scene in which one object partially occludes another nearby object, one eye will see portions of the occluded object that are invisible to the other eye. This viewing situation—illustrated schematically in the upper part of this figure—creates a local region where dissimilar images stimulate corresponding retinal areas, conditions normally sufficient for binocular rivalry. When this condition is simulated in the laboratory, the incidence of rivalry is reduced (Shimojo and Nakayama, 1990). The pair of half-images in the lower part of the figure may be used to compare binocular rivalry for a “valid” binocular con-

figuration and an “invalid” binocular configuration. If you combine the two half-images by crossing your eyes, use the middle and left-hand figures; if you combine the two half-images by uncrossing your eyes, use the middle and right-hand figures. When the two half-images are appropriately fused, the gray squares will stand out in depth from the textured background, and the partially occluded striped object will be valid in the upper part of the figure and invalid in the lower part. Compare how frequently the striped region disappears in the upper versus the lower field.

There are several conditions, however, in which dissimilar stimulation to corresponding areas of the two eyes does not seem to trigger rivalry. One of these is when the two eyes view rapidly flickering rival targets (O’Shea and Blake, 1986; O’Shea and Crassini, 1984), and another is when the two rival targets are very low in contrast, near their threshold for visibility (Liu et al., 1992). In both instances, people describe seeing the binocular superimposition of the two dissimilar targets. It has been noted that these two stimulus conditions—rapid flicker and low contrast—both favor activation of the magnocellular pathway (see Chapter 30), which could mean that rivalry is more tightly coupled to neural activity in the parvocellular pathway (Carlson and He, 2000; Ramachandran, 1991). Another interesting situation where dissimilar monocular stimulation does not yield rivalry has to do with the geome-

try of 3D space (Shimojo and Nakayama, 1990). Consider the two monocular images produced when you view a 3D scene in which one object (a gray rectangle in this example) partially occludes another (a striped rectangle), as illustrated by the drawing in the top part of Figure 88.2. In this situation, your right eye would be seeing parts of the occluded object that are invisible to your left eye. This means, in other words, that a region of the right eye’s image will contain a pattern of optical stimulation that is different from the optical stimulation falling on the corresponding region of the left eye’s image. (By the way, this geometrical consequence of looking at the world from two different perspectives was realized and discussed by Leonardo da Vinci.) Despite this conflicting stimulation to corresponding retinal areas, observers report that vision is relatively more stable when viewing pairs of images that mimic this situation, implying

:  

1315

F 88.3. Two stamps that are nearly identifcal in form but different in color. Free-fuse these two rival stamps and note the appearance of color. (See color plate 75.)

that the brain “understands” that the “conflict” is attributable to the 3D layout of objects in the world (Nakayama and Shimojo, 1990). Interestingly, when the images going to the two eyes are exchanged, observers experience more instances of binocular rivalry within this region of conflict. (Readers may experience the dependence of rivalry on scene interpretation using the pair of stereo images in the bottom part of Fig. 88.2.) These observations imply that the brain knows which eye is receiving which figure, which in turn determines whether one experiences stable fusion or unstable rivalry (see also Blake and Boothroyd, 1985). Finally, there is some evidence that different “colors” viewed by the left and right eyes do not necessarily rival each other in the same way that different forms do. This can be seen in Figure 88.3, which presents a version of one of the stimulus conditions studied by Creed (1935) in his widely cited (and sometimes misinterpreted) study of binocular fusion and binocular rivalry. Note that the two stamps are identical in form except for the numerals and words signifying the stamps’ value; the colors, however, are distinctly different. When these two images are viewed in binocular combination, however, the color differences do not alternate in dominance; instead, people describe seeing a “washedout” brown. Whatever the perceived color, it is clear—to put it in Creed’s words—that “we are not dealing with the common type of binocular rivalry in which the form that prevails brings with it the colour of its own object” (p. 383). Using these stamps, you might also try to compare the color of the numeral 1, when it is dominant, with the numeral 2, when it is dominant. Here, too, there is no sense that color changes over time, even though the form does. Again, to quote Creed, “Successive rivalry of forms is therefore not necessarily accompanied by successive rivalry of the corresponding colours” (p. 384). T  S D The alternations in dominance and suppression during rivalry are not periodic, like the oscillations of a metronome; instead, successive durations of visibility are independent, random events that

1316

, ,   

collectively conform to a gamma distribution (Fig. 88.4). In other words, one cannot predict exactly how long a given duration of dominance will last (Fox and Herrmann, 1967; Lehky, 1995; Levelt, 1965). However, it is possible to influence the overall predominance of one figure over another, where predominance is defined as the total percentage of time that a given figure is dominant during an extended viewing period. Thus, for example, a high-contrast rival figure will tend to predominate over a low-contrast one (Mueller and Blake, 1989), and this increased predominance comes about mostly because the durations of suppression of a highcontrast figure are shorter, on average, than those of a lowcontrast figure. It is as if a “stronger” stimulus manages to overcome suppression more quickly than does a “weaker” one (Blake, 1989; Fox and Rasche, 1969). When you stop and think about it, this also means that rivalry alternations should be faster, on average, between a pair of high-contrast rival targets than between a pair of low-contrast rival targets; this is, in fact, the case (Levelt, 1965). Besides contrast, other stimulus variables that “strengthen” a rival target and thereby enhance its predominance include spatial frequency (Fahle, 1982), motion velocity (Blake et al., 1998; Breese, 1899), and luminance (Levelt, 1965). A rival figure embedded in a larger meaningful context also tends to predominate over one viewed in isolation, but in this case increased predominance arises from a lengthening of the durations of dominance, not an abbreviation of suppression durations (Sobel and Blake, 2001). This dissociation between the effect of a rival figure’s context and the effect of the strength of the figure itself suggests that rivalry dynamics have multiple determinants (Blake, 2001). There are also reports in the literature that meaningful or familiar rival targets predominate in rivalry over less meaningful or unfamiliar ones (for a review of this literature, see Walker, 1978; for an alternative interpretation of the role of meaning in rivalry, see Blake, 1988). It would be informative to replicate these results and, moreover, to learn whether increases in predominance with meaning, if reliably demonstrable, come about through lengthened dominance durations. To the extent that meaning influences other aspects of visual perception (Raftopoulos, 2001), it would not be surprising to find that meaning influences dominance periods of rivalry, for the dominance phases of rivalry appear to be functionally equivalent to normal monocular vision (Fox, 1991). The potency of global context to influence the temporal dynamics of rivalry could reflect the involvement of higherlevel visual processes in rivalry. Is there other evidence for the involvement of such processes? There are several published studies pointing to a role for attention in the control of rivalry dynamics. Several decades ago, Lack (1978) showed that observers could be trained to prolong one eye’s view during rivalry without resorting to tricks such as moving

F 88.4. When an observer presses buttons to indicate successive periods of dominance of two rival targets, those successive durations are randomly distributed, independent variables (Levelt, 1965). The upper diagram shows a representative time line of successive periods of exclusive dominance of a vertical rival grating (viewed by the left eye in this case) and periods of exclusive dominance of a horizontal rival grating (viewed by the right eye). Intermixed among periods of exclusive dominance are occasional periods of “mixed” dominance during which portions of both rival

gratings are visible in a patchwork mosaic pattern. The incidence of mixed dominance tends to be greater with rival targets larger in angular subtense (Meenes, 1930). When dominance durations measured during extended tracking periods are plotted as a frequency histogram (lower portion of the figure), that distribution is well fit by a gamma distribution (solid line in the histogram). In general, the gamma distribution provides a close fit to rivalry alternation data when parameters are adjusted for individual differences (Fox and Herrmann, 1967; Logothetis, 1998).

the eyes in a manner that would favor that view. Instead, Lack’s observers purportedly directed attention to the favored target and thereby prolonged its visibility. It should be noted that rivalry alternations were not abolished under these conditions; despite focused attention, spontaneous reversals in dominance still occurred, albeit less frequently. More recently, Ooi and He (1999) showed that directing attention to a region of the visual field where a rivalry target is currently suppressed boosts the potency of visual motion at that region of the field to break suppression of that target. What happens when one views multiple rival patterns distributed throughout the visual field? Take a look at the rival display presented in Figure 88.5, a pair of arrays of black and white Gaussian (i.e., blurred) blobs, with each white blob in one array pitted against a black blob in the other array. View the two arrays dichoptically using the free-fusion technique, hold your gaze steadily on the small fixation cross, and concentrate on the pattern of dominance throughout the visual field. Note how often all the black blobs or all the white blobs are simultaneously dominant. When observers actually track these periods of simultaneous dominance, the total incidence turns out to be greater than what would be

predicted if the individual blobs were rivaling independently on their own (Logothetis, 1998). There is a tendency, in other words, for dominance periods of coherent patterns to become perceptually entrained. This tendency is even more pronounced in the color rival patterns shown in Figure 88.6, a variation of the display created by Diaz-Caneja (1928; see Alais et al., 2000). Here the addition of color further encourages grouping according to coherence, as you can confirm by comparing the rivalry associated with the gray-scale version and the color version. In both Figures 88.5 and 88.6, periods of coherent dominance can be achieved only by very specific combinations of left-eye and right-eye components that are dominant simultaneously. This means, then, that one eye alone cannot be responsible entirely for dominance at any given moment. Instead, dominance may consist of a patchwork of visible features (Blake et al., 1992; Meenes, 1930), in this case collated from left- and right-eye views. Interocular grouping during rivalry has been nicely documented by Kovács et al. (1997), and it has been studied by others as well (Alais and Blake, 1999; Dörrenhaus, 1975).

:  

1317

F 88.5. Rival display consisting of spatially distributed rival targets. Each eye’s view consists of an array of blurred circles (Gaussian blobs) differing in contrast polarity between left- and

right-eye views. Free-fuse these two targets and note how often all blobs of a given contrast polarity are simultaneously dominant. (This display is a version of one described by Logothetis, 1998.)

One other intriguing characteristic of rivalry deserves mention, one having to do with the appearance of rival figures during transitions from suppression to dominance. Looking again at rivalry produced by the rival pair in the second row of Figure 88.1, pay particular attention to the emerging appearance of the rings as they assume dominance. You will probably notice that dominance originates

within a restricted region of a ring and spreads from there to encompass the entire figure. These “waves” of dominance are typical of rivalry produced by all sorts of rival figures; they imply the existence of neural “cooperativity” wherein interconnections among neighboring ensembles of neurons promote the spread of activation over spatially extended regions of cortical tissue. Wilson et al. (2001) estimated the

F 88.6. Two versions of the well-known display of DiazCaneja (1928). Upon viewing the gray-scale version (upper pair of rival targets), observers often see a complete bulls-eye pattern or a complete pattern of vertical lines. This implies that portions of

each eye’s target are simultaneously dominant, presumably promoted by the spatial grouping that produces a coherent figure. This tendency toward grouping is even stronger in the color version of this figure shown at the bottom. (See color plate 76.)

1318

, ,   

speed with which dominance waves travel around the circumference of circular rival targets like those pictured in the second row of Figure 88.1, coming up with an average wave speed of about 4 degrees of visual angle per second. This estimated wave speed was even faster when the contours forming the rival target were collinear (i.e., the target was a concentric grating), presumably because waves of dominance travel more effectively along collinear contours. Wave speed (expressed in units of visual degrees per second) was also faster for larger rival targets whose contours were imaged in more peripheral regions of the retina. Expressed in units of cortical tissue (not degrees of visual angle), however, wave speed remained constant with retinal eccentricity, implying that these waves are being generated in a visual area whose magnification factor matches that for visual area V1 (i.e., the exaggerated neural representation of the fovea in the visual cortex). With these spatial and temporal characteristics of binocular rivalry in mind, we turn next to a consideration of the source of interest in rivalry. Why have vision scientists been fascinated with this phenomenon for almost two centuries, and why in recent years have neuroscientists become intrigued with rivalry?

Binocular rivalry and visual awareness Binocular rivalry was first systematically described by Sir Charles Wheatstone in his famous monograph on binocular stereopsis (Wheatstone, 1838). But it was Hermann von Helmholtz (1866/1925) and William James (1891) who brought rivalry to the forefront of psychology by treating it as a paradigm case for the study of attention. The noted British neurophysiologist Sir Charles Sherrington (1906) also exploited rivalry to make the case that the brain analyzed left- and right-eye views independently and merged the two views only after each had been fully elaborated. Expressed in contemporary terms, binocular rivalry was construed as a high-level cognitive activity involving competition between conflicting object descriptions. This conceptualization was maintained throughout much of the twentieth century (Walker, 1978), and it led to some unusual ideas about rivalry’s practical applications. For example, some psychologists thought that rivalry might provide a revealing indirect tool for assessing intelligence (Crain, 1961), personality (Kohn, 1960), and a tendency to perceive violence (Toch and Schulte, 1961). Those psychometric applications never really caught on, although recent work by Pettigrew (2001) suggests that the issue is not dead. What has caught on during the past decade or so is the idea that rivalry may provide a powerful tool for studying the neural concomitants of visual awareness (Blake, 1997; Crick and Koch, 1998; Logothetis, 1998). After all, a complex suprathreshold rival pattern can disappear from

visual awareness for seconds at a time, even though that pattern remains imaged on the eye; from a perceptual standpoint, a suppressed pattern temporarily ceases to exist. It should be possible, then, to find fluctuations in neural activity correlated with the intermittent appearance and disappearance of that pattern. And while the search for those neural events has not produced unambiguous answers, the initial results, summarized in the next paragraph, are tantalizing. Brain imaging studies have discovered modulations in hemodynamic responses and, by inference, neural activity in visual areas of the brain that are tightly correlated with fluctuations in dominance and suppression during rivalry. These response modulations are observed in brain areas as early as primary visual cortex (Lee and Blake, 2002; Polansky et al., 2000; Tong and Engel, 2001), and they are even more pronounced in higher cortical areas in the temporal lobe (Tong et al., 1998) and in the parietal lobe (Lumer et al., 1998). Visual evoked potential measurements (Brown and Norcia, 1997) and neuromagnetic responses (Srinivasan et al., 1999) corroborate the involvement of occipital lobe regions in rivalry but do not pinpoint exactly where in the occipital lobe neural activity is varying with perception. Single-unit studies have recorded action potentials from neurons in awake monkeys trained to report rivalry alternations, and here the evidence points to an increasingly tight coupling between neural firing rate and perception as one moves from primary visual areas to higher cortical regions in the temporal lobe (Leopold and Logothetis, 1996; Logothetis and Schall, 1989; Sheinberg and Logothetis, 1997). Besides neural firing rate, temporal patterns of neural activity also may be correlated with dominance and suppression states during rivalry (Fries et al., 1997), a possibility that could tie rivalry to the larger issue of the role of synchronized neural discharges in perceptual grouping (Engel et al., 1999). Although admittedly more indirect, perceptual data can also be used to draw inferences about the neural operations mediating binocular rivalry. Reiterating a point made earlier, a normally visible monocular stimulus is temporarily erased from sight during rivalry. To what extent does that stimulus remain visually effective despite its absence from phenomenal awareness? Answers to this question can shed light on the neural locus of rivalry relative to other visual processes (Blake, 1995). If suppression blocks input to a neural processing stage responsible for a given aspect of vision, that blockage should be reflected in visual perception. To illustrate this strategy in operation, consider the well-known motion aftereffect (MAE), that is, the illusory perception that a stationary object is moving produced by prolonged exposure, or adaptation, to real motion (see Chapter 83 for more on the MAE). It has been found that the MAE can be generated even when the adaptation motion itself is suppressed from awareness for a substantial portion of the adaptation

:  

1319

phase owing to binocular rivalry (Lehmkuhle and Fox, 1976; O’Shea and Crassini, 1981). The enduring effectiveness of motion signals in the face of suppression implies that the neural events underlying adaptation transpire prior to the site at which suppression occurs (or, alternatively, that the two occur within parallel pathways). In contrast, the cognitive process termed picture priming is disrupted by suppression, as evidenced by the failure of a normally effective picture presented to a suppressed eye to facilitate performance on a subsequent picture-naming task (Cave et al., 1998). This finding implies that suppression temporarily cuts off the flow of information to those neural mechanisms responsible for registering information crucial for object recognition. In general, suppression provides a potentially powerful inferential “landmark” for pinpointing the relative sites of neural processes within the hierarchical stages of visual information processing. Perceptual measures also disclose other potentially revealing properties of the neural events underlying rivalry. It is well known, for example, that visually presented probe targets are harder to detect when those probes are presented to an eye during suppression, compared to detection during dominance (Wales and Fox, 1970). Evidently, the neural events mediating suppression of a given rival figure generalize to other, new information presented to the same region of the eye viewing that suppressed figure. However, the loss in sensitivity accompanying suppression phases is surprisingly modest in view of the fact that during suppression a complex high-contrast figure is being erased from conscious awareness for several seconds at a time. It remains to be learned how such a profound loss of vision can be accomplished by neural operations that produce only a very modest loss in overall visual sensitivity.

Is rivalry occurring all the time? Binocular rivalry admittedly is experienced under rather artificial conditions that have no resemblance to those typifying everyday visual experience. Indeed, one leading authority on visual perception has characterized the conditions producing rivalry as “optical trickery” (Gibson, 1966). However, an enduring idea in vision science is that rivalry is occurring all the time, including during everyday viewing. Alternations in perception are not seen, the argument goes, because normally the two eyes are viewing the same visual scene and, therefore, these alternations are not visually conspicuous. Known as suppression theory, this idea, odd as it may seem, has been seriously advocated by a number of vision scientists over the years (Asher, 1953; Hochberg, 1964; Wolfe, 1986). How can we know whether normal vision entails chronic, inconspicuous binocular rivalry? Recall from the previous section that periods of suppression are accompanied by

1320

, ,   

decreased visual sensitivity relative to an eye’s sensitivity during dominance. If binocular vision always involves alternating suppression, even when the two eyes view the same scene, losses in visual sensitivity should be a symptom of that alternating suppression. Of course, during normal vision one would not know which eye was temporarily suppressed, but by chance alone, a series of brief test probes presented to one eye should find that eye in the suppression state approximately half of the time. Thus, probe sensitivity should suffer compared to a condition where viewing is monocular and, therefore, dominance is ensured. When these kinds of comparisons are made, however, normal single vision shows no evidence of being accompanied by intermittent losses in visual sensitivity (Blake and Camisa, 1978; O’Shea, 1987). For this and related reasons, it is generally recognized that binocular fusion takes precedence over binocular rivalry. Only when the two eyes view explicitly conflicting images does vision lapse into reciprocal periods of dominance and suppression. This is an important fact to know, for it implies that the neural concomitants of rivalry should be measurable only when a viewer is actually experiencing rivalry and not when experiencing stable binocular single vision. While suppression cannot be the sole mechanism of binocular single vision, there are still everyday viewing situations where suppression probably does contribute to perception. It is true that left and right foveas rarely receive dissimilar stimulation for any length of time; the oculomotor system seeks to correct this situation by altering the vergence angle until matching features are imaged on the two foveas. There are, however, locations on the two retinas where dissimilar monocular images strike corresponding retinal areas, an inevitable consequence of the geometry of binocular vision. Objects located well in front of or well behind the horopter do cast images on distinctly different areas of the two eyes, and the resulting disparities will be too large for the stereoscopic system to resolve. Yet one ordinarily does not experience the consequences—confusion or diplopia—of this dissimilar monocular stimulation. Binocular single vision, then, may be accomplished by two processes, binocular fusion and binocular suppression (Ono et al., 1977). The potent inhibitory process revealed during rivalry, in other words, may contribute to normal binocular single vision. For some individuals with chronic eye misalignment (i.e., the condition called strabismus, described in more detail in Chapters 12 and 14), suppression may indeed be operating all the time throughout the entire binocular visual field. After all, the stimulus conditions eliciting rivalry suppression and strabismic suppression are comparable: corresponding areas of the two eyes receive discrepant inputs. (In effect, the eyes “disagree” about the nature of the stimulus located at a given region of visual space.) Indeed, this assumption of

similarity of mechanisms appears in classic texts on ophthalmology (e.g., Duke-Elder, 1949). Moreover, there is some evidence suggesting that the two forms of suppression are comparable psychophysically (Blake and Lehmkuhle, 1976; Fahle, 1983; Holopigian, 1989). Inspired by this possible link, several laboratories are utilizing visual evoked potentials to examine the development of rivalry and binocular interactions in very young infants (Birch and Petrig, 1996; Brown et al., 1999), one motive being to relate those findings to the onset of strabismic suppression.

Is rivalry equivalent to other forms of multistable perception? Binocular rivalry is not the only phenomenon where perceptual experience fluctuates even though the physical input remains unchanged, a behavior sometimes called multistability. Other compelling examples include bistable figures (wherein an ambiguous figure fluctuates between alternative figure/ground organizations or between alternative perspective interpretations; see Logothetis, 1998), monocular rivalry (in which two physically superimposed patterns dissimilar in color and form rival for dominance; see Campbell et al., 1973), and motion-induced blindness (wherein a normally visible stationary pattern is erased from awareness by a superimposed global moving pattern; see Bonneh et al., 2001). In all these cases, binocular rivalry included, the brain is confronted with conflicting or ambiguous information about the nature of an object at a given location in visual space. And descriptively speaking, the brain resolves this conflict by “entertaining” alternative perceptual interpretations over time. Several investigators have commented on the similarities in temporal dynamics among these various forms of perceptual instability and have concluded, therefore, that all may stem from common neural operations (Andrews and Purves, 1997; Pettigrew, 2001). This commonality certainly deserves closer examination, perhaps using some of the established techniques (e.g., test probes) that have been used to characterize binocular rivalry’s properties.

Conclusions Binocular rivalry is inherently fascinating: visual experience fluctuates between alternative perceptual interpretations even though the input to vision remains unchanged. It is not surprising, then, that rivalry has enjoyed enduring interest over the past century and a half. But it’s really only during the past decade that the study of rivalry has become widely viewed as a serious, powerful “instrument” to be exploited by neuroscience in its attack on the mind/brain problem. Rivalry’s increased popularity has been accompanied by a number of clever psychophysical studies documenting the involvement of context, attention, and meaning in the pro-

motion of dominance. Indeed, concrete evidence of rivalry’s importance in visual neuroscience is the inclusion of a chapter on this subject in this volume.

Acknowledgment Preparation of this chapter was supported, in part, by a research grant from the National Institutes of Health (EY013358). REFERENCES Alais, D., and R. Blake, 1999. Grouping visual features during binocular rivalry, Vis. Res., 39:4341–4353. Alais, D., R. P. O’Shea, C. Mesana-Alais, and I. G. Wilson, 2000. On binocular alternation, Perception, 29:1437–1445. Andrews, T. J., and D. Purves, 1997. Similarities in normal and binocularly rivalrous viewing, Proc. Natl. Acad. Sci. USA, 94:9905–9908. Asher, H., 1953. Suppression theory of binocular vision, Br. J. Ophthalmol., 37:37–49. Birch, E. E., and B. Petrig, 1996. FPL and VEP measures of fusion, stereopsis and stereoacuity in normal infants, Vis. Res., 36:1321–1327. Blake, R., 1988. Dichoptic reading: the role of meaning on binocular rivalry, Percept. Psychophys., 44:133–141. Blake, R., 1989. A neural theory of binocular rivalry, Psychol. Rev., 96:145–167. Blake, R., 1995. Psychoanatomical strategies for studying human vision, in Early Vision and Beyond (T. Papathomas, C. Chubb, E. Kowler, and A. Gorea, eds.), Cambridge, MA: MIT Press, pp. 17–25. Blake, R., 1997. What can be perceived in the absence of visual awareness? Curr. Dir. Psychol. Sci., 6:157–162. Blake, R., 2001. A primer on binocular rivalry, including current controversies, Brain Mind, 2:5–38. Blake, R., and K. Boothroyd, 1985. The precedence of binocular fusion over binocular rivalry, Percept. Psychophys., 37:114–124. Blake, R., and J. Camisa, 1978. Is binocular vision always monocular? Science, 200:1497–1499. Blake, R., and S. Lehmkuhle, 1976. On the site of strabismic suppression, Invest. Ophthalmol., 15:660–663. Blake, R., R. P. O’Shea, and T. J. Mueller, 1992. Spatial zones of binocular rivalry in central and peripheral vision, Vis. Neurosci., 8:469–478. Blake, R., D. Westendorf, and R. Fox, 1990. Temporal perturbations of binocular rivalry, Percept. Psychophys., 48:593–602. Blake, R., K. Yu, M. Lokey, and H. Norman, 1998. Binocular rivalry and visual motion, J. Cogn. Neurosci., 10:46–60. Bonneh, Y. S., A. Cooperman, and D. Sagi, 2001. Motion-induced blindness in normal observers, Nature, 411:798–801. Breese, B. B., 1899. On inhibition, Psychol. Monogr., 3:1–65. Brown, R. J., T. R. Candy, and A. M. Norcia, 1999. Development of rivalry and dichoptic masking in human infants, Invest. Ophthalmol. Vis. Sci., 40:3324–3333. Brown, R. J., and A. M. Norcia, 1997. A method for investigating binocular rivalry in real-time with the steady-state VEP, Vis. Res., 37:2401–2408. Campbell, F. W., A. S. Gilinsky, E. R. Howell, L. A. Riggs, and J. Atkinson, 1973. The dependence of monocular rivalry on orientation, Perception, 2:123–125.

:  

1321

Carlson, T. A., and S. He, 2000. Visible binocular beats from invisible monocular stimuli during binocular rivalry, Curr. Biol., 10: 1055–1058. Cave, C., R. Blake, and T. McNamara, 1998. Binocular rivalry disrupts visual priming, Psychol. Sci., 9:299–302. Crain, K., 1961. Binocular rivalry: its relation to intelligence, and general theory of its nature and physiological correlates, J. Gen. Psychol., 64:259–283. Creed, R. S., 1935. Observations on binocular fusion and rivalry, J. Physiol., 84:381–392. Crick, F., and C. Koch, 1998. Consciousness and neuroscience, Cereb. Cortex, 8:97–107. Diaz-Caneja, E., 1928. Sur l’alternance binoculaire, Ann. Oculist, October:721–731. Dörrenhaus, W., 1975. Musterspezifischer visueller wettstreit, Naturwissenschaften, 62:578–579. Duke-Elder, W. S., 1949. Textbook in Ophthalmology, St. Louis: Mosby. Engel, A. A. K., P. Fries, P. Konig, M. Brecht, and W. Singer, 1999. Temporal binding, binocular rivalry and consciousness, Consciousness Cogn., 8:128–151. Fahle, M., 1982. Cooperation between different spatial frequencies in binocular rivalry, Biol. Cybern., 44:27–29. Fahle, M., 1983. Non-fusable stimuli and the role of binocular inhibition in normal and pathologic vision, especially strabismus, Doc. Ophthalmol., 55:323–340. Fox, R., 1991. Binocular rivalry, in Binocular Vision and Psychophysics (D. M. Regan ed.), London: Macmillan, pp. 93–110. Fox, R., and J. Herrmann, 1967. Stochastic properties of binocular rivalry alternations, Percept. Psychophys., 2:432–436. Fox, R., and F. Rasche, 1969. Binocular rivalry and reciprocal inhibition, Percept. Psychophys., 5:215–217. Fries, P., P. R. Roelfsema, A. K. Engel, P. Konig, and W. Singer, 1997. Synchronization of oscillatory responses in visual cortex correlates with perception in interocular rivalry, Proc. Natl. Acad. Sci. USA, 94:12699–12784. Gibson, J. J., 1966. The Senses Considered as Perceptual Systems, Boston: Houghton Mifflin. Helmholtz, H. von., 1866/1925. Treatise on physiological optics ( J. P. Southall ed.), New York: Dover. Hochberg, J., 1964. Depth perception loss with local monocular suppression: a problem in the explanation of stereopsis, Science, 145:1334–1335. Holopigian, K., 1989. Clinical suppression and binocular rivalry suppression: the effects of stimulus strength on the depth of suppression, Vis. Res., 29:1325–1334. Howard, I. P., and B. J. Rogers, 1995. Binocular Vision and Stereopsis, New York: Oxford University Press. James, W., 1891. The Principles of Psychology, London: Macmillan. Kohn, H., 1960. Some personality variables associated with binocular rivalry, Psychol. Rep., 10:9–13. Kovacs, I., T. V. Papathomas, M. Yang, and A. Fehér, 1997. When the brain changes its mind. Interocular grouping during binocular rivalry, Proc. Natl. Acad. Sci. USA, 93:15508–15511. Lack, L., 1978. Selective Attention and the Control of Binocular Rivalry, The Hague: Mouton. Lee, S. H., and R. Blake, 2002. V1 activity is reduced during binocular rivalry, J. Vis., 2:618–626 (online at: http://journalofvision.org/2/9/4/). Lehky, S. R., 1988. An astable multivibrator model of binocular rivalry, Perception, 17:215–228. Lehky, S. R., 1995. Binocular rivalry is not chaotic, Proc. R. Soc. Lond. B, 259:71–76.

1322

, ,   

Lehmkuhle, S., and R. Fox, 1976. Effect of binocular rivalry suppression on the motion aftereffect, Vis. Res., 15:855–859. Leopold, D., and N. Logothetis, 1996. Activity changes in early visual cortex reflect monkeys’ percepts during binocular rivalry, Nature, 379:549–553. Levelt, W., 1965. On Binocular Rivalry, Soesterberg, the Netherlands: Institute for Perception RVO-TNO. Liu, L., C. W. Tyler, and C. M. Schor, 1992. Failure of rivalry at low contrast: evidence of a suprathreshold binocular summation process, Vis. Res., 32:1471–1479. Logothetis, N. K., 1998. Single units and conscious vision, Philos. Trans. R. Soc. Lond. B, 353:1801–1818. Logothetis, N. K., and J. D. Schall, 1989. Neuronal correlates of subjective visual perception, Science, 245:761–763. Lumer, E. D., K. Friston, and G. Rees, 1998. Neural correlates of perceptual rivalry in the human brain, Science, 280:1930– 1934. Matsuoka, K., 1984. The dynamic model of binocular rivalry, Biol. Cybern., 49:201–208. Meenes, M., 1930. A phenomenological description of retinal rivalry, Am. J. Psychol., 42:260–269. Mueller, T. J., 1990. A physiological model of binocular rivalry, Vis. Neurosci., 4:63–73. Mueller, T. J., and R. Blake, 1989. A fresh look at the temporal dynamics of binocular rivalry, Biol. Cybern., 61:223–232. Nakayama, K., and S. Shimojo, 1990. Da Vinci stereopsis: depth and subjective occluding contours from unpaired image points, Vis. Res., 30:1811–1825. Ono, H., R. Angus, and P. Gregor, 1977. Binocular single vision achieved by fusion and suppression, Percept. Psychophys., 21:513–521. Ooi, T. L., and Z. J. He, 1999. Binocular rivalry and visual awareness: the role of attention, Perception, 28:551–574. O’Shea, R. P., 1987. Chronometric analysis supports fusion rather than suppression theory of binocular vision, Vis. Res., 27:781–791. O’Shea, R., and R. Blake, 1986. Dichoptic temporal frequency differences do not lead to binocular rivalry, Percept. Psychophys., 39:59–63. O’Shea, R. P., and B. Crassini, 1981. Interocular transfer of the motion aftereffect is not reduced by binocular rivalry, Vis. Res., 21:801–804. O’Shea, R. P., and B. Crassini, 1984. Binocular rivalry occurs without simultaneous presentation of rival stimuli, Percept. Psychophys., 36:266–276. Pettigrew, J. D., 2001. Searching for the switch: neural bases for perceptual rivalry alternations, Brain Mind, 2:85–118. Polansky, A., R. Blake, J. Braun, and D. Heeger, 2000. Neuronal activity in human primary visual cortex correlates with perception during binocular rivalry, Nat. Neurosci., 3:1153–1159. Raftopoulos, A., 2001. Is perception informationally encapsulated? The issue of the theory-ladenness of perception, Cogn. Sci., 25:423–451. Ramachandran, V. S., 1991. Form, motion, and binocular rivalry, Science, 251:950–951. Sheinberg, D. L., and N. K. Logothetis, 1997. The role of temporal cortical areas in perceptual organization, Proc. Natl. Acad. Sci. USA, 94:3408–3413. Sherrington, C. S., 1906. Integrative Action of the Nervous System, New Haven, CT: Yale University Press. Shimojo, S., and K. Nakayama, 1990. Real world occlusion constraints and binocular rivalry, Vis. Res., 30:69–80.

Sobel, K., and R. Blake, 2001. How context influences binocular rivalry predominance. Presented at the annual meeting of the Visual Sciences Society, Sarasota, FL. Srinivasan, R., D. P. Russell, G. M. Edelman, and G. Tononi, 1999. Increased synchronization of neuromagnetic responses during conscious perception, J. Neurosci., 19:5435–5448. Toch, H. H., and R. Schulte, 1961. Readiness to perceive violence as a result of polic training, Br. J. Psychol., 52:389–393. Tong, F., and S. Engel, 2001. Interocular rivalry revealed in the cortical blind-spot representation, Nature, 411:195–199. Tong, F., K. Nakayama, J. T. Vaughan, and N. Kanwisher, 1998. Binocular rivalry and visual awareness in human extrastriate cortex, Neuron, 21:753–759. van de Grind, W. A., P. van Hof, M. J. van der Smagt, and A. J. Verstraten, 2001. Slow and fast visual motion channels have independent binocular-rivalry stages, Proc. R. Soc. Biol. Sci., 268: 437–443.

Wade, N. J., 1974. The effect of orientation in binocular contour rivalry of real images and afterimages, Percept. Psychophys., 15: 227–232. Wales, R., and R. Fox, 1970. Increment detection thresholds during binocular rivalry suppression, Percept. Psychophys., 8:90–94. Walker, P., 1978. Binocular rivalry: central or peripheral selective processes? Psychol. Bull., 85:376–389. Walker, P., and D. J. Powell, 1979. The sensitivity of binocular rivalry to changes in the nondominant stimulus, Vis. Res., 19: 247–249. Wheatstone, C., 1838. On some remarkable, and hitherto unobserved, phenomena of binocular vision, Philos. Trans. R. Soc. Lond., 128:371–394. Wilson, H. R., R. Blake, and S.-H. Lee, 2001. Dynamics of travelling waves in visual perception, Nature, 412:907–910. Wolfe, J., 1986. Stereopsis and binocular rivalry, Psychol. Rev., 93:269–282.

:  

1323

89

Sensorimotor Transformation in the Posterior Parietal Cortex HANSJÖRG SCHERBERGER AND RICHARD A. ANDERSEN

T   cortex (PPC) of the primate brain is an important structure for sensorimotor integration. One of its roles is the formation of intentions, or high-level plans for movement. The PPC can be subdivided into several subregions that represent maps of intentions for the planning of saccadic eye movements, reach movements, and grasping movements. These intention-related regions in the PPC appear to participate in the multisensory integration and coordinate transformations necessary for the generation of movements, and these functions seem to be facilitated by a unique distributed code for space representation. In at least two regions, the response fields of neurons are coded in eye-centered coordinates, independent of the sensory input modality (vision or audition) and the motor action that was performed (reach or saccade). Moreover, these retinal response fields in the PPC are gain-modulated by the position of the eye, head, and limb. Hence, space is not coded in a single-defined spatial reference frame in the PCC, but in a distributed fashion that allows other groups of neurons to read out spatial target locations in a variety of reference frames. The PPC also seems to participate in decision processes for movement generation, consistent with its role in sensorimotor transformation. Adaptation processes may constantly optimize the alignment of neural representations in the PPC, and the circuits are even flexible enough to allow the representation of new stimuli when they become relevant for behavior. Knowing how intentions are encoded in the PPC could lead to a potential medical application. Intentions could be used to control a neural prosthesis for paralyzed patients. Such a device would read out the activity of PPC neurons, decode movement intentions with computer algorithms, and use the predictions to control an external device such as a cursor on a computer screen or a robotic limb. In preliminary investigations in healthy monkeys, the number of parietal recording sites needed to operate such a device has been estimated using single neurons and local field potentials. Recently, the intended movement activity in the PPC of monkeys was used to position a cursor on a computer screen without the actual movement of the monkey’s arm. This was obtained without extensive training, which strongly suggests that the neural signals in PPC are

1324

indeed highly cognitive and represent high-level plans for movement.

Coding of intention in the PPC We define intention as a high-level plan for a movement that specifies the type and goal of the movement, such as the wish “I want to pick up my cup of tea.” Such a high-level movement plan would not necessarily contain all the details of the movement, such as the movement path, muscle activation patterns, and joint angles, but it would encode the end-point location of the movement and what kind of movement is to be made. Intentions are derived from the integration of sensory input and may occur at the beginning of a sequence of ever more specific movement plans along the sensorimotor pathway. In the example of picking up a cup of tea, necessary specifications include which arm to use, the movement trajectory and speed, and the details of the muscle force pattern. These movement specifications may be included in the movement plan closer to the motor output stage and may not be present at the earlier, or higher-level, stages of movement planning. Evidence that the PPC is involved in high-level cognitive functions for sensorimotor transformation first came from the observation of neurological deficits after PPC lesions (Balint, 1909). Patients with PPC lesions do not have a primary sensory or motor deficit; however, they have difficulties in estimating the location of stimuli in space or may be unable to plan movements appropriately (Geshwind and Damasio, 1985). The observations suggest that a disconnection occurs between the sensory and motor systems in the sensorimotor pathway (Goodale and Milner, 1992). Human functional magnetic resonance imaging (fMRI) experiments and monkey electrophysiological recordings further supported the concept that the PPC is neither strictly sensory nor motor, but encodes high-level cognitive functions related to action (Andersen, 1987; Goodale and Milner, 1992; Mountcastle et al., 1975). The monkey is a particularly good model for the study of the PPC, since its sophisticated eyehand coordination is similar to that of humans and the PPC in both species seems to perform similar functions (Connolly et al., 2000; DeSouza et al., 2000; Rushworth et al., 2001).

a Delayed Reach Task Memorize target

Reach to target

Wait

Time

b

c 80 Hz 60 40 20

120 Hz 2.7 s

F 89.1. Spatial tuning of reach-related activity in PPC in a delayed reach task to separate sensory from motor components of behavior. a, Paradigm: animals memorized the location of a briefly flashed visual target, then waited in complete darkness for a go signal and made a reach to the remembered target location. b, Activity of a typical PPC cell with reach-related activity during stimulus presentation, waiting, and movement period of the task. Activity is maximal in the right-down direction for this cell. Each histogram depicts the cell’s activity in one of eight tested reach

directions (white arrows; left-down direction occluded). Each histogram shows a spike density histogram representing the average action potential firing rate of all trials in the particular reach direction, and the short horizontal bars indicate the time of the target flash and the reach. c, Spatial tuning of the average firing rate during the waiting period (same cell as in b). Mean firing rate is plotted as amplitude along the movement direction, illustrating the strongest responses for right-down. (Data from Batista et al., 1999.)

P A   PPC For separating sensory from motor components of behavior, the so-called memory task has been particularly useful (Hikosaka and Wurtz, 1983). In this task, a subject is first cued to the location of a movement by a briefly flashed stimulus, but must withhold the movement response until a go signal occurs (Fig. 89.1A). A typical PPC cell shows a burst of activity during the cue and at the movement period, indicating its relation to both sensory and motor components of the movement. During the memory period, cells in many parietal areas are also active, even in the dark (Gnadt and Andersen, 1988; Snyder et al., 1997). Figures 89.1B and 89.1C show the activity of a PPC neuron while the animal performed a delayed-reach task to targets in eight different directions (Fig. 89.1B). The histogram corresponding to each movement direction depicts the average firing rate of the neuron during the trial. The activity of the cell strongly increased during stimulus presentation, the waiting period, and the movement period of the task—in this example most strongly, in the right-down

direction. The increased activity during sensory, planning, and movement periods of the trial indicates that the PPC is neither a purely sensory nor a purely motor area but is involved in the high-level planning of movements, consistent with a role of this area in sensorimotor transformations. The neuron in Figure 89.1 shows strongest activity for stimuli and movements down and to the right while being essentially silent for movements in the opposite direction. This directional tuning of activity is illustrated in the polar plot in Figure 89.1C, with the mean firing rate during the waiting period plotted as amplitude along the movement direction. Directional tuning is very common for many PPC neurons, with different neurons coding for different preferred directions. Hence, the combined activity of many neurons can code the direction of movements quite precisely. To demonstrate that the activity during the delay period does not simply represent the sensory memory of the target, we used a paradigm in which the animals planned movements to two stimuli. For example, it has been shown that

  :  

1325

a

Delayed reach task cue into RF cue out of RF 80 Hz

b

Intervening reach task cue first in RF, then out of RF

c

Intervening reach task cue first out of RF, then in RF

F 89.2. Example of a PRR neuron during the intervening reach task. Each panel in a–c shows, from top to bottom: timing of the cue stimulus into ( filled bar) or out of (open bar) the response field (RF); spike rasters of 10 trials; spike density function (using a triangular kernel); timing of the button presses (horizontal bars) in one representative trial; and the acquired target ( filled: in RF; open: out of RF). Vertical bar: calibration of firing rate. a, Delayed reach task with strong activity of the cell during the stimulus, delay, and movement periods when movements are planned inside the RF (left panel) and no activity when planned out of the RF (right panel). b, Activity of the same cell during an intervening reach task, when the first cue was shown in the RF and a second cue was presented out of the RF. c, Same cell with an intervening reach task, with the first cue out of the RF and the second cue in the RF. The cell was active when a reach was planned to the target location in the RF, but not when the animal was remembering that location and planning a reach to a location out of the RF. (Modified from Batista and Andersen, 2001.)

the delay period activity of neurons in the lateral intraparietal area (LIP) represents only the next planned eye movement, even though the animal had to hold two cued locations in memory (Mazzoni et al., 1996a). More recently, a similar result was also found for reach movements in the parietal reach region (PRR) (Batista and Andersen, 2001). Figure 89.2 shows a typical PRR neuron during this exper-

1326

, ,   

iment. In the delayed-reach task (Fig. 89.2A), the cell showed strong activity during the stimulus, delay, and movement periods when the movement was to a target inside of the response field of the cell (left panel) and no activity when the movement was to a target outside of the response field (right panel ). On randomly chosen trials, an intervening reach task was used (Figs. 89.2B, C ). In one task (B), a cue was first shown within the response field of the cell, and the animal began to plan, but not execute, a reach movement to that target. The activity of the cell was highly elevated. However, a second target was briefly shown outside the response field, and the animal had to change his plan to reach to the second target first. During this period, the cell was not active. After reaching to the second target, the animal had to reach to the remembered location of the first target that had been presented within the response field, and the cell became vigorously active again. The cell was active when a reach movement was planned to the target location within the response field of the cell, but was not active when the animal was remembering that location but planning a reach elsewhere. Corresponding results were found for the intervening reach task when the first cue was presented outside of the response field and the second in the response field of the cell (Fig. 89.2C). The delay period activity of PRR neurons therefore represents only the next planned reach movement, even though the animal had to remember two cued locations. The finding that nearly all PRR cells showed this behavior in these double movement task experiments rules out the coding of sensory memory in the delay period activity for most PRR neurons. This memory is most likely represented elsewhere or in a very small subpopulation of PRR cells. T E  P A Further evidence supporting the view that the PPC is involved in sensorimotor transformations comes from studies that elucidate the dynamic evolution of PPC activity during the task, changing in nature from sensory to cognitive to motor with the evolution of the task. For example, it was demonstrated in a delayed eye movement task (Platt and Glimcher, 1999) that the early activity of LIP neurons varied as a function of the reward probability or the probability that a stimulus location became a saccade target. However, during later task periods, the cells coded only the direction of the upcoming eye movement. In another study of LIP (Breznen et al., 1999; Sabes et a., 2002), monkeys were trained to make eye movements to specific locations cued on an object, and the object was rotated between the extinction of the cue and the saccade. At the beginning of the trial, LIP cells were found to carry information about the location of the cue and about the orientation of the object, both of which are important to perform the task. However, near the time of the eye movement, the same neurons encoded the direction of the

intended movement. Finally, in a study investigating the neural activity of PRR neurons when monkeys reached to auditory versus visual targets in a memory reach task (Cohen and Andersen, 2000), it was found that, during the cue period, visually cued trials carried more information about target location than did auditory cued trials. However, the amount of spatial information increased during the auditory trials, and the activity in the visual and auditory trials was not significantly different when the reach occurred. These studies emphasize the temporal evolution of activity in the PPC to reflect sensory, cognitive, and motor signals at different stages during a task. D P  M P Recording experiments have found that the neural activity in LIP is related to the decision of a monkey to make an eye movement. Both the prior probability and the amount of reward associated with a particular movement influenced the neural representation of visual activity in LIP, which points to a role of this area in decision making (Platt and Glimcher, 1999). As monkeys accumulated sensory information for the planning of an eye movement, activity in LIP and the prefrontal cortex was found to increase, consistent with the idea that these areas are weighing decision variables for the purpose of eye movement planning (Coe et al., 2002; Gold and Shadlen, 2001; Shadlen and Newsome, 1996; Thompson et al., 1996). Monkeys and humans have been shown to choose between two targets for a reach, depending on eye position and the stimulus locations in space, essentially favoring targets that tend to center the reach with respect to the head (Scherberger et al., 2003). We recently investigated the neural activity in PRR during this choice paradigm in the monkey and found that the neural activity of single PRR cells was only transiently related to the visual stimulus at the beginning of the trial, which was essentially identical for both choices. Later in the trail, the activity closely reflected the animal’s choice of target for a reach (Scherberger and Andersen, 2001). This result suggests a role for PRR in decision processes for the generation of reach movements similar to that for LIP in decision processes for eye movements. Moreover, eye position gain effects have been shown in PRR and in LIP, and these gain effects may bias the decision of animals to choose targets based on the eye position. These and other findings suggest that decision making for movement planning is a distributed process that may involve many cortical areas including the PPC, and the particular areas involved may depend on the specific sensory input and motor actions considered. I  A Given the fact that the PPC sits at the interface between sensory and motor systems, it is perhaps not surprising that the issue of how to distinguish intention from attention in this area has been of considerate research interest. To separate sensory from movement

processing, antisaccade and antireach paradigms have been used in which animals were trained to make movements in the opposite direction to a briefly flashed visual stimulus. Activity in the medial intraparietal area (MIP) has been shown to code mostly the direction of the movement plan, not the location of the stimulus (Eskandar and Assad, 1999; Kalaska, 1996). In LIP, the reverse has been reported for eye movements (Gottlieb and Goldberg, 1999); however, recently it has been reported that most LIP cells also code the direction of the planned eye movement after a brief transient response linked to the stimulus (Zhang and Barash, 2000). The antisaccade and antireach results suggest that the PPC contains both sensory- and movement-related responses and is involved in the intermediate stages of sensorimotor transformations. In an experiment specifically designed to separate spatial attention from intention, monkeys were trained to attend to a flashed target and plan a movement to it during a delay period (Snyder et al., 1997). However, in one case the plan was for a saccadic eye movement, while in the other case the plan was for a reach. In other words, during the memory period, the only difference in the task was the type of movement the animal was planning. Figure 89.3 shows two intention-specific neurons from PPC, one from LIP (A) and one from PRR (B), while the animal planned an eye or arm movement to the same location in space. The activity of the LIP neuron showed a transient response due to the briefly flashed stimulus followed by activity during the delay period when the animal was planning an eye movement (left histogram) but not when the animal was planning an arm movement to the same location (right histogram). In contrast, the PRR neuron showed no elevated activity in the delay period when an eye movement was planned but a strong activity for the planning of an arm movement. Such results were typical in the PPC: LIP was much more active for the planning of eye movements, whereas PRR was more active during arm movement planning. PRR includes MIP, 7a, and the dorsal aspect of the parieto-occipital area (PO); however, MIP contains the highest concentration of reachrelated neurons. These results from LIP and PRR strongly argue for a role of the PPC in movement planning. I M The experiments mentioned above indicate a topographical separation of intentions within the PPC (Fig. 89.4). While area LIP seems to be specialized for the planning of saccades (Gnadt and Andersen, 1988), MIP and area 5 are more dedicated to the planning of reach movements (Buneo et al., 2002; Kalaska, 1996; Snyder et al., 1997). Other groups have identified areas PO, 7m, 7a, and PEc as additional reaching-related areas within the PPC (Battaglia-Mayer et al., 2000; Ferraina et al., 1997, 2001; MacKay, 1992). Furthermore, the anterior intraparietal area (AIP) seems to play a specialized role for grasping, as demon-

  :  

1327

a

Saccade trials

Reach trials

Cue

Cue

Saccade

Reach (Area 5)

Reach

Reach (MIP) Saccade (LIP)

185 Hz

LIP neuron

24

b

º

Grasp (AIP)

Cue

Saccade

Cue

Reach

F 89.4. Anatomical map of intentions in the PPC. AIP, anterior intraparietal area; LIP, lateral intraparietal area; MIP, medial intraparietal area.

PRR neuron

Sensory integration and space representation in the PPC 500 msec

F 89.3. LIP and PRR neuron activity during a delayed saccade and delayed reach movement task. a, LIP cell showing elevated activity during the delay period (150 to 600 msec after the cue) before a saccade (left) but not before a reach movement (right). b, PRR cell showing no saccade activity (left) but showing reach activity (right) during the delay period with both movements planned to the same location in space. The neural activity in the delay period depended specifically on the movement intention. In all panels, short horizontal bars indicate the timing of the target flash ( filled: saccade cue; open: reach cue), and long horizontal bars indicate the timing of the motor response (saccade or reach). Each panel shows a spike raster (eight trials aligned on cue presentation, every third action potential shown) and the corresponding spike density function (computed as in Fig. 89.2). Thin horizontal lines indicate the animal’s vertical eye position during each trial. Vertical bars indicate calibration of firing rate and eye position. (Modified from Snyder et al., 1997.)

strated by Sakata and colleagues (1995). Cells in AIP respond to the shape of objects and the formation of the hand during grasping. Recent results of fMRI studies in humans were found to be consistent with the electrophysiological findings in the monkey. Rushworth et al. (2001) found that a peripheral attention task activated the lateral bank of the intraparietal sulcus, while the planning of manual movements involved activity in the medial bank. Connolly et al. (2000) reported a similar result using event-related fMRI. A specialized area for grasping has also been identified in the anterior aspect of the intraparietal sulcus in humans, which may be homologous to the monkey area AIP (Binkofski et al., 1998). Simon et al. (2002) also found a systematic anterior-posterior organization of activations associated with grasping, pointing, and eye movements. All this suggests that the parietal cortex is composed of distinguishable subregions both in monkeys and in humans.

1328

, ,   

C F Sensorimotor transformations include the integration of different sensory modalities, such as vision, sound, and touch, to develop movement plans. Since different sensory modalities represent space in different reference frames, for example, vision in retinal or eyecentered coordinates, sound in head-centered coordinates, and touch in body-centered coordinates, the question of how these senses are integrated becomes particularly important. On the motor output side, different motor actions similarly require the representation of target location in different reference frames according to the natural coordinates of the moving muscles. The surprising finding that several movement planning areas in the PPC represent space in an eye-centered coordinate frame independent of the sensory input and motor output may provide a unifying scheme for multimodal sensory integration and the development of high-level movement plans. In area LIP, it was recently established that cells were involved in the planning of eye movements, regardless of whether the eye movements were triggered by a visual or an auditory stimulus (Grunewald et al., 1999; Linden et al., 1999; Mazzoni et al., 1996b). In further experiments directly addressing the question of the underlying reference frame, it was found that the majority of neurons coded auditory targets, similar to visual targets, in eye-centered coordinates, and with many response fields of LIP neurons gainmodulated by eye position (Stricanne et al., 1996). These findings suggest that LIP is involved in the sensory integration of visual and auditory signals and that many LIP neurons are encoding these signals in eye-centered coordinates. Earlier stages of auditory processing that project to LIP seem to encode auditory targets in head-centered coordinates, also with gain modulation by eye position (Wu and Andersen, 2001). Such a change of reference frame is con-

sistent with a current model of auditory target processing, where auditory signals are represented in head-centered coordinates, and gain-modulated by eye position, in order to transform them to an eye-centered representation at a subsequent stage (Xing and Andersen, 2000b). Based on the findings in LIP, one wonders, how are sensory stimuli encoded for reach movements in PRR. One prediction would be that PRR codes sensory stimuli as motor error, that is, in limb coordinates; another possibility would be that PRR also codes in eye-centered coordinates. To test this hypothesis, monkeys were trained to reach to visual targets on a reach board from two different initial arm positions while their gaze was fixating in two different directions (Batista et al., 1999). Figure 89.5A depicts the four conditions: on the left side with the same eye position but different initial hand positions, and on the right side with the same initial hand position but different eye positions. The activity of one typical PPR neuron is shown in Figure 89.5B. Gray-scale values on each panel represent the cell’s activity for reach movements from the illustrated initial hand position to the particular location on the board. As can be seen, the response field of the cell did not change with changes of the initial hand position (left column) but instead shifted with the gaze direction (right column). This finding was confirmed throughout the population of recorded neurons (Fig. 89.5C), where the response fields of the majority of neurons correlated better with the task when viewed in eye-centered as opposed to limb-centered coordinates. This result indicates that PRR codes intended limb movements to visual stimuli in eye-centered coordinates similar to how LIP codes for intended eye movements.

Task

a Same eye position Different initial hand position

Different eye position Same initial hand position

b

0

140 Hz



c Correlation coefficient in limb coordinates

F 89.5. Encoding of reach target locations in eye-centered coordinates in PRR for visual and auditory stimuli. a, Delayed reach task in four conditions: same eye position but different initial hand positions (left side) and different eye position but same initial hand position (right side). b, Activity of one typical PPR neuron during the delayed reach task to a visual stimulus. On each panel, gray-scale values represent the cell’s activity in the delay period for a reach from the illustrated initial hand position to the particular location on the board. The response field of the cell did not change with changes of the initial hand position (left column) but shifted with gaze direction (right column). c, Population response of the experiment in b, each circle represents one neuron. x-axis: correlation coefficient between responses for the same eye position but different hand positions (a, left side); y-axis: correlation coefficient between responses for the same initial hand position but different eye position (a, right side). Response fields of most neurons correlated better with eye-centered as opposed to limb-centered coordinates. d, Same experiment as in b and c, but with reaches made to auditory target locations in complete darkness. Axes and symbols as in c. Again, for most of the cells, the correlation was larger between response fields with the same eye position than between response fields with the same initial arm position. (a–c modified from Batista et al., 1999; d modified from Cohen and Andersen, 2000.)

1

Population Visual stimulus

1

0

0

-1 -1

d Population Auditory stimulus

0

1

Correlation coefficient in eye coordinates

-1 -1

0

1

Correlation coefficient in eye coordinates

  :  

1329

Sensory input Eye Coordinates

Head Coordinates

Body Coordinates

Visual

Auditory

Somatosensory

Eye Coordinates

Gain Fields

High-level movement planning

Eye Movement

Head Movement

Limb Movement

Eye Coordinates

Head Coordinates

Body Coordinates

Motor output F 89.6. Multisensory integration and coordinate transformation in the PPC. Sensory input is transformed into a common eye-centered coordinate frame for high-level planning of movements that are subsequently transformed into various motor output coordinate frames. Dotted arrows: speculated transformations. Gain fields may provide a mechanism for coordinate transformations.

The possibility that PRR codes reach intentions independent of the sensory stimuli, similar to LIP, led to the counterintuitive prediction that many PRR neurons would encode reaches to auditory stimuli in eye-centered coordinates as well. This idea was tested in another study (Cohen and Andersen, 2000), where monkeys performed reach movements to auditory targets in complete darkness while the animal’s initial hand position and gaze direction were systematically varied as described above (Fig. 89.5A). Again, in the majority of cells, the correlation between response profiles with the same eye position was found to be much larger than between response fields with the same initial arm position (Fig. 89.5D). This result confirms that a majority of PRR neurons code reaches to auditory stimuli in eyecentered coordinates, even though the sensory stimulus, which is initially coded in head-centered coordinates, could easily be converted to body- and then limb-centered coordinates without the involvement of any eye-centered reference frame. The above-mentioned results suggest that populations of LIP and PRR neurons both use a common eye-centered coordinate frame for space representation, independent of whether the sensory input is visual or auditory and regardless of whether the output is to move a limb or to make an eye movement. These findings lead to a general scheme of space representation in the PPC, where sensory input is converted to eye-centered representations in the PPC and highlevel movement plans are formed that can be read out by motor output structures in their natural coordinates (Fig. 89.6). Currently, we do not know if somatosensory signals

1330

, ,   

coding the position of the hand are also coded in eyecentered coordinates in LIP and PRR and how head movements are coded in the PPC (dotted arrows). These are interesting questions for future research. A possible reason for the use of a common reference frame in the PPC could be to facilitate the planning of coordinated movements, for example, during eye-hand coordination. A particular reason why an eye-centered representation has evolved in the PPC might be that vision is the most dominant and accurate spatial sense in primates. There is a problem when using an eye-centered representation for the encoding of intentions in the PPC. Movement plans would need to be updated in eye coordinates every time an eye movement occurred before the movement plan was executed. This updating of movement plans during intervening saccades was indeed found in the superior colliculus (SC) (Mays and Sparks, 1980) and in LIP (Gnadt and Andersen, 1988) for eye movement plans, as well as for reach movement plans in PRR (Batista et al., 1999) and in human psychophysical reach experiments (Henriques et al., 1998). G F The gain modulations of the eye-centered response fields in LIP and PRR by eye, head, and limb position signals may provide a general mechanism for converting stimuli from various reference frames into eye-centered coordinates and, likewise, may allow other areas to read out signals from LIP and PRR in different coordinate frames, including eye, head, body, and limb-centered schemes (Fig. 89.6). It has been shown computationally that gain effects could be an effective mechanism for coordinate transformations in neural networks (Salinas and Abbott, 1995; Zipser and Andersen, 1988). Such a mechanism would also be very flexible, since the information of a group of eye-centered neurons that is gain-modulated by various body part positions could be subsequently read out in multiple frames of reference (Pouget and Snyder, 2000; Xing and Andersen, 2000b). The number of cells necessary for representing all possible combinations of eye, head, and body positions can remain within reasonable limits if, in each area, only those variables are encoded that are necessary to perform its specific functions (Snyder et al., 1998). Space representation in at least some areas of the PPC is thus distributed and composed of retinotopic response fields that are gain-modulated. A similar gain mechanism could also provide the means for the remapping of eye-centered response fields in the PPC during intervening eye movements, which was demonstrated to occur during saccade planning in LIP in the double saccade paradigm (Gnadt and Andersen, 1988; Mazzoni et al., 1996a) and during reach planning in PRR in the intervening saccade paradigm (Batista et al., 1999). In a recent computational model, it was demonstrated that dynamical neural networks could be trained to perform the double saccade task (Xing and Andersen, 2000a). The model com-

prised eye-centered response fields that were gain-modulated by eye position and a shift of activity within eye-centered visual maps that corrected for intervening saccades, demonstrating that gain field mechanisms are sufficient for the saccade updating of eye-centered response fields in the PPC. We highlighted the role of gain fields in the PPC for the updating of neural maps and for the coordinate transformations between different reference frames. It is clear, however, that gain mechanisms play an important role for other brain functions as well, such as decision making, attention, and object recognition (Salinas and Thier, 2000) and may be a general strategy of neural computation. C T Sensorimotor transformation in the PPC converts sensory input signals into motor plans for action. One essential step in achieving this goal is the transformation of the target location from the coordinates of its sensory input into the coordinates of the motor output. While we have highlighted a common eye-centered reference frame for the representation of intentions in the PPC and its possible role for multisensory integration, it is less well understood how intentions are transformed further downstream into motor commands (Fig. 89.6). In the case of a reaching movement to a visual target, the target location of the reach has to be transformed from visual coordinates to the motor coordinates of the limb. This could be archived in various ways. In a sequential model, the eye-centered representation would be transformed sequentially first into a head-centered representation, then into a body-centered representation, and finally into the coordinates of the limb by sequentially considering the eye position, head position, and limb position. Alternatively, in a combinatorial model, the eye-centered target location could be combined at a single stage with the eye, head, body, and arm positions to compute the target in limb-centered coordinates (Battaglia-Mayer et al., 2000). However, such a combinatorial approach might require an unrealistically large number of cells for implementation in the PCC. In a third direct model, the current position of the limb is encoded in eye coordinates and compared with the eye-centered target position to directly generate the motor vector in the coordinates of the limb (Fig. 89.7A). This approach would require only a few computational stages and would rely only on variables in eye-centered coordinates. This direct model is supported by findings of a recent study of area 5, a somatosensory cortical area within the PPC, where single neurons were found to encode target locations simultaneously in eye- and limb-centered coordinates (Buneo et al., 2002). Figure 89.7B–E shows the activity of a typical neuron for a reach movement with the same motor error in four conditions with different initial hand and eye fixation positions. The activity of the cell varied substantially

a Direct model Eye-centered Target Position

Eye-centered Hand Position

-

Hand-centered Target Position

+

Variation of initial hand position

Variation of eye fixation position

b

d

c

e

140 Hz

1s

reach onset

F 89.7. Direct visuomotor transformation for reaching in area 5. a, Direct transformation model. b–e, Activity of an area 5 neuron that simultaneously codes the target location in eye- and limb-centered coordinates, shown in a delayed reach movement task with different initial eye and arm positions. Each panel illustrates on top the different initial arm and eye positions for the same reach movement vector (arrow), and below the neural activity (spike rasters and spike density function). Thin bars (in e) calibrate time and firing rate. The activity of the cell varies substantially in the task when the initial arm position (b, c) or the eye position (d, e) is varied. However, the neural activity is very similar when the initial hand positions and target locations are identical in eye coordinates b, e and c, d. (Modified from Buneo et al., 2002.)

when either the initial hand position (B, C) or the eye fixation position (D, E) was varied, but was very similar between conditions with the same eye fixation and initial hand position relative to the target (B, E and C, D). Hence, the activity of the cell could be best described when both the target location in eye coordinates and the initial hand location in eye coordinates were taken into account. This finding is consistent with the PPC transforming target locations directly between these two reference frames. Cells in PRR code the target location in eye-centered coordinates with a gain modulation by the initial hand

  :  

1331

position, also in eye-centered coordinates (Buneo et al., 2002). Therefore, a convergence of input from cells in PRR onto area 5 could perform this direct transformation from eye- to limb-centered coordinates by using a simple gain field mechanism and without having to rely on intermediate coordinate frames or a large number of retinal, head, and limb position signals. On the other hand, psychophysical evidence has supported the above-mentioned sequential model (Flanders et al., 1992; McIntyre et al., 1997, 1998). The different results may reflect an underlying context dependence of the coordinate transformations for reaching, where direct transformations are preferred when both the target location and the hand are visible, while a sequential scheme may be preferred otherwise. Only future experiments can reveal the mechanisms for generating motor commands under different sensory conditions.

% of cells

Plasticity: adaptation and behavioral significance

b

1332

, ,   

Spatial tuning 100 Auditory stimuli

Visual stimuli

80 n. s.

60 * 45%

* 46%

p