Time and the Brain (Conceptual Advances in Brain Research, Vol 3)

Time and the Brain

Conceptual Advances in Brain Research A series of books focusing on brain dynamics and information processing systems of the brain. Edited by Robert Miller, Otago Centre for Theoretical Studies in Psychiatry and Neuroscience, New Zealand (Editor-in-chief), Günther Palm, University of Ulm, Germany and Gordon Shaw, University of California at Irvine, USA.

Volume 1 Brain Dynamics and the Striatal Complex edited by R.Miller and J.R.Wickens Volume 2 Complex Brain Functions—Conceptual Advances in Russian Neuroscience edited by R.Miller, A.M.Ivanitsky and P.Balaban Volume 3 Time and the Brain edited by R.Miller Forthcoming Volume Sex Differences in Lateralization in the Animal Brain edited by V.L.Bianki and E.B.Filippova, translated by T.Ganf Volumes in Preparation Cortical Areas: Unity and Diversity The Female Brain Functional Memory and Brain Oscillations This book is part of a series. The publisher will accept continuation orders which may be cancelled at any time and which provide for automatic billing and shipping of each title in the series upon publication. Please write for details.

Time and the Brain Edited by

R.Miller Otago Centre for Theoretical Studies in Psychiatry and Neuroscience New Zealand

harwood academic publishers Australia • Canada • France • Germany • India • Japan Luxembourg • Malaysia • The Netherlands • Russia • Singapore Switzerland

This edition published in the Taylor & Francis e-Library, 2005. "To purchase your own copy of this or any of Taylor & Francis or Routledge's collection of thousands of eBooks please go to www.eBookstore.tandf.co.uk." Copyright © 2000 OPA (Overseas Publishers Association) N.V. Published by license under the Harwood Academic Publishers imprint, part of The Gordon and Breach Publishing Group. All rights reserved. No part of this book may be reproduced or utilized in any form or by any means, electronic or mechanical, including photocopying and recording, or by any information storage or retrieval system, without permission in writing from the publisher. Printed in Singapore. Amsteldijk 166 1st Floor 1079 LH Amsterdam The Netherlands British Library Cataloguing in Publication Data A catalogue record for this book is available from the British Library. ISBN 0-203-30457-8 Master e-book ISBN

ISBN 0-203-34338-7 (Adobe eReader Format) ISBN: 90-5823-060-0 (Print Edition) ISSN: 1029-2136

CONTENTS

Series Preface Preface List of Contributors 1 Empirical Evidence about Temporal Structure in Multi-unit Recordings Alessandro E.P.Villa 2 Cross-correlograms for Neuronal Spike Trains. Different Types of Temporal Correlation in Neocortex, their Origin and Significance Lionel G.Nowak and Jean Bullier 3 The Space-Time Continuum in Mammalian Sensory Pathways Asif A.Ghazanfar and Miguel A.L.Nicolelis 4 Information Flow along Neocortical Axons Harvey A.Swadlow 5 Psychophysics of Human Timing Thomas H.Rammsayer and Simon Grondin 6 Cortical Processing by Fast Synchronization: High Frequency Rhythmic and Non-rhythmic Signals in the Visual Cortex Point to General Principles of Spatiotemporal Coding Reinhard Eckhorn 7 EEG Alpha and Cognitive Processes Wolfgang Klimesch 8 Theta Frequency, Synchronization and Episodic Memory Performance Wolfgang Klimesch 9 Distributed Assemblies, High Frequencies and the Significance of EEG/MEG Recordings Friedemann Pulvermüller 10 Cell Assemblies, Associative Memory and Temporal Structure in Brain Signals Thomas Wennekers and Günther Palm 11 The Relation between EEG and Evoked Potentials Erol Başar , Sirel Karakas , Elke Rahn and Martin Schürmann 12 Coherence and Phase Relations between EEG Traces Recorded from Different Locations Peter Rappelsberger , S.Weiss and Baerbel Schack 13 Temporal Structure of Neural Activity and Models of Information Processing in the Brain Galina N.Borisyuk , Roman M.Borisyuk and Yakov B.Kazanovich

vii ix xi 1 63

112 150 181 196

236 262 281

293 320 346

389

Discussion Section POSTLUDE The Neuroanatomy of Time Valentino Braitenberg Index

410 454 460

SERIES PREFACE The workings of the brain, including the human brain are a source of endless fascination. In the last generation, experimental approaches to brain research have expanded massively, partly as a result of the development of powerful new techniques. However, the development of concepts which integrate and make sense of the wealth of available empirical data has lagged far behind the experimental investigation of the brain. This series of books entitled Conceptual Advances in Brain Research (CABR) is intended to provide a forum in which new and interesting conceptual advances can be presented to a wide readership in a coherent and lucid way. The series will encompass all aspects of the sciences of brain and behaviour, including anatomy, physiology, biochemistry and pharmacology, together with psychological approaches to defining the function of the intact brain. In particular, the series will emphasise modern attempts to forge links between the biological and the psychological levels of describing brain function. It will explore new cybernetic interpretations of the structure of nervous tissue; and it will consider the dynamics of brain activity, integrated across wide areas of the brain and involving vast numbers of nerve cells. These are all subjects which are expanding rapidly at present. Subjects relating to the human nervous system as well as clinical topics related to neurological or psychiatric illnesses will also make important contributions to the series. These volumes will be aimed at a wide readership within the neurosciences. However, brain research impinges on many other areas of knowledge. Therefore, some volumes may appeal to a readership, extending beyond the neurosciences. Books suitable for the series are monographs, edited multiauthor collections or books deriving from conferences, provided they have a clear underlying conceptual theme. In order to make these books widely accessible within the neurosciences and beyond, the style will emphasise broad scholarship comprehensible by readers in many fields, rather than descriptions in which technical detail of a particular speciality is dominant. The next decades promise to provide major new revelations about brain function, with far-reaching impact on the way we view ourselves. These great breakthroughs will require a broad interchange of ideas across many fields. We hope that the CABR series plays a significant part in the exploration of this important frontier of knowledge.

PREFACE Since the days of Galileo, time has been a fundamental variable in scientific attempts to understand the natural world. It has been clear since the first recordings of electrical activity in the brain, by Caton, and later by Danielewski, that electrical signals from the brain consist of very complex temporal patterns. This perspective has continued to the present day. It is now richly demonstrated by recordings of electrical activity at both the single unit level and by the methods of electroencephalography. When recording of the electrical activity from single nerve cells in the brain became possible, the emphasis was initially on events occurring on a more brief time scale than the fastest of psychological processes. However, in more recent years, single unit activity has also been widely studied on a larger time scale, corresponding to many psychological processes. In particular, in the last decade there has been a major shift in the way we view single unit activity in several brain structures: A generation ago, the emphasis was on mapping space (for instance in a sensory receptor surface) against spatial location in the brain. Nowadays we are more concerned with the temporal structure detectable in neuronal impulse trains. Temporal structure in impulse trains may thus be part of the way in which timing of sensory or motor events is represented in the brain; and indeed, it is possible that external events and inner cognitive processes may be represented in the brain in part using temporal coding, even when these events and processes, in themselves, lack detailed temporal structure. Analysis of temporal structure in the activity of large populations of nerve cells in the brain has also undergone major advances in the last ten or fifteen years. Functional brain imaging using metabolic activity gives good spatial resolution of brain processes. In contrast, mapping based on brain activity, based on the EEG or the MEG is superior in revealing the temporal structure of dynamic processes in the brain. For the EEG, the time scale needed to describe the basic rhythms of brain electrical activity corresponds roughly to that of the more rapid of psychological processes. That electrical activity is also known to correlate in many ways with psychological or behavioural events observed at the level of the whole organism. This book, the third in the series Conceptual Advances in Brain Research, explores modern approaches to these temporal aspects of brain electrical activity. The earlier chapters focus mainly on temporal structure revealed from trains of impulses recorded from single nerve cells, or from several such nerve cells recorded at the same time, and their possible relation to behaviour and psychological processes. In the later chapters, the emphasis shifts to temporal structure in the EEG, again with several essays exploring correlations with behaviour and psychology. There are also two chapters of a more theoretical nature. The final chapter forms an edited discussion between contributors, delving further into a variety of issues which arose during editing of the main chapters. In this, a significant question is also touched on—the relation (if any) between temporal structure discernible at the single unit level, and that seen in the EEG.

Study of the electrical activity of the brain by EEG recording and by the methods of single unit electrophysiology are often not well integrated: The two approaches tend to be adopted in different laboratories, and indeed in different countries. By bringing together within a single book modern information acquired using both approaches it is hoped that these two approaches can become better integrated as complementary windows on the information processing achieved by the brain. As editor of this book, I have to say that I am greatly honoured to have received such a fine set of chapters to edit, and I hope that my own contributions to the book are at the same standard as those of the chapter authors. R.Miller.Dunedin. December. 1999

LIST OF CONTRIBUTORS Erol Başar Institute of Physiology Medical University Lübeck Ratzeburger Allee 160 D-23538 Lübeck Germany Galina N.Borisyuk Institute of Mathematical Problems in Biology Russian Academy of Sciences Pushchino Moscow Region, 142292 Russia Roman M.Borisyuk Institute of Mathematical Problems in Biology Russian Academy of Sciences Pushchino Moscow Region, 142292 Russia Valentino Braitenberg Max Planck Institute for Biological Cybernetics Spemann Strasse 38 D-7400 Tübingen Germany Jean Bullier INSERM U371 Cerveau et Vision 18 Avenue du Doyen Lépine 69675 Bron Cedex France Reinhard Eckhorn Department of Physics

Neurophysics Group Philipps-University Renthof 7 D-35032 Marburg Germany Asif A.Ghazanfar Department of Neurobiology Box 3209 Duke University Medical Centre 101 Research Drive Bryan Research Building Durham, NC 27710 USA Simon Grondin Institute of Psychology University of Goettingen Gosslerstrasse 14 D-37073 Goettingen Germany Sirel Karakas TÜBITAK Brain Dynamics Research Unit Ankara Turkey Yakov B.Kazanovich Institute of Mathematical Problems in Biology Russian Academy of Sciences Pushchino Moscow Region 142292 Russia Wolfgang Klimesch Department of Physiological Psychology Institute of Psychology University of Salzburg Hellbrunnerstrasse 34 A-5020 Salzburg Austria

Miguel A.L.Nicolelis Department of Neurobiology Box 3209 Duke University Medical Centre 101 Research Drive Bryan Research Building Durham, NC 27710 USA Lionel G.Nowak Section of Neurobiology Yale University School of Medicine C303 Sterling Hall of Medicine 333 Cedar Street New Haven, CT 06510 USA Günther Palm Abteilung Informatik Fakultät für Informatik Universität Ulm Oberer Eselberg D-7900 Ulm Germany Friedemann Pulvermüller Department of Psychology Fachgruppe Psychologic Sozialwissenschaftliche Fakultät Universität Konstanz Postfach D23 D-78457 Konstanz Germany Elke Rahn Institute of Physiology Medical University Lübeck Ratzeburger Allee 160 D-23538 Lübeck Germany Thomas H.Rammsayer Institute of Psychology University of Goettingen

Gosslerstrasse 14 D-37073 Goettingen Germany Peter Rappelsberger Institute for Neurophysiology University of Vienna Waehringer Strasse 17 A-1090 Vienna Austria B.Schack Institute for Medical Statistics, Informatics and Documentation Friedrich Schiller University Jahn Strasse 3 D-07740 Jena Germany Martin Schürmann Institute of Physiology Medical University Lübeck Ratzeburger Allee 160 D-23538 Lübeck Germany Harvey A.Swadlow Department of Psychology (U-20) The University of Connecticut Storrs, CT-06269 USA Alessandro E.P.Villa Laboratory of Neuro-heuristics Institute of Physiology University of Lausanne 7 Rue du Bugnon CH-1005 Lausanne Switzerland S.Weiss Institute for Neurophysiology University of Vienna Waehringer Strasse 17 A-1090 Vienna

Austria Thomas Wennekers Abteilung Informatik Fakultät für Informatik Universität Ulm Oberer Eselberg D-7900 Ulm Germany

1 Empirical Evidence about Temporal Structure in Multi-unit Recordings Alessandro E.P.Villa Laboratory of Neuro-heuristics, Institute of Physiology, University of Lausanne, 7 Rue du Bugnon, CH-1005 Lausanne, Switzerland Tel: ++41+21–692–5532/X.5516/X.5500; FAX ++41–21–692.5532 e-mail: [email protected] URL: www-lnh.unil.ch

The brain is a highly interconnected network of neurones, in which the activity in any neurone is necessarily related to the combined activity in the neurones that are afferent to it. Due to the widespread presence of reciprocal connections between brain areas, reentrant activity through chains of neurones is likely to occur. Certain pathways through the network may be favoured by inhomogeneity in the number or efficacy of synaptic interactions between the neural elements as a consequence of developmental and/or learning processes. In cell assemblies interconnected in this way, some ordered sequences of intervals within spike trains of individual neurones, and across spike trains recorded from different neurones, will recur. Such recurring, ordered, and precise (in the order of few ms) interspike interval relationships are referred to as “spatiotemporal patterns” of discharges. This term encompasses both their precision in time, and the fact that they can occur across different neurones, even recorded from separate electrodes. This chapter introduces the fundamental assumptions and algorithms that lead to the detection of complex patterns of neural discharges and introduces a way of interpreting this activity within the framework of non-linear dynamics. Empirical results of experimental and simulation studies are provided in different sections of the chapter. KEYWORDS: Spatiotemporal firing patterns; Neural dynamics; Brain theory; Time code; Frequency code; Multi-unit recordings; Non-linear dynamics; Sensorimotor association

1. BRIEF HISTORICAL INTRODUCTION In 1753–1755 the physiologist Albrecht von Haller published in Göttingen an historical essay, the “Dissertation on the Irritable and Sensitive Parts of Animals” (original title: De partium corporis humani sensibilius et irritabilus). This work was based on numerous experiments of vivisection, and on stimulation of organs using the new knowledge

Time and the brain

2

offered to physiology by physics, chemistry and natural history. With a rudimentary technique of stimulation, Von Haller classified the parts as irritable, sensible or elastic and noted that the reactions varied between different parts of the brain. The historical importance of the work by Von Haller is not so much related to the results obtained, but rather in systematically developing a transdisciplinary approach to brain research using the new technologies of his time. In 1791 the Italian physician Luigi Galvani started the publication of a remarkable series of studies that demonstrated muscle twitch in a frog by touching its nerves with electrostatically charged metal, and later using two dissimilar uncharged metals. These observations led Galvani to postulate that the circulation of a particular body fluid, that exists naturally in the nerves in a state of disequilibrium, provided the stimuli for the muscle fibres to contract. In addition, normal muscular contraction without a source of electrostatic electricity was, in Galvani’s view, evidence for the existence of an additional and “natural” form of electricity, that he called “animal electricity”. This latter statement set the scene for the famous Galvani-Volta controversy. Alessandro Volta, a friend of Galvani and an Italian physicist who, in 1775, invented the electrophorus—a device to generate static electricity—gave an opposed and experimentally valid explanation of Galvani’s experiment. The electricity did not come from the animal tissue but was generated by the contact of different metals, brass and iron, in a moist environment. The interest in this controversy resides in the confrontation of two basic, irreducible interpretations of the same observation, derived from each scientist’s different background: Galvani saw the frog phenomenon as the work of biological organs, Volta as that of a physical apparatus. The outcome of the controversy was exceptional. On the one hand the challenge of Volta’s opinion led Galvani to perform a new series of experiments that demonstrated muscular contraction by touching the exposed muscle of one frog with the nerve of another frog, thus showing for the first time the existence of bioelectric forces. On the other hand, Volta focused his research efforts upon the study of electric fluids between dissimilar metals, and in 1800 he presented the first electric battery, providing future researchers with a stable source of electricity not dependent on electrostatic forces. With the introduction of currents of “Galvanic fluids” into the brain, a way was opened to making new discoveries in physiology at the beginning of the XIXth century. Electricity was used not just as an experimental intervention, which formed the basis of electrophysiology (the study of the connection between living organisms and electricity), but in addition, the proper characteristics of propagation and generation of this type of energy became the basis of fertile hypotheses. In the last quarter of the XIXth century, Eduard Hitzig and Gustav Fritsch discovered the localization of cortical motor areas in the dog, using electrical stimulation, and Richard Caton was the first to record electrical activity from the brain. Electrophysiology started to develop rapidly, and Edgar D.Adrian published a seminal study suggesting the all-or-none principle in nerve (1912). In the late 1920s Hans Berger in Germany demonstrated the first human electroencephalogram and opened the way to clinical applications of electrophysiology. Nevertheless, the English School led investigations in electrophysiology in the first part of our century, and for his specific research on the function of neurones, Adrian shared the 1932 Nobel Prize for Medicine with Sir Charles Sherrington. Although most remembered for his scientific contributions to neurophysiology, Sherrington’s research focused on spinal reflexes as

Empirical evidence about temporal structure

3

well as on the physiology of perception, reaction and behaviour. His approach was transdisciplinary, in the sense that disciplines were used not only “one next to the other”, but were really intermingled in his protocols, an extraordinary example of a nonreductionist view of neurophysiology. At present, and still based upon the work performed by Adrian (1934), most neurophysiologists make their deductions by observing mean frequencies of nervous discharges (spikes), i.e. whether there are a lot or only a few spikes over a relatively prolonged time interval. Important concepts and findings have been clarified by using the overall mean rate as a measure of neuronal activity, at both peripheral and central levels of the nervous system. In particular it is important to remind ourselves of the influence of the findings of Mountcastle et al. (1963) indicating that the relationship between frequency of firing of selected neurones in the ventrobasal thalamus of deeply anaesthetized monkeys and the angle of extension of the contralateral knee was a perfect power function. Hence, the relationship between log frequency and log angle is virtually a perfect linear function. Although this point will not be developed further in the present chapter, it is important to note that the effect, and the type of anaesthesia determine dramatically the neuronal discharge pattern, especially at thalamic and cortical levels (e.g., see Mukhametov et al., 1970; Zurita et al., 1994). This effect has not always been adequately taken into account by investigators eager to develop theoretical models of “neuromimetic” circuits. The slow integration time of nerve cells, operating in the range of a few milliseconds, roughly a million times slower than presently available supercomputers, and the huge number of connections established by a single neurone (for review, see Braitenberg and Schüz, 1991) has suggested that information in the nervous system might be transmitted by simultaneous discharge of a large set of neurones. Multiple dimensions of stimuli relevant to sensory function and behaviour are processed by thousands of neurones distributed over many areas of the brain. Indeed, the hypothesis that neurones (both individually and jointly) process information over time, and following precise time relationships, has pervaded the neurosciences ever since the nervous system was conceptualized as a set of dynamic networks of interacting neurones (McCulloch and Pitts, 1943; for review, see MacGregor, 1987). That neurones convey a temporal code has been well-known since electrophysiological studies in the 1960s led to the recognition that spike trains—the time series formed by the sequences of time intervals between spikes—were related to meaningful physiological variables (Bullock, 1961; Segundo et al., 1963; Perkel and Bullock, 1965; Segundo et al., 1966; Nafe, 1968). Subsequent work made an important contribution, by recognizing temporal coding in different experimental circumstances and animal species, and by proposing worthwhile quantification procedures (Perkel et al., 1967a,b; Segundo and Perkel, 1969; Legéndy, 1975; Eckhorn et al., 1976; Brillinger et al., 1976; Klemm and Sherry, 1981; Abeles, 1982; Tsukada et al., 1982; Rosenberg et al., 1989; Bialek et al., 1991; Rapp et al., 1994; Friston, 1995; Martignon et al., 1995; Rieke et al., 1995; de Ruyter van Steveninck et al., 1997). This provided new insights into how the association between simultaneously recorded spike trains, in the time and frequency domains, may reflect the degree of coordinated activity within a cell assembly. The integrative capabilities of neurones (for review, see Segundo, 1986) on the one hand, and parallel processing and redundancy

Time and the brain

4

inherent to neural pathways on the other hand, have become more obvious during the past two decades, thanks also to the advances in neuroanatomical methods using retrograde and/or anterograde tracers (e.g. horseradish peroxidase, fluorescent dyes, biocytin, etc.). However, for most functional systems the available electrophysiological data demonstrate merely the presence of temporally organized neural activity, so that during recent years investigations have mainly focused on establishing causal relations between the occurrence of precise temporal relationships and cognitive or motor processes (Fetz, 1997). Therefore, two hundred years after the Galvani-Volta controversy, two main classes of theories explaining information processing in the brain have been proposed: In one, neurones convey a precise temporal code (Abeles, 1982; Abeles, 1991), while the other is based on noisy rate coding (Shadlen and Newsome, 1994). According to the rationale of a noisy rate code, the question has been raised whether the variability of spike intervals carries information. This variability may depend on the role of the decay of the post-synaptic potential (PSP) in determining the prevailing operating mode of network processing (König et al., 1996). Softky and Koch (1993) have shown that, in order to find the same degree of variability seen in vivo, coincidence detection with fast integration of small excitatory PSPs is required in models. Alternatively, Shadlen and Newsome (1994) have proposed that the balance between inhibition and excitation plays a critical role in the variability of network behaviour with membrane time constants of 8–20 ms. If an irregular interspike interval results from integration of excitatory and inhibitory postsynaptic potentials, then the timing of postsynaptic spikes is random and can no longer reflect the timing of presynaptic events. Precise patterns of spikes—their intervals and coincidences—would fail to propagate (Shadlen and Newsome, 1994). König et al., (1996) have provided a summary of these two viewpoints and proposed that the interplay of excitation and inhibition (E-I) effectively reduces the time constant for synaptic integration, thus providing coincidence detection in the cortex, although the actual synaptic decay is relatively long. Indeed, a balance condition of excitatory and inhibitory post-synaptic potentials plays an important role in the activity of many network models (Wilson and Cowan, 1974; Douglas and Martin, 1991; Tsodyks and Sejnowski, 1995; Usher and Stemmler, 1995; Xing and Gerstein, 1996) and has many complex spatial and temporal influences (Thomson and Deuchars, 1994). Changes in cellular excitability modify the time to firing, thus altering the E-I dynamics. Some specific examples of time-varying factors influencing the balance of E-I levels are the cell threshold potential, PSP kinetics, receptor activity such as NMDA, and calcium currents (Edmonds et al., 1995). The inability to preserve information about the time of a spike suggests that the discharge rate cannot be modulated in a time-locked fashion to specific inputs. In a random walk model (Gerstein and Mandelbrot, 1964) this assumption may be wrong because the discharge rate can follow the activity of inputs, but, given a base rate, the time to the next spike would appear as random. Then, the average instantaneous discharge rate through an ensemble of neurones belonging to a functional assembly would be capable of transmitting changes in spike rate with a precision of 10–50 ms (Shadlen and Newsome, 1998). Recordings performed in several cortical areas of behaving monkeys support the hypothesis of a connection between fluctuations in neural discharge rate and behaviour (Georgopoulos et al., 1993; Celebrini and Newsome, 1994). However, the existence of the rate code

Empirical evidence about temporal structure

5

mechanism does not imply that temporal codes do not also exist in the brain. The cerebral cortex is a highly interconnected network of neurones, in which the activity of each cell is necessarily related to the combined activity in the neurones that are afferent to it. Due to the presence of reciprocal connections between cortical areas, reentrant activity occurs, through chains of neurones. Furthermore, certain pathways through the network may be favoured by inhomogeneity in the number or efficacy of synaptic interactions between the neural elements, as a consequence of developmental and/or learning processes. According to the rationale of a precise temporal code, in cell assemblies interconnected in this way, some ordered sequences of intervals within spike trains of individual neurones, and across spike trains recorded from different neurones, will recur. Such recurring and ordered interspike intervals, with a precision of the order of a few milliseconds, are referred to as “spatiotemporal patterns” of discharges. For true demonstration, such temporal firing patterns must occur to a statistically significant level (see Figure 1). If functional correlates of spatiotemporal neural coding exist, one would expect that whenever the same information is presented, the same temporal pattern of firing would be observed. Several lines of evidence exist showing Spatiotemporal patterns in vivo (Bair and Koch, 1996) and in vitro (Mainen and Sejnowski, 1995). Recent studies on active propagation of action potentials in dendrites have provided additional results supporting the existence of precise neuronal timing (Stuart and Sakmann, 1994). A synaptic response increases if the presynaptic spike precedes the postsynaptic spike, but if the order is reversed, the synaptic response decreases. The window for synaptic

Time and the brain

6

Figure 1 . Outline of the general procedure followed by pattern detection algorithms, (a) Analysis of a set of simultaneously recorded spike trains. Three cells, labelled A, B, and C, participate in patterned activity. Three occurrences of a precise pattern are detected. Each occurrence of the pattern has been labelled by a specific marker in order to help the reader to identify the corresponding spikes, (b) Estimation of the statistical significance of the detected pattern, (c) Display of pattern occurrences as a raster plot aligned on the start of the pattern.

plasticity is 100 ms wide, and a difference in spike timing of only 10 ms near coincidence switches plasticity from potentiation to depression (Markram et al., 1997). Neurophysiological measurements in human subjects performing a visual go/no-go categorization task indicate that speed of processing of visual information in human is in

Empirical evidence about temporal structure

7

the order of 150 ms (Thorpe et al., 1996). Taking into account the firing rates of neurones in the visual pathway, only few spikes at most can be generated by each neurone in performing this highly demanding task, thus providing strong arguments for the importance of precise temporal coding. Indeed, electrophysiological experiments in primates have established that correlated firing between single neurones recorded simultaneously in the frontal cortex may evolve within tens of milliseconds, in systematic relation to behavioural events, without modulation of the firing rates (Vaadia et al., 1995). Furthermore, in multiple electrode recordings performed in the primary motor cortex of monkeys trained in a delay-pointing task, spike synchronization occurred in relation to purely internal events (stimulus expectancy), during which modulations of firing rate were distinctly absent (Riehle et al., 1997). These findings provide an important support for the general view that a stronger synaptic influence is exerted by multiple converging neurones firing in coincidence, thus making synchrony of firing ideally suited for highlighting responses and for expressing relations among neurones with high temporal precision.

2. DETECTION OF SPATIOTEMPORAL FIRING PATTERNS An influential and remarkable model based on the assumptions of high temporal precision in brain processing is the “synfire” chain hypothesis. This model suggests how precise timing can be sustained in the central nervous system by means of feed-forward chains of convergent/divergent links and re-entrant loops between interacting neurones forming an assembly (Abeles, 1982). Structures like synfire chains may exhibit attractors in which a group of neurones excite themselves, maintaining elevated firing rates for long periods and allowing the same neurone to participate in many different synfire chains. A fundamental prediction of such a model is that simultaneous recording of activity of cells belonging to the same assembly and involved repeatedly in the same process should be able to reveal repeated occurrences of such spatiotemporal firing patterns. Note that the term “firing pattern” encompasses both their precision in time, and the fact that they can occur across different neurones, even when recorded from separate electrodes. The following example (Figure 2) illustrates schematically the recurrence of patterns in a network connected according to the “synfire” model. A chain of 56 model neurones is formed by 8 sets of 7 neurones connected by diverging/converging links. This means one neurone in a set (in black in Figure 2a) receives 7 inputs from the previous set and projects to all 7 neurones of the next set. In the example shown in Figure 2a the links are subdivided into 5 excitatory (open circle) and 2 inhibitory connections (gray circles). This network is embedded into a larger network (200 neurones) with random connections. Note that the last link of the chain is connected to the first link. The model neurones follow simple integrate-and-fire dynamics (Villa and Tetko, 1995; Hill and Villa, 1997). Each neurone integrates all post-synaptic potentials. If the integrated depolarization passes the threshold, the neurone fires, becoming refractory, whereas if firing does not take place, the membrane potential tends to return to the resting level. Synaptic strengths and time constants have been selected in

Time and the brain

8

order to allow “spontaneous activity” to propagate but also to prevent “epileptic” spikes to self-generate in a simulation run. Figure 2b shows 28 successive frames of the neuronal chain activity, in arbitrary time units. Let us assume that three selected neurones, i.e. c10, c36, and c46, are recorded by virtual electrodes. The corresponding spike trains are shown in Figure 2c, and a firing pattern formed by four spikes was detected in these spike trains. More generally, let the list of cell labels that appear in a pattern of c spikes be noted as S c =(i 1 ,...,i j ,...i c ). In this, any label of the recorded neurones can be assigned to i j . In particular, for a 4-tuple pattern of spikes (quadruplet) obtained from a simultaneous recording of ten spike trains, let us assume that we find S 4 =(c36, c36, c10, c46), meaning all patterns of complexity c equal to 4, formed by a spike recorded from cell c36, followed by another event of cell c36, then a spike recorded from cell c10, which in turn is followed by a spike recorded from cell c46. One specific example of such patterns with delays of 6 time units between the first two spikes from

Figure 2 . Example of spatiotemporal pattern detected in a simulated chain of diverging/converging links. (a) Basic structure of one set of the chain, formed by 7 neurones. One neurone in a set (black circle) receives 5 excitatory (white circles and solid lines) and 2 inhibitory (gray circles and dashed lines) inputs from the previous chain. The output projections of one neurone on the cells of the next set follow the same pattern. (b) The results of the simulation of the activity in a network formed by eight successive sets from time 61 to 88 in arbitrary time steps. The large dots indicate that a neurone is firing.

Empirical evidence about temporal structure

9

The events belonging to the pattern detected in panel (c) are enclosed by squares, (c) A significant pattern formed by four cells is detected. The pattern was: cell 36; 6 time units (t.u.) later cell 36 again; 8 t.u. later cell 10 and 11 t.u. later cell 46.

c36, 2 time units between c36 and c10 and 5 time units between c10 and c46 is notated as (see Figure 2). In addition, this notation assumes that the same time jitter was found for all events forming the pattern. An apparent weak point of the “synfire” theory is the requirement for neural mechanisms able to support a precise timing of spike patterns even after large time delays, and in the presence of various neuromodulators. For instance, a prominent effect of cholinergic modulation is to reduce adaptation of spike frequency. This results in an increase of neuronal excitability and a shortening of inter-spike intervals, so that the overall effect of such modulation can be a modification of spike timing. Differential actions of acetylcholine on the excitability of two subtypes of inhibitory cortical interneurones exist, so that cortical cholinergic activation may change the direction of information flow within cortical circuits (Xiang et al., 1998). Despite this, patch-clamp experiments in rat neocortical slices have shown that timing of spikes produced in response to fluctuating current injections may be preserved during cholinergic modulation (Tang et al., 1997). Transmission across neocortical synapses depends on the frequency of presynaptic activity and studies of synaptic depression have demonstrated that different aspects of the firing patterns of their afferents are transmitted depending on the average presynaptic frequency (Abbott et al., 1997; Tsodyks and Markram, 1997). When a single sensory stimulus drives many neurones to fire at elevated rates, the spikes of these neurones may become tightly synchronized, which could be involved in “binding” together individual firing-rate feature representations into a unified object percept. However, the response to elevated rates of firing may be weak, if any, and population coding based on relative spike timing can systematically signal stimulus features following the stimulus time course even where mean firing rate does not change (deCharms and Merzenich, 1996). These properties determine the fact that little information about steady-state frequency is transferred across synapses. This result is in contradiction with the rate code hypothesis, assuming that a higher input rate corresponds to a higher output rate. In this respect, information theory applied to the coding problem in the frog and house fly (Bialek and Rieke 1992; Rieke et al., 1995) also suggests that spike timing is important for reliable information transfer from receptors to brain, and may explain the subsequent behavioural events. In summary, several lines of evidence are in favour of precise temporal activity in the brain, but the development of analytical methods to detect reliably the transient temporal relations in sequences of spike intervals therefore represents a critical step to establish their function as a temporal code. One proposed technique aimed at detecting “favoured patterns”—i.e. patterns that occur more often than can reasonably be expected at random—consists of identifying template patterns even if the detailed timing of the patterns varies slightly (Dayhoff and Gerstein, 1983a). The advantages of this method, referred to as Favoured Pattern Detection (FPD), is that it can be used to detect other favoured patterns whose occurrences may have extra or missing spikes, and the firing patterns can be tested for

Time and the brain

10

significance by Monte Carlo algorithms. Unfortunately, this technique can be applied only for analysis of temporal patterns of activity generated by a single neurone, and for fixed jitter—i.e. the fluctuation in temporal precision allowed for each spike. Moreover, the choice for settings, e.g. the threshold level parameter, may have discouraged its use among electrophysiologists, and limited its application to experimental data (Dayhoff and Gerstein, 1983b). The use of multiple microelectrode recordings, and of large artificial neural networks of spiking neurones has challenged investigators to search for methods allowing the analysis of higher complexity firing patterns, formed by several spike trains and involving several events within a pattern. An approach to the exact calculation of the probability of randomly obtaining each individual recurring pattern was proposed for three and more distinct spike trains (Frostig et al., 1984, 1990). In this method the statistical evaluation is based on the use of a 2×2 contingency tables and the application of Fisher’s exact test. At the first step, the detection stage, all possible intervals between three spike trains are tested for significance, and at a second step, the expansion stage, all significant patterns found previously are checked for higher-order associations among spike trains. The main limitations of this method consist of its inability to detect recurring patterns involving less than three different spike trains and patterns of more than three spikes, if not all subpatterns formed by a triplet of spikes are statistically significant. Several investigators have pointed out that in most experimental preparations presently available the chance of observing patterns of discharges of higher-order complexity is very low, and specific analytical methods aimed at analyzing patterns formed by three spikes (triplets) were developed. The principles of three-fold correlation among spikes are illustrated in Figure 3. Suppose that the firing times of three neurones A, B, and C are recorded simultaneously (Figure 3a). One could ask whether the firing of cells B and C depends on the time elapsed from a spike of cell A. Let us consider the time delays tAB of the discharges of cell B after neurone A fired and similarly the time delays tAC. The joint distribution histogram may be constructed as shown in Figure 3b. At each discharge of cell A the spike train of neurone B is plotted in the X-axis and the spike train of neurone C is plotted in the Y-axis of the graph. The crossings of columns and rows, falling within a defined bin size, that correspond to the spike trains of neurones B and C recorded simultaneously are counted and a Z-axis, perpendicular to the plan of drawing, is plotted. The value of the compound rate plotted on this Z-axis may be coded in shades of gray (or on a colour scale). A graphic possibility for plotting all compound joint distributions between three neurones is based on a triangular coordinate system because the three ). The X-Y graph can be variables tAB, tAC, tCB, are not independent (since skewed by 60° (Figure 3b, right panel) so that all three joint distributions can be plotted together (Figure 3c). A practical application of this method in practice is shown in Figure 3d. In this example the data were recorded from the thalamus of an anaesthetized cat during spontaneous activity (Villa, 1988, 1990). Cell 1 (discharging at a rate of 2.8 spikes/s) was recorded in the posterior group of the thalamus, cell 3 (1.7 spikes/s) in the dorsal nucleus of the medial geniculate body and cell 7 (5.7 spikes/s) in the brachium of the inferior colliculus. A black triangular bin indicates a significant event corresponding to a spike of cell 3, then 8 ms later a spike from cell 7 and a spike of cell 1 occurring 116 ms after the onset of the pattern. The time accuracy of this estimation was limited to 15

Empirical evidence about temporal structure

11

ms so that spikes of cells 7 and 1 might be considered synchronous. Detection of triplets was first achieved using the “snowflake” triple-spike renewal histogram (Perkel et al., 1975; Abeles, 1983). However, triple-spike renewal histograms do not provide a reliable estimation of the significance of the detected patterns, and to solve this problem the method was developed further into the Joint PeriStimulus Time Histogram (JPSTH) (Aertsen et al., 1989). A number of applications of this technique to

Figure 3 . Outline of the general procedure followed to study three-fold correlation among spikes, (a) Three spike trains, labelled A, B and C, are recorded simultaneously. The time of occurrence of spikes A are

Time and the brain

12

successively taken as onset time for calculation of delays of simultaneously recorded spikes of cell B (tAB) and C (tAC). Three markers indicate three successive spikes of cell A. (b) A plot of all possible patterns formed by B and C spikes as a function of the A spikes marked on the top panel is presented in form of a X-Y plot where the abscissa represents the delays of B spikes after A firing and the ordinate the delays of C spikes after A firing. The left panel shows the same plot with the Y-axis skewed by 60°. (c) A three-cell correlogram is constructed by joining three graphs in order to represent all possible time relations among the firing of three neurones. The three intervals (tAB, tAC, tCB) associated with any point on the graph can be read by projecting the point onto the three external axes, (d) The activity of three neurones, labelled 1, 3 and 7, were recorded from three different electrodes in the cat thalamus. A significant pattern is detected as shown by the black triangular bin. The pattern was: cell 3; 8 ms later cell 7 and 108 ms later cell 1. The shades of gray correspond to the number of counts per bin on a relative scale.

the study of simultaneously recorded spike trains in behaving monkeys have been reported (Abeles et al., 1993b,c). Since then, the algorithm has been improved for statistical evaluation, in order to take into account the existence of significant crosscorrelation between cells. Analyses of multiple spike trains recorded in behaving monkeys using this most recent algorithm, referred to as Joint Triplet Histogram (JTH), have been published recently (Prut et al., 1998). A careful analysis of the compound activity of three-cells may suggest interesting hypotheses about functional connections between distinct nuclei. Let us consider the result illustrated in Figure 3d. One could argue that cells in the brachium of the inferior colliculus and cells in the dorsal nucleus of the medial geniculate body receive common afferent activation from the ascending auditory pathway, and synchronous activation of these nuclei is easy to accept. However, the cross-correlation between the neurones recorded in these nuclei was flat, as indicated by the absence of dark diagonal bands centered on time zero in the snowflake plot, and the quasi-synchronous activation of cells 3 and 7 was only detected when a spike in cell 1 occurred more than 100 ms later. This delay cannot be due to direct connections through the thalamus (according to the present knowledge of thalamic organization). The observation of a complex pattern of firing in different thalamic nuclei suggests that these anatomical regions may form distributed cell assemblies that are functionally connected by some kind of reverberating activity, probably involving re-entrant activity from the cerebral cortex. In chains of diverging/converging links there is no need for each of the synaptic connections to be particularly strong, since they become effective through coactivation with others. Then, several synapses can depolarize the postsynaptic cell to reach its firing threshold, by spatial and temporal summation. In the case of pure excitatory feed-forward chains of neurones (Abeles, 1982, 1991) it has been demonstrated that a strong stimulation (e.g. input fibres firing at a high asynchronous rate) applied to all the cells of a set in the chain would elicit a spike in all of them synchronously, so that the next set of neurones down the chain would also be activated synchronously. This synchronous

Empirical evidence about temporal structure

13

activity would then propagate along the whole chain, with a limited time jitter. Thus, interconversions from firing rate code to synchronous activity, and back, can be performed by a chain of diverging/converging links. This observation is related to the origin of the “synfire” chain model (Abeles, 1982). If these chains include excitatory and inhibitory links, the analysis of the propagating activity is not so simple. In addition when one considers that distinct sets of the chain can be activated independently , then the picture of a propagating wave becomes less clear. However, these chains will tend to produce unitary events, i.e. precise spike synchronization between neurones, which occur significantly more often than expected by chance. The temporal spacing between the spikes should not be regarded as corresponding to the relative location of the parent cells along the chain. Such characteristics of the model are illustrated by Figure 4. In this example the very same simulation results as in Figure 2 are considered, but different cells were sampled by the virtual electrodes. It is important to note that cells involved in unitary events would not necessarily show signs of synchronization in crosscorrelograms, because they do not receive either common excitatory inputs or common inhibitory inputs. In Figure 4a it can be seen that, due to the presence of other inputs in some occurrences of the temporal pattern, a spike belonging to the pattern may be missing because a cell is in a refractory state. However, due to the peculiar properties of diverging/converging links the pattern is sustained by the other neurones and may persist in the network. An increasing number of studies are now investigating unitary events and preliminary experimental evidence has been reported (Riehle et al., 1997; Prut et al., 1998). It is therefore possible that unitary events in the primary motor cortex are related to pattern generation, for instance as starting points for delayed synchronous patterned activity related to the precise activation of sequences of muscles involved in motor output.

3. EMPIRICAL EXAMPLES The Pattern Detection Algorithm (PDA) developed by Abeles and Gerstein (1988) allows one to perform a comprehensive search for firing patterns and to test if there is a statistical excess of patterns. According to the complexity of the patterns to be analyzed,

Time and the brain

14

Figure 4 . Example of a unitary event detected in a simulated chain of diverging/converging links, (a) Two events corresponding to the synchronous activation of cells 1, 17, 23 and 33 have been detected, plus an event corresponding to the synchronous activation of cells 1, 17 and 33. (b) Same simulation results of Figure 2b. The large dots indicate that a neurone is firing. The events belonging to the unitary events are enclosed by squares.

two methods (“ad-hoc” and “bootstrap” algorithms tested against simulated multineurone data) are used for estimation of the expected number of repeated patterns of spikes, regardless of their complexity. Improved and computationally effective versions for estimation of the significance of patterns of spikes in this algorithm, based on probabilistic and combinatorial algorithms, were recently elaborated (Tetko and Villa, 1997a). The main restriction of PDA when compared to the other methods is that it does not allow one to establish the statistical significance of an individual pattern which repeats a number of times, but only the existence of an overall excess of precise patterns beyond expectations. This limits its main applicability to cases when pattern occurrence is very rare. Assuming a Poisson distribution of point processes, the significance of the excess of detected patterns Z, of complexity c, and repeating exactly r times in the record,

Empirical evidence about temporal structure

15

over the expected number X is estimated by pr{Z,X}=P(Z,X) where P(Z,X) represents the incomplete gamma function (Press et al., 1994). This estimation is simplified to if only one firing pattern is detected. Let us fix the level of significance to be p 0 =0.001. For example, if the number of patterns detected by PDA is Z=6, and only one pattern (i.e., X=1) is expected by chance, then a significant excess of detected patterns is observed because . However, this estimation cannot indicate which ones out of the six detected patterns are significant. In some cases PDA allows one to detect an excess of patterns when all sequences of spike intervals are significant. Consider an example where the expected number of patterns is as low as X=0.001. If one pattern is detected, its significance is equal to thus indicating that the pattern is highly significant. If in the above example we detect six patterns instead of one, then any of these six can be considered as highly significant. (In such a case each of the six pattern can be considered significant by itself.) Routine studies using the classical PDA have demonstrated that this optimal case is exceptional. Most often, patterns that could be significant by themselves under appropriate values of jitter are detected by PDA as distinct patterns if the search is performed with maximum accuracy, i.e. with time jitter equal to 1 ms. The application of PDA to several experimental models, and to recordings from various brain areas, has revealed that complex Spatiotemporal firing patterns do indeed occur (Vaadia et al., 1989; Villa and Abeles, 1990; Abeles et al., 1993a; Villa and Fuster, 1992). In these studies the algorithm was usually set to find any repeating pattern of three or more spikes, provided that the entire pattern lasted not more than 900 ms at most, and was repeated with accuracy of 1 ms. Since high frequency bursts (200 spikes/s or more) may produce patterns which relate to intracellular processes, burst filtering should be performed. A filtering procedure was originally proposed by Abeles and Gerstein (1988). A good filtering algorithm should provide a minimal loss of information, for an acceptable rate of errors in the estimation method. This would require fine tuning of this procedure that includes several combinations of three independent parameters: the time window for burst detection, the time window and the number of counts in high-frequency filtering. An alternative filtering method based only on one parameter, the filtering frequency, has been recently proposed (Tetko and Villa, 1997a). This method, with filtering frequency ranging between 200 and 400 spikes/s, provides better estimates of the number of patterns, for a comparable loss of information. A record was considered as having an excess of repeating firing patterns only when the probability of finding as much (or more) repetitions by chance was less than the significance level p 0 . Significant patterns that repeated exactly, more than four times, may be used as templates for searching approximate patterns with jitter of ±5 ms. Figure 5 shows an example of the activity of six neurones recorded in the cat thalamus during spontaneous activity (Villa, 1990; Villa and Abeles, 1990). It illustrates a pattern of 3 spikes, repeating 25 times, starting with spike 2, then after 89 ms spike 4, and then after 305 ms spike 5. On the left panel (Figure 5a) the rasters are aligned on pattern start and on the right panel (Figure 5b) the corresponding histograms (we may call peri-pattern histograms) are computed. In addition to the pattern itself, lasting hundreds of ms and formed by spikes recorded from different electrodes, this example is important because it shows an additional feature. The histogram of Figure 5b shows that two bursty neurones, cells 1 and 6, do not

Time and the brain

16

discharge in correspondence to the last spike of the triplet (cell 5 at 394 ms), but both cells show a significant increased tendency to discharge, in bursts, near 410 ms after the pattern

Figure 5 . (a) Raster display of the activities of six neurones during spontaneous activity recorded in the auditory thalamus of the cat. The rasters are aligned by displaying the first spike in the pattern at time 0. The pattern was: cell 2; 89±5 ms later cell 4, and then 305±5 ms later cell 5. The units were recorded by four microelectrodes. The first, in the thalamic reticular nucleus, picked-up units 1–2; the second, in the medial geniculate body, unit 3; the third, also in the medial geniculate body, picked-up units 4–5 and the last, in thalamic reticular nucleus, unit 6. (b) Histograms of the activities of the units triggered by the pattern onset. The abscissa full scale is 800 ms and the ordinate full scale is 85 spikes/s. Note that units 1 and 6 had a significant higher firing rate 410 ms after the pattern onset. The dashed lines correspond to the 99% confidence levels and a solid line is plotted at the average firing level. The curves are smoothed by a moving Gaussian bin shape of 10 ms.

onset. Therefore we could say that Figure 5 illustrates in fact a very complex pattern that should be noted . This is true, but no algorithms are yet available to detect such a pattern because the firing of the bursty cells is only loosely,

Empirical evidence about temporal structure

17

yet significantly, synchronized with the pattern onset. Furthermore, the selective inhibition of cells 1 and 6 at the same time as the third event of the triplet represents per se a valuable piece of information, related to the highly precise temporal processing occurring in the thalamus. Unfortunately, complex temporal firing patterns mixing spike occurrences and “missing spikes” are not detected by the usual algorithms. This is partly due to the enormous increase in computations required by taking recursively the “negative” of one spike train with all other spike trains recorded simultaneously, and partly by the fact that it is not correct to simply take the “negative”. However, there is no doubt that selective absence of spikes at specific timings would contribute to a better understanding the temporal processing in the brain, and it is reasonable to forecast that in the next few years methods will appear to deal with this problem.

4. PATTERN RECURRENCE IN RELATION TO STIMULI AND RECORDING CONDITION The next two examples illustrate some relations of firing patterns elicited by sensory stimuli, in the sense that they would not be observed during spontaneous activity (Villa 1988; Villa et al., 1991). Recordings were made in the auditory thalamus of the cat. Figure 6 shows a triplet, occurring selectively during the presentation of a stimulus which was a white noise burst lasting 200 ms, delivered to both ears. The firing pattern always started after the stimulus onset, but it always ended before the stimulus offset. However, the pattern was not time locked with the stimulus onset, as clearly shown by the irregular alignment of the stimulus onset and the start of the pattern. Similarly, a pattern could be elicited by a stimulus, but occurring after the stimulus offset and without a tight locking to it. Such an example is illustrated by the triplet in Figure 7, repeating 12 times, starting by spike 3, then after 285 ms spike 1 and then after 189 ms spike 2. Note that spikes 1 and 2 were recorded from the same electrode, thus indicating that these cells lay within a distance of few tens of microns of each other. Nevertheless the delay between these correlated discharges was near to 200 ms, and suggested long reentrant loops from the cortex to the thalamic reticular nucleus or, alternatively, from the substantia nigra and/or from modulatory mesencephalic nuclei, that are known to project to this thalamic region. In order to establish the significance of firing patterns as signposts of distributed network activity related to specific brain processes, it is crucial to demonstrate that whenever the network is working in the same state, the same temporal pattern of firing is observed. Several above-mentioned examples have suggested that cortical activity may affect the recurrent activity in the thalamus, as shown by several corticofugal studies, either experimental (Villa et al., 1991; Payne et al., 1996; Villa et al., 1999b) or theoretical (Tetko and Villa, 1997b). The next examples report data recorded in the auditory thalamus of the rat during reversible cortical deactivation by cooling (Villa et al., 1999b). This type of study illustrates an additional problem that may arise in studies of temporal coding. The number of spikes necessary for a fair statistical evaluation of the

Time and the brain

18

Figure 6 . Raster display of the activities of three thalamic neurones: cells 1 and 2 recorded from the same electrode, in the auditory part of the reticular thalamic nucleus, and cell 3 from a different electrode, in the ventral division of the medial geniculate body. The distance between the two electrode tips was 4.0 mm. The lowermost raster shows the onset of the stimulus, as a thick tick, and the gray bars represent the time of stimulus duration. The rasters are aligned by displaying the first spike in the pattern at time 0. The pattern is repeated 19 times with a jitter of ±3 ms. The abscissa full scale is 600 ms.

significance of the patterns may not be met in all data-sets because the experimental protocol, in this case the cortical cooling, may produce a massive decrease in firing rates of selected single units, or alternatively a massive increase of discharges, and a major change in the pattern of discharges (i.e. from regular to bursty). In case of a decrease in firing rate a partial methodological compensation might be obtained by increasing the recording time. Figure 8 shows an example of a pattern formed by three spikes belonging to two single units recorded simultaneously from two different electrodes in the rat auditory thalamus. The pattern is formed by cell 7 firing, then after 350 ms by a discharge of cell 11 and then after 21 ms cell 11 firing again. This exact pattern was used as template for searching approximate matching patterns with a jitter of ±2ms. Note that several spikes may occur in between the spikes belonging to the pattern without affecting the precise intervals separating the occurrences of the cell discharges. In this example, the significant pattern was observed 9 times in 400 s of spontaneous activity recorded during the control condition. No patterns were observed in 300 s of recording during cooling, but the very

Empirical evidence about temporal structure

19

same precise pattern was observed 5 times in 300 s during the recovery period. Note that between the first and the last occurrences of the pattern more than 90 minutes had passed. Another example (Figure 9) illustrates that significant patterns observed during cortical cooling might be completely absent prior to and after cortical deactivation. This case

Figure 7 . Raster display of the activities of three thalamic neurones: cells 1 and 2 recorded from the same electrode, in the auditory part of the reticular thalamic nucleus, and cell 3 from a different electrode, in the ventral division of the medial geniculate body. The distance between the two electrode tips was 4.4 mm. The rasters are aligned by displaying the first spike in the pattern at time 0. The pattern is repeated 12 times with a jitter of ±2 ms. The abscissa full scale is 1000 ms.

shows a pattern of 3 spikes repeating 9 times (with a jitter of ±2ms) starting by a spike of cell 8, followed by a spike of cell 5 180 ms later, then after 280 ms by a spike of cell 6. Note that only cells 5 and 6 were recorded from the same electrode. This second example also illustrates the tendency observed in firing patterns during cortical cooling to exhibit a higher complexity than patterns observed during control and recovery conditions (Villa and Abeles, 1990).

5. PATTERNS IN RELATION TO BEHAVIOURAL STATES Although the previous examples have provided solid evidence for the existence of precise temporal structure in neural activity, this does not yet establish its function as a temporal code. What is needed is some demonstration that Spatiotemporal firing patterns occur

Time and the brain

20

reliably under particular behavioural conditions, and are related to cognitive activity. The following example shows experimental results recorded in the primate inferotemporal cortex, which is considered one of the last in an ascending hierarchy of cortical processing stages that begins in the striate cortex. Studies with behaving monkeys have led to the conclusion that the inferotemporal cortex is not only a higher order stimulusanalyzer, but is also involved in retention of visual information, and is modulated by attention (Fuster, 1990). In the data presented here, the monkey performed a visual delayed matching-to-sample task (Villa and Fuster, 1992). A trial in one such task consisted of the following:

Figure 8 . Spatiotemporal pattern of spikes detected in the rat auditory thalamus 9 times prior to and 5 times after recovery from cooling of the auditory cortex, but absent during cortical deactivation. The firing pattern was formed by three spikes, starting by spike 7, then after 359 ms spike 11 and then after 21 ms spike 11 again. The pattern is repeated with a jitter of ±2 ms. The spike occurrences corresponding to the pattern are indicated by thick ticks in the raster displays (right panels) aligned by displaying the first spike in the pattern at time 0. Note that spikes 7 and 11 were recorded from different electrodes. The analyzed records corresponded to 400 s of spontaneous activity

Empirical evidence about temporal structure

21

during control condition, 300 s during cortical cooling and 300 s in the recovery condition. Each block of 100 s is represented by a thick mark along the time axis of the experimental protocol, and the dots indicate the start event of the pattern.

(1) sample stimulus (a colour or a geometric figure on a luminescent button with a diameter of 25 mm); (2) 10–20 seconds of delay (retention period); and (3) choice of one stimulus (among 2 or 4) that matched the sample by a given feature (colour or symbol). Correct responses were reinforced with a squirt of juice. About 40% of all single units recorded in the inferotemporal cortex showed a sustained elevation of firing during the delay after the sample stimulus (which was retained in short-term memory for performance of the task). This sample was further subdivided into selective units (14% of all units) showing an activation during the delay which was selectively and significantly higher after one particular tested stimulus, and non-selective units (26%) which were unspecifically activated by all tested stimuli. The distribution of firing rates for both

Figure 9 . Triplet of spikes detected 9 times in the rat auditory thalamus, exclusively during cortical deactivation, starting by spike 8, then after 180 ms spike 5 and then after 280 ms spike 6. The pattern is repeated with a jitter of ±2 ms. The spike occurrences corresponding to the pattern are indicated as in Figure 8. Note that only spikes 5 and 6

Time and the brain

22

were recorded from the same electrode. The analyzed records corresponded to 300 s of spontaneous activity during control condition, 500 s during cortical cooling and 300 s in the recovery condition.

groups in spontaneous discharge was skewed and broad, but the median firing rate for non-selective units (4.4 spikes/s) was higher than that for selective units (2.5 spikes/s). In this study, only one spike train at a time was recorded and thus the temporal firing patterns were formed by multiple spikes of one neurone. Figure 10 illustrates two patterns formed by three spikes observed in two different neurones. The triplet of cell 14 repeated 6 times during the intertrial periods, which did not reach statistical significance, whereas its repetitions were significant (19 times out of 68 trials) during the delay period. The accuracy of the pattern was ±3 ms. The triplet of cell 47 repeated 7 and 19 times during the intertrial and delay periods, respectively, out of 75 trials. One pattern involved long intervals, of the order of hundreds of ms, whereas the other pattern involved relatively short intervals. However, the striking similarity between these two patterns is that both patterns appeared to a significant extent during the retention period of the sample stimulus, and that only one occurrence of the pattern (seldom two) was observed per trial (although in only 20–25% of all trials).

Figure 10 . Raster display (aligned by displaying the first spike in the pattern at time 0) of the activity of two non-selective units recorded in the

Empirical evidence about temporal structure

23

primate inferotemporal cortex during intertrial intervals (left panels) and during the delay (right panels). The measuring time was equal in both conditions. Cell 14 shows a pattern of 3 spikes repeating 6 and 19 times during the intertrial and delay periods, respectively: The second spike of the triplet occurred 165 ms after the pattern start and the third 195 ms after the second spike. Cell 47 shows a triplet repeating 7 and 19 times during the intertrial and delay periods, respectively. The first spike of cell 47 occurred at time 0 (aligning the rasters), the second 42 ms later, and the third 48 ms later. In these rasters the pattern is allowed an accuracy of ±3 ms.

In spike trains recorded in the inferotemporal cortex it was difficult to observe firing patterns composed of 4–6 spikes recurring more than twice within a record. Figure 11 shows an example of such a complex pattern formed by spikes of a selective unit during intertrial periods, and during the delay after the preferred stimulus (the only periods characterized by an increase in firing rate). The pattern was composed of 4 spikes: a spike at time 0, the second 95 ms later, the third 285 ms later, and the fourth and last spike 303 ms later. This quadruplet repeated 6 times during the intertrial periods. The four subpatterns corresponding to the triplets included in the original quadruplet are also displayed in Figure 11. The incidence of the highly organized pattern decreased when the cell was activated by its preferred stimulus, irrespective of an increase in the rate of discharges during the delay. However, it is important to note that one specific subpattern, i.e. was detected at a statistically-significant level during the intertrial and delay periods. This triplet occurred 5 and 9 times during the intertrial and delay periods, respectively. The example of Figure 11 is important, because it illustrates several additional characteristics of precisely time-structured activity in the cortex. Firstly, it illustrates the contribution of subpatterns to a higher order pattern, thus suggesting the robustness of a complex pattern even if at some occurrences a spike is missing. Secondly, it illustrates that subpatterns may express information processing different from that involving the superpattern (as shown by the disappearance of the event at a delay of 95 ms in Figure 11, and at the same time a relative increase in occurrences of the triplet ). On a more general level, the examples of Figures 10–11 raise an interesting problem. Notably, the incidence of firing patterns tended to be inversely related to the selectivity of the units for the stimulus that the animal held in short-term memory (Villa and Fuster, 1992). During the delay periods, and in comparison with intertrial periods, selective units showed a decrease of firing patterns, whereas non-selective units showed an increase. On the one hand, the more patterned activity of non-selective units may reflect their involvement in wide networks representing multiple and general attributes of that particular stimulus held in memory. On the other hand the term “selective unit” shows the long-held belief and

Time and the brain

24

Figure 11 . Raster display (aligned by displaying the first spike in the pattern at time 0) of the activity of a selective unit (cell 61) recorded in the primate inferotemporal cortex during intertrial intervals (25 lines) and during the delay (17 lines). The pattern was composed of 4 spikes, a spike at time 0, the second 95 ms later, the third 285 ms later, and the fourth and last spike 303 ms later. The occurrences of the exact subpatterns formed by 3 out of 4 spikes of the original pattern are also displayed on this raster. The quadruplet is allowed an accuracy of ±3 ms, whereas the subpatterns ±1 ms. Note that during the delay the average firing rate of the cell increases, while the quadruplet pattern disappears, although one particular subpattern remains visible.

prejudicial influence of the rate code on our assumption of what is “selective”. In fact, selective units were defined on the basis of an increased firing rate during the delay period that followed a particular stimulus of the tested set. It is not possible to rule out the possibility that in another paradigm or experimental condition the “selective” units would become “non-selective” and vice versa. Thus, one should bear in mind that the term “selective” should be restricted to a particular experimental situation. However, it is interesting to note that both populations of cells increased their rate of discharges during the retention period, but only the non-selective units increased the patterned activity. An attractive interpretation of these data could be the following: to retain all stimulus characteristics, some transfer of information occurs between the selective units, which encode the specific stimulus feature being retained, and the non-selective units, which are part of a wider network encoding contextual attributes of the stimulus that may be used for binding with other features. This model supports the hypothesis of conversion from asynchronous activity to synchronous activity (and back) through diverging/converging

Empirical evidence about temporal structure

25

connections, as proposed by Abeles (1982, 1991). One important limitation of the PDA method is its application to patterns significant by themselves, as explained above. An additional limitation is the use of a fixed jitter. A new method, called Pattern Grouping Algorithm (PGA), has recently been developed, aiming to identify and evaluate the statistical significance of temporal patterns of spikes formed by three or more different events (Villa and Tetko, 1999; Tetko and Villa, 1999). In particular, this new algorithm is able to recognize patterns of spikes with slight differences in spike timing, and to cluster them as one single group of patterns. Firstly, PGA considers those patterns detected by the classic pattern detection algorithm as templates for search of similar patterns in the spike train data. There are three adjustable parameters in PGA: (i) the maximal duration of the pattern, measured as a delay between the first and the last spike in the sequence of spikes (i.e. the window duration); (ii) the level of significance to be used for detection of significant groups; and (iii) the maximum allowed jitter for timing accuracy of time delays in a pattern group. Based on the idea developed in the template search method (Dayhoff and Gerstein, 1983a) and on selection of optimal jitter, PGA is able to minimize the jitters allowed for each spike in the template independently, according to the actual distribution of spikes. Thus, the probability of detection of significant patterns of spikes is optimized. PGA is not restricted to the analysis of patterns formed by only one cell, and may be applied to the identification of precise temporal patterns regardless of their complexity and number of different cells in a pattern. Therefore, PGA can estimate the significance of each spatiotemporal pattern of firing. The method (described in detail in Tetko and Villa, 1999) consists of an independent estimation by the previously developed PDA, FPD and JTH methods, that have been updated in order to consider the variable jitters. Only those patterns that are significant by all methods are considered for the final step of PGA. Eventually, the final output of PGA consists of a central template pattern with variable jitters corresponding to previously detected patterns grouped into one class. Several tests and validation analyses have demonstrated that PGA can detect true patterns and avoid the detection of spurious patterns, even in the presence of several non-stationarities in the spike trains. These non-stationarities are likely to occur in recordings from behaving and freely-moving animals. Even so, it must be kept in mind that for most of the functional systems the available data demonstrate the presence of precisely timed neural activity whose relationship with behaviour is not always evident. To prove that these temporal patterns do indeed serve for response selection, data are required that establish causal relations between the occurrence of precise temporal relationships and cognitive or motor processes. The next examples will provide solid evidence about these processes. Spike trains from up to 15 single neurones were recorded simultaneously in the temporal cortex of freely moving rats, while animals waited for acoustic cues in a Go/NoGo task (Villa et al., 1999c). The auditory stimulus contained two types of information: pitch (“high” or “low”), and position (“left” or “right”). During the first phase of training, the location was kept constant and the rats had to discriminate between tones of high and low pitch, with one signalling “Go” and the other “NoGo”. Reinforcement was given only after correct “Go” trials. In the second phase, pairs of tones were delivered (one tone from each location), with four possible tone-position combinations. We have shown that, despite the lack of reward for correct NoGo

Time and the brain

26

performance, and the lack of painful punishment for incorrect trials, all rats learn the task to a satisfactory level (performance in the range 70–90%). The paradigm allows analysis of activity during two functional epochs: the “wait” period prior to the stimulus, during which the animal must remain at the back of the cage, and restrain responding despite an increasing probability of imminent stimulus delivery; and the “processing” time, from stimulus delivery until beginning of movement, during which the information content of the signal is processed and motor output organized. Only the “wait” period is considered here for analysis. Significant patterns were detected using PGA from a data-set of 13 hours of recording, involving over one million spikes (Villa et al., 1998a, 1999a). “Go” responses resulted both from correct movements to the feeder in response to low pitch sound at the right speaker and from incorrect movements when a high pitch was delivered to this speaker. Conversely, “NoGo” responses could be correct (failure to move in response to the high pitch from the right speaker) or incorrect (failure to move in response to low pitch from right speaker). Of particular interest were the patterns that were significantly associated with the type of response the animal made later, independent of whether the response was that prompted by the cue. Figure 12 illustrates one such pattern, a triplet formed by the spikes of two cells recorded from distinct electrodes placed in the temporal cortex within the same hemisphere. Note that there were more than twice as many trials (n=27) with patterns that were followed by “Go” responses as there were trials (n=12) with patterns followed by “NoGo” responses, despite the fact that the total number of “Go” and “NoGo” responses was nearly the same (n=283 and 269, respectively). An alternative display of the spikes belonging to the triplets is shown in Figure 12b. In this illustration, the rasters are aligned to an external event (the stimulus onset) instead of being aligned to the start of the pattern. Of these “behaviour-predicting” patterns, half were associated with an enhanced tendency to “Go” in response to the stimulus, and for about 20% of these patterns, trials including the pattern were associated with a faster or slower reaction time than those lacking the pattern (Villa et al., 1998a, 1999a). Figure 13 illustrates one such pattern, and the corresponding reaction times. The triplet shown here is formed by three different neurones, two of which were recorded from the same electrode. It is important to notice that the patterns were associated in almost half (22/37) of the “Go” responses and that pattern occurrence during the wait period led to an accelerated reaction time, by 355 ms on average. Although the patterns could start at any time during the waiting period, we

Empirical evidence about temporal structure

27

Figure 12 . (a) Raster display of the activities of two neurones recorded from two different electrodes in the temporal cortex of a freely-moving behaving rat. The rasters are aligned by displaying the first spike in the pattern at time 0. The pattern was: cell 5; 320±3 ms later cell 11, then 342±3 ms later cell 5 again. The data shown include trials recorded on two consecutive days and in which both Go (n=27) and NoGo (n=12) responses occurred, (b) Raster display of the data corresponding only to the Go responses shown in (a) but on a compressed time scale, and aligned to the time of stimulus onset instead of to time of the first spike of the pattern. Spikes involved in generating instances of the pattern during the waiting period (time to the left of the stimulus onset) are displayed as bars instead of ticks. Three spikes constituting one instance of the pattern are picked out by empty circles. Note the stability of this pattern recorded over two consecutive days.

Time and the brain

28

observed that in several cases the time course of pattern appearance was related to the reaction time. Figure 14 illustrates this finding on a cumulative display of reaction time vs. timing of pattern occurrence. The reaction times of the subjects were normalized to 800 ms on average, in order to cumulate eleven examples on the same curves. Notice that the individual timing of each pattern was by itself significantly correlated with the reaction time on a quadratic regression analysis . The left

Figure 13 . Raster display of the activities of three rat cortical neurones participating in a spatiotemporal firing pattern, aligned by the occurrence of the first spike in the pattern. The time of stimulus presentation is marked by horizontal ticks in the lower window. The pattern was formed by spike #12, 867±2 ms later cell #11, and then after 29±1 ms cell #5. The timing of the three spikes forming the pattern is indicated by the arrows below the stimulus raster. Note that this pattern repeated 22 times (p